Information processing device, information processing method, and program

Abstract

The present disclosure relates to an information processing device, an information processing method, and a program which make it possible to improve object recognition accuracy. A feature quantity calculation unit of an own vehicle calculates, step by step, a feature quantity with respect to an object recognition result in each of a plurality of layers, an acquisition unit acquires a feature quantity with respect to an object recognition result calculated by a feature quantity calculation unit of other vehicle, and a recognition unit performs object recognition, on the basis of the feature quantity calculated by the own vehicle and the acquired feature quantity calculated by the other vehicle, and outputs the object recognition result. The present disclosure is applicable to moving bodies.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable improvement in object recognition accuracy. [Background technology]

[0002] Technologies have been proposed that sense the surroundings of a vehicle using various sensors such as cameras and LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) mounted on the vehicle, and recognize objects around the vehicle based on the sensing results.

[0003] However, sensing results solely from sensors installed in the vehicle in which the user is riding may result in sensing results in which the object is hidden depending on the positional relationship with other vehicles, or sensing results may only be obtained from a direction that makes it difficult to recognize the object, which can reduce the accuracy of object recognition.

[0004] Therefore, it is conceivable to apply a technology that improves the processing accuracy for a specific object by processing the specific object together with images captured for use in processing other objects (see Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-016161 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, even if the technology disclosed in Patent Document 1 is applied to use the sensing results obtained by sensing another vehicle, there is a risk that the recognition results will be inconsistent due to the difference in the sensors.

[0007] Furthermore, even if sensing results obtained by other vehicles are used, it is only possible to improve the recognition accuracy in the common observation area, and there is a risk that the overall recognition accuracy cannot be improved.

[0008] The present disclosure has been made in consideration of such circumstances, and in particular improves the accuracy of object recognition by acquiring, selecting, and combining features obtained from the sensing results of other vehicles, and then using them all at once for object recognition. [Means for solving the problem]

[0009] An information processing device and a program according to one embodiment of the present disclosure include: Based on the sensing results detected by the surrounding information detection sensor, a feature amount calculation unit that calculates a feature amount used for object recognition in a stepwise manner in each of a plurality of layers; Based on another sensing result detected by another surrounding information detection sensor different from the surrounding information detection sensor, The feature amount calculated by another feature amount calculation unit different from the feature amount calculation unit is used for the object recognition. Other features different from the above An acquisition unit for acquiring a feature amount; The above The feature amount, The other and a recognition unit that performs the object recognition based on the feature amount. The surrounding information detection sensor and the other surrounding information detection sensor are provided on different objects, The present invention relates to an information processing device and a program.

[0010] An information processing method according to one aspect of the present disclosure includes: An information processing method for an information processing device including a feature amount calculation unit, an acquisition unit, and a recognition unit, the information processing method including: the feature amount calculation unit performing a calculation process for stepwise calculating a feature amount used for object recognition in each of a plurality of hierarchical levels based on a sensing result detected by a surrounding information detection sensor; the acquisition unit performing an acquisition process for acquiring another feature amount different from the feature amount used for the object recognition calculated by another feature amount calculation process different from the feature amount calculation process based on another sensing result detected by another surrounding information detection sensor different from the surrounding information detection sensor; and the recognition unit performing an object recognition process for performing the object recognition based on the feature amount and the other feature amount, the surrounding information detection sensor and the other surrounding information detection sensor being provided on different objects. .

[0011] In one aspect of the present disclosure, a feature amount calculation unit: Based on the sensing results detected by the surrounding information detection sensor, In each of the multiple layers, features used for object recognition are calculated step by step, Based on another sensing result detected by another surrounding information detection sensor different from the surrounding information sensor, The feature amount calculated by another feature amount calculation unit different from the feature amount calculation unit is used for the object recognition. Other features different from the above The features are obtained, The above The feature amount, The other The object recognition is performed based on the feature amount. The surrounding information detection sensor and the other surrounding information detection sensor are provided on different objects, do. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an overview of an object recognition system according to the present disclosure. [Diagram 2] FIG. 1 is a diagram illustrating an example of the configuration of a vehicle control system according to the present disclosure. [Diagram 3] 1 is a block diagram illustrating a configuration example of an object recognition unit according to a first embodiment of the present disclosure. [Figure 4] 4 is a flowchart illustrating an object recognition process according to the first embodiment. [Diagram 5] 4 is a flowchart illustrating an object recognition process according to the first embodiment. [Figure 6] 13 is a flowchart illustrating an object recognition process according to a modified example of the first embodiment. [Figure 7] 13 is a flowchart illustrating an object recognition process according to a modified example of the first embodiment. [Figure 8] FIG. 13 is a block diagram illustrating a configuration example of an object recognition unit according to a second embodiment of the present disclosure. [Figure 9] 13 is a flowchart illustrating an object recognition process according to a second embodiment. [Figure 10] 13 is a flowchart illustrating an object recognition process according to a second embodiment. [Figure 11] FIG. 1 is a diagram illustrating an example of the configuration of a general-purpose computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0014] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Overview of this Disclosure 2. First embodiment 3. Modification of the First Embodiment 4. Second embodiment 5. Software implementation examples

[0015] <<1. Overview of this Disclosure>> The present disclosure improves the accuracy of object recognition by sensing the environment around each vehicle, extracting features from the sensing results, combining the features extracted for other vehicles with the features extracted for the vehicle itself, and recognizing objects based on the combined features.

[0016] First, an overview of the object recognition system of the present disclosure will be described with reference to FIG.

[0017] The object recognition system 11 in Fig. 1 is composed of a plurality of vehicles 31-1 to 31-5. In the object recognition system 11 in Fig. 1, when there is no need to particularly distinguish between the vehicles 31-1 to 31-5, they are simply referred to as vehicles 31. In addition, in the object recognition system 11 in Fig. 1, it is assumed that there are five vehicles 31, but the number of vehicles may be any other number as long as there is at least one more vehicle other than the host vehicle.

[0018] Moreover, the vehicle 31 in which the user is riding is also referred to as the user's own vehicle, and other vehicles 31 are also referred to as other vehicles.

[0019] In the object recognition system 11 of FIG. 1, the vehicle 31-1 is assumed to be the host vehicle, and the vehicles 31-2 to 31-5 are assumed to be other vehicles.

[0020] Although not shown, the vehicles 31-1 to 31-5 are configured to be capable of communicating with each other within a predetermined distance by so-called inter-vehicle communication.

[0021] Furthermore, the vehicles 31-1 to 31-5 are equipped with object recognition devices 41-1 to 41-5, respectively.

[0022] The object recognition devices 41-1 to 41-5 are equipped with object recognition units 41a-1 to 41a-5, respectively. The object recognition units 41a-1 to 41a-5 acquire information about their surroundings as point cloud information PC1 to PC5, and extract feature values ​​F1 to F5 from the point cloud information PC1 to PC5, respectively.

[0023] When no vehicle is present in the vicinity, the object recognition units 41a-1 to 41a-5 execute object recognition processing using the extracted feature amounts F1 to F5 alone, respectively, and output recognition results SR1 to SR5, respectively.

[0024] The vehicles 31-1 to 31-5 each execute various processes based on the recognition results SR1 to SR5.

[0025] In addition, when there is another vehicle in the vicinity within a predetermined distance, each of the object recognition units 41a-1 to 41a-5 requests and obtains the features of the other vehicle, and performs object recognition processing using the features of the other vehicle and the features of the own vehicle.

[0026] That is, in the case of vehicle 31-1 in FIG. 1, vehicles 31-2 and 31-4, for example, which are within a range Z of a predetermined distance indicated by a dotted line and capture images of the area ahead in the direction in which the vehicle is traveling, are recognized as being nearby vehicles to vehicle 31-1.

[0027] In this case, the object recognition unit 41a-1 of the vehicle 31-1 requests feature amounts from the surrounding vehicles 31-2 and 31-4.

[0028] In response to this request, the object recognition units 41a-2 and 41a-4 of the vehicles 31-2 and 31-4 transmit the feature amounts F2 and F4, respectively, to the vehicle 31-1.

[0029] As a result, the object recognition unit 41a-1 of the vehicle 31-1 acquires the feature amounts F2 and F4 transmitted from the object recognition units 41a-2 and 41a-4 of the vehicles 31-2 and 31-4.

[0030] The object recognition unit 41a-1 of the vehicle 31-1 then performs object recognition processing by combining and using the thus acquired feature amounts F2, F4 and the feature amount F1 that it has determined itself.

[0031] Through such processing, the object recognition unit 41a-1 of vehicle 31-1 performs object recognition processing using feature F1 based on point cloud information of the monitoring area of ​​the vehicle itself, as well as feature F2 and F4 based on point cloud information of the monitoring area of ​​other vehicles, thereby enabling improvement in the accuracy of object recognition.

[0032] Furthermore, since the transmitted and received features F2, F4 have a sufficiently small amount of information compared with the point cloud information PC2, PC4, it is possible to reduce the communication load in vehicle-to-vehicle communication and improve the communication speed, thereby enabling high-precision object recognition processing to be realized at high speed.

[0033] <<2. First embodiment>> <Example of vehicle control system configuration> 2 is a block diagram showing an example of a schematic functional configuration of a vehicle control system 100 of a vehicle 51, which is an example of a vehicle moving object control system to which the present technology can be applied. The vehicle 51 in FIG. 2 corresponds to the vehicle 31 in FIG. 1.

[0034] In addition, hereinafter, when the vehicle in which the vehicle control system 100 is installed is to be distinguished from other vehicles, it will be referred to as the host vehicle or the host vehicle.

[0035] The vehicle control system 100 includes an input unit 101, a data acquisition unit 102, a communication unit 103, an in-vehicle device 104, an output control unit 105, an output unit 106, a drive system control unit 107, a drive system 108, a body system control unit 109, a body system 110, a storage unit 111, and an automatic driving control unit 112. The input unit 101, the data acquisition unit 102, the communication unit 103, the output control unit 105, the drive system control unit 107, the body system control unit 109, the storage unit 111, and the automatic driving control unit 112 are connected to each other via a communication network 121. The communication network 121 is, for example, an in-vehicle communication network or a bus that complies with any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay (registered trademark). Note that each unit of the vehicle control system 100 may be directly connected without going through the communication network 121.

[0036] In the following description, when each unit of the vehicle control system 100 communicates with each other via the communication network 121, the description of the communication network 121 will be omitted. For example, when the input unit 101 and the autonomous driving control unit 112 communicate with each other via the communication network 121, it will be simply described that the input unit 101 and the autonomous driving control unit 112 communicate with each other.

[0037] The input unit 101 includes a device used by a passenger to input various data, instructions, and the like. For example, the input unit 101 includes operation devices such as a touch panel, a button, a microphone, a switch, and a lever, as well as an operation device that allows input by a method other than manual operation such as voice or gesture. In addition, for example, the input unit 101 may be a remote control device that uses infrared rays or other radio waves, or an externally connected device such as a mobile device or a wearable device that supports operation of the vehicle control system 100. The input unit 101 generates an input signal based on data, instructions, and the like input by the passenger, and supplies the input signal to each part of the vehicle control system 100.

[0038] The data acquisition unit 102 includes various sensors and the like that acquire data used for processing by the vehicle control system 100 , and supplies the acquired data to each unit of the vehicle control system 100 .

[0039] For example, the data acquisition unit 102 includes various sensors for detecting the state of the vehicle, etc. Specifically, for example, the data acquisition unit 102 includes a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU), and sensors for detecting the amount of operation of an accelerator pedal, the amount of operation of a brake pedal, the steering angle of a steering wheel, the engine rotation speed, the motor rotation speed, or the rotation speed of a wheel, etc.

[0040] Also, for example, the data acquisition unit 102 includes various sensors for detecting information outside the vehicle. Specifically, for example, the data acquisition unit 102 includes imaging devices such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. Also, for example, the data acquisition unit 102 includes an environment sensor for detecting weather or climate, and a surrounding information detection sensor for detecting objects around the vehicle. The environment sensor includes, for example, a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and the like. The surrounding information detection sensor includes, for example, an ultrasonic sensor, a radar, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), a sonar, and the like.

[0041] Furthermore, for example, the data acquisition unit 102 includes various sensors for detecting the current position of the vehicle. Specifically, for example, the data acquisition unit 102 includes a GNSS receiver that receives GNSS signals from Global Navigation Satellite System (GNSS) satellites.

[0042] Furthermore, for example, the data acquisition unit 102 includes various sensors for detecting information inside the vehicle. Specifically, for example, the data acquisition unit 102 includes an imaging device for capturing an image of the driver, a biosensor for detecting bioinformation of the driver, and a microphone for collecting sound inside the vehicle. The biosensor is provided, for example, on the seat or steering wheel, and detects bioinformation of a passenger sitting in the seat or a driver holding the steering wheel.

[0043] The communication unit 103 communicates with the in-vehicle device 104 as well as various devices, servers, base stations, and the like outside the vehicle, transmits data supplied from each part of the vehicle control system 100, and supplies received data to each part of the vehicle control system 100. Note that the communication protocol supported by the communication unit 103 is not particularly limited, and the communication unit 103 can also support multiple types of communication protocols.

[0044] For example, the communication unit 103 performs wireless communication with the in-vehicle device 104 by wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), WUSB (Wireless USB), etc. Also, for example, the communication unit 103 performs wired communication with the in-vehicle device 104 by USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), MHL (Mobile High-definition Link), etc. via a connection terminal (and a cable, if necessary) not shown.

[0045] Furthermore, for example, the communication unit 103 communicates with devices (for example, application servers or control servers) present on an external network (for example, the Internet, a cloud network, or a network specific to a business operator) via a base station or an access point. Also, for example, the communication unit 103 communicates with terminals (for example, pedestrian or store terminals, or MTC (Machine Type Communication) terminals) present in the vicinity of the vehicle using a P2P (Peer To Peer) technology. Furthermore, for example, the communication unit 103 performs V2X communication such as vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication. Also, for example, the communication unit 103 includes a beacon receiving unit, and receives radio waves or electromagnetic waves transmitted from a wireless station or the like installed on a road, and acquires information such as a current position, congestion, traffic regulations, or required time.

[0046] The in-vehicle devices 104 include, for example, mobile devices or wearable devices possessed by passengers, information devices carried into or attached to the vehicle, and navigation devices that search for a route to an arbitrary destination.

[0047] The output control unit 105 controls the output of various information to passengers of the vehicle or to the outside of the vehicle. For example, the output control unit 105 generates an output signal including at least one of visual information (e.g., image data) and audio information (e.g., audio data) and supplies the output signal to the output unit 106, thereby controlling the output of the visual information and audio information from the output unit 106. Specifically, for example, the output control unit 105 synthesizes image data captured by different imaging devices of the data acquisition unit 102 to generate an overhead image or a panoramic image, and supplies an output signal including the generated image to the output unit 106. Also, for example, the output control unit 105 generates audio data including a warning sound or a warning message against danger such as a collision, contact, or entry into a dangerous area, and supplies an output signal including the generated audio data to the output unit 106.

[0048] The output unit 106 includes a device capable of outputting visual or audio information to the passengers of the vehicle or to the outside of the vehicle. For example, the output unit 106 includes a display device, an instrument panel, an audio speaker, headphones, a wearable device such as a glasses-type display worn by the passenger, a projector, a lamp, etc. The display device included in the output unit 106 may be a device having a normal display, or may be a device that displays visual information within the field of view of the driver, such as a head-up display, a see-through display, or a device having an AR (Augmented Reality) display function.

[0049] The drive system control unit 107 generates various control signals and supplies them to the drive system 108, thereby controlling the drive system 108. In addition, the drive system control unit 107 supplies control signals to each unit other than the drive system 108 as necessary, and notifies the control state of the drive system 108, etc.

[0050] The drivetrain system 108 includes various devices related to the drivetrain of the vehicle, such as a drive force generating device for generating drive force from an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle, a braking device for generating braking force, an antilock brake system (ABS), an electronic stability control (ESC), and an electric power steering device.

[0051] The body system control unit 109 generates various control signals and supplies them to the body system 110, thereby controlling the body system 110. Furthermore, the body system control unit 109 supplies control signals to each unit other than the body system 110 as necessary, and notifies the control state of the body system 110, etc.

[0052] The body system 110 includes various body system devices mounted on the vehicle body, such as a keyless entry system, a smart key system, a power window device, a power seat, a steering wheel, an air conditioning device, and various lamps (e.g., head lamps, back lamps, brake lamps, blinkers, fog lamps, etc.).

[0053] The storage unit 111 includes, for example, a read only memory (ROM), a random access memory (RAM), a magnetic storage device such as a hard disc drive (HDD), a semiconductor storage device, an optical storage device, and a magneto-optical storage device. The storage unit 111 stores various programs and data used by each unit of the vehicle control system 100. For example, the storage unit 111 stores map data such as a three-dimensional high-precision map such as a dynamic map, a global map that is less accurate than the high-precision map and covers a wide area, and a local map that includes information about the surroundings of the vehicle.

[0054] The automatic driving control unit 112 performs control related to automatic driving such as autonomous driving or driving assistance. Specifically, for example, the automatic driving control unit 112 performs cooperative control for the purpose of realizing the functions of an ADAS (Advanced Driver Assistance System) including collision avoidance or impact mitigation of the host vehicle, following driving based on the distance between vehicles, maintaining vehicle speed, collision warning of the host vehicle, or lane departure warning of the host vehicle. In addition, for example, the automatic driving control unit 112 performs cooperative control for the purpose of automatic driving that runs autonomously without relying on the operation of the driver. The automatic driving control unit 112 includes a detection unit 131, a self-position estimation unit 132, a situation analysis unit 133, a planning unit 134, and an operation control unit 135.

[0055] The detection unit 131 detects various types of information necessary for controlling the autonomous driving. The detection unit 131 includes an outside-vehicle information detection unit 141, an inside-vehicle information detection unit 142, and a vehicle state detection unit 143.

[0056] The outside-vehicle information detection unit 141 performs a detection process of information outside the vehicle based on data or signals from each unit of the vehicle control system 100. For example, the outside-vehicle information detection unit 141 performs a detection process, a recognition process, and a tracking process of objects around the vehicle, as well as a detection process of the distance to the objects. Objects to be detected include, for example, vehicles, people, obstacles, structures, roads, traffic lights, traffic signs, road markings, and the like. In addition, for example, the outside-vehicle information detection unit 141 performs a detection process of the environment around the vehicle. The surrounding environment to be detected includes, for example, weather, temperature, humidity, brightness, and road surface conditions. The outside-vehicle information detection unit 141 supplies data indicating the results of the detection process to the self-position estimation unit 132, the map analysis unit 151, the traffic rule recognition unit 152, and the situation recognition unit 153 of the situation analysis unit 133, the emergency situation avoidance unit 171 of the operation control unit 135, and the like.

[0057] The vehicle exterior information detection unit 141 further includes an object recognition unit 181 and a vehicle detection unit 182 .

[0058] The object recognition unit 181 has a configuration corresponding to the object recognition unit 41a in FIG. 1, and performs object authentication processing based on a point cloud (point cloud information) supplied from surrounding information detection sensors such as an ultrasonic sensor, radar, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), stereo camera, and sonar of the data acquisition unit 102, and authenticates objects for each point of the point cloud around the vehicle.

[0059] The vehicle detection unit 182 detects the approach of other vehicles around the vehicle using images captured by an imaging device such as a camera of the data acquisition unit 102, or vehicle-to-vehicle communication between the vehicle and other vehicles via the communication unit 103.

[0060] More specifically, the object recognition unit 181 calculates features of the sensing result consisting of a point cloud using a DNN (Deep Neural Network), and performs object recognition processing based on the calculation result to recognize the object on a point-by-point basis.

[0061] At this time, when the vehicle detection unit 182 detects the approach of another vehicle around the own vehicle, the object recognition unit 181 requests and acquires the features calculated by the object recognition unit 181 of the other vehicle via vehicle-to-vehicle communication, etc., and combines them with the features calculated by the object recognition unit 181 itself to perform object authentication processing.

[0062] This enables the object recognition unit 181 to perform object recognition corresponding to the entire point cloud around the vehicle, thereby improving the accuracy of object recognition.

[0063] The configuration of the object recognition unit 181 will be described in detail later.

[0064] The in-vehicle information detection unit 142 performs detection processing of information inside the vehicle based on data or signals from each unit of the vehicle control system 100. For example, the in-vehicle information detection unit 142 performs authentication processing and recognition processing of the driver, detection processing of the driver's state, detection processing of passengers, detection processing of the environment inside the vehicle, etc. The driver's state to be detected includes, for example, physical condition, alertness level, concentration level, fatigue level, line of sight direction, etc. The environment inside the vehicle to be detected includes, for example, temperature, humidity, brightness, odor, etc. The in-vehicle information detection unit 142 supplies data indicating the result of the detection processing to the situation recognition unit 153 of the situation analysis unit 133, the emergency situation avoidance unit 171 of the operation control unit 135, etc.

[0065] The vehicle state detection unit 143 performs detection processing of the state of the vehicle based on data or signals from each unit of the vehicle control system 100. The vehicle state to be detected includes, for example, the speed, acceleration, steering angle, the presence or absence and content of an abnormality, the state of driving operation, the position and inclination of the power seat, the state of the door lock, and the state of other in-vehicle devices. The vehicle state detection unit 143 supplies data indicating the result of the detection processing to the situation recognition unit 153 of the situation analysis unit 133, the emergency situation avoidance unit 171 of the operation control unit 135, etc.

[0066] The self-position estimation unit 132 performs estimation processing of the position and attitude of the vehicle based on data or signals from each unit of the vehicle control system 100, such as the outside-vehicle information detection unit 141 and the situation recognition unit 153 of the situation analysis unit 133. The self-position estimation unit 132 also generates a local map (hereinafter referred to as a self-position estimation map) used for estimating the self-position as necessary. The self-position estimation map is a high-precision map using a technology such as SLAM (Simultaneous Localization and Mapping). The self-position estimation unit 132 supplies data indicating the result of the estimation processing to the map analysis unit 151, the traffic rule recognition unit 152, the situation recognition unit 153, and the like of the situation analysis unit 133. The self-position estimation unit 132 also stores the self-position estimation map in the storage unit 111.

[0067] The situation analysis unit 133 performs an analysis process of the situation of the vehicle itself and the surroundings. The situation analysis unit 133 includes a map analysis unit 151, a traffic rule recognition unit 152, a situation recognition unit 153, and a situation prediction unit 154.

[0068] The map analysis unit 151 performs analysis processing of various maps stored in the storage unit 111 while using data or signals from each unit of the vehicle control system 100, such as the self-position estimation unit 132 and the outside vehicle information detection unit 141, as necessary, to construct a map including information necessary for autonomous driving processing. The map analysis unit 151 supplies the constructed map to the traffic rule recognition unit 152, the situation recognition unit 153, the situation prediction unit 154, and the route planning unit 161, the action planning unit 162, the operation planning unit 163, and the like of the planner 134.

[0069] The traffic rule recognition unit 152 performs recognition processing of traffic rules around the vehicle based on data or signals from each unit of the vehicle control system 100, such as the self-position estimation unit 132, the outside vehicle information detection unit 141, and the map analysis unit 151. This recognition processing recognizes, for example, the positions and states of traffic signals around the vehicle, the details of traffic regulations around the vehicle, and lanes in which travel is possible. The traffic rule recognition unit 152 supplies data indicating the results of the recognition processing to the situation prediction unit 154 and the like.

[0070] The situation recognition unit 153 performs recognition processing of the situation related to the vehicle based on data or signals from each unit of the vehicle control system 100, such as the self-position estimation unit 132, the vehicle outside information detection unit 141, the vehicle inside information detection unit 142, the vehicle state detection unit 143, and the map analysis unit 151. For example, the situation recognition unit 153 performs recognition processing of the situation of the vehicle, the situation around the vehicle, the situation of the driver of the vehicle, and the like. In addition, the situation recognition unit 153 generates a local map (hereinafter referred to as a situation recognition map) used for recognizing the situation around the vehicle, as necessary. The situation recognition map is, for example, an occupancy grid map.

[0071] The vehicle's status to be recognized includes, for example, the vehicle's position, posture, and movement (e.g., speed, acceleration, moving direction, etc.), as well as the presence or absence of an abnormality and its contents, etc. The vehicle's surroundings to be recognized include, for example, the type and position of surrounding stationary objects, the type, position and movement (e.g., speed, acceleration, moving direction, etc.) of surrounding moving objects, the configuration of surrounding roads and the condition of the road surface, as well as the surrounding weather, temperature, humidity, brightness, etc. The driver's status to be recognized includes, for example, physical condition, alertness, concentration, fatigue, eye movement, driving operation, etc.

[0072] The situation recognition unit 153 supplies data indicating the result of the recognition process (including a situation recognition map, if necessary) to the self-position estimation unit 132, the situation prediction unit 154, etc. In addition, the situation recognition unit 153 stores the situation recognition map in the storage unit 111.

[0073] The situation prediction unit 154 performs prediction processing of the situation regarding the vehicle, based on data or signals from each unit of the vehicle control system 100, such as the map analysis unit 151, the traffic rule recognition unit 152, and the situation recognition unit 153. For example, the situation prediction unit 154 performs prediction processing of the situation of the vehicle, the situation around the vehicle, the situation of the driver, and the like.

[0074] The vehicle's status to be predicted includes, for example, the vehicle's behavior, the occurrence of an abnormality, and the driving distance, etc. The vehicle's surroundings to be predicted include, for example, the behavior of animals around the vehicle, changes in traffic signals, and changes in the environment such as weather, etc. The driver's status to be predicted includes, for example, the driver's behavior and physical condition, etc.

[0075] The situation prediction unit 154 supplies data indicating the results of the prediction process, together with data from the traffic rule recognition unit 152 and the situation recognition unit 153, to the route planning unit 161, the action planning unit 162, and the motion planning unit 163 of the planner 134.

[0076] The route planning unit 161 plans a route to the destination based on data or signals from each unit of the vehicle control system 100, such as the map analysis unit 151 and the situation prediction unit 154. For example, the route planning unit 161 sets a route from the current position to a specified destination based on a global map. In addition, for example, the route planning unit 161 appropriately changes the route based on conditions such as congestion, accidents, traffic restrictions, and construction, and the driver's physical condition, etc. The route planning unit 161 supplies data indicating the planned route to the action planning unit 162, etc.

[0077] The behavior planning unit 162 plans the behavior of the vehicle for safely traveling the route planned by the route planning unit 161 within the planned time based on data or signals from each unit of the vehicle control system 100, such as the map analysis unit 151 and the situation prediction unit 154. For example, the behavior planning unit 162 plans starting, stopping, traveling direction (for example, moving forward, backward, turning left, turning right, changing direction, etc.), traveling lane, traveling speed, overtaking, etc. The behavior planning unit 162 supplies data indicating the planned behavior of the vehicle to the operation planning unit 163, etc.

[0078] The action planning unit 163 plans the action of the vehicle to realize the action planned by the action planning unit 162, based on data or signals from each unit of the vehicle control system 100, such as the map analysis unit 151 and the situation prediction unit 154. For example, the action planning unit 163 plans acceleration, deceleration, a driving trajectory, etc. The action planning unit 163 supplies data indicating the planned action of the vehicle to the acceleration / deceleration control unit 172 and the direction control unit 173 of the action control unit 135, etc.

[0079] The operation control unit 135 controls the operation of the host vehicle. The operation control unit 135 includes an emergency situation avoidance unit 171, an acceleration / deceleration control unit 172, and a direction control unit 173.

[0080] The emergency avoidance unit 171 performs detection processing of an emergency such as a collision, contact, entry into a dangerous area, driver abnormality, or vehicle abnormality based on the detection results of the vehicle outside information detection unit 141, the vehicle inside information detection unit 142, and the vehicle state detection unit 143. When the emergency avoidance unit 171 detects the occurrence of an emergency, it plans an operation of the vehicle to avoid an emergency such as a sudden stop or a sharp turn. The emergency avoidance unit 171 supplies data indicating the planned operation of the vehicle to the acceleration / deceleration control unit 172, the direction control unit 173, etc.

[0081] The acceleration / deceleration control unit 172 performs acceleration / deceleration control to realize the operation of the host vehicle planned by the operation planning unit 163 or the emergency situation avoidance unit 171. For example, the acceleration / deceleration control unit 172 calculates a control target value of a driving force generation device or a braking device to realize the planned acceleration, deceleration, or sudden stop, and supplies a control command indicating the calculated control target value to the drive system control unit 107.

[0082] The direction control unit 173 performs direction control to realize the operation of the host vehicle planned by the action planning unit 163 or the emergency situation avoidance unit 171. For example, the direction control unit 173 calculates a control target value of a steering mechanism to realize a traveling trajectory or a sharp turn planned by the action planning unit 163 or the emergency situation avoidance unit 171, and supplies a control command indicating the calculated control target value to the drive system control unit 107.

[0083] <First configuration example of object recognition unit> Next, a first configuration example of the object recognition unit 181 will be described with reference to FIG.

[0084] The object recognition unit 181 performs object recognition processing based on the point cloud PC supplied from the data acquisition unit 102 via a sensor such as a stereo camera or LiDAR, and outputs the result as a global feature amount R for each point.

[0085] The point cloud PC is point cloud information consisting of position information of each of the points (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and so on (xn, yn, zn).

[0086] In addition, the global feature R is global feature R(x1, y1, z1), global feature R(x2, y2, z2), global feature R(x3, y3, z3), global feature R(xn, yn, zn) corresponding to (x1, y1, z1), (x2, y2, z2), and (x3, y3, z3), which are the object recognition results for each point.

[0087] More specifically, the object recognition unit 181 includes a local feature calculation unit 201, a feature combination unit 202, a global feature calculation unit 203, a recognition unit 204, a feature extraction unit 205, a feature transmission control unit 206, a feature reception control unit 207, and a feature selection unit 208.

[0088] The local feature amount calculation unit 201 calculates the feature amount for each point in the point cloud PC in a step-by-step manner, and includes a first feature amount calculation unit 221, a second feature amount calculation unit 222, and a third feature amount calculation unit 223.

[0089] The first feature calculation unit 221 is composed of MLPs (Multi Layer Perceptrons) 231-1 to 231-n, one for each point, and calculates a first feature, which is a first-stage feature, for each point of the point cloud PC, and outputs it to the second feature calculation unit 222 and the first feature extraction unit 281 of the feature extraction unit 205.

[0090] The second feature calculation unit 222 is composed of MLPs 241-1 to 241-n, one for each point, and calculates a second feature, which is a feature of the second stage, based on the first feature supplied by the first feature calculation unit 221 for each point of the point cloud PC, and outputs the second feature to the third feature calculation unit 223 and the second feature extraction unit 282 of the feature extraction unit 205.

[0091] The third feature calculation unit 223 is composed of MLPs 251-1 to 251-n, one for each point, and calculates a third feature, which is a feature of the third stage, based on the second feature supplied from the second feature calculation unit 222 for each point of the point cloud PC, and outputs the third feature to the feature combination unit 202 and the third feature extraction unit 283 of the feature extraction unit 205.

[0092] The feature combination unit 202 combines the first feature, the second feature, and the third feature supplied from other vehicles via the communication unit 103, the feature reception control unit 207, and the feature selection unit 208 with the third feature supplied from the local feature calculation unit 201, and outputs the combined result to the global feature calculation unit 203.

[0093] More specifically, the feature combination unit 202 includes a second feature calculation unit 271 and a third feature calculation unit 272 .

[0094] The second feature calculation unit 271 and the third feature calculation unit 272 have the same functions as the second feature calculation unit 222 and the third feature calculation unit 223, and calculate the second feature and the third feature from the first feature and the second feature supplied from the other vehicle as necessary, and finally calculate the third feature based on the feature of the other vehicle.

[0095] Then, the feature amount combination unit 202 combines the third feature amount determined from the other vehicle and the third feature amount calculated by the local feature amount calculation unit 201, for example, by a product-sum operation to which a weight is set.

[0096] The weights may be set equally or according to the distance from other vehicles and whether or not the DNN structure is the same.

[0097] The global feature calculation unit 203 calculates a global feature R at each point based on the feature (effectively the third feature) supplied from the feature combination unit 202, for example, by Maxpooling, and outputs the global feature R to the recognition unit 204. The recognition unit 204 performs object recognition processing based on the global feature R, and outputs an object recognition result that is the processing result.

[0098] That is, in the object recognition unit 181 of FIG. 3 of the present disclosure, the local feature calculation unit 201, the feature combination unit 202, and the global feature calculation unit 203 form a PointNet using a DNN.

[0099] Here, PointNet refers to a neural network that receives point cloud information as input and outputs a predetermined processing result based on the input point cloud information.

[0100] That is, the first feature amount calculation section 221, the second feature amount calculation section 222, and the third feature amount calculation section 223 in the local feature amount calculation section 201 are intermediate layers that extract local features.

[0101] For this reason, the local feature calculation unit 201 in FIG. 3 is configured to calculate features using a three-stage hierarchical structure, such as the first feature calculation unit 221 to the third feature calculation unit 223, but the number of layers may be other than two or three, so long as the hierarchical structure has at least two layers corresponding to the input layer and the output layer.

[0102] It is assumed that the local feature calculation unit 201 and the global feature calculation unit 203 have been trained (backpropagation) using, for example, position (x, y, z) and color (r, g, b) of a point cloud as input and semantic segmentation labels as correct answers.

[0103] The feature extraction unit 205 extracts features that are the calculation results of the local feature calculation unit 201 , and supplies them to a feature transmission control unit 206 .

[0104] More specifically, the feature amount extraction section 205 includes a first feature amount extraction section 281, a second feature amount extraction section 282, and a third feature amount extraction section 283.

[0105] The first feature extraction unit 281 extracts a first feature which is the calculation result of the first feature calculation unit 221 of the local feature calculation unit 201 , and outputs it to the feature transmission control unit 206 .

[0106] The second feature extraction unit 282 extracts a second feature that is the calculation result of the second feature calculation unit 222 of the local feature calculation unit 201 , and outputs it to the feature transmission control unit 206 .

[0107] The third feature extraction unit 283 extracts a third feature which is the calculation result of the third feature calculation unit 223 of the local feature calculation unit 201 , and outputs it to the feature transmission control unit 206 .

[0108] When a request for features is received from another vehicle around the vehicle, the feature transmission control unit 206 controls the communication unit 103 to transmit the first feature to the third feature extracted, for example, by vehicle-to-vehicle communication, to the requesting other vehicle.

[0109] At this time, the feature amount transmission control unit 206 adds information for identifying each feature amount as the first feature amount, the second feature amount, or the third feature amount, and transmits the added information.

[0110] Furthermore, the feature transmission control unit 206 transmits the feature by adding information indicating a DNN (Deep Neural Network) structure of the local feature calculation unit 201 of the vehicle. Here, the information indicating the DNN structure is, for example, the values ​​of weighting coefficients for the MLPs 231, 241, and 251 corresponding to each point of the first feature calculation unit 221 to the third feature calculation unit 223, which are each layer in the DNN.

[0111] When there is another vehicle within a predetermined distance around the vehicle, the feature reception control unit 207 controls the communication unit 103 to request the other vehicle in the vicinity of the vehicle to send its features, and also acquires the features transmitted from the other vehicle in response to the request and supplies them to the feature selection unit 208.

[0112] The feature selection unit 208 selects necessary feature amounts from among the feature amounts from other companies present around the vehicle supplied by the feature reception control unit 207, and outputs the selected feature amounts to the feature combination unit 202. That is, the feature selection unit 208 selects only feature amounts within a predetermined time range from the current time, for example, and outputs the selected feature amounts to the feature combination unit 202.

[0113] In addition, the feature selection unit 208 selectively extracts only features that are identical to the DNN structure of the host vehicle based on information indicating the DNN structure of the other vehicles that is assigned to each of the features supplied from the other vehicles, and outputs them to the feature combination unit 202.

[0114] The feature combination unit 202 combines the features supplied from the feature selection unit 208 in this manner and the features according to the DNN structure of other vehicles, and outputs the combined features to the global feature calculation unit 203.

[0115] Here, when the feature combination unit 202 combines the third feature calculated from the features supplied from the feature selection unit 208 and the third feature supplied from the local feature calculation unit 201 by product-sum calculation using a weight, the feature combination unit 202 may adjust the weight.

[0116] For example, in a case where the DNN structures of the own vehicle and the other vehicle are different, the feature combination unit 202 may combine the features of the other vehicle by reducing the weight of the features of the own vehicle and by increasing the weight of the features of the own vehicle.

[0117] In addition, when the DNN structures of the host vehicle and the other vehicle are the same, the feature combination unit 202 may combine the feature amounts of the other vehicle and the feature amounts of the host vehicle by making the weights of the feature amounts of the other vehicle the same.

[0118] Furthermore, the first feature amount calculation unit 221, the second feature amount calculation unit 222, and the third feature amount calculation unit 223 constituting the local feature amount calculation unit 201 constitute a local processing structure that calculates the feature amount for each point.

[0119] Furthermore, the global feature calculation unit 203 configures a global processing structure that performs processing that does not affect the global feature R as a whole even if the order of the points is changed.

[0120] That is, the object recognition unit 181 in FIG. 3 of the present disclosure comprises a local processing structure that performs the above-mentioned processing for each point, and a global processing structure in which the processing results are not affected even if the order of the points is changed, and as a whole, a PointNet is formed.

[0121] In the object recognition unit 181 of Figure 3 of the present disclosure, the local processing structure calculates features for each point in the point cloud PC, and the global structure uses global features R consisting of the calculated features to perform object recognition processing for each point.

[0122] <Object Recognition Processing in the First Embodiment> Next, the object recognition process in the first embodiment by the object recognition unit 181 in FIG. 3 will be described with reference to the flowcharts in FIG. 4 and FIG.

[0123] In step S11, the first feature calculation unit 221 of the local feature calculation unit 201 acquires a point cloud PC supplied from the data acquisition unit 102, the point cloud PC being detected by surrounding information detection sensors such as an ultrasonic sensor, a radar, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), a stereo camera, and a sonar.

[0124] In step S 12 , the first feature amount calculation unit 221 calculates a first feature amount from the point cloud PC, and outputs it to the second feature amount calculation unit 222 and the first feature amount extraction unit 281 of the feature amount extraction unit 205 .

[0125] In step S13, the first feature extraction unit 281 extracts the first feature supplied from the first feature calculation unit 221, and outputs it to the feature transmission control unit 206. As a result, the feature transmission control unit 206 stores the supplied first feature with a timestamp corresponding to the current time of a system clock (not shown).

[0126] In step S14, the second feature calculation unit 222 acquires the first feature supplied from the first feature calculation unit 221, calculates a second feature from the first feature, and outputs it to the third feature calculation unit 223 and the second feature extraction unit 282 of the feature extraction unit 205.

[0127] In step S15, the second feature extraction unit 282 of the feature extraction unit 205 extracts the second feature supplied from the second feature calculation unit 222, and outputs it to the feature transmission control unit 206. As a result, the feature transmission control unit 206 stores the supplied second feature with a timestamp corresponding to the current time of a system clock (not shown).

[0128] In step S16, the third feature calculation unit 223 acquires the second feature supplied from the second feature calculation unit 222, calculates a third feature from the second feature, and outputs the third feature as a local feature to the feature combination unit 202 and the third feature extraction unit 283 of the feature extraction unit 205.

[0129] In step S17, the third feature extraction unit 283 of the feature extraction unit 205 extracts the third feature supplied from the third feature calculation unit 223, and outputs it to the feature transmission control unit 206. As a result, the feature transmission control unit 206 stores the supplied third feature with a timestamp corresponding to the current time of a system clock (not shown).

[0130] In step S18, the feature transmission control unit 206 controls the communication unit 103 to determine whether or not a feature request has been transmitted from another vehicle in the vicinity.

[0131] If it is determined in step S18 that a request for features has been transmitted from another vehicle, the process proceeds to step S19.

[0132] In step S19, the feature transmission control unit 206 receives a feature request transmitted from another vehicle.

[0133] In step S20, the feature amount transmission control unit 206 selects the first feature amount to the third feature amount that are closest to the current time from among the stored first feature amount to third feature amount.

[0134] In step S21, the feature amount transmission control unit 206 controls the communication unit 103 to transmit the first to third feature amounts for the selected time closest to the current time to the other vehicle that has requested the feature amounts.

[0135] At this time, the feature amount transmission control unit 206 transmits information on the vehicle's own position detected by the self-position estimation unit 132 together with information on the vehicle's attitude to the other vehicle that has requested the feature amount.

[0136] Furthermore, the information on the vehicle's own position and attitude is information for converting the position information of each point in the point cloud PC, which consists of a coordinate system (local coordinate system) in the vehicle 51, into a global coordinate system or the local coordinate system of another vehicle.

[0137] Therefore, when the position information of each point in the point cloud PC is expressed in the global coordinate system, information on the self-position and information on the self-orientation are not necessary.

[0138] Moreover, if it is determined in step S18 that there is no request for sending the feature amount, the processes in steps S19 to S21 are skipped and the feature amount is not sent to the other vehicles.

[0139] In step S22 (FIG. 5), the vehicle detection unit 182 detects a vehicle (other vehicle) approaching the host vehicle by communicating with vehicles around the host vehicle through vehicle-to-vehicle communication or the like.

[0140] In step S23, the feature amount reception control unit 207 determines, based on the detection result of the vehicle detection unit 182, whether or not there is another vehicle near the host vehicle within a predetermined distance from the host vehicle.

[0141] In step S23, for example, when other vehicles 31-2, 31-4 are present within a range indicated by a dotted line indicating a predetermined distance from the host vehicle 31-1 as shown in Fig. 1, it is determined that other vehicles are present near the host vehicle. When it is determined that other vehicles are present near the host vehicle as shown in Fig. 1, the process proceeds to step S24.

[0142] In step S24, the feature amount reception control unit 207 controls the communication unit 103 to transmit the feature amount to other vehicles within a predetermined distance from the vehicle.

[0143] As a result of this process, a request from the other vehicle is received in the above-mentioned step S18 in the object recognition process in the other vehicle, and therefore features are transmitted from the other vehicle to the own vehicle through the processes of steps S19 to S21.

[0144] Therefore, in step S 25 , the feature amount reception control unit 207 controls the communication unit 103 to obtain the feature amount transmitted from the other vehicle, as well as the information on the position and attitude of the other vehicle, and supplies this to the feature amount selection unit 208 .

[0145] In step S26, the feature selection unit 208 selects, from the supplied features, the feature whose time stamp is closest to the current time, and outputs the feature to the feature combination unit 202. At this time, the feature selection unit 208 also supplies to the feature combination unit 202 information on the position and attitude of the other vehicle that transmitted the selected feature.

[0146] In step S27, the feature combination unit 202 combines the features transmitted from the other vehicles and supplied from the feature selection unit 208 with the features transmitted from the local feature calculation unit 201, and outputs the combined result to the global feature calculation unit 203.

[0147] Here, for the first feature amount and the second feature amount among the feature amounts transmitted from the other vehicles, the feature amount combination unit 202 uses the built-in second feature amount calculation unit 271 and third feature amount calculation unit 272 to determine the third feature amount, and combines the third feature amount supplied from the local feature amount calculation unit 201 for each point.

[0148] At this time, the feature combination unit 202 sets weights according to the positional relationship between the other vehicle that transmitted the feature and the own vehicle and the DNN structure, and combines the third feature by product-sum calculation using the weights. In addition, the feature combination unit 202 converts the third feature from the other vehicle into its own coordinate system based on the information on the position and posture of the other vehicle, and combines the third feature from the other vehicle and the third feature calculated by the local feature calculation unit 201.

[0149] In step S28, the global feature calculation unit 203 calculates a global feature R based on the third feature supplied from the feature combination unit by, for example, Maxpooling, and outputs the global feature R to the recognition unit 204.

[0150] In step S29, the recognition unit 204 performs object recognition processing for each point in the global feature amount R, and outputs the object recognition result.

[0151] In step S30, the object recognition unit 181 determines whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S11 (FIG. 4), and the subsequent processes are repeated.

[0152] Also, in step S23, if it is determined that there is no vehicle 51 within a predetermined distance from the host vehicle and therefore no other vehicle is present, the process proceeds to step S31.

[0153] In step S31, the feature combination unit 202 outputs the third feature calculated by the local feature calculation unit 201 to the global feature calculation unit 203 as is.

[0154] In other words, if there is no other vehicle near the own vehicle, this means that there is no supply of features from the other vehicles, so the process proceeds to step S30, where a global feature using only the third feature calculated by the own vehicle alone is calculated, and object recognition processing is performed.

[0155] By the above series of processes, when another vehicle is present within a predetermined distance from the own vehicle, the feature quantities are requested from the other vehicle, and the feature quantities calculated in the object recognition process of the other vehicle are acquired in response to the request. The feature quantities calculated in the own vehicle and the acquired feature quantities of the other vehicle are then combined, and the feature quantities of the own vehicle and the other vehicle are used together to perform the object recognition process.

[0156] As a result, the sensing results of other vehicles and the sensing results of the own vehicle are used to determine features, making it possible to achieve object recognition processing with recognition accuracy equivalent to that achieved when the sensing results of the entire point cloud are used.

[0157] In the above, an example has been described in which all three types of feature amounts, the first feature amount, the second feature amount, and the third feature amount, are transmitted to another vehicle or transmitted from another vehicle; however, the types of feature amounts transmitted to another vehicle or transmitted from another vehicle may be only a part of them, for example, any one of the three types of feature amounts, the first feature amount, the second feature amount, and the third feature amount, or a combination of two of them.

[0158] Furthermore, the feature amounts transmitted and received may be randomly sampled in accordance with the communication load to limit the amount of data, thereby reducing the amount of data.

[0159] Furthermore, in the above, an example has been described in which features are acquired from other vehicles in the vicinity of the vehicle and combined with the vehicle's own features to perform object recognition processing. However, as long as features can be obtained from a point cloud, the vehicle may be other than the vehicle in the vicinity of the vehicle. For example, features calculated based on a point cloud acquired by a LiDAR or stereo camera installed on the roadside may be requested and acquired, and the features may be combined with the features calculated by the vehicle itself to perform object recognition processing.

[0160] In other words, the feature amount other than that of the own vehicle may be acquired from something other than the vehicle not only through the V2V communication described above but also through V2X communication.

[0161] Furthermore, in the object recognition process, the more features that are acquired, the higher the object recognition accuracy can be, so by having more other vehicles in the vicinity of the vehicle and acquiring features from more of those other vehicles, it is possible to further improve the object recognition accuracy.

[0162] Furthermore, since the point cloud PC may be any of LiDAR, stereo camera, and depth image, the object recognition unit 181 of the present disclosure can easily realize sensor fusion.

[0163] Furthermore, the input point cloud PC may be in various forms as long as point cloud information is formed, and may be, for example, three-dimensional point cloud information or two-dimensional point cloud information.

[0164] <<3. Modification of the First Embodiment>> In the above, an example has been described in which, when there is another vehicle near the own vehicle and there is a request for features from the other vehicle, the vehicle transmits the features and also requests features from itself, acquires the features from the other vehicle, and combines the features of the own vehicle and the features of the other vehicle to perform object recognition processing all at once. However, it is also possible to set conditions for the features requested of other vehicles, and acquire from the other vehicle only those features that meet the set conditions.

[0165] That is, for example, consider a case where the DNN structure of the own vehicle is different from the DNN structure of another vehicle, i.e., where the weighting coefficients etc. related to the calculations of the first feature calculation unit 221, the second feature calculation unit 222, and the third feature calculation unit 223 are different.

[0166] In such cases, the features obtained for the host vehicle and the other vehicle are not calculated under the same conditions, so even if the two are combined to perform object recognition, it may not necessarily be possible to improve recognition accuracy.

[0167] Therefore, when a vehicle requests features from another vehicle, information on the vehicle's DNN structure (such as the value of the weighting coefficient when calculating the features) is also transmitted as a condition for the requested features. Then, the other vehicle that receives the feature request transmits the features in response to the request only if the DNN structures match and the conditions are met.

[0168] In this way, only features that satisfy the conditions of a DNN structure identical to the vehicle's own DNN structure are transmitted from other vehicles, and the features of the vehicle and other vehicles calculated using the same DNN structure are combined for object recognition processing, thereby improving the accuracy of object recognition.

[0169] In the above, the case where the condition of the feature is a DNN structure has been described, but other conditions may be used, for example, information specifying any one or two of the first feature, the second feature, and the third feature, or information on a time range indicating the timing when the feature is calculated. In addition, the condition of the feature may be limited to features from other vehicles that are outside the field of view of the vehicle, or may be a condition that excludes features of other vehicles that are behind the vehicle, or may be limited to only features of other vehicles facing the vehicle.

[0170] Here, referring to the flowcharts of Figs. 6 and 7, an object recognition process will be described in which, when a vehicle requests features from another vehicle, conditions for the features to be requested from the other vehicle are set and the other vehicle transmits only those features that meet the set conditions.

[0171] Incidentally, the processes of steps S51 to S58, and steps S64, S65, and S71 to S74 in the flowcharts of FIGS. 6 and 7 are the same as the processes of steps S11 to S18, and steps S28 to S31 in the flowcharts of FIGS. 4 and 5, and therefore description thereof will be omitted.

[0172] That is, if it is determined in step S58 that a request for sending a feature has been sent, the process proceeds to step S59.

[0173] In step S59, the feature amount transmission control unit 206 receives the feature amount request transmitted from the other vehicle together with the feature amount conditions.

[0174] In step S60, the feature transmission control unit 206 determines whether or not there is a feature that meets the conditions.

[0175] In step S60, if there is a feature amount that meets the conditions, the process proceeds to step S61.

[0176] In step S61, the feature amount transmission control unit 206 selects a feature amount that meets the conditions from among the first to third feature amounts stored.

[0177] In step S62, the feature amount transmission control unit 206 controls the communication unit 103 to transmit the selected feature amount that meets the conditions to the other vehicle that has requested the feature amount. At this time, the feature amount transmission control unit 206 transmits information on the self-position detected by the self-position estimation unit 132 together with information on the self-attitude to the other vehicle that has requested the feature amount.

[0178] If there is no feature amount that meets the conditions in step S60, the process proceeds to step S63.

[0179] In step S63, the feature transmission control unit 206 controls the communication unit 103 to transmit a notification indicating that there is no feature that meets the conditions to the other vehicle that has requested the feature.

[0180] With the above process, when a feature is requested from another vehicle, if there is a feature that meets the conditions received together with the request, the feature that meets the conditions is transmitted to the other vehicle, and if there is no feature that meets the conditions, a notification that there is no feature that meets the conditions is given.

[0181] Furthermore, if there is another vehicle in the vicinity of the own vehicle, in step S66 (FIG. 7), the feature reception control unit 207 controls the communication unit 103 to transmit a notification requesting the feature together with information on the conditions of the feature to the other vehicles in the vicinity within a predetermined distance from the own vehicle.

[0182] In step S67, the feature amount reception control unit 207 determines whether or not a notification indicating that there is no feature amount that meets the conditions has been transmitted from the other vehicle that requested the feature amount.

[0183] In step S67, for example, if a feature that meets the conditions is transmitted from the other vehicle that requested the feature, a notification indicating that there is no feature that meets the conditions is not transmitted from the other vehicle that requested the feature, and the process proceeds to step S68.

[0184] In step S68, the feature reception control unit 207 controls the communication unit 103 to obtain feature amounts that meet the conditions transmitted from the other vehicle, as well as information on the position and attitude of the other vehicle, and supplies these to the feature selection unit 208.

[0185] In step S69, the feature selection unit 208 selects the feature that satisfies the conditions and that has the time stamp that is closest to the current time, and outputs the feature to the feature combination unit 202. At this time, the feature selection unit 208 also supplies the feature combination unit 202 with information on the position and attitude of the other vehicle that transmitted the selected feature.

[0186] In step S70, the feature combination unit 202 combines the features that meet the conditions transmitted from the other vehicles and that are supplied from the feature selection unit 208 with the features of the subject vehicle transmitted from the local feature calculation unit 201, and outputs the combined feature to the global feature calculation unit 203.

[0187] Also, in step S67, if a notification is received from the other vehicle that requested the feature amount to indicate that there is no feature amount that meets the conditions, the process proceeds to step S74.

[0188] In other words, in this case, it is notified that there are no features that meet the conditions, and no features are transmitted, so the object recognition process is performed using only the features calculated by its own local feature calculation unit 201.

[0189] However, if there are multiple other vehicles in the vicinity of the vehicle, only the feature values ​​transmitted from the other vehicles that meet the conditions are received and used for object recognition processing.

[0190] Through the above processing, only features that satisfy the conditions are transmitted from other vehicles, and the features calculated for the vehicle itself and other vehicles are combined using only the features that meet the conditions to perform object recognition processing, thereby making it possible to improve the accuracy of object recognition.

[0191] In the above, an example has been described in which vehicle 51 requests features from other vehicles in the vicinity along with conditions for the requested features, and the other vehicles that receive the request transmit any features that meet the conditions, or notify the other vehicles that there are no features that meet the conditions, if there are no features that meet the conditions.

[0192] However, when there is another vehicle nearby, vehicle 51 may request features from the other vehicle nearby, and the other vehicle that receives the request may transmit features regardless of the conditions. Vehicle 51 that requested the features may select and use those features that meet the conditions from among the received features, and discard the other features.

[0193] <<4. Second embodiment>> <Second configuration example of object recognition unit> In the above, an example has been described in which the object recognition unit 181 is configured by PointNet. However, other configurations may be used as long as the configuration realizes a DNN structure, and for example, the object recognition unit 181 may be configured by PointNet++.

[0194] Here, PointNet++, like PointNet, refers to a neural network that receives point cloud information as input and outputs a predetermined processing result based on the input point cloud information.

[0195] FIG. 8 is a diagram showing a second configuration example of the object recognition unit 181 configured using PointNet++.

[0196] The object recognition unit 181 in FIG. 8 includes a local feature calculation unit 301, a feature combination unit 302, a global feature calculation unit 303, a recognition unit 304, a feature extraction unit 305, a feature transmission control unit 306, a feature reception control unit 307, and a feature selection unit 308.

[0197] The local feature amount calculation unit 301 calculates the local feature amount for each point in the point cloud PC in a step-by-step manner, and includes a first feature amount calculation unit 321, a second feature amount calculation unit 322, and a third feature amount calculation unit 323.

[0198] The first feature calculation unit 321 calculates a first feature, which is a feature of the first stage, for each group of multiple points that make up the point cloud PC, and outputs the first feature to the second feature calculation unit 322, the first feature combination unit 341 of the feature combination unit 302, and the first feature extraction unit 381 of the feature extraction unit 305.

[0199] In this case, a group that is a unit for calculating the first feature amount by the first feature amount calculation unit 321 is referred to as a first hierarchical group.

[0200] The second feature calculation unit 322 calculates second features, which are second-stage features, for each group consisting of multiple first features among the first features obtained by the first feature calculation unit 321, and outputs the second features to the third feature calculation unit 323, the second feature combination unit 342 of the feature combination unit 302, and the second feature extraction unit 382 of the feature extraction unit 305.

[0201] In this case, a group that is a unit for calculating the second feature amount by the second feature amount calculation section 322 is referred to as a second hierarchical group.

[0202] The third feature calculation unit 323 calculates a third feature, which is a feature of the third stage, for each group consisting of multiple second features among the second features obtained by the second feature calculation unit 322, and outputs the third feature to a third feature combination unit 343 of the feature combination unit 302 and a third feature extraction unit 383 of the feature extraction unit 305.

[0203] In this case, a group that is a unit for calculating the third feature amount by the third feature amount calculation unit 323 is called a third hierarchical group.

[0204] The feature combination unit 302 combines the first feature, the second feature, and the third feature supplied from other vehicles via the communication unit 103, the feature reception control unit 307, and the feature selection unit 308 with the first feature, the second feature, and the third feature supplied from the local feature calculation unit 301, and outputs the combined result to the global feature calculation unit 303.

[0205] More specifically, the feature amount combining unit 302 includes a first feature amount combining unit 341 , a second feature amount combining unit 342 , and a third feature amount combining unit 343 .

[0206] The first feature combination unit 341 combines the first feature supplied from the first feature calculation unit 321 and the first feature supplied from other vehicles supplied from the feature selection unit 308, and outputs the combined result to the second feature calculation unit 362 of the global feature calculation unit 303.

[0207] The second feature combination unit 342 combines the second feature supplied from the second feature calculation unit 322 with the second feature supplied from other vehicles supplied from the feature selection unit 308, and outputs the combined result to a third feature calculation unit 361 of the global feature calculation unit 303.

[0208] The third feature combination unit 343 combines the third feature supplied from the third feature calculation unit 323 and the third feature supplied from other vehicles supplied from the feature selection unit 308, and outputs the combined result to the third feature calculation unit 361 of the global feature calculation unit 303.

[0209] The global feature calculation unit 303 calculates and outputs a global feature R at each point based on the first feature, the second feature, and the third feature supplied from the feature combination unit 202.

[0210] More specifically, the global feature amount calculation unit 303 includes a third feature amount calculation unit 361 , a second feature amount calculation unit 362 , and a first feature amount calculation unit 363 .

[0211] The third feature calculation unit 361 reconstructs a feature consisting of a point group in second hierarchical group units based on the third feature supplied from the third feature combination unit 343 and the second feature supplied from the second feature combination unit 342, and outputs the reconstruction result to the second feature calculation unit 362 as a global feature in second hierarchical group units.

[0212] More specifically, the third feature amount calculation unit 361 expands the third feature amount supplied from the third feature amount combination unit 343 by using the second feature amount for each second hierarchical group supplied from the second feature amount combination unit 342, and restores it as a second feature amount that is a feature amount of a point cloud for each second hierarchical group. Then, the third feature amount calculation unit 361 outputs the restoration result to the second feature amount calculation unit 362 as a global feature amount R for each second hierarchical group.

[0213] The second feature calculation unit 362 reconstructs the features for the first hierarchical group unit based on the global feature R for the second hierarchical group unit supplied by the third feature calculation unit 361 and the first feature supplied by the first feature combination unit 341, and outputs the reconstruction result to the first feature calculation unit 363 as the global feature R for the first hierarchical group unit.

[0214] More specifically, the second feature amount calculation unit 362 expands the second feature amount supplied from the third feature amount calculation unit 361 by using the first feature amount for each first hierarchical group supplied from the first feature amount combination unit 341, and restores it as a first feature amount that is a feature amount of a point cloud for each first hierarchical group. Then, the second feature amount calculation unit 362 outputs the restoration result to the first feature amount calculation unit 363 as a global feature amount R for each first hierarchical group.

[0215] The first feature amount calculation unit 363 restores the feature amount for each point based on the global feature amount R for each first hierarchical group and the point cloud PC supplied from the second feature amount calculation unit 362, and outputs the restoration result to the recognition unit 304 as the global feature amount R of the object. The recognition unit 304 executes object recognition processing based on the global feature amount R, and outputs the object recognition result.

[0216] More specifically, the first feature amount calculation unit 363 expands the first feature amount supplied from the second feature amount calculation unit 362 by using the point cloud PC, and restores it as a global feature amount R based on the feature amount of the point group on a point-by-point basis. Then, the first feature amount calculation unit 363 outputs the restored result to the recognition unit 304 as a global feature amount R on a point-by-point basis.

[0217] That is, in the object recognition unit 181 of FIG. 8 of the present disclosure, the local feature calculation unit 301 and the global feature calculation unit 303 form PointNet++ using DNN.

[0218] That is, the first feature amount calculation unit 321, the second feature amount calculation unit 322, and the third feature amount calculation unit 323 in the local feature amount calculation unit 301 essentially constitute an input layer, a hidden layer, and an output layer in a DNN (Deep Neural Network).

[0219] For this reason, the local feature calculation unit 301 in FIG. 8 is configured to calculate features using a three-stage hierarchical structure like the first feature calculation unit 321 to the third feature calculation unit 323, but the number of layers may be other than two or three as long as the hierarchical structure has at least two layers corresponding to the input layer and the output layer.

[0220] The local feature calculation unit 301 and the global feature calculation unit 303 receive, for example, the position (x, y, z) and color (r, g, b) of a point cloud as input, and perform learning (backpropagation) using the semantic segmentation label as the correct answer.

[0221] The feature extraction unit 305 extracts features that are the calculation results of the local feature calculation unit 301 , and supplies them to a feature transmission control unit 306 .

[0222] More specifically, the feature amount extraction section 305 includes a first feature amount extraction section 381, a second feature amount extraction section 382, ​​and a third feature amount extraction section 383.

[0223] The first feature extraction unit 381 extracts a first feature that is the calculation result of the first feature calculation unit 351 of the local feature calculation unit 301 , and outputs it to the feature transmission control unit 306 .

[0224] The second feature extraction unit 382 extracts a second feature that is the calculation result of the second feature calculation unit 352 of the local feature calculation unit 301 , and outputs it to the feature transmission control unit 306 .

[0225] The third feature extraction unit 383 extracts a third feature which is the calculation result of the third feature calculation unit 353 of the local feature calculation unit 301 , and outputs it to the feature transmission control unit 306 .

[0226] When a request for features is received from another vehicle around the vehicle, the feature transmission control unit 306 controls the communication unit 103 to transmit the first feature to the third feature extracted by, for example, vehicle-to-vehicle communication to the other vehicle that has made the request.

[0227] At this time, the feature amount transmission control unit 306 adds information for identifying each feature amount as the first feature amount, the second feature amount, or the third feature amount, and transmits the added information.

[0228] In addition, the feature transmission control unit 306 adds information indicating a deep neural network (DNN) structure of the local feature calculation unit 301 of the host vehicle to the feature and transmits it.

[0229] When there is another vehicle in the vicinity of the vehicle, the feature reception control unit 307 controls the communication unit 103 to request the other vehicle in the vicinity of the vehicle to send its feature, and also acquires the feature transmitted from the other vehicle in response to the request and supplies it to the feature selection unit 308.

[0230] The feature selection unit 308 extracts only the features from other companies around the vehicle supplied by the feature reception control unit 307, for example, those within a predetermined time range from the current time, and outputs them to the feature combination unit 302.

[0231] In addition, the feature selection unit 308 selectively extracts only features that are identical to the DNN structure of the host vehicle based on information indicating the DNN structure of the other vehicles that is assigned to each of the features supplied from the other vehicles, and outputs them to the feature combination unit 302.

[0232] More specifically, the feature selection unit 308 outputs, of the selected features, the first feature to the first feature combination section 341 of the feature combination unit 302, the second feature to the second feature combination section 342, and the third feature to the third feature combination section 343.

[0233] The first feature combination unit 341, the second feature combination unit 342, and the third feature combination unit 343 of the feature combination unit 302 may adjust weights when combining the first feature, the second feature, and the third feature, respectively, supplied from the feature selection unit 308 with the first feature, the second feature, and the third feature, respectively, in accordance with the DNN structure of the other vehicle.

[0234] In other words, when the DNN structures of the own vehicle and the other vehicle are different, the feature combination unit 202 may combine the features of the other vehicle by reducing the weight of the features of the own vehicle and by increasing the weight of the features of the other vehicle.

[0235] In addition, when the DNN structures of the own vehicle and the other vehicle are the same, the feature combination unit 202 combines the feature amounts of the other vehicle and the own vehicle by making the weights of the feature amounts the same.

[0236] In addition, the first feature calculation unit 351, the second feature calculation unit 352, and the third feature calculation unit 353 that constitute the local feature calculation unit 301 constitute a local processing structure that calculates features for each of the first hierarchical group, the second hierarchical group, and the third hierarchical group, respectively, based on points in the point cloud PC.

[0237] Furthermore, the global feature calculation unit 303 configures a global processing structure that performs processing that does not affect the global feature R as a whole even if the order of the points is changed.

[0238] In other words, the object recognition unit 181 in FIG. 8 of the present disclosure comprises a local processing structure that performs the processing for each point described above, and a global processing structure in which the processing results are not affected even if the order of the points is changed, and as a whole, PointNet++ is formed.

[0239] In the object recognition unit 181 of Figure 8 of the present disclosure, processing is performed using a global processing structure in which features are calculated by processing each of the first hierarchical group, the second hierarchical group, and the third hierarchical group based on points in the point cloud PC using a local processing structure.

[0240] <Object Recognition Processing in the Second Embodiment> Next, the object recognition process in the second embodiment by the object recognition unit 181 in FIG. 8 will be described with reference to the flowcharts in FIG. 9 and FIG.

[0241] In step S91, the first feature amount calculation unit 221 of the local feature amount calculation unit 201 acquires a point cloud PC obtained by a stereo camera, a LiDAR, or the like, supplied from the data acquisition unit 102.

[0242] In step S92, the first feature calculation unit 321 calculates a first feature, which is a feature for each first hierarchical group, from the point cloud PC, and outputs it to the second feature calculation unit 322, the first feature combination unit 341 of the feature combination unit 302, and the first feature extraction unit 381 of the feature extraction unit 305.

[0243] In step S93, the first feature extraction unit 381 extracts the first feature supplied from the first feature calculation unit 321, and outputs it to the feature transmission control unit 306. As a result, the feature transmission control unit 306 stores the supplied first feature with a timestamp corresponding to the current time of a system clock (not shown).

[0244] In step S94, the second feature calculation unit 322 acquires the first feature supplied from the first feature calculation unit 321, calculates a second feature, which is a feature for each second hierarchical group, from the first feature, and outputs the second feature to the third feature calculation unit 323, the second feature combination unit 342 of the feature combination unit 302, and the second feature extraction unit 382 of the feature extraction unit 305.

[0245] In step S95, the second feature extraction unit 382 of the feature extraction unit 305 extracts the second feature supplied from the second feature calculation unit 322, and outputs it to the feature transmission control unit 306. As a result, the feature transmission control unit 306 stores the supplied second feature with a timestamp corresponding to the current time of a system clock (not shown).

[0246] In step S96, the third feature calculation unit 323 acquires the second feature supplied from the second feature calculation unit 322, calculates a third feature, which is a feature for each third hierarchical group, from the second feature, and outputs the third feature to a third feature combination unit 343 of the feature combination unit 302 and a third feature extraction unit 383 of the feature extraction unit 305.

[0247] In step S97, the third feature extraction unit 383 of the feature extraction unit 305 extracts the third feature supplied from the third feature calculation unit 323, and outputs it to the feature transmission control unit 306. As a result, the feature transmission control unit 306 stores the supplied third feature with a timestamp corresponding to the current time of a system clock (not shown).

[0248] In step S98, the feature transmission control unit 306 controls the communication unit 103 to determine whether or not a feature request has been transmitted from another vehicle in the vicinity.

[0249] If it is determined in step S98 that a feature request has been transmitted from another vehicle, the process proceeds to step S99.

[0250] In step S99, the feature transmission control unit 306 receives a feature request transmitted from another vehicle.

[0251] In step S100, the feature amount transmission control unit 306 selects the first feature amount to the third feature amount that are closest to the current time from among the stored first feature amount to third feature amount.

[0252] In step S101, the feature amount transmission control unit 306 controls the communication unit 103 to transmit the first to third feature amounts for the selected time closest to the current time to the other vehicle that has requested the feature amounts. At this time, the feature amount transmission control unit 306 transmits information on the vehicle's own position detected by the self-position estimation unit 132 together with information on the vehicle's own attitude to the other vehicle that has requested the feature amounts.

[0253] If it is determined in step S98 that there is no request for sending the feature amount, the processes in steps S99 to S101 are skipped and the feature amount is not sent to the other vehicles.

[0254] In step S102 (FIG. 10), the vehicle detection unit 182 detects vehicles approaching the host vehicle by communicating with vehicles (other vehicles) around the host vehicle through vehicle-to-vehicle communication or the like.

[0255] In step S103, the feature amount reception control unit 307 determines, based on the detection result of the vehicle detection unit 182, whether or not there is another vehicle nearby within a predetermined distance from the host vehicle.

[0256] If it is determined in step S103 that another vehicle is present near the host vehicle, the process proceeds to step S104.

[0257] In step S104, the feature amount reception control unit 307 controls the communication unit 103 to transmit the feature amount to other vehicles in the vicinity of the own vehicle.

[0258] As a result of this process, in the object recognition process in the other vehicle, a request from the other vehicle is received in the above-mentioned step S98, and therefore features are transmitted from the other vehicle to the host vehicle through the processes in steps S99 to S101.

[0259] Therefore, in step S 105 , the feature amount reception control unit 307 controls the communication unit 103 to acquire the feature amount transmitted from the other vehicle, as well as the position and attitude information of the other vehicle, and supplies them to the feature amount selection unit 308 .

[0260] In step S106, the feature selection unit 308 selects, from the supplied feature amounts, the feature amount having the time stamp attached thereto that is closest to the current time, and outputs the selected feature amount to the feature combination unit 302. At this time, the feature selection unit 308 also supplies, to the feature combination unit 302, information on the position and attitude of the other vehicle that transmitted the selected feature amount.

[0261] More specifically, of the supplied features, the feature selection unit 308 outputs the first feature to the first feature combination unit 341, the second feature to the second feature combination unit 342, and the third feature to the third feature combination unit 343.

[0262] In step S107, the first feature combination unit 341 of the feature combination unit 302 combines the first feature transmitted from the other vehicles and supplied from the feature selection unit 308 with the first feature transmitted from the first feature calculation unit 321 of the local feature calculation unit 301, and outputs the combined result to the second feature calculation unit 362 of the global feature calculation unit 303.

[0263] In addition, the second feature combination unit 342 and the third feature combination unit 343 of the feature combination unit 302 combine the second feature and the third feature transmitted from other vehicles supplied from the feature selection unit 308 with the second feature and the third feature transmitted from the second feature calculation unit 322 and the third feature calculation unit 323, respectively, and output the combined results to the third feature calculation unit 361 of the global feature calculation unit 303.

[0264] At this time, the first feature combination unit 341, the second feature combination unit 342, and the third feature combination unit 343 of the feature combination unit 302 set weights according to the positional relationship between the host vehicle and other vehicles that have transmitted features and the DNN structure, and combine the first feature, the second feature, and the third feature, respectively, by product-sum calculation using the weights.

[0265] In step S108, the third feature calculation unit 361 of the global feature calculation unit 303 calculates a global feature R for each third hierarchical group based on the third feature supplied from the third feature combination unit 343, expands it to a feature for each second hierarchical group using the second feature supplied from the second feature combination unit 342, and outputs it to the second feature calculation unit 362 as the global feature R for each second hierarchical group.

[0266] In step S109, the second feature calculation unit 362 of the global feature calculation unit 303 expands the global feature R of the second hierarchical group unit supplied from the third feature calculation unit 361 to a feature of the first hierarchical group unit using the first feature supplied from the first feature combination unit 341, and outputs it to the first feature calculation unit 363 as the global feature R of the first hierarchical group unit.

[0267] In step S110, the first feature calculation unit 363 of the global feature calculation unit 303 expands the global feature R in units of first hierarchical groups supplied from the second feature calculation unit 362 to features in units of points in the point cloud PC using the point cloud PC, and outputs it to the recognition unit 304 as the global feature R in units of points.

[0268] In step S111, the recognition unit 304 performs object recognition processing for each point in the point-by-point global feature R, and outputs the object recognition result.

[0269] In step S112, the object recognition unit 181 determines whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S91 (FIG. 9), and the subsequent processes are repeated.

[0270] Also, in step S103, if there is no vehicle within the predetermined distance from the host vehicle and it is determined that there is no other vehicle, the process proceeds to step S113.

[0271] In step S113, the first feature combination unit 341, the second feature combination unit 342, and the third feature combination unit 343 of the feature combination unit 302 output the first feature, the second feature, and the third feature calculated by the first feature calculation unit 321, the second feature calculation unit 322, and the third feature calculation unit 323, respectively, of the local feature calculation unit 301 directly to the third feature calculation unit 361 and the second feature calculation unit 362 of the global feature calculation unit 303.

[0272] In other words, if there is no other vehicle near the own vehicle, this means that no features are being supplied from the other vehicles, so the process proceeds to step S112, where object recognition processing is performed using the first feature amount, the second feature amount, and the third feature amount by the own vehicle alone, using only the calculated first feature amount, the second feature amount, and the third feature amount.

[0273] Through the above series of processes, even in the object recognition unit 181 configured by PointNet++ as shown in Fig. 8, when another vehicle is present in the vicinity within a predetermined distance of the own vehicle, a feature amount is requested from the other vehicle, and the feature amount calculated in the object recognition process of the other vehicle is acquired in response to the request. Then, the feature amount calculated in the own vehicle and the acquired feature amount of the other vehicle are combined, and the object is recognized by a collective object recognition process that combines the feature amount calculated by the own vehicle and the feature amount obtained by the other vehicle.

[0274] As a result, the sensing results of other vehicles and the sensing results of the own vehicle are used to determine features, making it possible to achieve object recognition processing with recognition accuracy equivalent to that achieved when the sensing results of the entire point cloud are used.

[0275] In the object recognition unit 181 of FIG. 8 as well, as in the modified example of the object recognition unit 181 of FIG. 3, when a feature is requested from another vehicle, conditions for the feature may also be transmitted so that only feature values ​​that meet the conditions are transmitted from the other vehicle, or features may be requested without any conditions, and only those feature values ​​that meet the conditions may be selected from the received feature values ​​and the rest may be discarded.

[0276] As for the modified example of the second embodiment, similarly to the object recognition processing in the modified example of the first embodiment, the only difference from the second embodiment is that when requesting features from other vehicles, conditions for the features are also transmitted, and only features that meet the conditions are transmitted from the other vehicles; however, as the explanation of this point is the same, it will be omitted.

[0277] 8 calculate features in groups for each hierarchy, the information amount of the features calculated in the order of the first feature amount, the second feature amount, and the third feature amount decreases. Therefore, when obtaining features from another vehicle, by selectively transmitting and receiving features in a higher hierarchy, it is possible to reduce the communication load of vehicle-to-vehicle communication and realize high-speed communication.

[0278] In other words, when requesting features from another vehicle, for example, by requesting only the second feature rather than only the first feature, the amount of information on the features to be acquired can be reduced, reducing the communication load and speeding up the processing related to acquiring the features.

[0279] Furthermore, when requesting features from another vehicle, by requesting only the third feature rather than only the second feature, the communication load can be further reduced and the processing related to obtaining the features can be further accelerated.

[0280] <<5. Example of execution by software>> The above-mentioned series of processes can be executed by hardware, but can also be executed by software. When the series of processes are executed by software, the programs constituting the software are installed from a recording medium into a computer built into dedicated hardware, or into, for example, a general-purpose computer capable of executing various functions by installing various programs.

[0281] 11 shows an example of the configuration of a general-purpose computer. This personal computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0282] An input unit 1006 consisting of input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 for outputting a processing operation screen and an image of the processing result to a display device, a storage unit 1008 consisting of a hard disk drive for storing programs and various data, and a communication unit 1009 consisting of a LAN (Local Area Network) adapter and the like for executing communication processing via a network such as the Internet are connected to the input / output interface 1005. Also connected is a drive 1010 for reading and writing data from and to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0283] The CPU 1001 executes various processes according to a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory and installed in a storage unit 1008, and loaded from the storage unit 1008 to a RAM 1003. The RAM 1003 also stores data and the like necessary for the CPU 1001 to execute various processes, as appropriate.

[0284] In a computer configured as described above, the CPU 1001 loads a program stored in the memory unit 1008, for example, into the RAM 1003 via the input / output interface 1005 and the bus 1004, and executes the program, thereby performing the series of processes described above.

[0285] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0286] In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by mounting the removable storage medium 1011 in the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be pre-installed in the ROM 1002 or the storage unit 1008.

[0287] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0288] 11 realizes the functions of the object recognition unit 181 in FIGS.

[0289] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0290] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure.

[0291] For example, the present disclosure can take the form of cloud computing in which a single function is shared and processed collaboratively by multiple devices via a network.

[0292] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0293] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0294] The present disclosure can also be configured as follows.

[0295] <1> a feature amount calculation unit that calculates a feature amount used for object recognition in a stepwise manner in each of a plurality of layers; an acquisition unit that acquires a feature amount used for the object recognition, the feature amount being calculated by another feature amount calculation unit different from the feature amount calculation unit; a recognition unit that performs the object recognition based on the calculated feature amount and the acquired feature amount; An information processing device comprising: <2> The feature amount calculation unit is provided in a moving body, and the other feature amount calculation unit is provided in another moving body different from the moving body. <1> The information processing device described in <3> The recognition unit performs the object recognition based on feature amounts of a plurality of hierarchical levels calculated in two or more stages by the feature amount calculation unit. <1> or <2> The information processing device described in <4> The acquisition unit acquires feature quantities of two or more hierarchical levels from among the feature quantities of the plurality of hierarchical levels calculated stepwise by the other feature quantity calculation unit. <1> or <2> The information processing device described in <5> The acquisition unit acquires a feature quantity of a specific layer from among the feature quantities of the plurality of layers calculated by the other feature quantity calculation unit. <1> or <2> The information processing device described in <6> The acquisition unit acquires, from among the features calculated by the other feature calculation units, features calculated by the other feature calculation units in a layer having the same neural network structure as that of the feature calculation unit. <2> The information processing device described in <7> When a distance between the moving object and the other moving object is within a predetermined distance, the acquisition unit requests, from the other moving object, a feature calculated by the other feature calculation unit of a hierarchical layer having the same neural network structure as that of the feature calculation unit; The acquisition unit acquires, in response to the feature request, a feature calculated by the other feature calculation unit at a layer having the same neural network structure as that of the feature calculation unit, from among the feature calculated by the other feature calculation unit transmitted from the other moving object. <6> The information processing device described in <8> The acquisition unit requests the other moving body for a feature calculated by the other feature calculation unit in a hierarchical layer having the same neural network structure as the feature calculation unit, and acquires, from the feature calculated by the other feature calculation unit transmitted from the other moving body in response to the feature request, the feature calculated by the other feature calculation unit in a hierarchical layer having the same neural network structure as the feature calculation unit. <7> The information processing device described in <9> The acquisition unit requests the other moving body for the features calculated by the other feature calculation unit, and in response to the feature request, extracts and acquires the feature calculated by the other feature calculation unit in a layer having the same neural network structure as that of the feature calculation unit from the feature calculated by the other feature calculation unit transmitted from the other moving body, and discards the feature calculated by the other feature calculation unit in a layer not having the same neural network structure as that of the feature calculation unit. <7> The information processing device described in <10> a combining unit for combining the calculated feature quantity and the acquired feature quantity; a global feature calculation unit that calculates a global feature based on the feature combined by the combination unit, The recognition unit performs the object recognition based on the feature amount calculated by the global feature amount calculation unit. <1> ~ <9> 2. An information processing device according to claim 1 . <11> In the feature amount calculation unit and the global feature amount calculation unit, the combining unit combines the feature amount of a final hierarchy calculated stepwise by the feature amount calculation unit with the feature amount of a final hierarchy calculated stepwise by the other feature amount calculation unit; The global feature calculation unit calculates the global feature based on a feature obtained by combining the calculated feature of the final layer and the acquired feature of the final layer by the combining unit. <10> The information processing device described in <12> The feature amount calculation unit and the global feature amount calculation unit form a PointNet structure. <11> The information processing device described in <13> In the feature amount calculation unit and the global feature amount calculation unit, the combining unit combines, for each layer, the feature amount for each layer calculated stepwise by the feature amount calculation unit and the feature amount for each layer calculated stepwise by the other feature amount calculation unit; The global feature calculation unit calculates the global feature based on a feature obtained by combining the calculated feature for each hierarchical layer and the acquired feature for each hierarchical layer by the combining unit for each hierarchical layer. <10> The information processing device described in <14> The feature amount calculation unit and the global feature amount calculation unit form a PointNet++ structure. <13> The information processing device described in <15> the feature amount calculation unit has a local processing structure for performing local processing for each point in input information consisting of a point cloud, and calculates a feature amount used for the object recognition in a stepwise manner in each of the plurality of hierarchical levels; The global feature calculation unit has a global processing structure that performs global processing that is not affected by the order of the points, and calculates the global feature based on the feature calculated by the feature calculation unit and the acquired feature. <10> The information processing device described in <16> A feature amount calculation unit; An acquisition unit; Recognition part and An information processing method of an information processing device comprising: the feature amount calculation unit calculates a feature amount used for object recognition in a stepwise manner in each of a plurality of hierarchical levels; the acquisition unit acquires a feature amount used for the object recognition, the feature amount being calculated by another feature amount calculation unit different from the feature amount calculation unit; The recognition unit performs the object recognition based on the calculated feature amount and the acquired feature amount. An information processing method comprising the steps of: <17> a feature amount calculation unit that calculates a feature amount used for object recognition in a stepwise manner in each of a plurality of layers; an acquisition unit that acquires a feature amount used for the object recognition, the feature amount being calculated by another feature amount calculation unit different from the feature amount calculation unit; a recognition unit that performs the object recognition based on the calculated feature amount and the acquired feature amount; programs that make a computer function. [Explanation of symbols]

[0296] 11 object recognition system, 31,51 vehicle, 141 outside vehicle information detection unit, 181 object recognition unit, 182 vehicle detection unit, 201 local feature calculation unit, 202 feature combination unit, 203 global feature calculation unit, 204 recognition unit, 205 feature extraction unit, 206 feature transmission control unit, 207 feature reception control unit, 208 feature selection unit, 221 first feature calculation unit, 222 second feature calculation unit, 223 third feature calculation unit, 271 second feature calculation unit, 272 third feature calculation unit, 281 first feature extraction unit, 282 second feature extraction unit, 283 third feature extraction unit, 301 feature calculation unit, 302 feature combination unit, 303 global feature calculation unit, 304 recognition unit, 305 feature extraction unit, 306 feature transmission control unit, 307 feature reception control unit, 308 feature selection unit, 321 first feature calculation unit, 322 second feature calculation unit, 323 third feature calculation unit, 361 third feature calculation unit, 362 second feature calculation unit, 363 first feature calculation unit, 381 first feature extraction unit, 382 second feature extraction unit, 383 third feature extraction unit

Claims

1. A feature amount calculation unit that calculates feature amounts to be used for object recognition in each of a plurality of hierarchical levels based on a sensing result detected by an ambient information detection sensor; an acquisition unit that acquires another feature amount, different from the feature amount calculated by another feature amount calculation unit different from the feature amount calculation unit, for use in the object recognition, based on another sensing result detected by another surrounding information detection sensor different from the surrounding information detection sensor; a recognition unit that performs the object recognition based on the feature amount and the other feature amount, The surrounding information detection sensor and the other surrounding information detection sensor are provided on different objects. Information processing device.

2. The surrounding information detection sensor and the feature amount calculation unit are provided on a moving body, and the other surrounding information detection sensor and the other feature amount calculation unit are provided on another moving body different from the moving body. The information processing device according to claim 1 .

3. The recognition unit performs the object recognition based on feature amounts of a plurality of hierarchical levels calculated in two or more stages by the feature amount calculation unit. The information processing device according to claim 1 .

4. The acquisition unit acquires, as the other feature amount, feature amounts of two or more hierarchical levels from among the feature amounts of the plurality of hierarchical levels calculated stepwise by the other feature amount calculation unit. The information processing device according to claim 1 .

5. The acquisition unit acquires, as the other feature amount, a feature amount of a specific layer from among the feature amounts of the plurality of layers calculated by the other feature amount calculation unit. The information processing device according to claim 1 .

6. The acquisition unit acquires, from among the features calculated by the other feature calculation units, a feature calculated by the other feature calculation unit in a layer having the same neural network structure as that of the feature calculation unit as the other feature. The information processing device according to claim 2 .

7. When the distance between the moving body and the other moving body is within a predetermined distance, the acquisition unit requests the other moving body to transmit features calculated by the other feature calculation unit in a hierarchical layer having the same neural network structure as the feature calculation unit, and acquires, as the other feature, the feature calculated by the other feature calculation unit in a hierarchical layer having the same neural network structure as the feature calculation unit among the features calculated by the other feature calculation unit transmitted from the other moving body in response to the feature request. The information processing device according to claim 6.

8. The acquisition unit requests the other moving body for feature amounts calculated by the other feature amount calculation unit in a hierarchical layer having the same neural network structure as that of the feature amount calculation unit, and acquires, as the other feature amounts, the feature amounts calculated by the other feature amount calculation unit in a hierarchical layer having the same neural network structure as that of the feature amount calculation unit, among the feature amounts calculated by the other feature amount calculation unit transmitted from the other moving body in response to the request for the feature amounts. The information processing device according to claim 7.

9. The acquisition unit requests the other moving body for the features calculated by the other feature calculation unit, and in response to the feature request, extracts the feature calculated by the other feature calculation unit in a layer having the same neural network structure as the feature calculation unit from the features calculated by the other feature calculation unit transmitted from the other moving body, and acquires it as the other feature, and discards the feature calculated by the other feature calculation unit in a layer not having the same neural network structure as the feature calculation unit. The information processing device according to claim 7.

10. A combining unit that combines the feature amount and the other feature amount; a global feature calculation unit that calculates a global feature based on the feature combined by the combination unit, The recognition unit performs the object recognition based on the feature amount calculated by the global feature amount calculation unit. The information processing device according to claim 1 .

11. In the feature amount calculation unit and the global feature amount calculation unit, the combining unit combines the feature amount of a final hierarchy calculated stepwise by the feature amount calculation unit with the feature amount of a final hierarchy calculated stepwise by the other feature amount calculation unit; The global feature calculation unit calculates the global feature based on a feature obtained by combining the calculated feature of the final layer and the acquired feature of the final layer by the combining unit. The information processing device according to claim 10.

12. The feature amount calculation unit and the global feature amount calculation unit form a PointNet structure. The information processing device according to claim 11.

13. In the feature amount calculation unit and the global feature amount calculation unit, the combining unit combines, for each layer, the feature amount for each layer calculated stepwise by the feature amount calculation unit and the feature amount for each layer calculated stepwise by the other feature amount calculation unit; The global feature calculation unit calculates the global feature based on a feature obtained by combining the calculated feature for each hierarchical layer and the acquired feature for each hierarchical layer by the combining unit for each hierarchical layer. The information processing device according to claim 10.

14. The feature amount calculation unit and the global feature amount calculation unit form a PointNet++ structure. The information processing device according to claim 13.

15. the feature amount calculation unit has a local processing structure for performing local processing for each point in input information consisting of a point cloud, and calculates feature amounts used for the object recognition in a stepwise manner in each of the plurality of hierarchical levels; The global feature calculation unit has a global processing structure that performs global processing that is not affected by the order of the points, and calculates the global feature based on the feature calculated by the feature calculation unit and the acquired feature. The information processing device according to claim 10.

16. A feature amount calculation unit; An acquisition unit; Recognition part and An information processing method of an information processing device comprising: the feature amount calculation unit performs a calculation process to calculate feature amounts used for object recognition in a stepwise manner in each of a plurality of hierarchical levels based on a sensing result detected by an ambient information detection sensor; an acquisition process in which the acquisition unit acquires another feature amount different from the feature amount used for the object recognition, the feature amount being calculated by another feature amount calculation unit different from the feature amount calculation unit, based on another sensing result detected by another surrounding information detection sensor different from the surrounding information detection sensor; The recognition unit performs an object recognition process to recognize the object based on the feature amount and the other feature amount, The surrounding information detection sensor and the other surrounding information detection sensor are provided on different objects. Information processing methods.

17. A feature amount calculation unit that calculates feature amounts used for object recognition in a stepwise manner in each of a plurality of hierarchical levels based on a sensing result detected by an ambient information detection sensor; an acquisition unit that acquires another feature amount different from the feature amount used for the object recognition, the feature amount being calculated by another feature amount calculation unit different from the feature amount calculation unit, based on another sensing result detected by another surrounding information detection sensor different from the surrounding information detection sensor; causing a computer to function as a recognition unit that performs the object recognition based on the feature amount and the other feature amount; The surrounding information detection sensor and the other surrounding information detection sensor are provided on different objects. program.

Images