Information processing device, information processing method, computer program, and mobile device

Abstract

An information processing device is provided that processes detection information from a sensor for identifying an external environment. The information processing device includes: a recognition unit that performs object recognition processing based on the detection signal from the sensor; and a processing unit that performs fusion processing on first data before recognition by the recognition unit with other data. The information processing device also includes a second recognition unit that performs object recognition processing based on the detection signal from the second sensor. The processing unit specifically performs fusion processing on third data before recognition by the second recognition unit with the first data, and on fourth data after recognition by the second recognition unit with the first data.

Description

Technical Field

[0001] The technology disclosed in this specification relates to an information processing device, an information processing method, a computer program, and a mobile device that process detection information by a plurality of sensors mainly for recognizing the external world. Background Art

[0002] To achieve autonomous driving and advanced driver assistance systems (ADAS), it is necessary to detect various objects, such as other vehicles, people, and lanes. Furthermore, object detection is necessary not only during daytime and in good weather, but also in various environments, such as rainy weather and at night. To this end, many different types of exterior recognition sensors, such as cameras, millimeter-wave radar, and lidar, have begun to be installed in vehicles.

[0003] Each sensor has its strengths and weaknesses. A sensor's recognition performance can deteriorate depending on factors such as the type and size of the object being detected, the distance to the object, and the weather conditions at the time of detection. For example, automotive radars have high range and relative velocity accuracy, but low angular accuracy and lack the ability to identify object types, or have low recognition accuracy. Cameras, on the other hand, have relatively low range and relative velocity accuracy, but good angular accuracy and recognition accuracy.

[0004] Therefore, not only using each sensor individually but also combining two or more sensors to utilize the characteristics of each sensor contributes to more accurate external recognition. The combination of two or more sensors will be referred to as "sensor fusion" or "fusion" hereinafter.

[0005] For example, a road traffic monitoring system has been proposed in which a combination of multiple sensors having different detection characteristics is switched and used based on environmental indicators such as temperature data, rainfall data, visibility data, and illuminance data (see Patent Document 1).

[0006] In addition, a vehicle driving control system has been proposed that prepares multiple fusion specifications for the external environment and notifies the driver of the detection area of ​​the sensor in which the recognition accuracy of the sensor is reduced due to the external environment and draws attention in the selected fusion specification (see Patent Document 2).

[0007] Citation List

[0008] Patent documents

[0009] Patent Document 1: Japanese Patent Application Laid-Open No. 2003-162795

[0010] Patent Document 2: Japanese Patent Application Laid-Open No. 2017-132285 Summary of the Invention

[0011] Problems to be solved by the present invention

[0012] An object of the technology disclosed in this specification is to provide an information processing device, an information processing method, a computer program, and a mobile device that perform fusion processing on a plurality of sensors mainly for recognizing the external world.

[0013] Solution to the problem

[0014] A first aspect of the technology disclosed in this specification is an information processing device including:

[0015] an identification unit configured to perform identification processing on the object based on the detection signal of the sensor; and

[0016] The processing unit is configured to perform fusion processing on the first data before identification by the identification unit and other data.

[0017] The sensor includes, for example, a millimeter-wave radar. The recognition unit then performs various processes for distance detection, speed detection, object angle detection, and object tracking based on the detection signal of the sensor before recognition, and the first data includes at least one of the detection signal, the object distance detection result, the speed detection result, the angle detection result, and the object tracking result. The processing unit can then perform at least one of a fusion process of fusing the third data with the first data before recognition by the second recognition unit, a fusion process of the fourth data with the first data after recognition by the second recognition unit, a fusion process of the first data with the second data after recognition by the recognition unit, or a fusion process of the fourth data with the second data.

[0018] Furthermore, the information processing device according to the first aspect further includes a second recognition unit that performs object recognition processing based on detection signals from a second sensor including at least one of a camera or a LiDAR. Then, if the recognition result of the second recognition unit is good but the recognition result of the recognition unit is poor, the processing unit performs fusion processing on the first data. Alternatively, if the recognition result of the second recognition unit is poor, the processing unit performs fusion processing on the first data.

[0019] Furthermore, a second aspect of the technology disclosed in this specification is an information processing method including:

[0020] a recognition step of performing recognition processing on the object based on the detection signal of the sensor; and

[0021] and a processing step of fusing the first data before identification in the identification step with other data. The processing step may include fusing the third data before identification by the second identification unit with the first data, fusing the fourth data after identification by the second identification unit with the first data, fusing the first data after identification by the identification unit with the second data, or fusing the fourth data with the second data.

[0022] Furthermore, a third aspect of the technology disclosed in this specification is a computer program described in a computer-readable format for causing a computer to function as:

[0023] an identification unit configured to perform identification processing on the object based on the detection signal of the sensor; and

[0024] The processing unit is configured to perform a fusion process on the first data before recognition by the recognition unit and other data. The processing unit may perform at least one of a fusion process of fusing the third data with the first data before recognition by the second recognition unit, a fusion process of fusing the fourth data with the first data after recognition by the second recognition unit, a fusion process of fusing the first data with the second data after recognition by the recognition unit, or a fusion process of fusing the fourth data with the second data.

[0025] The computer program according to the third aspect defines a computer program described in a computer-readable format for implementing predetermined processing on a computer. In other words, by installing the computer program according to the third aspect on a computer, cooperative actions are performed on the computer, and actions and effects similar to those of the information processing device according to the first aspect can be achieved.

[0026] In addition, a fourth aspect of the technology disclosed in this specification is a mobile device including:

[0027] mobile devices;

[0028] sensor;

[0029] an identification unit configured to perform identification processing on the object based on the detection signal of the sensor;

[0030] a processing unit configured to perform a fusion process on the first data before identification by the identification unit and other data; and

[0031] The control unit is configured to control the mobile device based on the processing result of the processing unit. The processing unit can perform at least one of a fusion process of fusing the third data before recognition by the second recognition unit with the first data, a fusion process of fusing the fourth data after recognition by the second recognition unit with the first data, a fusion process of fusing the first data with the second data after recognition by the recognition unit, or a fusion process of fusing the fourth data with the second data.

[0032] Effects of the Invention

[0033] The technology disclosed in this specification can provide an information processing device, an information processing method, a computer program, and a mobile device that perform fusion processing on a plurality of sensors mainly for recognizing the external world.

[0034] Note that the effects described in this specification are merely illustrative, and the effects of the present invention are not limited to these effects. Furthermore, in addition to the above-described effects, the present invention can also produce additional effects.

[0035] Other objects, features, and advantages of the technology disclosed in this specification will become apparent through more detailed description based on embodiments and accompanying drawings as described later. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a block diagram showing a schematic functional configuration example of the vehicle control system 100 .

[0037] Figure 2 is a diagram showing the functional configuration of the information processing apparatus 1000 .

[0038] Figure 3 is a diagram illustrating an image captured by a camera.

[0039] Figure 4 is a diagram illustrating a scene detected by a millimeter-wave radar.

[0040] Figure 5 The millimeter wave radar is used to detect Figure 4 Figure 1 shows the results for the scenario shown in .

[0041] Figure 6 is a diagram showing an internal configuration example of the radar recognition processing unit 1020 .

[0042] Figure 7 is a diagram showing one example of a scene to be recognized.

[0043] Figure 8 It shows Figure 7 A diagram of the scene data before recognition processing is shown in FIG.

[0044] Figure 9 It shows Figure 7 Figure 2 shows the data after scene recognition processing.

[0045] Figure 10 is a view showing another example of a scene to be recognized.

[0046] Figure 11 It shows Figure 10 A diagram of the scene data before recognition processing is shown in FIG.

[0047] Figure 12 It shows Figure 10 Figure 2 shows the data after scene recognition processing.

[0048] Figure 13 is a diagram showing an example of a fusion processing result (only the late fusion processing) by the information processing apparatus 1000 .

[0049] Figure 14 is a diagram illustrating an example of a fusion processing result (including early fusion processing) by the information processing device 1000 .

[0050] Figure 15 is a diagram illustrating an example in which results differ between the late fusion process and the early fusion process.

[0051] Figure 16 is a diagram illustrating an example in which results differ between the late fusion process and the early fusion process.

[0052] Figure 17 is a diagram showing a configuration example of an information processing device 1000 configured to adaptively perform early fusion processing.

[0053] Figure 18 It is shown in Figure 17 Flowchart of a processing procedure for performing target recognition in the information processing device 1000 shown in FIG.

[0054] Figure 19 is a diagram showing another configuration example of the information processing device 1000 configured to adaptively perform early fusion processing.

[0055] Figure 20 It is shown in Figure 19 Flowchart of a processing procedure for performing target recognition in the information processing device 1000 shown in FIG.

[0056] Figure 21 This is a diagram for describing a process of recognizing an object that cannot be recognized by the recognizer 1023 based on the RAW data of the millimeter wave radar 1080.

[0057] Figure 22This is a diagram for describing a process of recognizing an object that cannot be recognized by the recognizer 1023 based on the RAW data of the millimeter wave radar 1080.

[0058] Figure 23 This is a diagram for describing a process of recognizing an object that cannot be recognized by the recognizer 1023 based on the RAW data of the millimeter wave radar 1080. DETAILED DESCRIPTION

[0059] Hereinafter, embodiments of the technology disclosed in this specification will be described in detail with reference to the accompanying drawings.

[0060] Figure 1 1 is a block diagram showing a schematic functional configuration example of a vehicle control system 100 which is one example of a moving body control system to which the present technology can be applied.

[0061] Note that, hereinafter, in the case of distinguishing a vehicle provided with the vehicle control system 100 from other vehicles, the vehicle is referred to as a host vehicle or own vehicle.

[0062] The vehicle control system 100 includes an input unit 101, a data acquisition unit 102, a communication unit 103, an in-vehicle device 104, an output control unit 105, an output unit 106, a drive system control unit 107, a drive system 108, a body system control unit 109, a body system 110, a storage unit 111, and an autonomous driving control unit 112. The input unit 101, the data acquisition unit 102, the communication unit 103, the output control unit 105, the drive system control unit 107, the body system control unit 109, the storage unit 111, and the autonomous driving control unit 112 are connected to each other via a communication network 121. The communication network 121 includes, for example, an in-vehicle communication network or bus conforming to any standard such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), or FlexRay (registered trademark). Note that each unit of the vehicle control system 100 may be directly connected without passing through the communication network 121.

[0063] Note that, hereinafter, in the case where each unit of the vehicle control system 100 performs communication via the communication network 121, description of the communication network 121 will be omitted. For example, in the case where the input unit 101 and the automatic driving control unit 112 communicate with each other via the communication network 121, it will be simply described that the input unit 101 and the automatic driving control unit 112 communicate with each other.

[0064] The input unit 101 includes devices used by the occupant to input various data, instructions, and the like. For example, the input unit 101 includes operating devices such as a touch panel, buttons, microphones, switches, and joysticks, as well as operating devices that allow input by methods other than manual operation (such as voice and gestures). In addition, for example, the input unit 101 may be a remote control device using infrared or other radio waves, or may be an externally connected device that supports the operation of the vehicle control system 100, including a mobile device, a wearable device, and the like. The input unit 101 generates an input signal based on the data, instructions, and the like input by the occupant, and supplies the input signal to each unit of the vehicle control system 100.

[0065] The data acquisition unit 102 includes various sensors and the like that acquire data to be used for processing of the vehicle control system 100 , and supplies the acquired data to each unit of the vehicle control system 100 .

[0066] For example, the data acquisition unit 102 includes various sensors for detecting the state of the vehicle, etc. Specifically, for example, the data acquisition unit 102 includes a gyroscope sensor, an acceleration sensor, an inertial measurement unit (IMU), and sensors for detecting the amount of accelerator pedal operation, the amount of brake pedal operation, the steering wheel angle, the engine speed, the motor speed, the wheel speed, and the like.

[0067] In addition, for example, the data acquisition unit 102 includes various sensors for detecting information outside the vehicle. Specifically, for example, the data acquisition unit 102 includes image capture devices such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. In addition, for example, the data acquisition unit 102 includes environmental sensors for detecting weather, atmospheric phenomena, etc., and surrounding information detection sensors for detecting objects around the vehicle. Environmental sensors include, for example, raindrop sensors, fog sensors, sunlight sensors, snow sensors, etc. The surrounding information detection sensors include, for example, ultrasonic sensors, millimeter-wave radars, light detection and ranging, laser imaging detection and ranging (LiDAR), sonar, etc.

[0068] The data acquisition unit 102 includes, for example, various sensors for detecting the current position of the vehicle. Specifically, for example, the data acquisition unit 102 includes a Global Navigation Satellite System (GNSS) receiver that receives GNSS signals from GNSS satellites.

[0069] Furthermore, for example, the data acquisition unit 102 includes various sensors for detecting in-vehicle information. Specifically, for example, the data acquisition unit 102 includes an image capture device for capturing an image of the driver, a biometric sensor for detecting biometric information about the driver, and a microphone for collecting sounds from the vehicle interior. The biometric sensor is provided on, for example, a seat surface or a steering wheel, and detects biometric information about an occupant sitting in the seat or the driver steering the wheel.

[0070] The communication unit 103 communicates with the in-vehicle device 104 and various devices, servers, base stations, and the like outside the vehicle. The communication unit 103 transmits data supplied from each unit of the vehicle control system 100 and supplies received data to each unit of the vehicle control system 100. Note that the communication protocol supported by the communication unit 103 is not particularly limited, and the communication unit 103 may support multiple types of communication protocols.

[0071] For example, the communication unit 103 performs wireless communication with the in-vehicle device 104 through wireless LAN, Bluetooth (registered trademark), near field communication (NFC), wireless USB (WUSB), etc. In addition, for example, the communication unit 103 performs wired communication with the in-vehicle device 104 through a connection terminal (not shown) (and a cable if necessary) through a universal serial bus (USB), a high-definition multimedia interface (HDMI), or a mobile high-definition link (MHL).

[0072] Furthermore, for example, the communication unit 103 performs communication with a device (e.g., an application server or a control server) existing on an external network (e.g., the Internet, a cloud network, or a network specific to a business operator) via a base station or an access point. Furthermore, for example, the communication unit 103 performs communication with a terminal (e.g., a pedestrian terminal or a store terminal, or a machine type communication (MTC) terminal) existing near the vehicle by using a peer-to-peer (P2P) technology. Furthermore, for example, the communication unit 103 performs V2X communication including vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, vehicle-to-pedestrian communication, and the like. Furthermore, for example, the communication unit 103 includes a beacon receiving unit that receives radio waves or electromagnetic waves transmitted from a wireless station installed on a road, etc., and acquires information including the current position, traffic congestion, traffic regulations, required time, and the like.

[0073] The in-vehicle devices 104 include, for example, mobile devices or wearable devices owned by passengers, information devices carried in or attached to the vehicle, a navigation device for searching for a route to an arbitrary destination, and the like.

[0074] The output control unit 105 controls the output of various information to the occupants of the vehicle or to the exterior of the vehicle. For example, the output control unit 105 generates an output signal including at least one of visual information (e.g., image data) or auditory information (e.g., voice data) and supplies the output signal to the output unit 106, thereby controlling the output of the visual and auditory information from the output unit 106. Specifically, for example, the output control unit 105 combines image data captured by the different image capture devices of the data acquisition unit 102 to generate a bird's-eye view image, a panoramic image, etc., and supplies the output signal including the generated image to the output unit 106. Furthermore, for example, the output control unit 105 generates voice data including warning sounds, warning messages, etc. for hazards such as collision, contact, and entering a danger zone, and supplies the output signal including the generated voice data to the output unit 106.

[0075] The output unit 106 includes devices that can output visual or auditory information to the occupants of the vehicle or to the exterior of the vehicle. For example, the output unit 106 includes a display device, an instrument panel, audio speakers, headphones, wearable devices including glasses-type displays worn by occupants, projectors, lights, and the like. In addition to devices with conventional displays, the display device included in the output unit 106 may also be a device that displays visual information within the driver's field of view, including a head-up display, a transmissive display, a device with augmented reality (AR) display functionality, and the like.

[0076] The drive system control unit 107 generates various control signals and supplies the control signals to the drive system 108, thereby controlling the drive system 108. In addition, the drive system control unit 107 supplies control signals to each unit other than the drive system 108 as needed, and performs notification of the control state of the drive system 108, etc.

[0077] The drive system 108 includes various devices related to the vehicle's drive system. For example, the drive system 108 includes a drive force generating device for generating drive force, including an internal combustion engine and a drive motor; a drive force transmission mechanism for transmitting drive force to the wheels; a steering mechanism for adjusting the steering angle; a brake device for generating braking force; an anti-lock braking system (ABS); an electronic stability control system (ESC); and an electric power steering device.

[0078] The body system control unit 109 generates various control signals and supplies the control signals to the body system 110, thereby controlling the body system 110. In addition, the body system control unit 109 supplies control signals to each unit other than the body system 110 as needed, and performs notification of the control state of the body system 110, etc.

[0079] The vehicle body system 110 includes various vehicle body system devices. For example, the vehicle body system 110 includes a keyless entry system, a smart key system, power windows, power seats, a steering wheel, an air conditioner, and various lights (e.g., headlights, backup lights, parking lights, turn signals, fog lights, etc.).

[0080] The storage unit 111 includes, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, and the like. The storage unit 111 stores various programs, data, and the like to be used by each unit of the vehicle control system 100. For example, the storage unit 111 stores map data including a three-dimensional high-precision map (such as a dynamic map), a global map with lower precision than a high-precision map and covering a large area, a local map including information around the vehicle, and the like.

[0081] The autonomous driving control unit 112 controls autonomous driving, including autonomous driving and driving assistance. Specifically, for example, the autonomous driving control unit 112 performs cooperative control to implement the functions of an advanced driver assistance system (ADAS), including collision avoidance or impact mitigation for the vehicle, follow-up driving based on the distance between vehicles, driving while maintaining vehicle speed, collision warnings for the vehicle, lane departure warnings for the vehicle, and the like. Furthermore, for example, the autonomous driving control unit 112 performs cooperative control for autonomous driving, in which the vehicle drives autonomously without relying on the driver's operation. The autonomous driving control unit 112 includes a detection unit 131, a self-position estimation unit 132, a situation analysis unit 133, a planning unit 134, and an action control unit 135.

[0082] The detection unit 131 detects various types of information required for controlling automatic driving. The detection unit 131 includes an external vehicle information detection unit 141, an internal vehicle information detection unit 142, and a vehicle state detection unit 143.

[0083] The vehicle exterior information detection unit 141 detects information outside the vehicle based on data or signals from each unit in the vehicle control system 100. For example, the vehicle exterior information detection unit 141 detects, identifies, and tracks objects around the vehicle, and detects the distance to the objects. Objects to be detected include, for example, vehicles, people, obstacles, buildings, roads, traffic lights, traffic signs, road markings, and the like. Furthermore, the vehicle exterior information detection unit 141 detects the environment surrounding the vehicle. The detected environment includes, for example, weather, temperature, humidity, brightness, road conditions, and the like. The vehicle exterior information detection unit 141 supplies data indicating the results of the detection processing to the vehicle position estimation unit 132, the map analysis unit 151 of the situation analysis unit 133, the traffic rule recognition unit 152 and situation recognition unit 153, the emergency avoidance unit 171 of the action control unit 135, and other units.

[0084] The in-vehicle information detection unit 142 performs detection processing on information about the vehicle interior based on data or signals from each unit of the vehicle control system 100. For example, the in-vehicle information detection unit 142 performs driver authentication and identification processing, driver status detection processing, occupant detection processing, and in-vehicle environment detection processing. Driver status to be detected includes, for example, physical condition, arousal level, concentration level, fatigue level, and line of sight. In-vehicle environment to be detected includes, for example, temperature, humidity, brightness, and odor. The in-vehicle information detection unit 142 supplies data indicating the results of the detection processing to the situation identification unit 153 of the situation analysis unit 133, the emergency avoidance unit 171 of the action control unit 135, and the like.

[0085] The vehicle state detection unit 143 performs detection processing on the state of the host vehicle based on data or signals from each unit of the vehicle control system 100. The state of the host vehicle to be detected includes, for example, speed, acceleration level, steering angle, presence or absence of an abnormality and details of the abnormality, driving operation status, power seat position and tilt, door lock status, status of other in-vehicle equipment, etc. The vehicle state detection unit 143 supplies data indicating the results of the detection processing to the situation recognition unit 153 of the situation analysis unit 133, the emergency avoidance unit 171 of the action control unit 135, and the like.

[0086] The own position estimation unit 132 performs estimation processing on the vehicle's position, orientation, and other aspects based on data or signals from each unit of the vehicle control system 100 (such as the vehicle exterior information detection unit 141 and the situation recognition unit 153 of the situation analysis unit 133). Furthermore, the own position estimation unit 132 generates a local map (hereinafter referred to as the own position estimation map) to be used for own position estimation as needed. The own position estimation map is a high-precision map using, for example, techniques such as simultaneous localization and mapping (SLAM). The own position estimation unit 132 supplies data indicating the results of the estimation processing to the map analysis unit 151, traffic regulation recognition unit 152, and situation recognition unit 153 of the situation analysis unit 133. Furthermore, the own position estimation unit 132 stores the own position estimation map in the storage unit 111.

[0087] The situation analysis unit 133 performs analysis processing on the vehicle and the surrounding situation. The situation analysis unit 133 includes a map analysis unit 151, a traffic regulation recognition unit 152, a situation recognition unit 153, and a situation prediction unit 154.

[0088] The map analysis unit 151 performs analysis processing on the various maps stored in the storage unit 111 and, when necessary, constructs a map including information required for autonomous driving processing using data or signals from each unit of the vehicle control system 100 (such as the own position estimation unit 132 and the vehicle exterior information detection unit 141). The map analysis unit 151 supplies the constructed map to the traffic regulation recognition unit 152, the situation recognition unit 153, the situation prediction unit 154, and the route planning unit 161, the behavior planning unit 162, and the action planning unit 163 of the planning unit 134.

[0089] The traffic regulation recognition unit 152 performs recognition processing on traffic regulations surrounding the vehicle based on data or signals from each unit of the vehicle control system 100 (such as the own position estimation unit 132, the vehicle exterior information detection unit 141, and the map analysis unit 151). Through this recognition processing, for example, the position and status of traffic lights surrounding the vehicle, detailed traffic regulations surrounding the vehicle, and drivable lanes are recognized. The traffic regulation recognition unit 152 supplies data indicating the results of the recognition processing to the situation prediction unit 154 and the like.

[0090] The situation recognition unit 153 performs recognition processing on the vehicle's situation based on data or signals from each unit of the vehicle control system 100 (such as the vehicle position estimation unit 132, the vehicle exterior information detection unit 141, the vehicle interior information detection unit 142, the vehicle state detection unit 143, and the map analysis unit 151). For example, the situation recognition unit 153 recognizes the situation of the vehicle, the situation surrounding the vehicle, and the situation of the driver of the vehicle. Furthermore, the situation recognition unit 153 generates a local map (hereinafter referred to as a situation recognition map) to be used for recognizing the situation surrounding the vehicle, as needed. The situation recognition map is, for example, an occupancy grid map.

[0091] The vehicle's conditions to be identified include, for example, its position, orientation, movement (e.g., speed, acceleration level, direction of movement, etc.), the presence of any abnormality, and the details of any abnormality. The vehicle's surrounding conditions to be identified include, for example, the type and position of surrounding stationary objects, the type, position, and movement (e.g., speed, acceleration level, direction of movement, etc.) of surrounding moving objects, the structure and road conditions of surrounding roads, and the surrounding weather, temperature, humidity, and brightness. The driver's state to be identified includes, for example, physical condition, arousal level, concentration level, fatigue level, eye movement, and driving operation.

[0092] The situation recognition unit 153 supplies data indicating the result of the recognition process (including a situation recognition map as necessary) to the own position estimation unit 132 , the situation prediction unit 154 , etc. Furthermore, the situation recognition unit 153 stores the situation recognition map in the storage unit 111 .

[0093] The situation prediction unit 154 performs situation prediction processing regarding the vehicle based on data or signals from each unit of the vehicle control system 100, such as the map analysis unit 151, the traffic regulation recognition unit 152, and the situation recognition unit 153. For example, the situation prediction unit 154 performs prediction processing on the situation of the vehicle, the situation around the vehicle, the situation of the driver, and the like.

[0094] The vehicle's conditions to be predicted include, for example, the vehicle's movements, the occurrence of abnormalities, and the drivable distance. The vehicle's surrounding conditions to be predicted include, for example, the behavior of moving objects around the vehicle, changes in traffic light status, and changes in the environment (such as weather). The driver's conditions to be predicted include, for example, the driver's movements and physical condition.

[0095] The situation prediction unit 154 supplies data indicating the result of the prediction process to the route planning unit 161 , the behavior planning unit 162 , the action planning unit 163 and the like of the planning unit 134 together with the data from the traffic regulation recognition unit 152 and the situation recognition unit 153 .

[0096] The route planning unit 161 plans a route to a destination based on data or signals from each unit of the vehicle control system 100, such as the map analysis unit 151 and the situation prediction unit 154. For example, the route planning unit 161 sets a route from the current location to the specified destination based on a global map. Furthermore, the route planning unit 161 appropriately changes the route based on, for example, traffic congestion, accidents, traffic regulations, roadwork, the driver's physical condition, and the like. The route planning unit 161 supplies data indicating the planned route to the behavior planning unit 162 and the like.

[0097] The behavior planning unit 162 plans the behavior of the host vehicle based on data or signals from each unit of the vehicle control system 100 (such as the map analysis unit 151 and the situation prediction unit 154) so ​​that the host vehicle can safely travel on the route planned by the route planning unit 161 within the planned time. For example, the behavior planning unit 162 plans starting, stopping, driving direction (e.g., forward, backward, left turn, right turn, turning, etc.), driving lane, driving speed, overtaking, etc. The behavior planning unit 162 supplies data indicating the planned behavior of the host vehicle to the action planning unit 163 and the like.

[0098] The action planning unit 163 plans the operation of the host vehicle for implementing the behavior planned by the behavior planning unit 162 based on data or signals from each unit of the vehicle control system 100 (such as the map analysis unit 151 and the situation prediction unit 154). For example, the action planning unit 163 plans acceleration, deceleration, driving trajectory, etc. The action planning unit 163 supplies data indicating the planned operation of the host vehicle to the acceleration-deceleration control unit 172, the direction control unit 173, etc. of the action control unit 135.

[0099] The action control unit 135 controls the operation of the vehicle and includes an emergency avoidance unit 171 , an acceleration / deceleration control unit 172 , and a direction control unit 173 .

[0100] The emergency avoidance unit 171 detects emergency situations such as collision, contact, entry into a dangerous zone, driver abnormality, and vehicle abnormality based on the detection results of the vehicle exterior information detection unit 141, the vehicle interior information detection unit 142, and the vehicle state detection unit 143. Upon detecting the occurrence of an emergency, the emergency avoidance unit 171 plans the vehicle's operation to avoid such an emergency, such as a sudden stop or a sharp turn. The emergency avoidance unit 171 supplies data indicating the planned vehicle operation to the acceleration / deceleration control unit 172, the direction control unit 173, and the like.

[0101] The acceleration-deceleration control unit 172 performs acceleration-deceleration control for realizing the operation of the host vehicle planned by the action planning unit 163 or the emergency situation avoidance unit 171. For example, the acceleration-deceleration control unit 172 calculates a control target value of a driving force generating device or a braking device for realizing the planned acceleration, deceleration, or sudden stop, and supplies an instruction to the drive system control unit 107 to calculate the control target value.

[0102] The direction control unit 173 performs direction control for achieving the operation of the host vehicle planned by the action planning unit 163 or the emergency avoidance unit 171. For example, the direction control unit 173 calculates a control target value of a steering mechanism for achieving the travel trajectory or quick turn planned by the action planning unit 163 or the emergency avoidance unit 171, and supplies a control command instructing the calculated control target value to the drive system control unit 107.

[0103] To achieve higher-precision exterior recognition for autonomous driving and ADAS, many different types of exterior recognition sensors are being installed in vehicles. However, each sensor has its strengths and weaknesses. For example, cameras that capture visible light are less effective in dark areas, while radars that detect reflected radio waves are less effective against objects that don't readily reflect them, such as people and animals.

[0104] The strengths and weaknesses of each sensor also depend on the detection principle. The strengths and weaknesses of radar (millimeter wave radar), camera, and laser radar (LiDAR) are summarized in Table 1 below. In the table, ⊙ indicates a very strong point (high accuracy), ○ indicates a strong point (good accuracy), and △ indicates a weak point (lack of accuracy). However, the detection principle of radar is to reflect radio waves and measure the distance to the object, the detection principle of camera is to capture the visible light reflected from the subject, and the detection principle of LiDAR is to reflect light and measure the distance to the object, etc.

[0105] [Table 1]

[0106] Sensor Type radar camera LiDAR Measuring distance ○ Δ ⊙ Angle, resolution Δ ⊙ ○ Performance in bad weather ⊙ △ ○ Nighttime performance ⊙ ○ ⊙ Classification of objects △ ⊙ ○

[0107] Most sensor fusion technologies in the past supplement a sensor whose recognition accuracy has deteriorated due to changes in the external environment with another sensor or by switching the combination of sensors to be used, rather than using data detected by a sensor whose recognition accuracy has deteriorated due to changes in the external environment, etc.

[0108] However, even if a sensor has low recognition accuracy, it does not mean that nothing can be recognized from the sensor's detected data. For example, a camera is not good in dark places, but it is possible to recognize nearby objects or places illuminated by street lamps or city lights from the image captured by the camera to some extent.

[0109] Therefore, this specification proposes a technology for further improving recognition accuracy by effectively using detection signals from sensors with low recognition accuracy among a plurality of sensors such as radar, camera, and LiDAR mounted on a vehicle.

[0110] Figure 2 The functional configuration of the information processing apparatus 1000 to which the technology disclosed in this specification is applied is schematically shown.

[0111] The illustrated information processing device 1000 includes a camera recognition processing unit 1010 that processes the detection signal of the camera 1070, a radar recognition processing unit 1020 that processes the detection signal of the millimeter wave radar 1080, a LiDAR recognition processing unit 1030 that processes the detection signal of the LiDAR 1090, and a fusion processing unit 1040 that performs fusion processing on the processing results of each of the above-mentioned recognition processing units 1010 to 1030.

[0112] External recognition sensors such as the camera 1070, millimeter-wave radar 1080, and LiDAR 1090 are installed on the same vehicle after calibration of each installation position so that the detection range becomes almost the same. Furthermore, it is assumed that external recognition sensors other than the aforementioned sensors 1070 to 1090 are further installed on the same vehicle. Furthermore, it is assumed that the camera 1070 includes a plurality of cameras, and at least a portion of the cameras are installed so as to have a different detection range than the millimeter-wave radar 1080 and LiDAR 1090. The outputs of the plurality of cameras may be fused by the fusion processing unit 1040.

[0113] The camera recognition processing unit 1010 includes a RAW data processing unit 1011 that processes the RAW data input from the camera 1070, a signal processing unit 1012 that performs signal processing on the RAW data, and a recognizer 1013 that recognizes objects from the camera image after the signal processing. The RAW data referred to here is data that records the light information captured by the image sensor as is. The RAW data processing unit 1011 performs front-end processing (amplification, noise removal, A / D conversion, etc.) on the RAW data, while the signal processing unit 1012 performs back-end processing. The recognizer 1013 can be hardware that implements a predetermined image recognition algorithm or software that executes the recognition algorithm. The recognizer 1013 outputs information (a target list) regarding the shape classification of the recognized objects (targets). Examples of object shape classifications recognized by the recognizer 1013 include people, cars, signs, sky, buildings, roads, sidewalks, etc. The recognizer 1013 can be either hardware that implements a predetermined image recognition algorithm or software that executes the recognition algorithm.

[0114] As the recognition result of the recognizer 1013, the object shape classification (person, car, sign, sky, building, road, sidewalk, etc.) is output from the camera recognition processing unit 1010 to the subsequent fusion processing unit 1040. However, the recognizer 1013 does not output a recognition result with a low probability. Therefore, in a case where the recognition performance of the recognizer 1013 is reduced (such as in bad weather or at night), the amount of information output from the recognizer 1013 is reduced. In addition, in this embodiment, early data before the recognizer 1013 obtains the final recognition result is also output to the fusion processing unit 1040. The early data mentioned here includes the captured image (RAW data) input from the camera 1070, the output data, the data during the signal processing by the signal processing unit 1012, the data during the recognition by the recognizer 1013, etc. All or part of the early data is output to the fusion processing unit 1040. Since the probability of recognition is low, it is assumed that the data during the recognition process by the recognizer 1013 includes information about objects that are not ultimately output from the recognizer 1013 (for example, information about pedestrians that are hidden behind objects such as a vehicle body or a fence and can only be partially or sporadically recognized). Hereinafter, for convenience, the early data before the recognition process by the recognizer 1013 will be collectively referred to as "RAW data" of the camera 1070.

[0115] In addition, the radar recognition processing unit 1020 includes a RAW data processing unit 1021 that processes the RAW data input from the millimeter wave radar 1080, a signal processing unit 1022 that performs signal processing on the RAW data, and an identifier 1023 that identifies an object from the radar detection results after signal processing. The identifier 1023 can be in the form of hardware that implements a predetermined recognition algorithm or in the form of software that executes the recognition algorithm. The identifier 1023 tracks the identified targets (people, cars, signs, buildings, etc.) and outputs recognition results (such as the distance, elevation angle, azimuth angle, speed, and reflection intensity of each target).

[0116] The final recognition result of the identifier 1023 is output from the radar recognition processing unit 1020 to the subsequent fusion processing unit 1040. However, the identifier 1013 does not output a recognition result with a low probability. Therefore, it is assumed that the information output from the identifier 1023 regarding objects with weak radar reflection intensity (such as non-metal) will be reduced. In addition, in this embodiment, early data before the identifier 1023 obtains the final recognition result is also output to the fusion processing unit 1040. The early data mentioned here includes the captured image (RAW data) input from the millimeter wave radar 1080, output data, data during signal processing by the signal processing unit 1022, data during recognition by the identifier 1023, and the like. All or part of the early data is output to the fusion processing unit 1040. Due to the low probability of recognition, it is assumed that the data during recognition by the identifier 1023 includes information that is not ultimately output from the identifier 1023 (for example, a motorcycle whose reflection intensity is weakened due to the influence of reflected radio waves from nearby objects such as fences or signs). Hereinafter, for convenience, the early data before the identifier 1023 performs recognition processing is collectively referred to as the “RAW data” of the millimeter wave radar 1080 .

[0117] In addition, the LiDAR recognition processing unit 1030 includes a RAW data processing unit 1031 that processes the RAW data input from the LiDAR 1090, a signal processing unit 1032 that performs signal processing on the RAW data, and an identifier 1033 that identifies an object from the LiDAR detection results after signal processing. The identifier 1033 can be in the form of hardware that implements a predetermined recognition algorithm or in the form of software that executes the recognition algorithm. The identifier 1033 tracks the recognized targets (people, cars, signs, buildings, etc.) and outputs recognition results (such as the distance, elevation, azimuth, altitude, and reflectivity of each target).

[0118] The final recognition result of the identifier 1033 is output from the LiDAR recognition processing unit 1030 to the subsequent fusion processing unit 1040. However, the identifier 1033 does not output recognition results with a low probability. Furthermore, in this embodiment, early data before the identifier 1033 obtains the final recognition result is also output to the fusion processing unit 1040. The early data referred to here includes captured images (RAW data) input from the LiDAR 1090, output data, data during signal processing by the signal processing unit 1032, and data during recognition by the identifier 1033. All or part of this early data is output to the fusion processing unit 1040. Because the probability of recognition is low, it is assumed that the data during recognition by the identifier 1033 includes information that is not ultimately output from the identifier 1033 (pedestrian information that can only be partially or sporadically recognized). Hereinafter, for convenience, early data before recognition processing by the identifier 1033 will be collectively referred to as "RAW data" of the LiDAR 1090.

[0119] For convenience, Figure 2 The configuration of each of the recognition processing units 1010 to 1030 is schematically depicted. It should be understood that the detailed internal configuration is determined depending on the type of sensor, the model of the sensor, the design specifications, etc. For example, a configuration is also assumed in which some or all components of the camera recognition processing unit 1010 are installed in the unit of the camera 1070, some or all components of the radar recognition processing unit 1020 are installed in the unit of the millimeter wave radar 1080, or some or all components of the lidar recognition processing unit 1030 are installed in the unit of the lidar 1090.

[0120] In addition, in the case where external recognition sensors (not shown) other than the camera 1070, the millimeter-wave radar 1080, and the LiDAR 1090 are installed on the same vehicle, the information processing device 1000 may further include a recognition processing unit, which includes a RAW data processing unit, a signal processing unit, and an identifier for performing recognition processing based on the detection signals of the sensors. Similarly, in these cases, the final recognition result is output from each recognition processing unit to the subsequent fusion processing unit 1040, and the RAW data of the sensor is output.

[0121] Furthermore, while there is a trend toward installing multiple external recognition sensors on a vehicle to achieve autonomous driving and ADAS (as described above), it is, of course, assumed that only one external recognition sensor among the camera 1070, millimeter-wave radar 1080, and LiDAR 1090 is installed on a single vehicle. For example, it is also assumed that the millimeter-wave radar 1080 is not used when sufficient external recognition performance can be obtained using only the LiDAR 1090, and that the video captured by the camera 1070 is used only for viewing and not for external recognition. In this case, it should be understood that the information processing device 1000 is equipped with only the functional modules corresponding to the sensors to be used among the above-mentioned recognition processing units 1010 to 1030, or is equipped with all the functional modules corresponding to each sensor, but only the corresponding functional modules (or functional modules having input signals from the sensors) operate and send the output to the subsequent fusion processing unit 1040.

[0122] The fusion processing unit 1040 performs fusion processing on the recognition results based on each sensor of the camera 1070, millimeter wave radar 1080 and LiDAR 1090 installed on the same vehicle to perform external recognition. In the case where another external recognition sensor (not shown) is installed on the same vehicle, the fusion processing unit 1040 also performs further fusion processing on the detection signal from the sensor. In this embodiment, the fusion processing unit 1040 not only performs fusion processing on the recognition results of each sensor, but also performs fusion processing on the RAW data before recognition to perform external recognition. The fusion processing unit 1040 then outputs the external recognition result obtained by performing the fusion processing to the vehicle control system.

[0123] exist Figure 2In the example shown, the vehicle control system includes an electronic control unit (ECU) 1050 and an actuator (hereinafter referred to as "ACT") 1060 that moves the vehicle. ECU 1050 determines autonomous driving or driving assistance, such as adaptive cruise control (ACC), lane departure warning (LDW), lane keeping assist (LKA), autonomous emergency braking (AEB), and blind spot detection (BSD), based on the external recognition results of fusion processing unit 1040. ACT 1060 then performs driving control for each driving unit, i.e., vehicle operation, such as active cornering lights (ACL), brake actuator (BRK), and steering (STR), according to instructions from ECU 1050. For example, if fusion processing unit 1040 identifies a road lane, the vehicle control system controls the vehicle's travel to prevent it from deviating from the lane. Furthermore, if fusion processing unit 1040 identifies an obstacle, such as a surrounding vehicle, pedestrian, roadside fence, or sign, the vehicle control system controls the vehicle's travel to avoid collision with the obstacle. Autonomous driving generally involves three steps: "recognition -> determination -> action." The recognition step identifies that an object exists, and the determination step determines what was recognized to determine the vehicle's route plan. Figure 2 In the configuration example shown, the processing of the recognition step is mainly performed in the information processing device 1000, the processing of the determination step is mainly performed by the ECU 1050 in the vehicle control system, and the processing of the operation step is mainly performed by the ACT 1060. However, the distinction between the recognition step and the determination step is not strict, and part of the recognition step described in this embodiment can be positioned as a determination step. In addition, in the future, a design is also expected in which some or all of the functions of processing the recognition step will be installed in each sensor unit such as the camera 1070, the millimeter wave radar 1080, and the LiDAR 1090.

[0124] In this embodiment, the fusion processing unit 1040 includes a late fusion processing unit 1041, an early fusion processing unit 1042, and a hybrid fusion processing unit 1043. The late fusion processing unit 1041 performs fusion processing on the final output (late data) of each of the recognition processing units 1010 to 1030, that is, performs fusion processing on the recognition results of each of the recognizers 1013, 1023, and 1033 to perform external recognition. In addition, the early fusion processing unit 1042 performs fusion processing on the early data before the recognition distance of each of the recognition processing units 1010 to 1030, that is, performs fusion processing on the RAW data of each sensor in the camera 1070, the millimeter wave radar 1080, and the LiDAR 1090 (as described above) to perform external recognition. In addition, the hybrid fusion processing unit 1043 performs fusion processing on any one or more of the final outputs (late data) of each of the recognition processing units 1010 to 1030 and the RAW data of any one or more of the recognition processing units 1010 to 1030 to perform external recognition. Even if the probability of the final recognition result of the sensor's identifier is low, the hybrid fusion processing unit 1043 has the effect of enhancing recognition performance by performing fusion processing with the RAW data of another sensor or the same sensor. Then, the fusion processing unit 1040 further performs fusion processing on the recognition results of the late fusion processing unit 1041, the early fusion processing unit 1042, and the hybrid fusion processing unit 1043, or selectively selects the recognition results of the fusion processing units 1041 to 1043, and outputs the processing results to the subsequent ECU 1050.

[0125] The late fusion processing unit 1041, which obtains the final recognition results of the recognizers 1013, 1023, and 1033 from the recognition processing units 1010 to 1030, processes information with high reliability of the recognition results. However, there is a problem that the amount of information that can be used is small because only recognition results with high probability are output from the recognizers 1013, 1023, and 1033 and only information with high reliability can be obtained.

[0126] On the other hand, since the early fusion processing unit 1042 receives RAW data from the recognition processing units 1010 to 1030 before passing through the recognizers 1013, 1023, and 1033, respectively, the amount of information required is very large. However, RAW data, or data close to RAW data, contains noise. For example, when the camera 1070 captures an image in a dark place at night, various information items in addition to the object are included, which increases the possibility of false detection and reduces the reliability of the information. Furthermore, due to the large amount of information, the processing load is also large.

[0127] For example, Figure 3As shown in , assume that when camera 1070 captures an image in a dark location at night, the captured image (i.e., RAW data) contains images of multiple people. Since pedestrian 301, which is brightly projected and reflected by streetlights, etc., can be recognized with a high recognition rate, recognizer 1013 outputs a result that recognizes pedestrian 301 as a target. Meanwhile, pedestrians 302 and 302, which are darkly projected and not exposed to streetlights, etc., have a low recognition rate, and recognizer 1013 outputs a result that such pedestrians 302 and 302 are not recognized as targets. That is, although multiple pedestrians 301 to 303 are projected in the original camera image (i.e., RAW data), only information with high reliability is output by recognizer 1013, and the amount of information is reduced. Therefore, only information regarding pedestrian 301 with high reliability is input to post-fusion processing unit 1041, and unnecessary information such as noise is ignored. However, information regarding pedestrians 302 and 303 with low reliability is also ignored. At the same time, information on all the pedestrians 301 to 303 is input to the early fusion processing unit 1042 , and various information items such as noise are also input.

[0128] In addition, the RAW data of the detection result of the millimeter wave radar 1080 includes the intensity distribution of the radio waves reflected at the position (direction and distance) of each reflecting object within the predetermined detection range in front of the receiving unit of the reflected radio waves. The RAW data contains the intensity data of the radio waves reflected from various objects. However, when passing through the identifier 1033, the intensity data with low reflection intensity of the radio waves is ignored, and only digital information such as the direction, distance (including depth and width) and speed of the identified target is extracted, and the amount of information is reduced.

[0129] For example, Figure 4 As shown in FIG, in a scene where a pedestrian 403 walks between two vehicles 401 and 402, it is desirable to detect the pedestrian 403 as well as the vehicles 401 and 402. However, when using the millimeter wave radar 1080, the reflection intensity of an object sandwiched between strong reflectors tends to weaken. Therefore, when measuring the Figure 4 When the scenario shown in Figure 5As shown in , RAW data 500 contains strong reflected waves 501 and 502 from vehicles 401 and 402, respectively, and weak reflected wave 503 from pedestrian 403. If the detection result of this millimeter wave radar 1080 passes through the identifier 1033, then the reflected wave 503 with low reliability as information is ignored, and only the reflected waves 501 and 502 with high reliability are identified as targets, so the amount of information is reduced. Therefore, only the information with high reliability about vehicles 401 and 402 is input into the late fusion processing unit 1041, and unnecessary information such as noise is ignored. However, the information with low reliability about pedestrian 403 is also ignored. On the other hand, information about pedestrian 403 and vehicles 401 and 402 is input into the early fusion processing unit 1042, but various information items such as noise are also input.

[0130] In short, the results obtained by performing fusion processing on the recognition results of each of the identifiers 1013, 1023, and 1033 by the late fusion processing unit 1041 are narrowed down to information with high reliability. Therefore, there is a possibility that information with low reliability but high importance is ignored. On the other hand, the results obtained by performing fusion processing on the RAW data of each of the recognition processing units 1010 to 1030 by the early fusion processing unit 1042 have a large amount of information, but there is a possibility that information with low reliability, such as noise, is absorbed.

[0131] Therefore, the information processing device 1000 according to this embodiment is configured to obtain an external recognition result with sufficient information content and high reliability by supplementing the processing result of the late fusion processing unit 1041, which has a high reliability but reduced information content, with the processing result of the early fusion processing unit 1042, which has a large amount of information but also has noise. In addition, the hybrid fusion processing unit 1043 performs fusion processing 1030 on any one or more of the final output (late data) of each of the recognition processing units 1010 to 1030 and the RAW data of any one or more of the recognition processing units 1010 to 1030. Even if the final recognition result of the recognizer of one sensor has a low probability, the hybrid fusion processing unit 1043 can improve recognition performance by performing fusion processing with the RAW data of another sensor or the same sensor. That is, the information processing device 1000 is configured to restore important information that would be ignored by the identifier 1013, 1023 or 1033 based on the result of fusion processing of the RAW data of each external recognition sensor such as the camera 1070, the millimeter wave radar 1080 and the LiDAR 1090 by the early fusion processing unit 1042.

[0132] Note that at the time of this application, the accuracy of external recognition by the LiDAR 1090 is significantly higher than that of cameras or millimeter-wave radars. However, the recognition results of the LiDAR recognition processing unit 1030 can be fused together with the recognition results of another fusion processing unit 1010 or 1020 by the late fusion processing unit 1041, and the recognition results of the LiDAR recognition processing unit 1030 can be supplemented with the processing results of the RAW data of the early fusion processing unit 1042. Furthermore, the LiDAR recognition processing unit 1030 can be fused together with the recognition results of another recognition processing unit 1010 or 1020 by the hybrid fusion processing unit 1043. Furthermore, if the recognition results of the recognizer 1033 of the LiDAR recognition processing unit 1030 alone are sufficient, the late fusion processing unit 1041 does not need to perform fusion processing on the recognition results or RAW data of the recognizers 1013 and 1023 of the camera recognition processing unit 1010 and radar recognition processing unit 1020, respectively.

[0133] On the other hand, at the technological level at the time of this application, the LiDAR 1090 is much more expensive than other external recognition sensors such as the camera 1070 and the millimeter-wave radar 1080. Therefore, when the LiDAR 1090 is not used (in other words, when the LiDAR 1090 is not installed on the vehicle), the information processing device 1000 can be configured to supplement the results of the fusion processing performed by the late fusion processing unit 1040 on the recognition results respectively performed by the recognizers 1013 and 1023 of the camera recognition processing unit 1010 and the radar recognition processing unit 1020 with the results of the fusion processing performed by the early fusion processing unit 1042 on the RAW data of each of the camera 1070 and the millimeter-wave radar 1080.

[0134] In addition, according to the nature of the reflected light waves used by the LiDAR 1090, there is a concern that reliability will deteriorate in weather that blocks light (such as rain, snowfall, and fog) and in dark places (such as at night and in tunnels). In addition, similar problems also apply to the camera 1070. On the other hand, the reliability of the millimeter wave radar 1080 is not so dependent on the weather and is relatively stable. Therefore, when the fusion processing unit 1040 performs fusion processing on the information from each of the sensors 1070 to 1090 based on environmental information (such as weather and other external information), the information processing device 1000 can adjust the weighting. For example, in good weather, the late fusion processing unit 1041 and the early fusion processing unit 1042 in the fusion processing unit 1040 use the recognition result of the recognizer 1033 of the LiDAR recognition processing unit 1030 and the RAW data of the LiDAR 1090 with a high weighting. In the case of rain, snow, or fog or in a dark place such as at night or in a tunnel, fusion processing is performed by using or not using the recognition result of the recognizer 1033 of the LiDAR recognition processing unit 1030 and the RAW data of the LiDAR 1090 with low weight.

[0135] Figure 6 1020 shows an internal configuration example of the radar recognition processing unit 1020. The radar recognition processing unit 1020 includes a RAW data processing unit 1021, a signal processing unit 1022, and an identifier 1023.

[0136] The RAW data processing unit 1021 receives the RAW data of the millimeter wave radar 1080 and performs processing such as amplification, noise removal, and AD conversion. The RAW data or data after any of amplification, noise removal, or AD conversion is output from the RAW data processing unit 1021 to the early fusion processing unit 1042.

[0137] exist Figure 6 In the illustrated example, the signal processing unit 1022 includes a distance detection unit 601 for detecting the distance to each target captured by the radar, a speed detection unit 602 for detecting the speed at which each target is moving, an angle detection unit 603 for detecting the orientation of each target, a tracking unit 604 for tracking the target, and a MISC processing unit 605 for performing other processing. The algorithm used to detect the distance, orientation, size, and speed of a target from the RAW data of the millimeter wave radar 1080 is not particularly limited. For example, an algorithm developed by the manufacturer of the millimeter wave radar 1080 or the like can be applied as is.

[0138] When all processing by corresponding units 601 to 605 is completed in signal processing unit 1022, target information including distance, direction, size, speed, etc. detected by the radar is output to subsequent stage identifier 1023. Target information for which distance, direction, size, and speed cannot be accurately detected is not output to identifier 1023 and is ignored because it is unrecognizable. Furthermore, the signal processed by at least one functional module of corresponding units 601 to 605 is also output to early fusion processing unit 1042.

[0139] Note that the order in which the respective units 601 to 605 perform processing on the input data from the RAW data processing unit 1021 is not necessarily fixed, and it is assumed that the order will be appropriately changed according to product design specifications, etc. In addition, not all of the functional modules 601 to 605 described above are essential for detecting signals from the millimeter wave radar 1080. It is also assumed that the functional modules 601 to 605 are selected according to product design specifications, etc., or that the signal processing unit 1022 will be equipped with functional modules other than those shown.

[0140] The identifier 1023 performs external recognition processing based on the signal after being processed by the signal processing unit 1022 according to a predetermined recognition algorithm.

[0141] For example, in Figure 7 The image captured by the vehicle-mounted camera 1070 is a street scene. Figure 8 , data before being processed by the identifier 1023 of the radar identification processing unit 1020 is shown in FIG. Figure 9 , which shows the data after the recognition process by the recognizer 1023. Figure 8 is an image of the RAW data of the millimeter wave radar 1080 or the data during processing by the signal processing unit 1022. Figure 9 The recognition result of the recognizer 1023 of the radar recognition processing unit 1020 is shown in black blocks. For comparison, Figure 9 The recognition results of the recognizer 1033 of the LiDAR recognition processing unit 1030 are also shown with gray blocks.

[0142] exist Figure 7 In the scene shown, it is preferable to identify the motorcycle 701 traveling on the road in front of the vehicle as an obstacle. However, houses and fences 702 and 703 are arranged on both sides of the road (or motorcycle 701). Figure 8As shown in , the RAW data of the millimeter wave radar 1080 before being processed by the identifier 1023 contains various information items. Although the RAW data of the millimeter wave radar 1080 has side lobes, a strong reflection 801 from the motorcycle 701 can be confirmed. Note that relatively weak reflections 802 and 803 from the left and right fences, etc. can also be confirmed. The millimeter wave radar 1080 has high sensitivity to metals and low sensitivity to non-metals such as concrete. These objects with weak reflection intensity cannot be identified after passing through the identifier 1023, but their existence can be confirmed from the RAW data. In addition, referring to Figure 9 The recognition result of the identifier 1023 of the radar recognition processing unit 1020 shown in , together with the object 901 that appears to correspond to the motorcycle 701 near 20 meters ahead, objects 902 and 903 that appear to correspond to the house and fences 702 and 703 are also recognized on both sides of the road (or motorcycle 701). In particular, in the RAW data, the recognized object 901 and the recognized object 902 overlap with each other, but since the data has not yet been processed for recognition, the recognized object 901 and the recognized object 902 are recognized in the data even if the reflection intensity is weak. Therefore, the fusion of the RAW data and the recognition result of the identifier 1023 makes it possible to identify the recognized object 901 and the recognized object 902 as separate objects. Of course, in many scenarios, the identifier 1023 of the millimeter wave radar 1080 can only recognize the motorcycle. However, the reflection intensity of the motorcycle is weaker than that of the vehicle. As Figure 7 As shown in , if there are other reflectors near the motorcycle, it will be difficult to capture the motorcycle using only millimeter wave radar 1080. In the data after the recognition process for identifying fruits with a certain reflection intensity or higher, recognized object 901 and recognized object 902 are output as one data block. Figure 21 It schematically shows how a motorcycle 2102 approaches a wall 2101 within the detection range 2100 of the millimeter-wave radar 1080 . Figure 22 The result of identifying the reflected wave of the millimeter wave radar 1080 obtained from the detection range 2100 by the identifier 1023 is schematically shown. With the identifier 1023, the reflection intensity less than the predetermined value is ignored, and the reflection intensity equal to or greater than the predetermined value is identified as an object. Figure 22 In the example shown, a block 2201 integrating a wall 2101 and a motorcycle 2102 is identified as an object. On the contrary, based on the RAW data of the millimeter wave radar 1080, even weak reflection intensities that would be ignored by the identifier 1023 can be identified. Figure 23 As shown in , it is possible to identify the reflection from the wall 2101 and the reflection from the motorcycle 2102 as separate objects 2301 and 2302.

[0143] In addition, in Figure 10 The image captured by the vehicle-mounted camera 1070 is a street scene. Figure 11 , data before being processed by the identifier 1023 of the radar identification processing unit 1020 is shown, and Figure 12 , which is the data after the recognition process by the recognizer 1023. Figure 11 is an image of the RAW data of the millimeter wave radar 1080 or the data during processing by the signal processing unit 1022. Figure 12 The recognition result of the recognizer 1023 of the radar recognition processing unit 1020 is shown in black blocks. For comparison, Figure 12 The recognition results of the recognizer 1033 of the LiDAR recognition processing unit 1030 are shown together with gray blocks.

[0144] Figure 10 The scene is a scene of driving in a narrow alley, which is sandwiched between fences 1001 and 1002 on both sides. Preferably, the fences 1001 and 1002 on both sides can be identified as obstacles. Figure 11 As shown in , the RAW data of the millimeter wave radar 1080 before being processed by the identifier 1023 contains various information items. The fences 1001 and 1002 themselves are not metal and are difficult to be captured by the millimeter wave radar 1080, but reflections 1101 and 1102 that are believed to be caused by cracks or steps in the fences 1001 and 1002 can be confirmed. In addition, referring to Figure 12 According to the recognition result of the identifier 1023 of the radar recognition processing unit 1020 shown in FIG, the identifier 1023 can only discretely recognize some parts 1201 to 1204 in which reflections 1002 from cracks or steps are scattered on the corresponding fences 1001 and 100. However, it is difficult to recognize the fences 1001 and 1002 as a whole, and if the recognition result of the identifier 1033 of the LiDAR 1090 is not used (i.e., fusion processing), it is difficult to recognize the fences 1001 and 1002 as a whole. On the other hand, according to Figure 11 In the RAW data shown in , even if the reflected wave is weak, reflection information 1101 and 1102 indicating the presence of the fence can be acquired.

[0145] Figure 13An example of a target recognition result when the late fusion processing unit 1041 performs fusion processing on the recognition results of the camera recognition processing unit 1010 and the radar recognition processing unit 1020 in the information processing device 1000 is shown. However, "○" is input in the recognition result in which the target is recognized, and "×" is input in the recognition result in which the target is not recognized. In the case where each of the camera recognition processing unit 1010 and the radar recognition processing unit 1020 performs recognition processing on the same target, four modes are assumed: when both processing units can recognize the target (mode 1), when only one processing unit can recognize the target (modes 2 and 3), or when neither processing unit can recognize the target (mode 4). The late fusion processing unit 1041 outputs the target that can be recognized by the camera recognition processing unit 1010 and the radar recognition processing unit 1020 (modes 2 and 3). Figure 13 Meanwhile, targets that can be recognized by only one of the camera recognition processing unit 1010 and the radar recognition processing unit 1020 and targets that cannot be recognized by both are output as unrecognizable (in Figure 13 , enter “×”).

[0146] at the same time, Figure 14 1070 and the millimeter wave radar 1080 in the information processing device 1000. Figure 14 In the corresponding modes 1 to 4, we try to identify Figure 13 . Furthermore, "○" is entered for the recognition result where the target was recognized, and "×" is entered for the recognition result where the target was not recognized. There are also objects that were ignored by the late fusion processing at the determination threshold of the identifier 113 or 123, but could have been recognized by the early fusion processing using RAW data before being ignored at the determination threshold. However, it should be noted that objects with different recognition results between the late fusion processing and the early fusion processing are less likely to be actual objects.

[0147] In mode 1, in which the target can be recognized by both the recognizer 1013 of the camera recognition processing unit 1010 and the recognizer 1023 of the radar recognition processing unit 1020, the target can be recognized similarly using the RAW data of the camera 1070 and the RAW data of the millimeter wave radar 1080. Therefore, the early fusion processing unit 1042 outputs the target that can be recognized (in Figure 14 That is, when there is no difference between the recognition results of the classifiers 1013 and 1023 and the recognition result of the RAW data, the early fusion processing unit 1042 outputs a recognition result similar to that of the late fusion processing unit 1041.

[0148] Furthermore, in mode 2 in which a target can be recognized by the recognizer 1013 of the camera recognition processing unit 1010 but cannot be recognized by the recognizer 1023 of the radar recognition processing unit 1020, in the case where the target can be recognized based on the RAW data of the millimeter wave radar 1080, the early fusion processing unit 1042 outputs that the target can be recognized. For example, in the case where a target with weak reflection intensity and ignored by the recognizer 1023 can be recognized based on the RAW data, etc. Therefore, even a target that cannot be recognized by the late fusion processing unit 1041 can be recognized using the early fusion processing unit 1042 (see Figure 15 ) for recognition. It can be said that the recognition rate of the target is improved by using RAW data with rich information content for early fusion processing.

[0149] Furthermore, in mode 3, in which a target cannot be recognized by the recognizer 1013 of the camera recognition processing unit 1010 but can be recognized by the recognizer 1023 of the radar recognition processing unit 1020, the early fusion processing unit 1042 outputs that the target cannot be recognized if the target still cannot be recognized from the RAW data of the camera 1070. That is, in mode 3, the early fusion processing unit 1042 outputs a recognition result similar to the recognition result of the late fusion processing unit 1041.

[0150] Furthermore, in mode 4 in which neither the recognizer 1013 of the camera recognition processing unit 1010 nor the recognizer 1023 of the radar recognition processing unit 1020 can recognize a target, in a case where the target still cannot be recognized from the RAW data of the camera 1070 but can be recognized based on the RAW data of the millimeter wave radar 1080, the early fusion processing unit 1042 outputs the possibility that a target exists. For example, in a case where a target whose reflection intensity is weak and which is ignored by the recognizer 1023 can be recognized based on the RAW data, etc. Therefore, even a target that cannot be recognized by the late fusion processing unit 1041 can be recognized by using the early fusion processing unit 1042 (see Figure 16 ) for recognition. It can be said that using the rich information of RAW data for early fusion processing improves the recognition rate of the target. However, even with early fusion processing, the recognition rate is not high enough, so "Δ" is input instead of "○".

[0151] Therefore, in Figure 14 In the examples shown, in each of Mode 2 and Mode 4, it can be said that the recognition rate of the target is improved by supplementing the processing result of the late fusion processing unit 1041 with the final recognition result with high reliability, but based on the result processed by the early fusion processing unit 1042, the amount of information is reduced using RAW data that has a large amount of information but also has noise.

[0152] However, as in Figure 14 In the case of Mode 1 and Mode 3 in

[104] , in some cases, the recognition result of the late fusion processing unit 1041 does not change even when the early fusion processing unit 1042 is used. If the early fusion processing unit 1042 is always in operation, there is a concern about adverse effects such as an increase in the processing load and power consumption of the information processing device 1000. Therefore, the recognition processing of the early fusion processing unit 1042 should be activated only when necessary.

[0153] Figure 17 The configuration example of the information processing device 1000 configured to adaptively perform early fusion processing is schematically shown. Figure 17 In, with Figure 1 The same functional modules are denoted by the same reference numerals.

[0154] The determination processing unit 1701 in the fusion processing unit 1042 determines whether the RAW data of the millimeter wave radar 1080 is necessary. If the RAW data is necessary, the determination processing unit 1701 requests the radar recognition processing unit 1020 to output the RAW data of the millimeter wave radar 1080. For example, the determination processing unit 1701 compares the recognition results of the recognizer 1013 of the camera recognition processing unit 1010 with the recognition results of the recognizer 1023 of the radar recognition processing unit 1020. Figure 13 In the case where the camera recognition processing unit 1010 and the LiDAR recognition processing unit 1030 correspond to mode 2 or mode 4, the determination processing unit 1701 determines that RAW data is also necessary and requests the radar recognition processing unit 1020 to output the RAW data 1080 of the millimeter wave radar. Alternatively, the determination processing unit 1701 inputs environmental information (such as weather or other external information). When a phenomenon (such as rain, snow, fog) in which the recognition rate of the recognizers 1013 and 1033 of the camera recognition processing unit 1010 and the LiDAR recognition processing unit 1030 is reduced (or the reliability of recognition is deteriorated) is detected, as well as in a dark place (such as at night or in a tunnel), the determination processing unit 1701 may request the radar recognition processing unit 1020 to output the RAW data of the millimeter wave radar 1080.

[0155] In response to a request from the determination processing unit 1701, the radar recognition processing unit 1020 outputs the RAW data of the millimeter-wave radar 1080. The early fusion processing unit 1042 then performs early fusion processing using the RAW data, or the hybrid fusion processing unit 1043 performs hybrid fusion processing using the RAW data. The fusion processing unit 1040 then outputs a final recognition result, referring to the recognition results of the early fusion processing unit 1042 or the hybrid fusion processing unit 1043 in addition to the recognition result of the late fusion processing unit 1041.

[0156] Figure 18The flowchart shows the Figure 17 The information processing device 1000 shown here performs a target recognition process. However, for simplicity of description, the process is limited to the case where the information processing device 1000 performs a fusion process of the two sensors, the camera 1070 and the millimeter wave radar 1080.

[0157] When the object detection process is started, the camera recognition processing unit 1010 performs image processing on the RAW data (captured image) of the camera 1070 (step S1801 ), and outputs the recognition result of the recognizer 1013 (step S1802 ).

[0158] Furthermore, the radar identification processing unit 1020 performs signal processing on the RAW data of the millimeter wave radar 1080 (step S1803 ) Then, the radar identification processing unit 1020 checks whether an output request for the RAW data has been received (step S1804 ).

[0159] The determination processing unit 1701 in the fusion processing unit 1042 compares the recognition results of the recognizer 1013 of the camera recognition processing unit 1010 with the recognition results of the recognizer 1023 of the radar recognition processing unit 1020 to determine whether the RAW data of the millimeter wave radar 1080 is necessary. In the case that the RAW data is necessary, the determination processing unit 1701 requests the radar recognition processing unit 1020 to output the RAW data of the millimeter wave radar 1080 (as described above). Specifically, in the case of the RAW data of the radar recognition processing unit 1020, the RAW data of the millimeter wave radar 1080 is output. Figure 13 In the case corresponding to mode 2, the determination processing unit 1701 determines that the RAW data of the millimeter wave radar 1080 is necessary.

[0160] Here, if no request for outputting RAW data is received (step S1804: No), the radar identification processing unit 1020 outputs the identification result of the identifier 1023 (step S1805). Furthermore, if a request for outputting RAW data is received (step S1804: Yes), the radar identification processing unit 1020 outputs the RAW data of the millimeter-wave radar 1080 (step S1806) and requests the subsequent-stage fusion processing unit 1040 to perform early fusion processing or hybrid fusion processing using the RAW data of the millimeter-wave radar 1080 (step S1807).

[0161] The fusion processing unit 1040 then performs fusion processing on the processing details of the camera recognition processing unit 1010 and the radar recognition processing unit 1020 (step S1808). If a request for early fusion processing or hybrid fusion processing has been received (step S1809: Yes), the fusion processing unit 1040 performs early fusion processing by the early fusion processing unit 1041 or hybrid fusion processing by the hybrid fusion processing unit 1043 (step S1810). On the other hand, if a request for early fusion processing or hybrid fusion processing has not been received (step S1809: No), the fusion processing unit 1040 performs late fusion processing by the late fusion processing unit 1041 (step S1811).

[0162] Figure 19 Another configuration example of the information processing device 1000 configured to adaptively perform early fusion processing is schematically shown. Figure 19 In, with Figure 1 The same functional modules are denoted by the same reference numerals.

[0163] When the recognizer 1013 cannot recognize the target, or when the recognition rate of the target is insufficient, the camera recognition processing unit 1010 determines that the RAW data of the millimeter wave radar 1080 is necessary, and requests the radar recognition processing unit 1020 to output the RAW data of the millimeter wave radar 1080. In response to the request from the camera recognition processing unit 1010, the radar recognition processing unit 1020 outputs the RAW data of the millimeter wave radar 1080. Then, the early fusion processing unit 1042 performs early fusion processing using the RAW data, or the hybrid fusion processing unit 1043 performs hybrid fusion processing using the RAW data. Then, in addition to the recognition result of the late fusion processing unit 1041, the recognition result of the early fusion processing unit 1042 or the hybrid fusion processing unit 1043 is also referred to, and the fusion processing unit 1040 outputs the final recognition result.

[0164] Figure 20 The flowchart shows the Figure 19 The information processing device 1000 shown here performs a target recognition process. However, for simplicity of description, the process is limited to the case where the information processing device 1000 performs a fusion process of the two sensors, the camera 1070 and the millimeter wave radar 1080.

[0165] When the object detection process is started, the camera recognition processing unit 1010 performs image processing on the RAW data (captured image) of the camera 1070 (step S2001). Then, the camera recognition processing unit 1010 checks whether the image processing result is good (step S2002).

[0166] Here, when the image processing result is good (step S2002: Yes), the camera recognition processing unit 1010 outputs the recognition result of the recognizer 1013 (step S2003). In addition, when the image processing result is not good (step S2002: No), the camera recognition processing unit 1010 requests the radar recognition processing unit 1020 to output the RAW data of the millimeter wave radar 1080 (step S2004). Specifically, in the case of Figure 13 In the case of mode 4, the image processing result is not good.

[0167] Furthermore, the radar identification processing unit 1020 performs signal processing on the RAW data of the millimeter wave radar 1080 (step S2005 ) Then, the radar identification processing unit 1020 checks whether an output request for the RAW data has been received (step S2006 ).

[0168] Here, if no request for outputting RAW data is received (step S2006: No), the radar identification processing unit 1020 outputs the identification result of the identifier 1023 (step S2007). Furthermore, if a request for outputting RAW data is received (step S2006: Yes), the radar identification processing unit 1020 outputs the RAW data of the millimeter-wave radar 1080 (step S2008) and requests the subsequent-stage fusion processing unit 1040 to perform early fusion processing or hybrid fusion processing on the RAW data of the millimeter-wave radar 1080 (step S2009).

[0169] The fusion processing unit 1040 then performs fusion processing on the processing details of the camera recognition processing unit 1010 and the radar recognition processing unit 1020 (step S3010). If a request for early fusion processing or hybrid fusion processing has been received (step S3011: Yes), the fusion processing unit 1040 performs early fusion processing by the early fusion processing unit 1041 or hybrid fusion processing by the hybrid fusion processing unit 1043 (step S3012). On the other hand, if a request for early fusion processing or hybrid fusion processing has not been received (step S3011: No), the fusion processing unit 1040 performs late fusion processing via the late fusion processing unit 1041 (step S3013).

[0170] Industrial Applicability

[0171] The technology disclosed in this specification has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can modify or replace the embodiments without departing from the spirit of the technology disclosed in this specification.

[0172] This specification primarily describes embodiments related to the fusion of vehicle-mounted sensors, but the scope of application of the technology disclosed in this specification is not limited to vehicles. The technology disclosed in this specification can be similarly applied to various types of mobile devices, such as unmanned vehicles (such as drones), robots that move autonomously in a predetermined workspace (home, office, factory, etc.), ships, aircraft, etc. Of course, the technology disclosed in this specification can also be similarly applied to information terminals installed in mobile devices and various non-mobile devices.

[0173] In short, the technology disclosed in this specification has been described in an illustrative form, and the details of the description of this specification should not be interpreted in a limiting manner. In order to determine the spirit of the technology disclosed in this specification, the claims should be considered.

[0174] Note that the technology disclosed in this specification can also have the following configurations.

[0175] (1) An information processing device comprising:

[0176] an identification unit configured to perform identification processing on the object based on the detection signal of the sensor; and

[0177] The processing unit is configured to perform fusion processing on the first data before recognition by the recognition unit and other data. This information processing device has the following effect: by using the first data including information before being ignored by the recognition unit for fusion processing, more objects can be recognized.

[0178] (2) The information processing device according to (1) above, wherein

[0179] The sensor includes a millimeter-wave radar. This information processing device has the following effect: after the identifier used for the millimeter-wave radar has been identified, it can perform fusion processing on the identification results with high probability but small information volume using the rich raw data previously ignored by the identifier based on a threshold, thereby recognizing more objects.

[0180] (3) The information processing device according to (2) above, wherein

[0181] The recognition unit performs each process of distance detection, speed detection, object angle detection, and object tracking based on the detection signal of the sensor before recognition, and

[0182] The first data includes at least one of a detection signal, an object's distance detection result, a speed detection result, an angle detection result, and an object tracking result. This information processing device has the following effect: by fusing the raw data of the millimeter-wave radar, information obtained at each stage of signal processing of the raw data (such as distance, speed, and angle), the object tracking result, and the recognition result of the recognizer, more objects can be recognized.

[0183] (4) The information processing device according to any one of (1) to (3) above, further comprising a second recognition unit configured to perform recognition processing on the object based on the detection signal of the second sensor,

[0184] The processing unit performs at least one of a fusion process of fusing the third data before recognition by the second recognition unit with the first data, a fusion process of fusing the fourth data after recognition by the second recognition unit with the first data, a fusion process of fusing the first data with the second data after recognition by the recognition unit, or a fusion process of fusing the fourth data with the second data. This information processing device has the following effect: by fusing the data before recognition by the recognizers of the first sensor and the second sensor, and fusing the data after recognition by the recognizers of the first sensor and the second sensor, a larger number of objects can be recognized.

[0185] (5) The information processing device according to (4) above, wherein

[0186] The second sensor includes at least one of a camera or a LiDAR. This information processing device has the following effect: by performing fusion processing on the recognition results of the millimeter-wave radar and the camera or LiDAR, and performing fusion processing on the RAW data of the millimeter-wave radar and the camera or LiDAR, it is possible to recognize more objects.

[0187] (6) The information processing device according to (4) or (5) above, wherein

[0188] The processing unit determines a method for using the first data in a fusion process based on the recognition result of the recognition unit and the recognition result of the second recognition unit. This information processing device has the following effect: by adaptively using the first data in a fusion process based on the recognition result of the recognition unit and the recognition result of the second recognition unit, more objects can be recognized.

[0189] (7) The information processing device according to (6) above, wherein

[0190] When the second recognition unit has a high probability of recognition but the recognition unit has a low probability of recognition, the processing unit uses the first data in the fusion process. This information processing device has the following effect: by adaptively using the first data in the fusion process, more objects can be recognized while avoiding unnecessary fusion processing.

[0191] (8) The information processing apparatus according to any one of (4) to (7) above, wherein

[0192] The processing unit determines a method of using the first data in the fusion process based on the recognition result of the second recognition unit. This information processing device has the following effect: by adaptively using the first data in the fusion process, more objects can be recognized while avoiding unnecessary fusion processing.

[0193] (9) The information processing device according to (8) above, wherein

[0194] When the second recognition unit has a low probability of recognition, the processing unit uses the first data in the fusion process. This information processing device has the following effect: by adaptively using the first data in the fusion process, more objects can be recognized while avoiding unnecessary fusion processing.

[0195] (10) An information processing method comprising:

[0196] a recognition step of performing recognition processing on the object based on the detection signal of the sensor; and

[0197] The processing step includes fusing the first data before identification in the identification step with other data. This information processing method has the following effect: by fusing the second data after identification in the identification step with the first data including information before the identification step, which is ignored by the threshold, more objects can be identified.

[0198] (11) The information processing method according to (10) above, wherein

[0199] In the processing step, at least one of the following fusion processing is performed: fusion processing of the third data with the first data before the second identification unit is identified, fusion processing of the fourth data with the first data after the second identification unit is identified, fusion processing of the first data with the second data after the identification unit is identified, or fusion processing of the fourth data with the second data.

[0200] (12) A computer program described in a computer-readable format for causing a computer to:

[0201] an identification unit configured to perform identification processing on the object based on the detection signal of the sensor; and

[0202] The processing unit is configured to perform fusion processing on the first data before identification by the identification unit and other data.

[0203] (13) The computer program according to (12) above, wherein

[0204] The processing unit performs at least one of the following fusion processing: fusion processing of the third data with the first data before the second recognition unit is recognized, fusion processing of the fourth data with the first data after the second recognition unit is recognized, fusion processing of the first data with the second data after the recognition unit is recognized, or fusion processing of the fourth data with the second data.

[0205] (14) A mobile device comprising:

[0206] mobile devices;

[0207] sensor;

[0208] an identification unit configured to perform identification processing on the object based on the detection signal of the sensor;

[0209] a processing unit configured to perform a fusion process on the first data before identification by the identification unit and other data; and

[0210] The control unit is configured to control the mobile device based on a processing result of the processing unit. This mobile device has the following effect: by performing a fusion process on the second data after recognition by the recognition unit and the first data including information before being ignored by the recognition unit based on the threshold, more objects can be recognized and the mobile device can be completed to avoid collision with the objects.

[0211] (15) The mobile device according to (14) above, wherein

[0212] The processing unit performs at least one of the following fusion processing: fusion processing of the third data with the first data before the second recognition unit is recognized, fusion processing of the fourth data with the first data after the second recognition unit is recognized, fusion processing of the first data with the second data after the recognition unit is recognized, or fusion processing of the fourth data with the second data.

[0213] Reference Signs List

[0214] 100 Vehicle Control Systems

[0215] 101 Input Unit

[0216] 102 Data Acquisition Unit

[0217] 103 Communication Unit

[0218] 104 In-vehicle equipment

[0219] 105 Output Control Unit

[0220] 106 output units

[0221] 107 Drive system control unit

[0222] 108 drive system

[0223] 109 Body system control unit

[0224] 110 Body System

[0225] 111 storage units

[0226] 112 Automatic driving control unit

[0227] 121 Communication Network

[0228] 131 Detection Unit

[0229] 132 Self-position estimation unit

[0230] 133 Situation Analysis Unit

[0231] 134 planning units

[0232] 135 Motion Control Unit

[0233] 141 External vehicle information detection unit

[0234] 142 In-vehicle information detection unit

[0235] 143 Vehicle status detection unit

[0236] 151 Map Analysis Unit

[0237] 152 Traffic rules recognition unit

[0238] 153 Situation Identification Unit

[0239] 154 Situation Prediction Unit

[0240] 161 Route Planning Unit

[0241] 162 Behavior Planning Unit

[0242] 163 Action Planning Unit

[0243] 171 Emergency Avoidance Unit

[0244] 172 Acceleration-deceleration control unit

[0245] 173 Direction Control Unit

[0246] 1000 Information Processing Equipment

[0247] 1010 Camera Recognition Processing Unit

[0248] 1011 RAW data processing unit

[0249] 1012 Signal Processing Unit

[0250] 1013 Identifier

[0251] 1020 Radar Identification Processing Unit

[0252] 1021 RAW data processing unit

[0253] 1022 signal processing unit

[0254] 1023 Recognizer

[0255] 1030 LiDAR recognition processing unit

[0256] 1031 RAW data processing unit

[0257] 1032 Signal Processing Unit

[0258] 1033 Identifier

[0259] 1040 Fusion Processing Unit

[0260] 1041 Late Fusion Processing Unit

[0261] 1042 Early Fusion Processing Unit

[0262] 1043 Hybrid Fusion Processing Unit

[0263] 1050 ECT

[0264] 1060 Actuator (ACT)

[0265] 1070 Camera

[0266] 1080 mmWave radar

[0267] 1090 LiDAR

[0268] 601 distance detection unit

[0269] 602 Speed ​​Detection Unit

[0270] 603 Angle Detection Unit

[0271] 604 Tracking Unit

[0272] 605 MISC processing unit.

Claims

1. An information processing device, comprising: a first recognition unit configured to perform a first recognition process on the object based on a detection signal of a first sensor, wherein the first sensor is a millimeter wave radar, and the first recognition process generates a first recognition result; a second recognition unit configured to perform a second recognition process on the object based on a detection signal of a second sensor, wherein the second sensor is a camera, and the second recognition process generates a second recognition result; as well as A processing unit, wherein the processing unit is configured to: comparing the first recognition result and the second recognition result; and Based on the comparison indicating that first raw data from the first sensor is necessary, early fusion processing is performed on the first raw data from the first sensor and the second raw data from the second sensor, wherein the first raw data from the first sensor is necessary when the object can be recognized by the second recognition unit but not by the first recognition unit, and the object can be recognized based on the first raw data from the first sensor.

2. The information processing device according to claim 1, wherein Before executing the first recognition process, the first recognition unit is configured to perform each of the processes of distance detection, speed detection, angle detection of the object, and tracking of the object based on the detection signal of the first sensor, and The first original data includes at least one of a detection signal of the first sensor, a distance detection result of the object, a speed detection result, an angle detection result, and a tracking result of the object.

3. The information processing device according to claim 1, The processing unit is configured to perform hybrid fusion processing on the second recognition result and the first original data and / or the second original data. The information processing device according to claim 3 , wherein The processing unit is configured to determine a method of using the first original data in the hybrid fusion process based on at least one of a first recognition result of the first recognition unit and a second recognition result of the second recognition unit. The information processing device according to claim 3 , wherein The processing unit is configured to determine a method of using the first original data in the hybrid fusion process based on the second recognition result of the second recognition unit.

6. An information processing method, comprising: a first recognition step, wherein the first recognition step performs a first recognition process on the object based on a detection signal of a first sensor, wherein the first sensor is a millimeter wave radar, and the first recognition process generates a first recognition result; a second recognition step, wherein the second recognition step performs a second recognition process on the object based on a detection signal of a second sensor, wherein the second sensor is a camera, and the second recognition process generates a second recognition result; as well as Processing steps include: comparing the first recognition result and the second recognition result; as well as Based on the comparison indicating that first raw data from the first sensor is necessary, early fusion processing is performed on the first raw data from the first sensor and the second raw data from the second sensor, wherein the first raw data from the first sensor is necessary when the object can be identified by the second identification step but not by the first identification step, and the object can be identified based on the first raw data from the first sensor.

7. The information processing method according to claim 6, The processing step includes performing a mixed fusion process on the second recognition result and the first original data and / or the second original data.

8. A computer program product recorded on a medium in a computer-readable format, for causing a computer to: a first recognition unit configured to perform a first recognition process on the object based on a detection signal of a first sensor, wherein the first sensor is a millimeter wave radar, and the first recognition process generates a first recognition result; a second recognition unit configured to perform a second recognition process on the object based on a detection signal of a second sensor, wherein the second sensor is a camera, and the second recognition process generates a second recognition result; as well as A processing unit, wherein the processing unit is configured to: comparing the first recognition result and the second recognition result; and Based on the comparison indicating that first raw data from the first sensor is necessary, early fusion processing is performed on the first raw data from the first sensor and the second raw data from the second sensor, wherein the first raw data from the first sensor is necessary when the object can be recognized by the second recognition unit but not by the first recognition unit, and the object can be recognized based on the first raw data from the first sensor.

9. The computer program product of claim 8, wherein The processing unit is configured to perform a hybrid fusion process on the second recognition result and the first original data and / or the second original data.

10. A mobile device comprising: Mobile devices; First sensor; a first recognition unit configured to perform a first recognition process on the object based on a detection signal of a first sensor, wherein the first sensor is a millimeter wave radar, and the first recognition process generates a first recognition result; a second recognition unit configured to perform a second recognition process on the object based on a detection signal of a second sensor, wherein the second sensor is a camera, and the second recognition process generates a second recognition result; as well as A processing unit, wherein the processing unit is configured to: comparing the first recognition result and the second recognition result; and performing early fusion processing on the first raw data from the first sensor and the second raw data from the second sensor based on an indication that the comparison is necessary, wherein the first raw data from the first sensor is necessary when the object can be recognized by the second recognition unit but not by the first recognition unit, and the object can be recognized based on the first raw data from the first sensor; as well as A control unit is configured to control the mobile device based on a processing result of the processing unit. The mobile device according to claim 10 , wherein The processing unit is configured to perform a hybrid fusion process on the second recognition result and the first original data and / or the second original data.

Images