Information processing equipment, information processing methods and procedures
By combining the recognition methods of camera, millimeter-wave radar and LiDAR, the problem of erroneous obstacle identification on tilted roads is solved, accurate collision judgment is achieved in harsh environments, and the safety of the driver assistance system is improved.
Patent Information
- Application Number
- CN201980082175.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-12-18
- Filing Date
- 2019-12-05
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2039-12-05
AI Technical Summary
Existing driver assistance systems are prone to incorrect judgments when identifying obstacles on tilted roads, especially in harsh environments, which leads to inaccurate possible collision judgments and even inaccurate inclination.
Using a combination of multiple recognition methods, including camera, millimeter wave radar and LiDAR, the collision determination is selectively performed using the respective recognition results according to the environment around the vehicle, and the coordinated work of the object recognition unit, the tilt recognition unit and the collision determination unit, Identify the inclination of the road and the existence of obstacles.
It effectively suppresses erroneous judgments caused by environmental changes, improves the accuracy of obstacle identification on inclined roads, and ensures the safe driving of the vehicle.
Smart Images

Figure CN113196106B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and particularly to an information processing device, an information processing method, and a program that enable appropriate collision determination with respect to an obstacle on a sloped road. Background Art
[0002] In advanced driver assistance systems (ADAS) such as adaptive cruise control (ACC) and collision avoidance safety (PCS) designed to improve convenience and safety, it is necessary to determine whether the vehicle can drive after confirming the presence or absence of an obstacle to prevent system failure.
[0003] Particularly in the PCS, it may be erroneously determined that there is a possibility of collision with respect to a maintenance hole or a fence on a sloping road surface, and this may result in sudden braking.
[0004] Therefore, it is necessary to perform collision determination after recognizing the inclination on the road surface.
[0005] A technology for estimating the inclination based on a change in the vertical position of a rear slope portion of a preceding vehicle using a camera has been proposed as a technology for recognizing the inclination (see Patent Document 1).
[0006] Citation List
[0007] Patent Literature
[0008] Patent Document 1: Japanese Patent Application Laid-Open No. 2012-088217 Summary of the invention
[0009] Technical issues
[0010] However, since a camera is used, the technique of Patent Document 1 may be affected by changes in the environment in which the image is captured, such as at night or during bad weather. Therefore, the accuracy of recognizing the preceding vehicle may be reduced, and this may lead to erroneous determination.
[0011] Furthermore, when there is no preceding vehicle, the inclination cannot be estimated because a change in the vertical position of the taillight portion of the preceding vehicle needs to be detected.
[0012] The present disclosure has been achieved in view of the above circumstances, and in particular, the present disclosure aims to perform appropriate collision determination with respect to an obstacle on a sloped road without being affected by the environment or the presence or absence of a preceding vehicle.
[0013] Solution to the problem
[0014] An information processing device according to one aspect of the present disclosure is the following information processing device, which includes: an object recognition unit that uses multiple recognition methods to recognize an object in front of a vehicle and outputs the results of the object recognition of each of the multiple recognition methods; and a collision determination unit that selects the result of the object recognition from the results of the object recognition of the multiple recognition methods according to the environment around the vehicle, and determines whether there is a possibility of collision with the object based on the selected result of the object recognition.
[0015] An information processing method and a program according to an aspect of the present disclosure correspond to an information processing device.
[0016] In this aspect of the present disclosure, multiple recognition methods are used to recognize objects in front of a vehicle; the results of object recognition of each of the multiple recognition methods are output; the result of object recognition is selected from the results of object recognition of the multiple recognition methods based on the environment around the vehicle; and whether there is a possibility of collision with the object is determined based on the selected result of object recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a diagram for describing false detection in collision determination.
[0018] Figure 2 An outline of collision determination using a millimeter wave radar is shown.
[0019] Figure 3 An outline of LiDAR collision determination is shown.
[0020] Figure 4 is a block diagram showing an example of the configuration of a vehicle control system of the present disclosure.
[0021] Figure 5 is a graph showing that erroneous decisions are suppressed Figure 4 A block diagram of an example of a configuration of a vehicle control system.
[0022] Figure 6 This is a diagram for describing collision determination using LiDAR.
[0023] Figure 7 This is a diagram for describing collision determination using LiDAR.
[0024] Figure 8 is a flowchart showing the drive control process.
[0025] Fig. 9 An example of the configuration of a general-purpose computer is shown. DETAILED DESCRIPTION
[0026] Advantageous embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in the specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals to omit repeated description.
[0027] The following describes an embodiment for implementing the present technology. The description is made in the following order.
[0028] 1. Overview of the Disclosure
[0029] 2. Example of Configuration of Vehicle Control System for Controlling the Vehicle of the Present Disclosure
[0030] 3. Configuration that prevents misjudgment of obstacles on inclined roads
[0031] 4. Collision determination and processing
[0032] 5. Example of series processing using software
[0033] <<1. Overview of the Disclosure>>
[0034] <Error detection in collision determination>
[0035] An overview of the present disclosure is described.
[0036] The vehicle of the present disclosure performs appropriate collision determination with respect to an obstacle on a sloped road using an advanced driver assistance system (ADAS) aimed at improving convenience and safety.
[0037] Discuss the following situations: e.g. Figure 1 As shown in the upper part of FIG. 1 , a vehicle 1 - 1 including an automatic driving function or a driving assist function travels on a flat road FL1 in the right direction in the figure. Here, Figure 1 It is shown that in the upper part, the vehicle 1 - 1 is traveling in the right direction in the figure as viewed from the side of the vehicle 1 - 1 , where the inclined road SL1 is connected to the right end of the flat road FL1 in the figure.
[0038] Furthermore, a maintenance hole 2-1 is buried in the road surface of the inclined road SL1.
[0039] The vehicle 1-1 includes an external recognition sensor 1a-1 in order to perform an automatic driving function or a driving assistance function. In addition, the vehicle 1-1 uses the external recognition sensor 1a-1 to sense in front of the driving direction as shown by the dotted line. The vehicle 1-1 detects the presence or absence of an obstacle on the road surface, and when an obstacle is detected, the vehicle 1-1 performs an emergency stop movement or an emergency avoidance movement to ensure safety.
[0040] Examples of the exterior recognition sensor 1 a - 1 include a camera, a millimeter wave radar, and LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).
[0041] Here, when the exterior recognition sensor 1a-1 is a camera, even the size and type of the object in front can be recognized by using image recognition in a bright environment during the daytime in clear weather. Figure 1 In this case, it can be identified that the maintenance hole 2-1 is buried in the inclined road SL1.
[0042] Therefore, the maintenance hole 2-1 is recognized as existing on the road surface, but the maintenance hole 2-1 is not recognized as an obstacle.
[0043] However, in the case where the exterior recognition sensor 1a-1 is a camera and uses a captured image, when the surrounding environment changes, such as at night or during bad weather, there is a possibility that it cannot be recognized that the inclined road SL1 is located ahead and the maintenance hole 2-1 is buried in the inclined road SL1. In this case, the automatic driving function or the driving assistance function of the vehicle 1-1 cannot work.
[0044] Note that, in the case where the external recognition sensor 1a-1 is a camera, Figure 1 In the state shown in , the fact that the maintenance hole 2-1 cannot be detected is only a problem when the surrounding environment changes, such as at night or during bad weather. Problems due to, for example, false detection do not occur.
[0045] Next, the case where the exterior recognition sensor 1a-1 is a millimeter wave radar is discussed.
[0046] When the exterior recognition sensor 1a-1 is a millimeter wave radar, it can recognize that something exists at the location of the maintenance hole 2-1, for example, at night or during bad weather, but cannot recognize whether the target object is, for example, the inclined road SL1 or the maintenance hole 2-1.
[0047] However, when the exterior recognition sensor 1a-1 is a millimeter wave radar, the exterior recognition sensor 1a-1 can recognize that a certain target object is located ahead in the traveling direction, but cannot recognize that the maintenance hole 2-1 is buried in the road surface of the inclined road SL1.
[0048] Therefore, when the exterior recognition sensor 1a-1 is a millimeter wave radar, Figure 1 In the upper state, the presence of an obstacle on the flat road FL1 may be erroneously detected, the obstacle being an obstacle having a certain height and which may collide with the vehicle 1 - 1 .
[0049] Therefore, for example, the vehicle 1-1 erroneously detects that there is a possibility of collision with an obstacle, and performs an emergency stop movement or an emergency avoidance movement to avoid the collision. Such an emergency stop movement or such an emergency avoidance movement may result in, for example, an accident.
[0050] Note that similar phenomena may occur in the following situations: Figure 1 As shown in the lower part of , vehicle 1-2 includes an external recognition sensor 1a-2 which is a camera or a millimeter wave radar and travels downhill on the inclined road SL2 in the right direction in the figure; a flat road FL2 is connected to the front end of the inclined road SL2; and a maintenance hole 2-2 is set in the flat road FL2.
[0051] Note that when there is a preceding vehicle, the millimeter wave radar irradiates radio waves further ahead of the preceding vehicle, while causing reflected waves from the road surface under the body of the preceding vehicle, and receives the reflected waves. This makes it possible to detect the presence or absence of an object further ahead of the preceding vehicle.
[0052] Here, the following situations are discussed: Figure 2 As shown in the upper part of , the vehicle 11 includes an external recognition sensor 11a as a millimeter wave radar, and travels on a flat road FL in the right direction in the figure while sensing forward relative to the traveling direction; a sloped road SL is located in front; and a maintenance hole 12 is buried in the sloped road SL.
[0053] exist Figure 2 In the upper part, since the wave irradiated by the external recognition sensor 11a is reflected by the maintenance hole 12 in the front, the vehicle 11 can only recognize that there may be an obstacle, but the vehicle 11 cannot recognize the type of the object.
[0054] Therefore, such information alone does not allow the vehicle 11 to determine whether the object ahead is a maintenance hole 12 buried in the inclined road SL and with which there is no possibility of collision or whether the object ahead is an obstacle with which there is a possibility of collision.
[0055] In this case, in order to suppress the occurrence of erroneous detection, the vehicle 11 has no option but to notify the driver that a certain object exists, but it is impossible to determine what kind of object the certain object is.
[0056] However, if Figure 2 As shown in the lower part of FIG. 1 , when the preceding vehicle 11′ exists in front of the vehicle 11 and travels in the right direction in the figure, it is possible to detect whether the preceding vehicle 11′ has passed the same position as the inclined road SL or the maintenance hole 12. In this case, it is unknown whether the maintenance hole 12 is an obstacle. However, when the preceding vehicle 11′ has passed the position where some kind of object exists, the vehicle 11 can at least recognize that the detected object is not an obstacle with which there is a possibility of collision.
[0057] Therefore, in this disclosure, as in Figure 2 As in the case of the vehicle 11, the millimeter wave radar is used as the exterior recognition sensor 11a. Figure 2As shown in the lower part of , when the vehicle 11 continues to travel to identify an object that may be an obstacle, the vehicle 11 determines whether there is a possibility of collision with the object identified as a possible obstacle based on whether the preceding vehicle 11 ′ has passed the position of the detected object.
[0058] In other words, when the vehicle 11 of the present disclosure recognizes an object ahead, when the preceding vehicle 11' passes the position of the object, the vehicle 11 determines that the recognized object is not an obstacle. This results in suppressing the occurrence of erroneous detection.
[0059] However, when the exterior recognition sensor 11 a is a millimeter wave radar, unless there is a preceding vehicle that travels through a position where there is an object that may be an obstacle, the occurrence of erroneous detection cannot be suppressed.
[0060] Here, a case where LiDAR is used as the exterior recognition sensor 11 a is discussed.
[0061] LiDAR emits laser light in the direction in which ranging is performed, receives reflected light, obtains the distance using the round-trip time period of light, and acquires point group information at specified intervals in the horizontal and vertical directions.
[0062] In other words, e.g. Figure 3 As shown in FIG. 1 , when the vehicle 11 travels on the flat road FL in the right direction in the figure, the exterior recognition sensor 11 a emits, for example, five laser beams in the vertical direction. Then, the exterior recognition sensor 11 a obtains the distance from the difference between the timing of emitting each beam of light and the timing of receiving the beam reflected from the maintenance hole 12 provided on the front inclined road SL to obtain the point group information about the maintenance hole 12.
[0063] In this case, when the vehicle 11 travels in the right direction in the figure, as time passes in the order of time t1, time t2, and time t3, the distance between the inclined road SL and the vehicle 11 decreases.
[0064] Here, at time t1, two of the five laser beams are reflected from the maintenance hole 12 buried in the inclined road SL. Then, at time t2, three of the five laser beams are reflected from the maintenance hole 12 buried in the inclined road SL, and at time t3, four of the five laser beams are reflected from the maintenance hole 12 buried in the inclined road SL.
[0065] In other words, as the vehicle 11 gets closer to the inclined road SL, the number of beams of laser light reflected from the maintenance hole 12 increases.
[0066] In other words, as the external recognition sensor 11a as LiDAR gets closer to the maintenance hole 12, the number of beams of laser light reflected from the maintenance hole 12 changes. This makes it possible to identify the size of the maintenance hole 12 based on the three-dimensional point group information, and to identify the inclination of the inclined road SL based on the time series change of the size of the maintenance hole 12.
[0067] Therefore, using the point group information enables the vehicle 11 to recognize that the object ahead is the inclined road SL and the maintenance hole 12 is buried in the inclined road SL, thus suppressing the occurrence of erroneous detection such as erroneously detecting that the object ahead is an obstacle.
[0068] Therefore, in the present disclosure, three types of external recognition sensors 11a, which are a camera, a millimeter wave radar, and a LiDAR, are combined, and respective sensing results are selectively used according to the environment. This results in suppressing erroneous detection in object detection.
[0069] In other words, for example, in bright daylight in clear weather, the result of recognition based on the image captured by the camera used as the exterior recognition sensor 11a is used. For example, when there is a preceding vehicle at night or during bad weather, the result of sensing by the millimeter wave radar is used, and when there is no preceding vehicle at night or during bad weather, the result of sensing by the LiDAR is used.
[0070] As described above, when the sensing result obtained by the exterior recognition sensor 11 a including a plurality of sensors is selectively used according to the environment, this makes it possible to appropriately perform collision determination according to the characteristics of each of the plurality of sensors and thus suppress the occurrence of erroneous detection.
[0071] <<2. Example of Configuration of Vehicle Control System for Controlling Vehicle of the Present Disclosure>>
[0072] Next, refer to Figure 4 A block diagram depicts a vehicle control system for a vehicle of the present disclosure.
[0073] Figure 4 1 is a block diagram showing an example of a schematic functional configuration of a vehicle control system 100 , which is an example of a mobile object control system to which the present technology is applied. The vehicle control system 100 is mounted on a vehicle 11 .
[0074] Note that when distinguishing a vehicle provided with the vehicle control system 100 from other vehicles, the vehicle provided with the vehicle control system 100 will be referred to as an own car or an own vehicle hereinafter.
[0075] The vehicle control system 100 includes an input unit 101, a data acquisition unit 102, a communication unit 103, an on-board equipment 104, an output controller 105, an output unit 106, a powertrain controller 107, a powertrain system 108, a vehicle body-related controller 109, a vehicle body-related system 110, a storage unit 111, and an automatic driving controller 112. The input unit 101, the data acquisition unit 102, the communication unit 103, the output controller 105, the powertrain controller 107, the vehicle body-related controller 109, the storage unit 111, and the automatic driving controller 112 are connected to each other through a communication network 121. For example, the communication network 121 includes a bus or an on-board communication network conforming to any standard, such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), or FlexRay (registered trademark). Note that the respective structural elements of the vehicle control system 100 may be directly connected to each other without using the communication network 121.
[0076] Note that the description of the communication network 121 will be omitted below when the various structural elements of the vehicle control system 100 communicate with each other through the communication network 121. For example, when the input unit 101 and the automatic driving controller 112 communicate with each other through the communication network 121, it will be simply described as the input unit 101 and the automatic driving controller 112 communicating with each other.
[0077] The input section 101 includes devices used by the passenger to input various data, instructions, and the like. For example, the input section 101 includes operating devices such as a touch panel, a button, a microphone, a switch, and a lever; operating devices that can input by methods other than manual operation such as voice or gestures, and the like. Alternatively, for example, the input section 101 may be equipment connected externally using infrared or other radio waves, such as a remote control device; or mobile equipment or wearable equipment compatible with the operation of the vehicle control system 100. The input section 101 generates an input signal based on the data, instructions, and the like input by the passenger, and supplies the generated input signal to each structural element of the vehicle control system 100.
[0078] The data acquisition section 102 includes various sensors and the like for acquiring data used for processing performed by the vehicle control system 100 , and supplies the acquired data to the various structural elements of the vehicle control system 100 .
[0079] For example, the data acquisition unit 102 includes various sensors for detecting the state of the own car, for example. Specifically, for example, the data acquisition unit 102 includes: a gyroscope; an acceleration sensor; an inertial measurement unit (IMU); and sensors for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the number of revolutions of the engine, the number of revolutions of the motor, the rotation speed of the wheels, etc.
[0080] In addition, for example, the data acquisition unit 102 includes various sensors for detecting information about the outside of the own car. Specifically, for example, the data acquisition unit 102 includes image capture devices such as a time-of-flight (TOF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. In addition, for example, the data acquisition unit 102 includes an environmental sensor for detecting weather, meteorological phenomena, etc., and a surrounding information detection sensor for detecting objects around the own car. For example, the environmental sensor includes a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, etc. The surrounding information detection sensor includes an ultrasonic sensor, a radar, a LiDAR (light detection and ranging, laser imaging detection and ranging), a sonar, etc.
[0081] In addition, for example, the data acquisition unit 102 includes various sensors for detecting the current position of the own car. Specifically, for example, the data acquisition unit 102 includes, for example, a global navigation satellite system (GNSS) receiver that receives GNSS signals from GNSS satellites.
[0082] In addition, for example, the data acquisition section 102 includes various sensors for detecting information about the interior of the vehicle. Specifically, for example, the data acquisition section 102 includes an image capture device that captures an image of the driver, a biometric sensor that detects biometric information of the driver, a microphone that collects sounds inside the vehicle, etc. For example, the biometric sensor is provided on a seat surface, a steering wheel, etc., and detects biometric information of a passenger sitting on the seat or a driver holding the steering wheel.
[0083] The communication unit 103 communicates with the vehicle-mounted equipment 104 and various off-vehicle equipment, servers, base stations, etc., transmits data supplied by various structural elements of the vehicle control system 100, and supplies received data to various structural elements of the vehicle control system 100. Note that there is no particular limitation on the communication protocol supported by the communication unit 103. The communication unit 103 may also support multiple types of communication protocols.
[0084] For example, the communication section 103 wirelessly communicates with the in-vehicle equipment 104 using wireless LAN, Bluetooth (registered trademark), near field communication (NFC), wireless USB (WUSB), etc. In addition, for example, the communication section 103 communicates with the in-vehicle equipment 104 by wire through a connection terminal (not shown) (and a cable if necessary) using a universal serial bus (USB), a high-definition multimedia interface (HDMI) (registered trademark), a mobile high-definition link (MHL), etc.
[0085] In addition, for example, the communication unit 103 communicates with equipment (e.g., an application server or a control server) in an external network (e.g., the Internet, a cloud network, or a carrier-specific network) through a base station or an access point. In addition, for example, the communication unit 103 communicates with a terminal (e.g., a terminal of a pedestrian or a store, or a machine-type communication (MTC) terminal) near the own car using a peer-to-peer (P2P) technology. In addition, for example, the communication unit 103 performs V2X communication such as vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication between the own car and the home, and vehicle-to-pedestrian communication. In addition, for example, the communication unit 103 includes a beacon receiver that receives radio waves or electromagnetic waves transmitted from, for example, a radio station installed on a road, and obtains information about, for example, the current position, traffic congestion, traffic regulations, or required time.
[0086] Examples of the in-vehicle equipment 104 include mobile equipment or wearable equipment of passengers, information equipment introduced into or attached to one's own car, and a navigation device that searches for a route to any destination.
[0087] The output controller 105 controls the output of various information to the passengers of the own car or to the outside of the own car. For example, the output controller 105 generates an output signal including at least one of visual information (such as image data) or audio information (such as sound data), supplies the output signal to the output section 106, and thereby controls the output of the visual information and the audio information from the output section 106. Specifically, for example, the output controller 105 combines the data of images captured by different image capturing devices of the data acquisition section 102, generates a bird's-eye view image, a panoramic image, etc., and supplies the output signal including the generated image to the output section 106. In addition, for example, the output controller 105 generates sound data including, for example, a warning beep or a warning message that warns of dangers such as collision, contact, or entry into a danger zone, and supplies the output signal including the generated sound data to the output section 106.
[0088] The output unit 106 includes a device capable of outputting visual information or audio information to a passenger of the own car or outside the own car. For example, the output unit 106 includes a display device, an instrument panel, an audio speaker, headphones, a wearable device such as a glasses-type display for wearing on a passenger, a projector, a lamp, etc. Instead of a device including a conventional display, the display device included in the output unit 106 may be a device such as a head-up display, a transparent display, or a device including an augmented reality (AR) display function that displays visual information in the driver's field of view.
[0089] The powertrain controller 107 generates various control signals and supplies them to the powertrain 108, thereby controlling the powertrain 108. In addition, the powertrain controller 107 supplies control signals to structural elements other than the powertrain 108 as necessary to, for example, notify them of the state of controlling the powertrain 108.
[0090] The powertrain 108 includes various devices related to the powertrain of the own vehicle. For example, the powertrain 108 includes: a driving force generating device that generates driving force, such as an internal combustion engine and a driving motor; a driving force transmission mechanism for transmitting driving force to wheels; a steering mechanism that adjusts the steering angle; a braking device that generates braking force; an anti-lock braking system (ABS); an electronic stability control (ESC) system; an electric steering device, etc.
[0091] The body-related controller 109 generates various control signals and supplies them to the body-related system 110, thereby controlling the body-related system 110. In addition, the body-related controller 109 supplies control signals to structural elements other than the body-related system 110 as necessary to, for example, notify them of the state of controlling the body-related system 110.
[0092] The vehicle body related system 110 includes various vehicle body related devices provided to the vehicle body. For example, the vehicle body related system 110 includes a keyless entry system, a smart key system, a power window device, a power seat, a steering wheel, an air conditioner, various lights (such as headlights, taillights, brake lights, flashers, and fog lights), etc.
[0093] For example, the storage unit 111 includes a read-only memory (ROM), a random access memory (RAM), a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. The storage unit 111 stores therein various programs, data, etc. used by various structural elements of the vehicle control system 100. For example, the storage unit 111 stores therein map data such as a three-dimensional high-precision map, a global map, and a local map. A high-precision map is a dynamic map, etc. A global map is less accurate and covers a wider area than a high-precision map. A local map includes information about the surrounding environment of the own car.
[0094] The automatic driving controller 112 performs control related to automatic driving such as automatic driving or driving assistance. Specifically, for example, the automatic driving controller 112 performs cooperative control aimed at realizing the functions of an advanced driver assistance system (ADAS), including collision avoidance or impact mitigation of the own car, driving behind a preceding vehicle based on the distance between vehicles, driving while maintaining a vehicle speed, warning of a collision of the own car, warning of a lane deviation of the own car, etc. In addition, for example, the automatic driving controller 112 performs cooperative control aimed at realizing automatic driving such as autonomous driving without an operation performed by a driver. The automatic driving controller 112 includes a detector 131, an ego position estimator 132, a state analyzer 133, a planning unit 134, and a mobile controller 135.
[0095] The detector 131 detects various information required for controlling the automatic driving. The detector 131 includes a vehicle exterior information detector 141 , a vehicle interior information detector 142 , and a vehicle state detector 143 .
[0096] The vehicle external information detector 141 performs a process of detecting information about the outside of the own car based on data or signals from each structural element of the vehicle control system 100. For example, the vehicle external information detector 141 performs a process of detecting, identifying and tracking objects around the own car and a process of detecting the distance of the object. Examples of detection target objects include vehicles, people, obstacles, structures, roads, traffic lights, traffic signs and road signs. In addition, for example, the vehicle external information detector 141 performs a process of detecting the environment around the own car. Examples of detection target surrounding environments include weather, temperature, humidity, brightness and road conditions. The vehicle external information detector 141 supplies data indicating the result of the detection process to, for example, the self-position estimator 132, the map analyzer 151 of the state analyzer 133, the traffic rule recognition unit 152 and the state recognition unit 153, and the emergency avoidance unit 171 of the mobile controller 135.
[0097] The vehicle interior information detector 142 performs processing for detecting information about the interior of the vehicle based on data or signals from each structural element of the vehicle control system 100. For example, the vehicle interior information detector 142 performs processing for authenticating and identifying the driver, processing for detecting the driver's state, processing for detecting passengers, and processing for detecting the interior environment of the vehicle. Examples of the driver's detection target state include physical condition, degree of arousal, degree of concentration, degree of fatigue, and direction of sight. Examples of detection target interior vehicle environments include temperature, humidity, brightness, and odor. The vehicle interior information detector 142 supplies data indicating the result of the detection processing to, for example, the state recognition unit 153 of the state analyzer 133 and the emergency avoidance unit 171 of the mobile controller 135.
[0098] The vehicle state detector 143 performs a process of detecting the state of the own vehicle based on data or signals from each structural element of the vehicle control system 100. Examples of the detection target state of the own vehicle include speed, acceleration, steering angle, presence or absence of abnormality and details thereof, drive operation state, position and inclination of an electric seat, state of a door lock, and state of other vehicle-mounted equipment. The vehicle state detector 143 supplies data indicating the result of the detection process to, for example, the state recognition section 153 of the state analyzer 133 and the emergency avoidance section 171 of the mobile controller 135.
[0099] The self-position estimator 132 performs processing for estimating the position, posture, etc. of the own car based on data or signals from various structural elements such as the vehicle external information detector 141 and the state recognition unit 153 of the state analyzer 133 of the vehicle control system 100. In addition, the self-position estimator 132 generates a local map (hereinafter referred to as a self-position estimation map) for estimating the self-position as needed. For example, the self-position estimation map is a high-precision map using a technology such as simultaneous localization and mapping (SLAM). The self-position estimator 132 supplies data indicating the result of the estimation processing to the map analyzer 151, the traffic rule recognition unit 152, and the state recognition unit 153 of the state analyzer 133. In addition, the self-position estimator 132 stores the self-position estimation map in the storage unit 111.
[0100] The state analyzer 133 performs processing for analyzing the state of the own vehicle and its surrounding environment. The state analyzer 133 includes a map analyzer 151 , a traffic regulation recognition unit 152 , a state recognition unit 153 , and a state prediction unit 154 .
[0101] The map analyzer 151 performs processing of analyzing various maps stored in the storage section 111 and constructs a map including information required for the automatic driving process, using data or signals from various structural elements such as the self-position estimator 132 and the vehicle external information detector 141 of the vehicle control system 100 as needed. The map analyzer 151 supplies the constructed map to, for example, the traffic rule recognition section 152, the state recognition section 153, and the state prediction section 154, and the route planning section 161, the behavior planning section 162, and the movement planning section 163 of the planning section 134.
[0102] The traffic regulation recognition section 152 performs a process of recognizing traffic regulations around the own car based on data or signals from various structural elements such as the self-position estimator 132, the vehicle external information detector 141, and the map analyzer 151 of the vehicle control system 100. The recognition process makes it possible to recognize the position and state of traffic lights around the own car, the details of traffic control performed around the own car, and the drivable lanes. The traffic regulation recognition section 152 supplies data indicating the result of the recognition process to, for example, the state prediction section 154.
[0103] The state recognition unit 153 performs processing for recognizing the state related to the own car based on the data or signals from the various structural elements such as the self-position estimator 132, the vehicle external information detector 141, the vehicle internal information detector 142, the vehicle state detector 143 and the map analyzer 151 of the vehicle control system 100. For example, the state recognition unit 153 performs processing for recognizing the state of the own car, the state of the surrounding environment of the own car, the state of the driver of the own car, etc. In addition, the state recognition unit 153 generates a local map (hereinafter referred to as a state recognition map) for recognizing the state of the surrounding environment of the own car as needed. The state recognition map is, for example, an occupancy grid map.
[0104] Examples of the recognition target state of the own car include the position, posture, and movement (such as speed, acceleration, and direction of movement) of the own car, and the presence or absence of abnormalities and their details. Examples of the recognition target state of the surrounding environment of the own car include: the type and position of stationary objects around the own car; the type, position, and movement (such as speed, acceleration, and direction of movement) of moving objects around the own car; the structure of the road around the own car and the condition of the road surface; and the weather, temperature, humidity, and brightness around the own car. Examples of the recognition target state of the driver include physical condition, arousal level, concentration level, fatigue level, line of sight movement, and driving operation.
[0105] The state recognition section 153 supplies data indicating the result of the recognition processing (including a state recognition map as necessary) to, for example, the self-position estimator 132 and the state prediction section 154. Furthermore, the state recognition section 153 stores the state recognition map in the storage section 111.
[0106] The state prediction unit 154 performs a process of predicting a state related to the own vehicle based on data or signals from various structural elements such as the map analyzer 151, the traffic rule recognition unit 152, and the state recognition unit 153 of the vehicle control system 100. For example, the state prediction unit 154 performs a process of predicting the state of the own vehicle, the state of the surrounding environment of the own vehicle, the state of the driver, and the like.
[0107] Examples of the predicted target state of the own car include the behavior of the own car, the occurrence of abnormalities in the own car, and the drivable distance of the own car. Examples of the predicted target state of the surrounding environment of the own car include the behavior of moving objects, changes in the state of traffic lights, and changes in the environment such as weather around the own car. Examples of the predicted target state of the driver include the behavior and physical condition of the driver.
[0108] The state prediction section 154 supplies data indicating the result of the prediction processing together with the data from the traffic regulation recognition section 152 and the state recognition section 153 to, for example, the route planning section 161 , the behavior planning section 162 , and the movement planning section 163 of the planning section 134 .
[0109] The route planning section 161 plans a route to a destination based on data or signals from various structural elements such as the map analyzer 151 and the state prediction section 154 of the vehicle control system 100. For example, the route planning section 161 sets a route from a current position to a specified destination based on a global map. In addition, for example, the route planning section 161 appropriately changes the route based on the state of, for example, traffic congestion, accidents, traffic regulations and structures, and the physical condition of the driver. The route planning section 161 supplies data indicating the planned route to, for example, the behavior planning section 162.
[0110] Based on data or signals from various structural elements such as the map analyzer 151 and the state prediction section 154 of the vehicle control system 100, the behavior planning section 162 plans the behavior of the own car so that the own car safely travels on the route planned by the route planning section 161 within the time planned by the route planning section 161. For example, the behavior planning section 162 makes plans about, for example, starting to move, stopping, driving directions (such as forward movement, backward movement, left turn, right turn, and change of direction), lanes for driving, driving speed, and overtaking. The behavior planning section 162 supplies data indicating the planned behavior of the own car to, for example, the movement planning section 163.
[0111] Based on data or signals from various structural elements such as the map analyzer 151 and the state prediction section 154 of the vehicle control system 100, the movement planning section 163 plans the movement of the own car to realize the behavior planned by the behavior planning section 162. For example, the movement planning section 163 formulates a plan regarding, for example, acceleration, deceleration, and a driving trajectory. The movement planning section 163 supplies data indicating the planned movement of the own car to, for example, the acceleration / deceleration controller 172 and the direction controller 173 of the movement controller 135.
[0112] The movement controller 135 controls the movement of the own car. The movement controller 135 includes an emergency avoidance unit 171, an acceleration / deceleration controller 172, and a direction controller 173.
[0113] Based on the results of the detection of the vehicle external information detector 141, the vehicle internal information detector 142 and the vehicle state detector 143, the emergency avoidance unit 171 performs processing to detect emergency events such as collision, contact, entering a dangerous area, driver abnormality and vehicle abnormality. When the emergency avoidance unit 171 detects the occurrence of an emergency, the emergency avoidance unit 171 plans the movement of the own car, such as sudden stops or quick turns to avoid the emergency. The emergency avoidance unit 171 supplies data indicating the planned movement of the own car to, for example, the acceleration / deceleration controller 172 and the direction controller 173.
[0114] The acceleration / deceleration controller 172 controls the acceleration / deceleration to realize the movement of the own vehicle planned by the movement planning unit 163 or the emergency avoidance unit 171. For example, the acceleration / deceleration controller 172 calculates a control target value for the driving force generating device or the braking device to realize the planned acceleration, planned deceleration, or planned sudden stop, and supplies a control instruction indicating the calculated control target value to the powertrain controller 107.
[0115] The direction controller 173 controls the direction to realize the movement of the own car planned by the movement planning unit 163 or the emergency avoidance unit 171. For example, the direction controller 173 calculates a control target value for the steering mechanism to realize the driving trajectory planned by the movement planning unit 163 or the fast turn planned by the emergency avoidance unit 171, and supplies a control instruction indicating the calculated control target value to the powertrain controller 107.
[0116] <<3. Configuration that suppresses erroneous determination of obstacles on inclined roads>>
[0117] Next, refer to Figure 5 The block diagram describes a configuration that suppresses erroneous determination of obstacles on a sloped road. Note that Figure 5 shows the implementation reference Figure 4 The configuration of components for suppressing erroneous determination of an obstacle on a sloped road among the examples of the functional configuration of the vehicle control system 100 is described.
[0118] The data acquisition unit 102 includes a camera 201, a millimeter wave radar 202, and a LiDAR 203. The camera 201, the millimeter wave radar 202, and the LiDAR 203 all detect objects in front of the traveling direction of the vehicle 11, such as the front of the vehicle 11 and the upper part of the windshield of the vehicle 11. For example, the camera 201, the millimeter wave radar 202, and the LiDAR 203 correspond to Figure 2 and Figure 3 The exterior recognition sensor 11 a of the vehicle 11 is configured to be configured to be operable to identify the exterior of the vehicle 11 .
[0119] The camera 201 captures an image ahead in the traveling direction, and outputs the captured image to the vehicle exterior information detector 141 of the detector 131 in the automatic driving controller 112 .
[0120] The millimeter wave radar 202 irradiates radio waves in the millimeter wave band to the front in the driving direction, receives reflected waves from objects to detect information about the position and speed of the objects, and outputs the detected information to the vehicle external information detector 141 of the detector 131 in the automatic driving controller 112.
[0121] LiDAR 203 emits laser horizontally and vertically in front of the driving direction at specified intervals, and receives reflected light from objects to detect point group information according to the position and shape of the objects, and outputs the detected point group information to the vehicle external information detector 141 of the detector 131 in the automatic driving controller 112.
[0122] Based on the image captured by the camera 201, the detection result of the millimeter wave radar 202, and the sensing result of the LiDAR 203, the vehicle external information detector 141 identifies the object in front of the vehicle 11, and performs a collision determination based on the identification result. More specifically, the vehicle external information detector 141 includes an object recognition unit 221, a tilt recognition unit 222, and a collision determination unit 223.
[0123] The object recognition section 221 recognizes an object based on respective results of sensing by the camera 201 , the millimeter wave radar 202 , and the LiDAR 203 .
[0124] More specifically, the object recognition unit 221 recognizes each single object located in the path and appearing in the image for each pixel using, for example, semantic segmentation, based on the captured image as a result of sensing by the camera 201. Therefore, for example, when the road surface in the path is an inclined road surface, or when there is a maintenance hole or a fence, the object recognition unit 221 can recognize from the image that the inclined road is located ahead or the maintenance hole or fence is located ahead.
[0125] In addition, the object recognition unit 221 recognizes the presence or absence of an object in the path based on the sensing result of the millimeter wave radar 202. In this case, among the radio waves in the millimeter wave band irradiated to the front, the object recognition unit 221 recognizes the radio waves with high reflection intensity as reflected waves from the object. However, when only the sensing result of the millimeter wave radar 202 is used, the object recognition unit 221 can recognize the position of the object (the distance to the object and the direction of the object), but cannot recognize, for example, whether the object is a vehicle or a pedestrian, that is, the type of the object and the size of the object, etc.
[0126] Note that even when there is a preceding vehicle, a portion of the radio waves in the millimeter wave band is reflected from the road surface under the body of the preceding vehicle, and this causes a reflected wave from an object further ahead than the preceding vehicle to be received. This enables the object recognition unit 221 to recognize the presence of an object further ahead of the preceding vehicle, for example, based on the result of sensing by the millimeter wave radar 202.
[0127] In addition, the object recognition unit 221 recognizes an object in the path based on the sensing result of the LiDAR 203. In this case, as in the case of the millimeter wave radar, the object recognition unit 221 can recognize the object in front because the sensing result of the LiDAR 203 is represented by three-dimensional point group information. However, when the sensing result of the LiDAR 203 is used, the object recognition unit 221 cannot recognize what kind of object the detected object is. In addition, for example, when there is a leading vehicle, it is difficult to obtain information about the front of the leading vehicle because the sensing result of the LiDAR 203 is represented by laser. Therefore, in the object recognition based on the sensing result of the LiDAR 203, when there is a leading vehicle, object recognition required for collision determination cannot be performed. Note that in the case of using the sensing result of the LiDAR 203, and when there is an object and the LiDAR 203 gets closer to the object as time passes, the number of beams of reflected laser light increases as the LiDAR 203 gets closer to the object, as shown in reference. Figure 3 This enables the object recognition section 221 to recognize the size and inclination of the object based on the change in the number of beams of reflected laser light.
[0128] The inclination recognition unit 222 recognizes the inclination of the road surface in the path based on the result of object recognition based on the image captured by the camera 201 and the result of object detection based on the result of sensing by the LiDAR 203 among the results of recognition by the object recognition unit 221. For example, when the result of object recognition based on the image captured by the camera 201 is used, the inclination recognition unit 222 recognizes the inclination of the road surface from the information on the road surface based on the result of object recognition.
[0129] In addition, for example, when the result of object recognition based on the result of sensing by LiDAR is used, the tilt recognition unit 222 recognizes the object based on the time series change of the object recognized in the three-dimensional point group information. More specifically, when the result of sensing by the millimeter wave radar is used, as the millimeter wave radar is closer to the object, the number of beams of reflected laser light from the object (the area of the laser light) increases. Therefore, the object recognition unit 221 recognizes the size of the object based on the number of beams of reflected laser light from the object (the area of the laser light). In addition, the tilt recognition unit 222 recognizes the inclination of the road surface based on the time series change of the number of beams of reflected laser light from the object.
[0130] Note that it is difficult to obtain the inclination of the road surface from the result of sensing by, for example, a millimeter wave radar.
[0131] Based on the result of the object recognition by the object recognition unit 221 , the collision determination unit 223 performs collision determination to determine whether there is a possibility of collision with an object in the path.
[0132] The collision determination unit 223 selectively uses the result of object recognition to perform collision determination according to the environmental conditions.
[0133] During the daytime in clear weather, the collision determination unit 223 performs collision determination based on the result of object recognition of the image captured by the camera 201 .
[0134] Furthermore, at night or during bad weather, the collision determination section 223 performs collision determination using the result of object recognition based on the result of sensing by the millimeter wave radar 202 and using the result of object recognition based on the result of sensing by the LiDAR 203 .
[0135] In this case, when there is a leading vehicle, the collision determination unit 223 identifies the presence of an object in the path based on the result of object recognition of the sensing result of the millimeter wave radar 202, and the collision determination unit 223 performs a collision determination based on whether the leading vehicle has passed the position where the object is identified.
[0136] In other words, when performing collision determination based on the result of object recognition of the sensing result of the millimeter wave radar 202 , the collision determination section 223 can recognize the presence of the preceding vehicle and the presence of an object further ahead of the preceding vehicle.
[0137] However, when the result of object recognition based on the result of sensing by the millimeter wave radar 202 is used, the existence of the object can be recognized, but it is difficult to distinguish whether the object is an object for which the possibility of collision needs to be considered.
[0138] Therefore, when the collision determination unit 223 performs a collision determination based on the result of object recognition based on the sensing result of the millimeter wave radar 202, and when there is a leading vehicle, the collision determination unit 223 determines that there is no possibility of a collision with the identified object when the leading vehicle has passed the position where the object is identified.
[0139] In addition, when the collision determination unit 223 performs a collision determination based on the result of object recognition based on the sensing result of the millimeter wave radar 202, and when there is a preceding vehicle, the collision determination unit 223 determines that when the preceding vehicle has not passed the position of the identified object, that is, when it can be identified that the preceding vehicle has passed while avoiding the identified object, there is a possibility of a collision with the identified object.
[0140] Furthermore, when there is no preceding vehicle, the collision determination section 223 recognizes that there is an object in the path based on the result of object recognition of the sensing result of the LiDAR 203 , and performs collision determination.
[0141] In other words, when collision determination is performed based on the result of object recognition as a result of sensing by the LiDAR 203, the collision determination unit 223 can identify the size and inclination of the object based on changes in the number of reflected laser beams from the object among the emitted lasers, because as the LiDAR 203 gets closer to the object, the number of reflected laser beams from the object increases in time sequence.
[0142] More specifically, assuming that Figure 6 As shown in FIG. 1 , a vehicle 11 travels on a flat road FL in the right direction in the figure, the right end of the flat road FL is connected to a sloped road SL, and a maintenance hole 12 is provided on the sloped road SL. Here, an external recognition sensor 11a is provided at the front of the vehicle 11, in which components of the data acquisition unit 102 as a camera 201, a millimeter wave radar 202, and a LiDAR 203 are combined. Assuming that, for example Figure 6 As shown in FIG. 1 , the LiDAR 203 among the external recognition sensors 11 a as the data acquisition section 102 emits laser beams L1 to L13 to the front in the traveling direction, and receives reflected light of each beam to acquire three-dimensional point group information as an object recognition result.
[0143] here, Figure 6 A state is shown in which six laser beams L3 to L8 among the laser beams L1 to L13 are irradiated onto the maintenance hole 12 and are received as reflected beams.
[0144] When the vehicle 11 approaches the maintenance hole 12 on the inclined road SL as time passes in this state, as shown in FIG. Figure 7 As shown in FIG. 1 , the number of beams of reflected laser light from the maintenance hole 12 gradually increases from the time t1 to the time t3 .
[0145] In other words, at time t1, when Figure 6 When the state has passed the specified time period, eight laser beams L2 to L9 among the laser beams L1 to L13 are irradiated onto the maintenance hole 12 and are received by the LiDAR 203 as reflected beams, as shown in FIG. Figure 7 shown in the upper part of .
[0146] Furthermore, thereafter, at time t2, ten laser beams L1 to L10 among the laser beams L1 to L13 are irradiated onto the maintenance hole 12 and are received by the LiDAR 203 as reflected beams, as shown in FIG. Figure 7 shown in the middle part of .
[0147] Furthermore, thereafter, at time t3, all thirteen laser beams L1 to L13 are irradiated onto the maintenance hole 12 and are received by the LiDAR 203 as reflected beams, as shown in FIG. Figure 7 shown in the lower part of .
[0148] As described above, the number of beams of reflected laser light from the maintenance hole 12 among the laser beams irradiated by the LiDAR 203 provided as the exterior recognition sensor 11a of the vehicle 11 increases chronologically as the vehicle 11 gets closer to the maintenance hole 12. In addition, the magnitude of the chronological change in the number of beams of reflected laser light differs depending on the inclination of the surface of the maintenance hole 12. In other words, if the inclination is large, the chronological change in the number of laser beams is large, and if the inclination is small, the chronological change in the number of laser beams is small.
[0149] Therefore, the collision determination unit 223 obtains the inclination of the surface of the object based on the time series change of the number of beams of reflected laser light from the object, and performs collision determination based on whether the size of the object is larger than a specified size and whether the inclination of the object is larger than a specified inclination.
[0150] <<4. Collision determination processing>>
[0151] Next, refer to Figure 8 The collision determination process of the vehicle 11 of the present disclosure is described with reference to a flowchart of FIG.
[0152] In step S11 , the camera 201 , the millimeter wave radar 202 , and the LiDAR 203 of the data acquisition section 102 output respective sensing results (with respect to the captured image of the camera 201 ) to the detector 131 of the automatic driving controller 112 .
[0153] In step S12, the object recognition section 221 determines whether the object can be recognized based on the image captured by the camera 201. More specifically, the object recognition section 221 determines whether the object can be appropriately recognized based on the image captured by the camera 201, taking into account the influence of an environment with insufficient brightness such as at night, during bad weather, etc.
[0154] When it is determined in step S12 that the object can be recognized based on the image captured by the camera 201 , the process moves to step S13 .
[0155] In step S13 , based on the captured image as a result of sensing by the camera 201 , the object recognition section 221 recognizes an object for each pixel using, for example, semantic segmentation.
[0156] In step S14 , the collision determination section 223 determines whether there is an object that may be an obstacle in the path based on the result of object recognition performed by the object recognition section 221 based on the captured image as a result of sensing by the camera 201 .
[0157] For example, when it is determined in step S14 that there is an object that may be an obstacle in the path, the process moves to step S15.
[0158] In step S15 , the inclination recognition unit 222 recognizes the inclination of the road surface recognized by the object recognition unit 221 .
[0159] In step S16 , based on the inclination of the road surface and information on an object in the path that may be an obstacle, the collision determination section 223 determines whether there is a possibility of collision with the object.
[0160] For example, in step S16, when the object is, for example, a large-sized person or a parked vehicle, or when the road surface is greatly inclined and there is a possibility that the object is a wall or a fence, it is determined that there is a possibility of collision with the object, and the processing moves to step S17.
[0161] In step S17 , the collision determination unit 223 notifies the emergency avoidance unit 171 of the mobile controller 135 that there is a possibility of a collision.
[0162] In step S18 , the emergency avoidance section 171 controls the acceleration / deceleration controller 172 and the direction controller 173 based on the notification from the collision determination section 223 so that the powertrain 108 is driven by the powertrain controller 107 to avoid collision with an object that may be an obstacle.
[0163] In step S19, it is determined whether an instruction to terminate the process is given. When it is determined that an instruction to terminate has not been given, the process returns to step S11, and the subsequent processes are repeated. When it is determined in step S19 that an instruction to terminate has been given, the process is terminated.
[0164] Furthermore, when it is determined in step S14 that there is no object that may be an obstacle in the path, or when it is determined in step S16 that there is no possibility of a collision, the process moves to step S26 .
[0165] In step S26, the collision determination unit 223 notifies the emergency avoidance unit 171 of the mobile controller 135 that there is no possibility of a collision. Note that when there is no possibility of a collision, notification to the emergency avoidance unit 171 is not indispensable. Therefore, the processing of step S26 can be skipped as needed or the processing of step S26 itself can be deleted.
[0166] When it is determined in step S12 that the object recognition based on the image cannot be performed, the process moves to step S20 .
[0167] In step S20, the object recognition section 221 determines whether the object can be recognized based on the result of sensing by the LiDAR 203. More specifically, the LiDAR 203 is capable of sensing even in a difficult situation where the amount of light is insufficient to capture an image using the camera 201 due to, for example, nighttime or bad weather. Therefore, the object recognition section 221 determines whether sensing can be performed without, for example, a malfunction in the LiDAR 203.
[0168] When it is determined in step S20 that the object can be recognized based on the result of sensing by the LiDAR 203 , the process moves to step S21 .
[0169] In step S21 , the object recognition unit 221 recognizes an object based on the sensing result of the LiDAR 203 .
[0170] In step S22, the collision determination unit 223 determines whether there is a preceding vehicle based on the result of object recognition of the sensing result of the LiDAR 203. Based on the object recognition result, the collision determination unit 223 determines that there is a preceding vehicle when, for example, there is an object in front of the own vehicle and the positional relationship between the object and the own vehicle remains unchanged in a specified state.
[0171] When it is determined in step S22 that there is no preceding vehicle, the process moves to step S23.
[0172] In step S23 , the collision determination unit 223 determines whether there is an object that may be an obstacle in the path based on the result of object recognition of the sensing result of the LiDAR 203 .
[0173] When it is determined in step S23 that there is no object that may be an obstacle in the path, the process moves to step S24.
[0174] In step S24 , the inclination recognition unit 222 recognizes the inclination of the road surface based on the result of object recognition of the sensing result of the LiDAR 203 .
[0175] In step S25, based on the result of object recognition of the sensing result of LiDAR 203 and based on the inclination of the road surface, the collision determination unit 223 determines whether there is a possibility of collision with the object based on whether the object has a size greater than a specified size or whether the inclination is greater than a specified inclination.
[0176] When it is determined in step S25 that there is a possibility of a collision, or when it is determined in step S23 that there is no object that may be an obstacle in the path, the process moves to step S17, and a notification is given that there is a possibility of a collision.
[0177] When it is determined in step S25 that there is no possibility of a collision, the process moves to step S26 , and a notification is given that there is no possibility of a collision.
[0178] Furthermore, when it is determined in step S22 that a preceding vehicle exists, the process moves to step S27 .
[0179] In other words, in the case of the result of object recognition based only on the sensing result of LiDAR 203, when there is a preceding vehicle, the preceding vehicle will be an obstacle, and thus an obstacle that may be located further in front of the preceding vehicle will not be detected. Therefore, when there is a preceding vehicle, in the case of a malfunction in LiDAR 203, a collision is undeterminable.
[0180] In step S27, the object recognition section 221 determines whether the object can be recognized based on the result of sensing by the millimeter wave radar 202. More specifically, the millimeter wave radar 202 is capable of sensing even in difficult conditions such as when the amount of light due to, for example, nighttime or bad weather is insufficient to capture an image using the camera 201. Therefore, the object recognition section 221 determines whether sensing can be performed without, for example, a malfunction in the millimeter wave radar 202.
[0181] When it is determined in step S27 that the object can be identified based on the result of sensing by the millimeter wave radar 202 , the process moves to step S28 .
[0182] In step S28 , the object recognition unit 221 recognizes the object based on the result of the sensing by the millimeter wave radar 202 .
[0183] In step S29 , the collision determination unit 223 determines whether there is an object that may be an obstacle in the path based on the result of object recognition as a result of sensing by the millimeter wave radar 202 .
[0184] When it is determined in step S29 that there is no object that may be an obstacle in the path, the process moves to step S26 , and a notification is given that there is no possibility of a collision.
[0185] On the other hand, when it is determined in step S29 that an object that may be an obstacle exists in the path, the process moves to step S30 .
[0186] In step S30 , the collision determination section 223 determines whether the preceding vehicle has passed a position where an object that may be an obstacle exists.
[0187] When it is determined in step S30 that the preceding vehicle has passed a position where there is an object that may be an obstacle, the process moves to step S26 , and a notification is given that there is no possibility of a collision.
[0188] Furthermore, when it is determined in step S30 that, for example, the preceding vehicle has passed while avoiding the position of an object that may be an obstacle, the process moves to step S17 , and a notification is given that there is a possibility of a collision.
[0189] In other words, in this case, it is determined that the preceding vehicle has recognized the object as an obstacle and has passed through while avoiding the object to avoid collision with the object. Therefore, it is determined that there is a possibility of collision with the detected object.
[0190] When it is determined in step S20 that the object cannot be recognized based on the result of sensing by the LiDAR 203 , the process moves to step S31 .
[0191] In step S31 , the object recognition unit 221 determines whether or not an object can be recognized based on the result of sensing by the millimeter wave radar 202 .
[0192] When it is determined in step S31 that the object can be identified based on the result of sensing by the millimeter wave radar 202 , the process moves to step S32 .
[0193] In step S32 , the object recognition unit 221 recognizes the object based on the result of the sensing by the millimeter wave radar 202 .
[0194] In step S33 , the collision determination unit 223 determines whether there is a preceding vehicle based on the result of object recognition as a result of sensing by the millimeter wave radar 202 .
[0195] When it is determined in step S33 that there is a preceding vehicle, the process moves to step S29. It is determined whether there is an object that may be an obstacle in the path, and a collision determination is made based on whether the preceding vehicle has passed the position of the object.
[0196] On the other hand, when it is determined in step S33 that there is a preceding vehicle, or when it is determined in step S31 that the result of object recognition based on the sensing result of the millimeter wave radar 202 cannot recognize an object, the process moves to step S34.
[0197] In step S34, the collision determination unit 223 outputs information indicating that the collision is undeterminable. In other words, a notification is given that the collision is undeterminable based on the result of the object recognition based on the image captured by the camera 201, the result of the object recognition based on the sensing result of the millimeter wave radar 202, and the result of the object recognition based on the sensing result of the LiDAR 203.
[0198] In the above-described process, for example, when it is bright in clear weather, collision determination is performed based on the result of sensing (captured image) of the camera 201. This makes it possible to perform accurate collision determination based on the result of accurate object recognition of the image.
[0199] Furthermore, when object recognition cannot be performed based on an image, that is, during bad weather, at night, etc., and when there is no preceding vehicle, collision determination is performed based on the result of sensing by LiDAR.
[0200] In addition, when object recognition cannot be performed based on images, that is, during bad weather, at night, etc., and when there is a leading vehicle, a collision determination is made based on the results of sensing by the millimeter wave radar and based on whether the leading vehicle has passed the position of the detected object.
[0201] Therefore, even in a state where a collision is not determinable based on an image, collision determination can be performed regardless of the presence or absence of a preceding vehicle.
[0202] Therefore, appropriate collision determination can be made in any of these situations without being affected by environmental changes.
[0203] <<5. Example of series processing using software>
[0204] Note that the above series of processing can be performed using hardware or software. When the series of processing is performed using software, the program included in the software is installed on the computer from the recording medium. Examples of computers include computers incorporated into dedicated hardware and computers such as general-purpose computers that can perform various functions by various programs installed thereon.
[0205] Fig. 9 An example of a configuration of a general-purpose computer is shown. The computer includes a central processing unit (CPU) 1001 . An input / output interface 1005 is connected to the CPU 1001 via a bus 1004 . A read-only memory (ROM) 1002 and a random access memory (RAM) 1003 are connected to the bus 1004 .
[0206] An input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009 are connected to the input / output interface 1005. The input unit 1006 includes an input device such as a keyboard and a mouse that a user uses to input an operation command. The output unit 1007 outputs an image of a processing operation screen and a processing result to a display device. The storage unit 1008 includes, for example, a hard disk drive in which programs and various data are stored. The communication unit 1009 includes, for example, a local area network (LAN) adapter, and performs communication processing through a network represented by the Internet. In addition, a drive 1010 is connected to the input / output interface 1005. The drive 1010 reads data from a removable recording medium 1011 and writes data to the removable recording medium 1011, such as a magnetic disk (including a floppy disk), an optical disk (including a compact disk read-only memory (CD-ROM) and a digital versatile disk (DVD)), a magneto-optical disk (including a mini disk (MD)), or a semiconductor memory.
[0207] The CPU 1001 performs various processes in accordance with the program stored in the ROM 1002 or in accordance with the program read from a removable recording medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory to be installed on the storage section 1008 and loaded from the storage section 1008 into the RAM 1003. Data required for the CPU 1001 to perform various processes is also stored in the RAM 1003 as necessary.
[0208] In the computer having the above configuration, the above series of processing is performed by the CPU 1001 loading a program stored in, for example, the storage section 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing the program.
[0209] For example, the program executed by the computer (CPU 1001) may be provided by being stored in the removable recording medium 1011 serving as, for example, a package medium. Furthermore, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
[0210] In the computer, the program can be installed on the storage section 1008 via the input / output interface 1005 by the removable recording medium 1011 mounted on the drive 1010. In addition, the program can be received by the communication section 1009 via a wired or wireless transmission medium to be installed on the storage section 1008. In addition, the program can be installed in advance on the ROM 1002 or the storage section 1008.
[0211] Note that the program executed by the computer may be a program that is processed chronologically in the order described herein, or may be a program that is processed in parallel or at necessary timing such as call timing.
[0212] Notice, Figure 4 The functions of the autopilot controller 112 are as follows: Fig. 9 The CPU 1001 implements. In addition, Figure 4 The storage unit 111 is composed of Fig. 9 The storage unit 1008 is implemented.
[0213] In addition, the system used in this article refers to a collection of multiple components such as devices and modules (components), and it is not important whether all components are in a single housing. Therefore, multiple devices housed in separate housings and connected to each other via a network and a single device in which multiple modules are housed in a single housing are both systems.
[0214] Note that the embodiments of the present disclosure are not limited to the above-described examples, and various modifications may be made thereto without departing from the scope of the present disclosure.
[0215] For example, the present disclosure may also have a configuration of cloud computing in which a single function is shared to be collaboratively processed by a plurality of devices via a network.
[0216] Furthermore, the various steps described using the above flowcharts may be shared to be performed by a plurality of devices in addition to being performed by a single device.
[0217] Furthermore, when a single step includes a plurality of processes, in addition to being performed by a single device, the plurality of processes included in the single step may be shared to be performed by a plurality of devices.
[0218] Note that the present disclosure can also adopt the following configurations.
[0219] <1> An information processing device, comprising:
[0220] an object recognition unit that recognizes an object in front of the vehicle using a plurality of recognition methods and outputs results of object recognition by each of the plurality of recognition methods; and
[0221] The collision determination unit selects a result of object recognition from among the results of object recognition by a plurality of recognition methods according to the environment around the vehicle, and determines whether there is a possibility of collision with the object based on the selected result of object recognition.
[0222] <2> according to <1> The information processing device further includes:
[0223] A camera that captures images in front of the vehicle;
[0224] millimeter wave radar that irradiates radio waves in the millimeter wave band forward and detects reflected waves from objects; and
[0225] LiDAR shines laser light in front of the vehicle and detects reflected light from objects.
[0226] The object recognition unit outputs a result of image object recognition based on an image captured by a camera, a result of millimeter wave radar object recognition based on a result of detection by a millimeter wave radar, and a result of LiDAR object recognition based on a result of detection by a LiDAR.
[0227] <3> according to <2> The information processing device, wherein
[0228] When object recognition is possible based on a result of the image object recognition according to the environment around the vehicle, the collision determination unit determines whether there is a possibility of collision with the object based on the result of the image object recognition.
[0229] <4> according to <2> or <3> The information processing device also includes
[0230] The tilt recognition unit recognizes the tilt of the road surface in front of the vehicle based on the result of the image object recognition, wherein
[0231] The collision determination unit determines whether there is a possibility of collision with an object based on a result of image object recognition and an inclination of a road surface.
[0232] <5> according to <4> The information processing device, wherein
[0233] The inclination recognition unit recognizes the inclination of a road surface in front of the vehicle based on a time-series change in the number of beams of reflected light detected by the LiDAR as a time-series change in distance from an object occurs.
[0234] <6> according to <2> to <5> The information processing device according to any one of
[0235] When object recognition cannot be performed based on the result of image object recognition due to the environment around the vehicle, the collision determination unit determines whether there is a possibility of collision with the object based on the result of millimeter wave radar object recognition or the result of LiDAR object recognition.
[0236] <7> according to <6> The information processing device, wherein
[0237] When object recognition cannot be performed based on the result of image object recognition due to the environment around the vehicle, and when object recognition can be performed based on the result of LiDAR object recognition, the collision determination unit determines whether there is a possibility of collision with the object based on the result of LiDAR object recognition.
[0238] <8> according to <7> The information processing device, wherein
[0239] When the presence of the preceding vehicle cannot be recognized based on the result of LiDAR object recognition, it is deemed that object recognition based on the result of LiDAR object recognition is possible, and the collision determination unit determines whether there is a possibility of collision with the object based on the result of LiDAR object recognition.
[0240] <9> according to <8> The information processing device, wherein
[0241] When the presence of the preceding vehicle is recognized based on the result of LiDAR object recognition, it is deemed that object recognition based on the result of LiDAR object recognition cannot be performed, and the collision determination unit determines whether there is a possibility of collision with the object based on the result of millimeter wave radar object recognition.
[0242] <10> according to <9> The information processing device, wherein
[0243] The collision determination section determines whether there is a possibility of collision with the object based on a result of the millimeter wave radar object recognition and based on whether a preceding vehicle has passed through a position where the object has been recognized.
[0244] <11> according to <6> The information processing device, wherein
[0245] When the object recognition cannot be performed based on the result of the LiDAR object recognition, the collision determination unit determines whether there is a possibility of collision with the object based on the result of the millimeter wave radar object recognition.
[0246] <12> according to <11> The information processing device, wherein
[0247] When the presence of the preceding vehicle is recognized based on the result of the millimeter wave radar object recognition, the collision determination unit determines whether there is a possibility of collision with the object based on the result of the millimeter wave radar object recognition.
[0248] <13> according to <12> The information processing device, wherein
[0249] The collision determination section determines whether there is a possibility of collision with the object based on a result of the millimeter wave radar object recognition and based on whether a preceding vehicle has passed through a position where the object has been recognized.
[0250] <14> according to <2> The information processing device, wherein
[0251] When object recognition based on the results of image object recognition cannot be performed due to the environment around the vehicle, and object recognition based on the results of millimeter-wave radar object recognition and object recognition based on the results of LiDAR object recognition cannot be performed, the collision determination unit outputs information indicating that it is impossible to determine whether there is a possibility of collision with an object.
[0252] <15> An information processing method, comprising:
[0253] performing object recognition processing, including using a plurality of recognition methods to recognize an object in front of the vehicle, and outputting object recognition results of each of the plurality of recognition methods; and
[0254] The collision determination process includes selecting a result of object recognition from results of object recognition by a plurality of recognition methods according to the environment around the vehicle, and determining whether there is a possibility of collision with the object based on the selected result of object recognition.
[0255] <16> A program for causing a computer to operate as an object recognition unit and a collision determination unit,
[0256] The object recognition unit recognizes an object in front of the vehicle using a plurality of recognition methods, and outputs object recognition results of each of the plurality of recognition methods.
[0257] The collision determination unit selects a result of object recognition from results of object recognition by a plurality of recognition methods according to an environment around the vehicle, and determines whether there is a possibility of collision with the object based on the selected result of object recognition.
[0258] Reference numerals list
[0259] 11 Vehicles
[0260] 100 Vehicle Control Systems
[0261] 102 Data Acquisition Department
[0262] 112 Autopilot Controller
[0263] 133 Status Analyzer
[0264] 134 Planning Department
[0265] 135 Mobile Controller
[0266] 153 Status Identification Unit
[0267] 154 Status Prediction Department
[0268] 172 Acceleration / deceleration controller
[0269] 173 Direction Controller
[0270] 201 Camera
[0271] 202 millimeter wave radar
[0272] 203 LiDAR
[0273] 221 Object Recognition Department
[0274] 222 Tilt recognition unit
[0275] 223 Collision Detection Unit
Claims
1. An information processing device, comprising: An object recognition unit that uses a plurality of recognition methods to recognize an object in front of the vehicle and outputs object recognition results of each of the plurality of recognition methods; a collision determination unit that selects a result of object recognition from results of object recognition by a plurality of recognition methods according to an environment surrounding the vehicle, and determines whether there is a possibility of collision with the object based on the selected result of object recognition; A camera that captures images in front of the vehicle; Millimeter-wave radar, which irradiates radio waves in the millimeter-wave band forward and detects reflected waves from objects; as well as LiDAR shines laser light in front of the vehicle and detects reflected light from objects. The object recognition unit outputs the result of image object recognition based on the image captured by the camera, the result of millimeter wave radar object recognition based on the result of millimeter wave radar detection, and the result of LiDAR object recognition based on the result of LiDAR detection. Among them, when there is no leading vehicle, the collision determination unit identifies the size and inclination of the object based on the time series change of the number of reflected laser beams from the object detected by the LiDAR among the emitted lasers, which changes with the time series change of the distance to the object, and determines whether there is a possibility of collision with the object based on whether the object has a size greater than a specified size and whether the inclination is greater than a specified inclination.
2. The information processing device according to claim 1, wherein When object recognition is possible based on a result of the image object recognition according to the environment around the vehicle, the collision determination unit determines whether there is a possibility of collision with the object based on the result of the image object recognition.
3. The information processing device according to claim 1, wherein When object recognition cannot be performed based on the result of image object recognition due to the environment around the vehicle, the collision determination unit determines whether there is a possibility of collision with the object based on the result of millimeter wave radar object recognition or the result of LiDAR object recognition.
4. The information processing device according to claim 3, wherein When object recognition cannot be performed based on the result of image object recognition due to the environment around the vehicle, and when object recognition can be performed based on the result of LiDAR object recognition, the collision determination unit determines whether there is a possibility of collision with the object based on the result of LiDAR object recognition.
5. The information processing device according to claim 4, wherein When the presence of the preceding vehicle cannot be recognized based on the result of LiDAR object recognition, it is deemed that object recognition based on the result of LiDAR object recognition is possible, and the collision determination unit determines whether there is a possibility of collision with the object based on the result of LiDAR object recognition.
6. The information processing device according to claim 5, wherein When the presence of the preceding vehicle is recognized based on the result of LiDAR object recognition, it is deemed that object recognition based on the result of LiDAR object recognition cannot be performed, and the collision determination unit determines whether there is a possibility of collision with the object based on the result of millimeter wave radar object recognition.
7. The information processing device according to claim 6, wherein The collision determination section determines whether there is a possibility of collision with the object based on a result of the millimeter wave radar object recognition and based on whether a preceding vehicle has passed through a position where the object has been recognized.
8. The information processing device according to claim 3, wherein When the object recognition cannot be performed based on the result of the LiDAR object recognition, the collision determination unit determines whether there is a possibility of collision with the object based on the result of the millimeter wave radar object recognition.
9. The information processing device according to claim 8, wherein When the presence of the preceding vehicle is recognized based on the result of the millimeter wave radar object recognition, the collision determination unit determines whether there is a possibility of collision with the object based on the result of the millimeter wave radar object recognition.
10. The information processing device according to claim 9, wherein The collision determination section determines whether there is a possibility of collision with the object based on a result of the millimeter wave radar object recognition and based on whether a preceding vehicle has passed through a position where the object has been recognized.
11. The information processing device according to claim 1, wherein When object recognition based on the results of image object recognition cannot be performed due to the environment around the vehicle, and object recognition based on the results of millimeter-wave radar object recognition and object recognition based on the results of LiDAR object recognition cannot be performed, the collision determination unit outputs information indicating that it is impossible to determine whether there is a possibility of collision with an object.
12. An information processing method, comprising: Performing object recognition processing, including using a plurality of recognition methods to recognize an object in front of the vehicle, and outputting object recognition results of each of the plurality of recognition methods; and performing collision determination processing, including selecting a result of object recognition from results of object recognition by a plurality of recognition methods according to an environment surrounding the vehicle, and determining whether there is a possibility of collision with the object based on the selected result of object recognition, Among them, the object recognition processing outputs the result of image object recognition based on the image taken by the camera, the result of millimeter wave radar object recognition based on the result of millimeter wave radar detection, and the result of LiDAR object recognition based on the result of LiDAR detection, In which, when there is no leading vehicle, the size and inclination of the object are identified based on the time series change of the number of reflected laser beams from the object detected by the LiDAR among the emitted lasers, which changes with the time series change of the distance to the object, and whether there is a possibility of collision with the object is determined based on whether the object has a size greater than a specified size and whether the inclination is greater than a specified inclination.
13. A computer program product causing a computer to execute the following processing, object recognition processing, including using a plurality of recognition methods to recognize an object in front of the vehicle, and outputting the results of the object recognition of each of the plurality of recognition methods; and collision determination processing, including selecting a result of object recognition from results of object recognition by a plurality of recognition methods according to an environment surrounding the vehicle, and determining whether there is a possibility of collision with the object based on the selected result of object recognition, in, Outputting, through object recognition processing, a result of image object recognition based on an image captured by a camera, a result of millimeter-wave radar object recognition based on a result of detection by a millimeter-wave radar, and a result of LiDAR object recognition based on a result of detection by a LiDAR, In which, when there is no leading vehicle, the size and inclination of the object are identified based on the time series change of the number of reflected laser beams from the object detected by the LiDAR among the emitted lasers, which changes with the time series change of the distance to the object, and whether there is a possibility of collision with the object is determined based on whether the object has a size greater than a specified size and whether the inclination is greater than a specified inclination.
