Determination device, vehicle control device, determination method, and storage medium

Abstract

The present invention provides a determination device, a vehicle control device, a determination method, and a storage medium. The determination device according to an embodiment includes: an acquisition unit that acquires a primary determination result of a surrounding environment determined based on a first surrounding image of a vehicle and a reliability of the primary determination result; and a determination unit that performs a secondary determination of the primary determination result based on both a comparison result between the reliability and a preset threshold value and a past primary determination result of a surrounding image of the vehicle captured earlier than the first surrounding image, and outputs a secondary determination result having a higher accuracy than the primary determination result.

Description

Technical Field

[0001] The present invention relates to a determination device, a vehicle control device, a determination method, and a storage medium. Background Art

[0002] Conventionally, there has been known a technique of detecting the driving state of other vehicles based on a peripheral image captured by an in-vehicle camera of the own vehicle and performing automatic driving control of the own vehicle. Summary of the Invention

[0003] Problems to be Solved by the Invention

[0004] In the prior art, in a case where the image quality of the captured peripheral image is poor due to backlighting or the like, the driving state of other vehicles cannot be accurately detected, and over-detection sometimes occurs. When such over-detection occurs, problems sometimes arise in the automatic driving control, such as the own vehicle erroneously performing an emergency braking operation through the automatic driving control. There is known a technique of using vehicle-to-vehicle communication in order to improve the detection accuracy of other vehicles (for example, refer to Japanese Patent Laid-Open No. 2013-168019), but on the premise of such vehicle-to-vehicle communication, in addition to the limited number of vehicles equipped with the vehicle-to-vehicle communication function, it may also involve complication of the system structure and an increase in cost.

[0005] The present invention has been made in view of such circumstances, and one of its purposes is to provide a determination device, a vehicle control device, a determination method, and a storage medium that can detect the surrounding environment by an accurate and simple method.

[0006] Means for Solving the Problems

[0007] The determination device, the vehicle control device, the determination method, and the storage medium of the present invention adopt the following configuration.

[0008] (1) A determination device according to one aspect of the present invention includes: an acquisition unit that acquires a primary determination result of a surrounding environment determined based on a first peripheral image of a vehicle and a reliability of the primary determination result; and a determination unit that performs a secondary determination on the primary determination result based on both a comparison result between the reliability and a preset threshold value and a past primary determination result of a peripheral image of the vehicle captured earlier than the first peripheral image, and outputs a secondary determination result having a higher accuracy than the primary determination result.

[0009] Based on the determination device in the above-mentioned solution (1), in solution (2), the threshold includes a first threshold and a second threshold lower than the first threshold. When the confidence level is greater than the first threshold, the determination unit performs a first process of outputting the primary determination result as the secondary determination result. When the confidence level is less than the second threshold, the determination unit performs a second process of retaining the secondary determination. When the confidence level is between the first threshold and the second threshold, the determination unit performs a third process of determining whether to adopt the primary determination result as the secondary determination result based on the past primary determination results.

[0010] Based on the determination device in the above-mentioned solution (2), in solution (3), in the third process, when the number of the past primary determination results indicating the same determination result as the primary determination result among the specified number of the past primary determination results is equal to or greater than the specified value, the determination unit outputs the primary determination result as the secondary determination result. When the number of the past primary determination results indicating the same determination result as the primary determination result among the specified number of the past primary determination results is less than the specified value, the determination unit retains the secondary determination.

[0011] Based on the determination device in the above-mentioned solution (3), in solution (4), the specified number of the past primary determination results is an integer N of 2 or more, and the specified value is a value equal to or greater than N / 2 and equal to or less than N.

[0012] Based on the determination device in any one of the above-mentioned solutions (2) to (4), in solution (5), when the determination unit retains the secondary determination, the determination unit outputs information indicating that the secondary determination cannot be performed.

[0013] Based on the determination device in any one of the above-mentioned solutions (1) to (5), in solution (6), the primary determination result is a determination result of the driving state of the surrounding vehicle based on the action information of the lamp body of the surrounding vehicle included in the first surrounding image.

[0014] Based on the determination device in the above-mentioned solution (6), in solution (7), the lamp body includes at least one of a brake lamp and a direction indicator.

[0015] The vehicle control device according to another aspect of the present invention includes: a determination device according to any one of the first to seventh aspects; and a control unit that determines whether it is necessary to change the behavior control of the vehicle based on the secondary determination result output from the determination device.

[0016] Based on the vehicle control device of the above solution (8), when the control unit determines that it is necessary to change the behavior control of the vehicle based on the secondary determination result, the control unit changes the behavior control of the vehicle.

[0017] Based on the vehicle control device of the above solution (8) or (9), when the control unit determines that it is not necessary to change the behavior control of the vehicle based on the secondary determination result, the control unit does not change the behavior control of the vehicle.

[0018] Based on the vehicle control device of any one of the above solutions (8) to (10), when the determination unit retains the secondary determination, the control unit determines whether it is necessary to change the behavior control of the vehicle based on the surrounding environment information obtained by a detection unit different from the camera that captured the first surrounding image.

[0019] Based on the vehicle control device of any one of the above solutions (8) to (11), the control unit determines whether it is necessary to change at least one of the vehicle speed control, acceleration control, steering control, and stop control.

[0020] (13) In another determination method of the present invention, a computer mounted on a vehicle performs the following processing: obtaining a primary determination result of the surrounding environment determined based on a first surrounding image of the vehicle and the credibility of the primary determination result, and performing a secondary determination on the primary determination result based on both the comparison result of the credibility with a preset threshold and the past primary determination results of the surrounding images of the vehicle captured earlier than the first surrounding image, and outputting a secondary determination result with higher accuracy than the primary determination result.

[0021] (14) A storage medium of another solution of the present invention stores a program that causes a computer mounted on a vehicle to perform the following processing: obtaining a primary determination result of the surrounding environment determined based on a first surrounding image of the vehicle and the credibility of the primary determination result, and performing a secondary determination on the primary determination result based on both the comparison result of the credibility with a preset threshold and the past primary determination results of the surrounding images of the vehicle captured earlier than the first surrounding image, and outputting a secondary determination result with higher accuracy than the primary determination result.

[0022] Advantages of the Invention

[0023] According to the above solutions (1) to (14), it is possible to detect the surrounding environment by an accurate and simple method and suppress the occurrence of over-detection of the surrounding environment.

[0024] According to the above solutions (6) and (7), the detection rate of the driving state of surrounding vehicles can be improved.

[0025] According to the above solutions (8) to (12), by determining whether to change the behavior control of the vehicle based on the secondary determination result, stable autonomous driving control can be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a structural diagram of a vehicle system using the determination device of the embodiment.

[0027] Figure 2 It is a functional structural diagram of the first control unit and the second control unit of the embodiment.

[0028] Figure 3 It is a flowchart showing an example of the determination process performed by the recognition unit of the autonomous driving control device of the embodiment.

[0029] Figure 4A It is a diagram showing an example of a surrounding image of the front of the vehicle.

[0030] Figure 4B It is a diagram showing another example of a surrounding image of the front of the vehicle.

[0031] Figure 5A It is a diagram showing another example of a surrounding image of the front of the vehicle.

[0032] Figure 5B It is a diagram showing another example of a surrounding image of the front of the vehicle.

[0033] Figure 6 It is a diagram showing another example of a surrounding image of the front of the vehicle.

[0034] Figure 7 It is a diagram showing another example of a surrounding image of the front of the vehicle.

[0035] Figure 8 It is a diagram for explaining the determination process performed by the determination unit of the embodiment.

[0036] Figure 9 It is a diagram for explaining the determination process performed by the determination unit of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0037] Hereinafter, embodiments of the determination device, vehicle control device, determination method, and storage medium of the present invention will be described with reference to the drawings.

[0038] [SUMMARY]

[0039] The determination device of the present invention obtains a primary determination result of the surrounding environment (e.g., the driving state of other vehicles) determined based on the surrounding image of the host vehicle and the credibility of the primary determination result, and performs a secondary determination of the primary determination result based on both the comparison result between the credibility and a threshold value and the past primary determination results of the surrounding images captured earlier than the surrounding image of the determination target. Thereby, a secondary determination result with higher accuracy than the primary determination result can be obtained, and the surrounding environment can be detected by an accurate and simple method.

[0040] [Overall Structure]

[0041] Figure 1 FIG. 1 is a structural diagram of a vehicle system 1 using the determination device of the embodiment. The vehicle equipped with the vehicle system 1 is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using the generated electric power generated by a generator connected to the internal combustion engine, or the discharge electric power of a secondary battery or a fuel cell.

[0042] The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driving operation member 70, an autonomous driving control device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are interconnected via a multi-channel communication line such as a CAN (Controller Area Network) communication line, a serial communication line, or a wireless communication network. It should be noted that Figure 1 the structure shown is merely an example, and a part of the structure may be omitted, and other structures may also be added.

[0043] The camera 10 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is installed at an arbitrary position of the vehicle (hereinafter referred to as the host vehicle M) equipped with the vehicle system 1. When shooting the front, the camera 10 is installed on the upper part of the windshield or the back of the interior rearview mirror, etc. When shooting the rear, the camera 10 is installed on the upper part of the rear windshield or the back door, etc. When shooting the side, the camera 10 is installed on the door-mounted rearview mirror, etc. The camera 10 periodically repeats shooting the surroundings of the host vehicle M. The camera 10 may also be a stereo camera.

[0044] The radar device 12 radiates radio waves such as millimeter waves to the periphery of the host vehicle M, and detects the radio waves (reflected waves) reflected by an object to detect at least the position (distance and azimuth) of the object. The radar device 12 is installed at an arbitrary part of the host vehicle M. The radar device 12 can also detect the position and speed of an object by the FM-CW (Frequency Modulated Continuous Wave) method.

[0045] The LIDAR 14 irradiates light (or an electromagnetic wave having a wavelength close to that of light) to the periphery of the host vehicle M and measures the scattered light. The LIDAR 14 detects the distance to an object based on the time from light emission to light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 is installed at an arbitrary part of the host vehicle M.

[0046] The object recognition device 16 performs sensor fusion processing on the detection results of some or all of the camera 10, the radar device 12, and the LIDAR 14, and recognizes the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition result to the autonomous driving control device 100. The object recognition device 16 can directly output the detection results of the camera 10, the radar device 12, and the LIDAR 14 to the autonomous driving control device 100. The object recognition device 16 can also be omitted from the vehicle system 1.

[0047] The communication device 20 communicates with other vehicles existing in the periphery of the host vehicle M, for example, using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), etc., or communicates with various server devices via a radio base station.

[0048] The HMI 30 presents various information to the passengers of the host vehicle M and accepts input operations performed by the passengers. The HMI 30 includes various display devices, speakers, buzzers, touch panels, switches, buttons, etc.

[0049] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the host vehicle M, an acceleration sensor that detects acceleration, an azimuth sensor that detects the direction of the host vehicle M, and the like.

[0050] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as an HDD (Hard Disk Drive) or a flash memory.

[0051] The GNSS receiver 51 determines the position of the host vehicle M based on signals received from GNSS satellites (radio waves coming from artificial satellites). The position of the host vehicle M can also be determined or supplemented by INS (Inertial Navigation System) using the output of the vehicle sensor 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, etc. Part or all of the navigation HMI 52 can also be made common with the above-mentioned HMI 30. The route determination unit 53 determines, for example, with reference to the first map information 54, a route (hereinafter referred to as a map route) from the position of the host vehicle M determined by the GNSS receiver 51 (or an arbitrary input position) to the destination input by the passenger using the navigation HMI 52.

[0052] The first map information 54 is, for example, information representing the road shape by lines indicating roads and nodes connected by the lines. The first map information 54 may also include road curvature, POI (Point Of Interest) information, etc. The map route is output to the MPU 60. The navigation device 50 can also perform route guidance using the navigation HMI 52 based on the map route. The navigation device 50 can be realized, for example, by the functions of a terminal device such as a smartphone or a tablet terminal held by the passenger. The navigation device 50 can also send the current position and the destination to the navigation server via the communication device 20 and obtain a route equivalent to the map route from the navigation server.

[0053] The MPU 60 includes, for example, a recommended lane determination unit 61, and the second map information 62 is stored in a storage device such as an HDD or a flash memory. The recommended lane determination unit 61 is realized by a hardware processor (computer) such as a CPU (Central Processing Unit) executing a program (software). In addition, the recommended lane determination unit 61 can also be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), and can also be realized by the cooperation of software and hardware. The program can be pre-stored in the storage device (a storage device having a non-transitory storage medium) of the MPU 60, and can also be stored in a removable storage medium such as a DVD or a CD-ROM, and is installed in the storage device of the MPU 60 by mounting the storage medium (non-transitory storage medium) on a drive device.

[0054] The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into a plurality of sections (for example, divided every 100 [m] in the vehicle traveling direction), refers to the second map information 62, and determines the recommended lane for each section. The recommended lane determination unit 61 makes a determination as to which lane from the left the vehicle should travel in. When there is a branch point on the route on the map, the recommended lane determination unit 61 determines the recommended lane so that the own vehicle M can travel on a reasonable route for traveling to the branch destination.

[0055] The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of the lane (road center line, center line) or information on the boundaries of the lane (road marking lines, marking lines). In addition, the second map information 62 may include road information (road signs, traffic signal machines, etc., road structures), traffic restriction information, address information (address / zip code), facility information, telephone number information, etc. The second map information 62 can be updated at any time by communicating with other devices through the communication device 20.

[0056] The driving operation member 70 includes, for example, in addition to the steering wheel 72, an accelerator pedal, a brake pedal, a shift lever, and other operation members. A sensor for detecting the operation amount or the presence or absence of an operation is installed on the driving operation member 70. The detection result of this sensor is output to the automatic driving control device 100, or is output to a part or all of the driving force output device 200, the braking device 210, and the steering device 220. The steering wheel 72 does not have to be circular, and may be in the form of an irregularly shaped steering wheel, a joystick, a button, etc. A steering grip sensor 74 is installed on the steering wheel 72. The steering grip sensor 74 is implemented by an electrostatic capacitance sensor or the like, and outputs a signal capable of detecting whether the driver is gripping the steering wheel 72 (which means contacting in a state where a force is applied) to the automatic driving control device 100.

[0057] The automatic driving control device 100 includes, for example, a first control unit 120 and a second control unit 160. The first control unit 120 and the second control unit 160 are respectively implemented by a hardware processor (computer) such as a CPU executing a program (software). In addition, a part or all of these components can also be implemented by hardware such as LSI, ASIC, FPGA, GPU (including circuitry), and can also be implemented by the cooperation of software and hardware. The program can also be pre-stored in a storage device (a storage device having a non-transitory storage medium) such as an HDD or a flash memory of the automatic driving control device 100, or can be stored in a removable storage medium such as a DVD or a CD-ROM, and is installed in the HDD or flash memory of the automatic driving control device 100 by mounting the storage medium (non-transitory storage medium) on a driving device.

[0058] Figure 2 It is a functional structure diagram of the first control unit 120 and the second control unit 160. The first control unit 120 includes, for example, an identification unit 130 and an action plan generation unit 140. The automatic driving control device 100, the first control unit 120, or the identification unit 130 is an example of a "determination device". The automatic driving control device 100 is an example of a "vehicle control device".

[0059] The first control unit 120 implements functions based on AI (Artificial Intelligence) and functions based on a pre-given model in parallel. For example, the function of "identifying an intersection" can be implemented by: executing in parallel the identification of the intersection based on deep learning, etc. and the identification based on pre-given conditions (signals with pattern matching, road signs, etc.), scoring both and comprehensively evaluating. Thereby, the reliability of automatic driving is ensured.

[0060] The identification unit 130 identifies the position, speed, acceleration, and other states of an object located around the host vehicle M based on at least a part of the information input from the camera 10, the radar device 12, and the LIDAR 14. The position of the object is, for example, identified as a position on the absolute coordinates with the representative point (center of gravity or center of the drive shaft, etc.) of the host vehicle M as the origin, and is used for control. The position of the object can also be represented by a representative point such as the center of gravity or a corner of the object, or can be represented by a region. The "state" of the object can also include the acceleration, jerk, or "action state" of the object (for example, whether a lane change is in progress or whether a lane change is about to occur).

[0061] In addition, the identification unit 130, for example, identifies the lane (travel lane) in which the host vehicle M is traveling. For example, the identification unit 130 identifies the travel lane by comparing the pattern of the road dividing line obtained from the second map information 62 (for example, the arrangement of solid lines and dashed lines) and the pattern of the road dividing line around the host vehicle M identified from the image captured by the camera 10. It should be noted that, not limited to the road dividing line, the identification unit 130 can also identify the travel lane by identifying the travel road boundary (road boundary) including the road dividing line, the road shoulder, the road edge, the median strip, the guardrail, etc. In this identification, the position of the host vehicle M obtained from the navigation device 50 or the processing result based on the INS can also be added. In addition, the identification unit 130 identifies the temporary stop line, obstacles, red lights, toll booths, and other road matters.

[0062] When the recognition unit 130 recognizes the driving lane, it recognizes the position and attitude of the host vehicle M relative to the driving lane. For example, the recognition unit 130 may also recognize the deviation of the reference point of the host vehicle M from the center of the lane and the angle formed by the traveling direction of the host vehicle M with respect to the line connecting the centers of the lanes as the relative position and attitude of the host vehicle M relative to the driving lane. Instead, the recognition unit 130 may also recognize the position of the reference point of the host vehicle M relative to any one of the side ends (road dividing lines or road boundaries) of the driving lane as the relative position of the host vehicle M relative to the driving lane.

[0063] The recognition unit 130 includes, for example, a primary determination unit 131 and a secondary determination unit 135. The primary determination unit 131 performs a primary determination of the surrounding environment of the host vehicle M based on the surrounding image of the host vehicle M input from the camera 10. The surrounding environment includes, for example, the driving states of surrounding vehicles (other vehicles). For example, the primary determination unit 131 determines the driving state of other vehicles based on the image information of the light bodies of other vehicles included in the surrounding image of the host vehicle M. The light bodies include, for example, light sources for various lights such as brake lights (stop lights) as brake lights, direction indicator lights (turn signals) as direction indicators, headlights as headlights, and rear lights as reverse lights. The surrounding image can be a black-and-white image or a color image. For example, the primary determination unit 131 determines whether another vehicle is about to stop by determining the operation state (lighting or extinguishing of the brake light) of the brake light of another vehicle. Additionally, for example, the primary determination unit 131 determines whether another vehicle is about to change its travel route, such as a right or left turn, by determining the operation state (flashing of the direction indicator light) of the direction indicator light of another vehicle. By performing such a primary determination, the primary determination unit 131 can determine the operation intention (operation intention of the light body) of the driver of another vehicle.

[0064] The primary determination unit 131 includes, for example, an object feature extraction unit 132 and an intention determination unit 133. The object feature extraction unit 132 extracts the feature information of the image corresponding to the light body of other vehicles included in the surrounding image by performing rule-based image analysis processing such as pattern matching on the surrounding image input from the camera 10. For example, the object feature extraction unit 132 extracts the feature information of the image corresponding to the brake light and the direction indicator light of other vehicles included in the surrounding image.

[0065] The intention determination unit 133 determines the operation intention (the operation intention of the lamp unit) of the driver of another vehicle based on the feature information extracted by the object feature extraction unit 132, and outputs an intention determination result (a primary determination result) and the confidence level of this intention determination result. The intention determination result is, for example, information indicating the action state of the lamp unit of another vehicle, such as the right turn indicator blinking, the left turn indicator blinking, or the brake lamp being lit. That is, the intention determination result is a determination result of the driving state of the surrounding vehicle based on the action information of the lamp unit of the surrounding vehicle included in the surrounding image.

[0066] The confidence level of the intention determination result is an index value indicating the reliability (reliability) of the intention determination result. For example, the confidence level can be a real value between 0 and 1, or can be discrete values such as high, medium, and low. The intention determination unit 133, for example, uses a learned model (for example, a neural network) obtained through machine learning to derive an intention determination result and a confidence level for the feature information. This learned model is a model that has been learned in the following manner: using a data set in which an intention determination result and a confidence level are labeled for the feature information as teaching data, and when the feature information is input, outputting an intention determination result and a confidence level.

[0067] The intention determination unit 133 outputs an intention determination result and a confidence level based on the feature information extracted from a single surrounding image (single frame) or the feature information extracted from a plurality of consecutive surrounding images (consecutive frames). For example, when determining whether the brake lamp is lit, the intention determination unit 133 outputs an intention determination result and a confidence level based on the feature information extracted from a single surrounding image (single frame). In addition, for example, when determining whether the turn indicator is blinking, the intention determination unit 133 calculates an intention determination result and a confidence level based on the feature information extracted from a plurality of consecutive surrounding images (consecutive frames).

[0068] It should be noted that the above-mentioned learned model can also be a model that has been learned in the following manner: using a data set in which an intention determination result and a confidence level are labeled for the surrounding image as teaching data, and when the surrounding image is input, outputting an intention determination result and a confidence level. In this case, it is also possible to omit the rule-based image analysis processing such as pattern matching performed by the above-mentioned primary determination unit 131.

[0069] The secondary determination unit 135, for example, includes an acquisition unit 136 and a determination unit 137. The acquisition unit 136 acquires the intention determination result and the confidence level output from the primary determination unit 131. That is, the acquisition unit 136 acquires the primary determination result of the surrounding environment determined based on the surrounding image (the first surrounding image) of the own vehicle M (vehicle) and the confidence level of this primary determination result. The acquisition unit 136 is an example of an "acquisition unit". The determination unit 137 is an example of a "determination unit".

[0070] The determination unit 137 performs a secondary determination on the intention determination result based on both the comparison result between the credibility obtained by the acquisition unit 136 and a preset threshold value, and the past intention determination results for the surrounding images of the host vehicle M captured earlier than the surrounding images that are the determination targets of the intention determination unit 133, and outputs a secondary determination result with a higher accuracy than the intention determination result. The determination unit 137 outputs the secondary determination result to the action plan generation unit 140. The processing of the determination unit 137 will be described in detail later.

[0071] The action plan generation unit 140 generates a target trajectory for the future travel of the host vehicle M automatically (independent of the driver's operation) so as to travel in the recommended lane determined by the recommended lane determination unit 61 in principle and be able to cope with the surrounding conditions of the host vehicle M. The target trajectory includes, for example, a speed element. For example, the target trajectory is expressed as a trajectory formed by arranging in parallel the points (trajectory points) that the host vehicle M should reach in sequence. The trajectory points are the points that the host vehicle M should reach at regular driving distances (for example, about several [m]) in the along-the-way distance. In addition, the target speed and target acceleration at regular sampling times (for example, about zero point several [sec]) are generated as part of the target trajectory. Alternatively, the trajectory points can be the positions that the host vehicle M should reach at each sampling time at regular sampling times. In this case, the information on the target speed and target acceleration is represented by the interval of the trajectory points. The action plan generation unit 140 is an example of a "control unit".

[0072] When generating the target trajectory, the action plan generation unit 140 can set events for autonomous driving. The events for autonomous driving include a constant-speed driving event, a low-speed following driving event, a lane change event, a branch event, a merging event, a takeover event, and the like. The action plan generation unit 140 generates a target trajectory corresponding to the started event.

[0073] The action plan generation unit 140 determines whether to change the behavior control of the host vehicle M based on the secondary determination result output by the recognition unit 130. When it is determined based on the secondary determination result that a change in the behavior control of the host vehicle M is required, the action plan generation unit 140 changes the behavior control of the host vehicle M (such as changing the target trajectory). In addition, when it is determined based on the secondary determination result that no change in the behavior control of the host vehicle M is required, the action plan generation unit 140 does not change the behavior control of the host vehicle M. The processing of the action plan generation unit 140 will be described in detail later.

[0074] The second control unit 160 controls the driving force output device 200, the braking device 210, and the steering device 220 so that the host vehicle M passes through the target trajectory generated by the action plan generation unit 140 at a predetermined time.

[0075] The second control unit 160 includes, for example, an acquisition unit 162, a speed control unit 164, and a steering control unit 166. The acquisition unit 162 acquires information on a target trajectory (trajectory points) generated by the action plan generation unit 140 and stores it in a memory (not shown). The speed control unit 164 controls the driving force output device 200 or the braking device 210 based on the speed elements attached to the target trajectory stored in the memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target trajectory stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is realized, for example, by a combination of feedforward control and feedback control. As an example, the steering control unit 166 executes in combination feedforward control corresponding to the curvature of the road ahead of the own vehicle M and feedback control based on the deviation from the target trajectory.

[0076] The driving force output device 200 outputs a driving force (torque) for the vehicle to travel to the drive wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above structure according to the information input from the second control unit 160 or the information input from the driving operation member 70.

[0077] The braking device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to the information input from the second control unit 160 or the information input from the driving operation member 70, and outputs a braking torque corresponding to the braking operation to each wheel. The braking device 210 may include a mechanism that transmits the hydraulic pressure generated by the operation of the brake pedal included in the driving operation member 70 to the cylinder via a master hydraulic cylinder as a backup. It should be noted that the braking device 210 is not limited to the structure described above, and may also be an electronically controlled hydraulic braking device that controls an actuator according to the information input from the second control unit 160 and transmits the hydraulic pressure of the master hydraulic cylinder to the cylinder.

[0078] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies a force to a rack and pinion mechanism to change the direction of the steering wheel. The steering ECU drives the electric motor according to the information input from the second control unit 160 or the information input from the driving operation member 70 to change the direction of the steering wheel.

[0079] [Determination Process]

[0080] Hereinafter, the determination process of the embodiment will be described using a flowchart. Figure 3This is a flowchart showing an example of the determination process performed by the recognition unit 130 of the autonomous driving control device 100 according to an embodiment. In the following description, an example will be described in which the host vehicle M performs autonomous driving under the control of autonomous driving based on the autonomous driving control device 100. Figure 3 The determination process of the shown flowchart is repeatedly executed during the period when the host vehicle M performs autonomous driving.

[0081] First, the object feature extraction unit 132 of the primary determination unit 131 extracts object feature information from the surrounding image input by the camera 10 (step S101). For example, the object feature extraction unit 132 performs rule-based image analysis processing such as pattern matching on the surrounding image, thereby extracting an image of a part corresponding to the lamp body (brake lamp, turn signal lamp, etc.) of another vehicle included in the surrounding image.

[0082] Figure 4A This is a diagram showing an example of a surrounding image captured in front of the host vehicle M. In Figure 4A the shown surrounding image F1, another vehicle V1 traveling in front of the host vehicle M is included. In this case, the object feature extraction unit 132 extracts the brake lamp image BL of the part corresponding to the brake lamp of the other vehicle V1 from the surrounding image F1 as object feature information. It should be noted that in addition to the brake lamp image BL, the object feature extraction unit 132 can also extract an image of a part corresponding to a lamp body other than the brake lamp, such as the turn signal lamp of the other vehicle V1, from the surrounding image F1.

[0083] Figure 4B This is a diagram showing another example of a surrounding image captured in front of the host vehicle M. Figure 4B The shown surrounding image F2 is an image captured at a different timing from Figure 4A the shown surrounding image F1. In this case, the object feature extraction unit 132 extracts the brake lamp image BL of the part corresponding to the brake lamp of the other vehicle V1 from the surrounding image F2 as object feature information. It should be noted that in addition to the brake lamp image BL, the object feature extraction unit 132 can also extract an image of a part corresponding to a lamp body other than the brake lamp, such as the turn signal lamp of the other vehicle V1, from the surrounding image F2.

[0084] Figure 5A This is a diagram showing another example of a surrounding image captured in front of the host vehicle M. In Figure 5AThe surrounding image F3 shown includes another vehicle V2 traveling ahead in the left lane of the host vehicle M. In this case, the target feature extraction unit 132 extracts a turn signal lamp image TS of a part corresponding to the turn signal lamp of the other vehicle V2 from the surrounding image F3 as target feature information. It should be noted that in addition to the turn signal lamp image TS, the target feature extraction unit 132 may also extract an image of a part corresponding to a lamp body other than the turn signal lamp, such as the brake lamp of the other vehicle V2, from the surrounding image F3.

[0085] Figure 5B FIG. is a diagram showing another example of a surrounding image captured in front of the host vehicle M. Figure 5B The surrounding image F4 shown is an image captured at a timing different from that of Figure 5A the surrounding image F3 shown. In this case, the target feature extraction unit 132 extracts a turn signal lamp image TS of a part corresponding to the turn signal lamp of the other vehicle V2 from the surrounding image F4 as target feature information. It should be noted that in addition to the turn signal lamp image TS, the target feature extraction unit 132 may also extract an image of a part corresponding to a lamp body other than the turn signal lamp, such as the brake lamp of the other vehicle V2, from the surrounding image F4.

[0086] Next, the intention determination unit 133 of the primary determination unit 131 determines the operation intention (lamp body operation intention) of the driver of the other vehicle based on the target feature information extracted by the target feature extraction unit 132 (step S103). The intention determination unit 133 outputs an intention determination result and the reliability of the intention determination result. For example, the intention determination unit 133 outputs an intention determination result indicating whether the brake lamp is lit or whether the turn signal lamp is flashing, and the reliability of the intention determination result.

[0087] When the brake lamp image BL is extracted from the Figure 4A surrounding image F1 shown by the target feature extraction unit 132, the intention determination unit 133 determines that the brake lamp is off based on the brake lamp image BL, and outputs an intention determination result indicating that the brake lamp is off. In addition, in this case, it can be determined that the image quality of the brake lamp image BL is high and the reliability of the intention determination result is high. Therefore, the intention determination unit 133 outputs "0.95" as a value of high reliability.

[0088] On the other hand, even when an image of the same scene as the Figure 4A surrounding image F1 shown is captured, if the image quality of the brake lamp image BL is low or a part of the image is missing as in the Figure 6 surrounding image F5 shown, or in a case where there is a concern about a problem with the accuracy of the intention determination result, a value of low reliability is output. That is, when the brake lamp image BL is extracted from the Figure 6When the brake light image BL with low image quality is extracted from the peripheral image F5 shown, the intention determination unit 133 determines that the brake light is off based on the brake light image BL and outputs an intention determination result indicating that the brake light is off. However, in this case, it can be determined that the image quality of the brake light image BL is low and the credibility of the intention determination result is low. Therefore, the intention determination unit 133 outputs "0.40", which is a value with low credibility.

[0089] In addition, when the object feature extraction unit 132 extracts Figure 4B the brake light image BL from the peripheral image F2 shown, the intention determination unit 133 determines that the brake light is on based on the brake light image BL and outputs an intention determination result indicating that the brake light is on. In addition, in this case, it can be determined that the image quality of the brake light image BL is high and the credibility of the intention determination result is high. Therefore, the intention determination unit 133 outputs "0.92", which is a value with high credibility.

[0090] On the other hand, even when an image of the same scene as the Figure 4B peripheral image F2 shown is captured, if there are concerns about the accuracy of the intention determination result, such as the case where the image quality of the brake light image BL is low or a part of the image is missing as in the Figure 7 peripheral image F20 shown, a value with low credibility is output. That is, when the object feature extraction unit 132 extracts Figure 7 the brake light image BL with low image quality from the peripheral image F20 shown, the intention determination unit 133 determines that the brake light is on based on the brake light image BL and outputs an intention determination result indicating that the brake light is on. However, in this case, it can be determined that the image quality of the brake light image BL is low and the credibility of the intention determination result is low. Therefore, the intention determination unit 133 outputs "0.60", which is a value with low credibility.

[0091] In addition, when the object feature extraction unit 132 extracts Figure 5A the turn signal image TS from the peripheral image F3 shown, the intention determination unit 133 determines that the turn signals (right turn signal and left turn signal) are off based on the turn signal image TS and outputs an intention determination result indicating that the turn signals are off. In addition, in this case, it can be determined that the image quality of the turn signal image TS is high and the credibility of the intention determination result is high. Therefore, the intention determination unit 133 outputs "0.98", which is a value with high credibility. In addition, when the object feature extraction unit 132 extracts Figure 5BWhen the direction indicator light image TS is extracted from the surrounding image F4 shown, the intention determination unit 133 determines, based on the direction indicator light image TS, that the direction indicator light (right direction indicator light) is on, and outputs an intention determination result indicating that the right direction indicator light is on. Further, in this case, it is possible to determine that the image quality of the direction indicator light image TS is high and the reliability of the intention determination result is high. Therefore, the intention determination unit 133 outputs "0.91", which is a value with high reliability.

[0092] Next, the acquisition unit 136 of the secondary determination unit 135 acquires the intention determination result and the reliability output by the intention determination unit 133 (step S105).

[0093] Next, the determination unit 137 of the secondary determination unit 135 determines whether the acquired reliability is greater than the first threshold (step S107). This first threshold is preset as a criterion for determining whether the reliability is sufficiently high. For example, the lower limit value of the reliability that can be determined to be able to use the corresponding intention determination result for autonomous driving control is set for this first threshold. When the reliability is defined by a real value between 0 and 1, a value of 0.5 or more, such as 0.9, 0.8, 0.7, 0.6, 0.5, etc., is set for this first threshold. It should be noted that the determination unit 137 may also determine whether the acquired reliability is greater than or equal to the first threshold.

[0094] When the determination unit 137 determines that the acquired reliability is greater than the first threshold (step S107, "Yes"), it outputs the acquired intention determination result as a secondary determination result to the action plan generation unit 140 (step S113). Next, the action plan generation unit 140 determines whether it is necessary to change the behavior control of the own vehicle M based on the secondary determination result output by the recognition unit 130 (step S115). For example, the action plan generation unit 140 determines whether it is necessary to change at least one of the speed control, acceleration control, steering control, and stop control of the own vehicle M.

[0095] In the above step S115, when the action plan generation unit 140 determines that it is necessary to change the behavior control of the own vehicle M (step S115, "Yes"), it changes the behavior control (step S117). For example, Figure 4B in the case where the intention determination result indicates that the brake light of another vehicle V1 is on as in the example, in order to avoid a collision with the other vehicle V1, the action plan generation unit 140 determines that it is necessary to change the behavior control of the own vehicle M, changes the behavior control such as decreasing the speed of the own vehicle M and changing lanes, and regenerates the target trajectory. The action plan generation unit 140 outputs the generated target trajectory to the second control unit 160.

[0096] In the above step S115, when the action plan generation unit 140 determines that there is no need to change the behavior control of the own vehicle M (step S115, "No"), it does not change the behavior control. For example, in the case where the secondary determination result indicates that the brake light of another vehicle V1 is "off" as in the example of Figure 4A since it is okay to maintain the current driving state, the action plan generation unit 140 determines that there is no need to change the behavior control of the own vehicle M and does not change the behavior control.

[0097] That is, when the action plan generation unit 140 determines that it is necessary to change the behavior control of the own vehicle M, it changes the behavior control of the own vehicle M. On the other hand, when the action plan generation unit 140 determines that there is no need to change the behavior control of the own vehicle M, it does not change the behavior control of the own vehicle.

[0098] In the above step S107, when the determination unit 137 determines that the obtained credibility is not greater than the first threshold (step S107, "No"), it determines whether the credibility is less than the second threshold (step S109). This second threshold is preset as a criterion for determining whether the credibility is sufficiently low. For example, an upper limit value of the credibility that can immediately determine that the corresponding intention determination result cannot be used for autonomous driving control is set for this second threshold. When the credibility is defined by a real value between 0 and 1, a value less than 0.5 such as 0.1, 0.2, 0.3, or 0.4 is set for this second threshold. This second threshold is set to a value smaller than the first threshold. It should be noted that the determination unit 137 may also determine whether the credibility is below the second threshold.

[0099] In the above step S109, when the determination unit 137 determines that the obtained credibility is not less than the second threshold (step S109, "No"), it determines whether the number of past intention determination results indicating the same judgment result as the current intention determination result among a specified number of past intention determination results is equal to or greater than a specified value (step S111).

[0100] For example, when the first threshold is set to "0.9" and the second threshold is set to "0.5", for the Figure 7 peripheral image F20 (credibility "0.60") shown, the determination unit 137 performs the determination process of the above step S111. Figure 8 And Figure 9 are diagrams for explaining the determination process of step S111. In Figure 8In this case, the peripheral image F20 of the current determination object and the peripheral images F12 to F19 of the amount of eight frames captured earlier than the peripheral image F20 are shown. Among these past peripheral images F12 to F19, the past peripheral images that have the same determination result as the determination result of "brake light on" for the peripheral image F20 are the five peripheral images F15 to F19. Here, for example, when the specified value is set to "5", the determination unit 137 determines that the number of past intention determination results indicating the same determination result as the obtained intention determination result is equal to or greater than the specified value.

[0101] On the other hand, in Figure 9 this example, among the past peripheral images F12 to F19, the past peripheral images that have the same determination result as the determination result of "brake light on" for the peripheral image F20 are the three peripheral images F12, F13, and F15. Here, for example, when the specified value is set to "5", the determination unit 137 determines that the number of past intention determination results indicating the same determination result as the obtained intention determination result is not equal to or greater than the specified value. It should be noted that the determination unit 137 can also determine whether the number of past intention determination results indicating the same determination result as the intention determination result of the current determination object is greater than the specified value.

[0102] That is, when the confidence level is greater than the first threshold, the determination unit 137 performs the first process of outputting the primary determination result as the secondary determination result. When the confidence level is less than the second threshold, the determination unit 137 performs the second process of retaining the secondary determination. When the confidence level is equal to or less than the first threshold and equal to or greater than the second threshold, the determination unit 137 performs the third process of determining whether to adopt the primary determination result as the secondary determination result based on the past primary determination result.

[0103] In addition, in the above-mentioned third process, when the number of past primary determination results indicating the same determination result as the primary determination result among the specified number of past primary determination results is equal to or greater than the specified value, the determination unit 137 outputs the primary determination result as the secondary determination result. When the number of past primary determination results indicating the same determination result as the primary determination result among the specified number of past primary determination results is less than the specified value, the secondary determination is retained. The specified number of past primary determination results is an integer N of 2 or more, and the specified value is a value equal to or greater than N / 2 and equal to or less than N.

[0104] In the above step S111, when the determination unit 137 determines that the number of past willingness determination results indicating the same determination result as the obtained willingness determination result is equal to or greater than a specified value (step S111, "Yes"), the obtained willingness determination result is output to the action plan generation unit 140 as a secondary determination result (step S113). In this way, the determination unit 137 adopts the willingness determination result with high persistence as the secondary determination result. Next, based on the secondary determination result output by the recognition unit 130, the action plan generation unit 140 determines whether it is necessary to change the behavior control of the own vehicle M (step S115). When the action plan generation unit 140 determines that it is necessary to change the behavior control of the own vehicle M, it changes the behavior control (step S117).

[0105] On the other hand, when the determination unit 137 determines in the above step S109 that the obtained credibility is less than the second threshold (step S109, "Yes"), or when it determines in the above step 111 that the number of past willingness determination results indicating the same determination result as the obtained willingness determination result is not equal to or greater than the specified value (step S111, "No"), the obtained willingness determination result is not adopted, the secondary determination is retained, and information indicating that the secondary determination cannot be performed (for example, an Unknown signal) is output to the action plan generation unit 140 (step S119). In this case, the determination unit 137 may output the willingness determination result and the credibility output by the willingness determination unit 133 to the action plan generation unit 140.

[0106] Next, the action plan generation unit 140 performs error control based on the Unknown signal output by the recognition unit 130 (step S121). For example, as error control, the action plan generation unit 140 determines the surrounding environment based on the surrounding environment information (information such as the direction, speed, and acceleration of other vehicles) obtained from detection units (radar device 12, LIDAR 14, etc.) different from the camera 10 that captured the surrounding image, and determines whether it is necessary to change the behavior control of the own vehicle M.

[0107] Alternatively, the action plan generation unit 140 may also perform a control process of maintaining the current driving state of the own vehicle M and waiting until the output of the next secondary determination result. Alternatively, as error control, the action plan generation unit 140 may also perform a control process of changing to the safe side of the driving state according to the current driving state of the own vehicle M. For example, when the speed of the current own vehicle M is equal to or greater than a specified threshold, the action plan generation unit 140 may perform a control process such as decreasing the speed as a control process for changing to the safe direction of the driving state. Thus, the processing of this flowchart ends.

[0108] According to the embodiments described above, there are provided: an acquisition unit 136 (acquisition unit) that acquires a primary determination result of the surrounding environment determined based on a first surrounding image of the own vehicle M and the reliability of the primary determination result; and a determination unit 137 (determination unit) that performs a secondary determination on the primary determination result based on both the comparison result between the reliability and a preset threshold value and the past primary determination results of the surrounding images of the own vehicle M captured earlier than the first surrounding image, and outputs a secondary determination result with a higher accuracy than the primary determination result. Thus, it is possible to detect the surrounding environment of the own vehicle M by an accurate and simple method and suppress the occurrence of over-detection of the surrounding environment. In addition, by determining whether to change the behavior control of the own vehicle M based on the secondary determination result, stable autonomous driving control can be performed.

[0109] The embodiments described above can be expressed as follows.

[0110] A determination device, wherein,

[0111] The determination device includes a storage device storing a program and a hardware processor,

[0112] By executing the program by the hardware processor,

[0113] acquire a primary determination result of the surrounding environment determined based on a first surrounding image of the vehicle and the reliability of the primary determination result,

[0114] perform a secondary determination on the primary determination result based on both the comparison result between the reliability and a preset threshold value and the past primary determination results of the surrounding images of the vehicle captured earlier than the first surrounding image, and output a secondary determination result with a higher accuracy than the primary determination result.

[0115] As described above, specific embodiments of the present invention have been described using the embodiments, but the present invention is in no way limited to such embodiments, and various modifications and substitutions can be made without departing from the gist of the present invention.

Claims

1. A determination device, wherein: the determination device includes: an acquisition unit that acquires a primary determination result of the surrounding environment determined based on a first surrounding image of a vehicle and a confidence level of the primary determination result; and a determination unit that performs a secondary determination on the primary determination result based on both a comparison result between the confidence level and a preset threshold value and a past primary determination result of a surrounding image of the vehicle captured earlier than the first surrounding image, and outputs a secondary determination result with higher accuracy than the primary determination result; the threshold value includes a first threshold value and a second threshold value lower than the first threshold value; when the confidence level is greater than the first threshold value, the determination unit performs a first process of outputting the primary determination result as the secondary determination result; when the confidence level is less than the second threshold value, the determination unit performs a second process of withholding the secondary determination; when the confidence level is less than or equal to the first threshold value and greater than or equal to the second threshold value, the determination unit performs a third process of determining whether to adopt the primary determination result as the secondary determination result based on the past primary determination result.

2. The determination device according to claim 1, wherein: in the third process, the determination unit outputs the primary determination result as the secondary determination result when the number of the past primary determination results indicating the same determination result as the primary determination result among a specified number of the past primary determination results is equal to or greater than a specified value; withholds the secondary determination when the number of the past primary determination results indicating the same determination result as the primary determination result among the specified number of the past primary determination results is less than the specified value.

3. The determination device according to claim 2, wherein: the specified number of the past primary determination results is an integer N of 2 or more; the specified value is a value greater than or equal to N / 2 and less than or equal to N.

4. The determination device according to any one of claims 1 to 3, wherein: when withholding the secondary determination, the determination unit outputs information indicating that the secondary determination cannot be performed.

5. The determination device according to any one of claims 1 to 3, wherein: the primary determination result is a determination result of a driving state of a surrounding vehicle based on operation information of a lamp body of the surrounding vehicle included in the first surrounding image.

6. The determination device according to claim 5, wherein: the lamp body includes at least one of a brake lamp and a direction indicator.

7. A vehicle control device, wherein: the vehicle control device includes: the determination device according to any one of claims 1 to 6; and a control unit that determines whether to change the behavior control of the vehicle based on the secondary determination result output from the determination device.

8. The vehicle control device according to claim 7, wherein: when it is determined based on the secondary determination result that the behavior control of the vehicle needs to be changed, the control unit changes the behavior control of the vehicle.

9. The vehicle control device according to claim 7 or 8, wherein when it is determined based on the secondary determination result that there is no need to change the vehicle behavior control, the control unit does not change the vehicle behavior control.

10. The vehicle control device according to claim 7 or 8, wherein when the determination unit holds the secondary determination, the control unit determines whether it is necessary to change the vehicle behavior control based on the surrounding environment information obtained by a detection unit different from the camera that captured the first surrounding image.

11. The vehicle control device according to claim 7 or 8, wherein the control unit determines whether it is necessary to change at least one of the vehicle speed control, acceleration control, steering control, and stop control.

12. A determination method, wherein a computer mounted on a vehicle performs the following processing: obtaining a primary determination result of the surrounding environment determined based on a first surrounding image of the vehicle and the credibility of the primary determination result, performing a secondary determination on the primary determination result based on both the comparison result of the credibility with a preset threshold and the past primary determination result of the surrounding image of the vehicle captured earlier than the first surrounding image, and outputting a secondary determination result with higher accuracy than the primary determination result, the threshold includes a first threshold and a second threshold lower than the first threshold, when the credibility is greater than the first threshold, performing a first process of outputting the primary determination result as the secondary determination result, when the credibility is less than the second threshold, performing a second process of holding the secondary determination, when the credibility is less than or equal to the first threshold and greater than or equal to the second threshold, performing a third process of determining whether to adopt the primary determination result as the secondary determination result based on the past primary determination result.

13. A storage medium, wherein the storage medium stores a program that causes a computer mounted on a vehicle to perform the following processing: obtaining a primary determination result of the surrounding environment determined based on a first surrounding image of the vehicle and the credibility of the primary determination result, performing a secondary determination on the primary determination result based on both the comparison result of the credibility with a preset threshold and the past primary determination result of the surrounding image of the vehicle captured earlier than the first surrounding image, and outputting a secondary determination result with higher accuracy than the primary determination result, the threshold includes a first threshold and a second threshold lower than the first threshold, when the credibility is greater than the first threshold, performing a first process of outputting the primary determination result as the secondary determination result, when the credibility is less than the second threshold, performing a second process of holding the secondary determination, In the case where the confidence level is below the first threshold and above the second threshold, a third process is performed to determine whether to adopt the one-time determination result as the secondary determination result based on the past one-time determination result.

Images