Determination device

Abstract

To provide a determination device capable of determining whether or not a pedestrian enters a road based on the pedestrian's state, as well as a state of a driver or a vehicle that the pedestrian can recognize.SOLUTION: A determination device includes: a determination unit configured to execute a determination process to determine whether or not a pedestrian enters a road based on road information representing a road within a predetermined range in a vehicle's traveling direction and pedestrian information representing the pedestrian; a generation unit configured to generate state information, based on information representing the driver of the vehicle or information representing the operation of the vehicle, the state information representing a state of the driver or the vehicle during a past predetermined period, the state being recognizable by the pedestrian; and a modification unit configured to modify a criterion of the determination process of the determination unit so that it becomes easier to determine that the pedestrian enters the road or that the pedestrian does not enter the road, based on the state information generated by the generation unit, wherein the determination unit executes the determination process of whether or not the pedestrian enters the road using the criterion of the determination process modified by the modification unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a determination device. [Background technology]

[0002] There are cases where a pedestrian tries to cross the road in front of a moving vehicle. The driver of the vehicle drives the vehicle while watching the pedestrian's behavior (see, for example, Patent Document 1). The pedestrian also watches the vehicle's behavior to determine the timing to cross the road.

[0003] In this way, the movement of the vehicle may affect the movement of the pedestrian, and the movement of the pedestrian may affect the movement of the vehicle.

[0004] By repeatedly considering the pedestrian's movements and the vehicle's movements, it is believed that it is possible to accurately determine whether a pedestrian is crossing the road. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-332297 Summary of the Invention [Problem to be solved by the invention]

[0006] However, currently, the behavior of pedestrians and the behavior of vehicles are not repeatedly considered. Whether a pedestrian enters a road is determined based on the state of the pedestrian up to the time of determination.

[0007] Therefore, it has been difficult to accurately determine whether a pedestrian is entering the road. Therefore, it is believed that by incorporating the interaction between the pedestrian's movement and the vehicle's movement, it is possible to more accurately determine whether a pedestrian is entering the road.

[0008] A pedestrian observes the movement of the driver or the vehicle and determines the timing to cross the road. Therefore, the state of the driver or the vehicle that can be confirmed by the pedestrian becomes information for determining the pedestrian's actions.

[0009] An object of the present disclosure is to provide a determination device that can determine whether a pedestrian enters a road based on the state of the pedestrian and the state of the driver or the vehicle that can be confirmed by the pedestrian.

Means for Solving the Problems

[0010] (1) According to one embodiment, a determination device is provided. This determination device includes a determination unit that executes a determination process of whether a pedestrian enters a road based on road information representing a road in a predetermined range in the traveling direction of the vehicle and pedestrian information representing the pedestrian, and information representing the driver of the vehicle or information representing the operation of the vehicle. A generation unit that generates state information representing the state of the driver or the vehicle that can be confirmed by the pedestrian and the state of the vehicle in a predetermined period in the past, and based on the state information generated by the generation unit, it is easier to determine that the pedestrian enters the road, or it is easier to determine that the pedestrian does not enter the road. And a change unit that changes the criterion of the determination process of the determination unit, and the determination unit executes the determination process of whether the pedestrian enters the road using the criterion of the determination process changed by the change unit.

[0011] (2) In the determination device of (1), the determination unit estimates the entry probability that the pedestrian enters the road after a predetermined time based on the pedestrian information. When the entry probability exceeds a predetermined reference value, the determination unit determines that the pedestrian enters the road. It is preferable that the change unit changes the reference value based on the state information generated by the generation unit.

[0012] (3) In the determination device of (2), it is preferable that the pedestrian information includes the age of the pedestrian or information representing the luggage of the pedestrian.

[0013] (4) In any of the determination devices according to (1) to (3), when the state information generated by the generation unit indicates that the vehicle has decelerated, the driver's line of sight is directed at a pedestrian, or the vehicle has been steered away from the position of the pedestrian, it is preferable that the change unit changes the criteria for the determination process so that it is easier to determine that the pedestrian is about to enter the road.

[0014] (5) In any of the determination devices according to (1) to (3), when the state information generated by the generation unit indicates that the vehicle has accelerated or the driver's line of sight is not directed at a pedestrian, it is preferable that the change unit changes the criteria for the determination process so that it is easier to determine that the pedestrian will not enter the road.

Advantages of the Invention

[0015] The determination device according to the present disclosure can accurately determine whether a pedestrian will enter the road based on the state of the pedestrian and the state of the driver or vehicle that can be confirmed by the pedestrian.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0017] FIG. 1 is a diagram for explaining an overview of the operation of the determination device according to the present embodiment. Hereinafter, with reference to FIG. 1, an overview of the operation related to the determination process of the determination device 13 according to the present embodiment disclosed in this specification will be described.

[0018] As shown in FIG. 1, the vehicle 10 is traveling on the road 50. The vehicle 10 includes an object detection device 11, an automatic control device 12, and a determination device 13. The object detection device 11 detects pedestrian information representing a pedestrian 60 in a predetermined range in the traveling direction of the vehicle based on environmental information representing the environment around the vehicle 10 such as a camera image. In the example shown in FIG. 1, the determination device 13 detects the position of the pedestrian 60 with respect to the vehicle 10 as the pedestrian information.

[0019] In addition, the object detection device 11 detects road information representing a road in a predetermined range in the traveling direction of the vehicle based on the environmental information. The object detection device 11 detects the position of the lane dividing line of the road 50 as the road information. The automatic control device 12 controls the vehicle 10 based on the road information, the pedestrian information, and the like. The vehicle 10 may be an autonomous vehicle.

[0020] The determination device 13 executes a determination process as to whether or not the pedestrian 60 enters the road based on the road information and the pedestrian information. The determination device 13 notifies the automatic control device 12 of the result of the determination process. The automatic control device 12 controls the vehicle 10 based on the result of the determination process.

[0021] The determination device 13 detects state information representing the state of a driver (not shown) who can confirm the pedestrian 60 and drives the vehicle 10 during a predetermined period in the past. In addition, the determination device 13 detects state information representing the state of the vehicle 10 during a predetermined period in the past that can be confirmed from the pedestrian 60.

[0022] When the state information represents, for example, that the driver's line of sight is directed at the pedestrian 60, that the vehicle 10 has reduced its speed, or that the vehicle 10 has been steered away from the position of the pedestrian 60, the determination device 13 changes the criteria for the determination process so that it is more likely to be determined that the pedestrian 60 enters the road 50.

[0023] When the pedestrian 60 confirms that the vehicle 10 has reduced its speed, or that the driver's line of sight is directed at the pedestrian 60, or that the vehicle 10 has been steered away from the pedestrian 60's position, the pedestrian 60 will likely determine that the vehicle 10 is allowing the pedestrian 60 to cross the road 50. The pedestrian will actively attempt to cross the road 50.

[0024] On the other hand, if the status information indicates, for example, that the driver's gaze is not directed at the pedestrian or that the vehicle 10 has increased its speed, the determination device 13 changes the criteria for the determination process so that it becomes easier to determine that the pedestrian 60 will not enter the road 50.

[0025] When the pedestrian 60 sees that the vehicle 10 has increased its speed or that the driver's gaze is not directed at the pedestrian 60, the pedestrian 60 may conclude that the vehicle 10 does not allow the pedestrian 60 to cross the road 50. The pedestrian becomes reluctant to cross the road.

[0026] Therefore, the determination device 13 takes into consideration the interaction between the pedestrian's movements and the vehicle's movements based on the state information, and also performs a determination process to determine whether the pedestrian 60 will enter the road using the changed criteria for the determination process based on the road information and pedestrian information.

[0027] As described above, the determination device 13 can accurately determine whether the pedestrian 60 will enter the road 50 based on the state of the pedestrian 60 as well as the state of the driver or vehicle 10 that can be seen by the pedestrian 60.

[0028] When the determination device 13 determines that the pedestrian 60 is entering the road 50, the determination device 13 may notify the driver. Furthermore, when the determination device 13 determines that the pedestrian 60 is entering the road 50, the automatic control device 12 may slow down or stop the vehicle 10. This ensures the safety of the pedestrian 60 and the vehicle 10.

[0029] FIG. 2 is a hardware configuration diagram of a vehicle 10 in which the determination device 13 of the present embodiment is implemented. The vehicle 10 includes a front camera 2, a millimeter-wave radar 3, a monitoring camera 4, an operation unit 5, a vehicle speed sensor 6, a user interface (UI) 7, an object detection device 11, an automatic control device 12, a determination device 13, and the like. Note that the vehicle 10 may further include another distance measurement sensor such as a LiDAR sensor.

[0030] The front camera 2, the millimeter-wave radar 3, the monitoring camera 4, the operation unit 5, the vehicle speed sensor 6, the UI 7, the object detection device 11, the automatic control device 12, and the determination device 13 are communicably connected via an in-vehicle network 14 compliant with a standard such as a controller area network.

[0031] The front camera 2 is an example of an imaging unit provided in the vehicle 10. The front camera 2 is attached to the vehicle 10 so as to face the front of the vehicle 10. The front camera 2 captures a camera image representing the environment of an area within a predetermined field of view in front of the vehicle 10, for example, at a camera image capture time set at a predetermined cycle. The camera image may represent a road included in a predetermined area in front of the vehicle 10 and road features such as lane dividing lines on the road surface. The front camera 2 includes a two-dimensional detector configured by an array of photoelectric conversion elements sensitive to visible light, such as a CCD or a C-MOS. The front camera 2 also includes an imaging optical system that forms an image of an area to be photographed on the two-dimensional detector. The field of view of the front camera 2 is an example of a predetermined range.

[0032] Each time the front camera 2 captures a camera image, the front camera 2 outputs the camera image and the camera image capture time to the object detection device 11 and the like via the in-vehicle network 14. The camera image is used in the object detection device 11 for processing to detect objects and road features around the vehicle 10.

[0033] The millimeter-wave radar 3 is attached to, for example, the outer surface of the vehicle 10 so as to face the front of the vehicle 10. The millimeter-wave radar 3 emits millimeter waves so as to scan a predetermined field of view in front of the vehicle 10 at a reflection wave information acquisition time set at a predetermined cycle. Then, the millimeter-wave radar 3 receives the reflected waves reflected by the reflecting object. The time required for the reflected waves to return has distance information between the vehicle 10 and other objects located in the direction in which the millimeter waves are irradiated. The millimeter-wave radar 3 outputs the reflected wave information to the object detection device 11 and the like via the in-vehicle network 14 together with the reflection wave information acquisition time when the millimeter waves are emitted. The reflected wave information includes the irradiation direction of the millimeter waves and the time required for the reflected waves to return. The reflection wave information acquisition time represents the time when the millimeter waves are emitted. The reflected wave information is used in the object detection device 11 for the process of detecting the objects around the vehicle 10. The field of view of the millimeter-wave radar 3 preferably overlaps with the field of view of the front camera 2.

[0034] The monitoring camera 4 is disposed in the vehicle interior so as to be able to acquire a monitoring image including the face of the driver driving the vehicle 10. The monitoring camera 4 is disposed, for example, on the steering column. The monitoring camera 4 acquires a monitoring image representing the periphery of the driver's seat, for example, at a monitoring image acquisition time having a predetermined cycle. The monitoring image is an example of information representing the state of the driver.

[0035] The monitoring camera 4 includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared rays, such as a CCD or a C-MOS, and an imaging optical system that forms an image of an area to be acquired on the two-dimensional detector. Each time the monitoring camera 4 acquires a monitoring image, the monitoring camera 4 outputs the monitoring image and the image acquisition time when the monitoring image is acquired to the determination device 13 via the in-vehicle network 14. The monitoring image is used in the determination device 13 to represent state information that can be confirmed by a pedestrian and represents the state of the driver driving the vehicle 10 for a predetermined period in the past.

[0036] The operation unit 5 includes a steering wheel, an accelerator pedal, and a brake pedal. The steering wheel generates a signal corresponding to the steering angle by the driver and outputs it to the automatic control device 12 via the in-vehicle network 14. The accelerator pedal generates a signal corresponding to the accelerator opening by the driver and outputs it to the automatic control device 12 via the in-vehicle network 14. The brake pedal generates a signal corresponding to the braking amount by the driver and outputs it to the automatic control device 12 via the in-vehicle network 14.

[0037] The vehicle speed sensor 6 detects speed information representing the speed of the vehicle 10. The vehicle speed sensor 6 has, for example, a measuring unit that measures the rotation speed of the tires of the vehicle 10. The vehicle speed sensor 6 outputs the speed information to the object detection device 11, the automatic control device 12, the determination device 13, etc. via the in-vehicle network 14. The speed information is used in the object detection device 11, the automatic control device 12, and the determination device 13 for processing to obtain the speed of the vehicle 10. The speed information is an example of information representing the operation of the vehicle.

[0038] The UI 7 is an example of a notification unit. The UI 7 is controlled by the automatic control device 12, the determination device 13, etc., and notifies the driver of the driving information or warnings of the vehicle 10. The driving information of the vehicle 10 includes the current position of the vehicle, notifications to the driver, etc. The UI 7 has a display device 7a such as a liquid crystal display or a touch panel for displaying the driving information, etc. Also, the UI 7 may have an acoustic output device (not shown) for notifying the driver of the driving information, etc.

[0039] The object detection device 11 detects the objects around the vehicle 10 and their types based on the camera image. The objects include moving objects such as pedestrians or vehicles. Also, the object detection device 11 detects road features such as lane dividing lines and traffic lights based on the camera image. The object detection device 11 may detect the lighting state of the traffic light. Also, the object detection device 11 may be configured to detect the road edge.

[0040] The object detection device 11 has, for example, a discriminator that detects objects and road features represented in an image by inputting a camera image. As the discriminator, for example, a deep neural network (DNN) pre-trained to detect objects and road features represented in the input image can be used. The object detection device 11 may use a discriminator other than the DNN.

[0041] Also, the object detection device 11 may detect an object around the vehicle 10 based on the reflected wave information. Further, the object detection device 11 may obtain the orientation of the object with respect to the vehicle 10 based on the position of the object in the camera image, and obtain the distance between the object and the vehicle 10 based on this orientation and the reflected wave information. The object detection device 11 estimates the position of the object, for example, represented in the vehicle coordinate system, based on the current position of the vehicle 10, the distance and orientation to the object with respect to the vehicle 10. Further, the object detection device 11 may track the object detected from the latest camera image by associating the object detected from the latest camera image with the object detected from the past image according to the tracking process based on the optical flow. The tracked object is assigned an object identification number. Then, the object detection device 11 may obtain the trajectory of the object being tracked based on the position of the object in the past image to the latest image. The object detection device 11 can estimate the speed of the object with respect to the vehicle 10 based on the change in the position of the object over time. Also, the object detection device 11 can estimate the acceleration of the object based on the change in the speed of the object over time. The object detection device 11 may obtain the position of the road feature in the same manner as described above. The position of the road feature is represented, for example, in the vehicle coordinate system.

[0042] The object detection device 11 notifies the automatic control device 12 and the like of object detection information including information representing an object and road feature information representing road features. The object detection information includes information indicating the type of the detected object, information indicating its position, speed, acceleration, and information indicating the driving lane. For a tracked object, the object detection information includes an object identification number. Further, the object detection device 11 outputs road information including the position of the lane dividing line representing the road to the determination device 13 via the in-vehicle network 14. The road information may further include the position of a traffic signal and the lighting state of the traffic signal. Also, when a pedestrian is detected, the object detection device 11 generates pedestrian information including the position of the pedestrian tracked within a recent predetermined time and outputs it to the determination device 13 via the in-vehicle network 14. Incidentally, the pedestrian information may include the speed, acceleration, and object identification number of the pedestrian.

[0043] The automatic control device 12 controls the operation of the vehicle 10. The automatic control device 12 has an automatic driving mode in which the vehicle 10 is driven in automatic driving and a manual driving mode in which the operation of the vehicle 10 is controlled based on the driver's operation. In the automatic driving mode, the automatic control device 12 mainly drives the vehicle 10. In the automatic driving mode, the automatic control device 12 controls operations such as steering, driving, and braking based on information detected by the front camera 2 and the millimeter-wave radar 3 mounted on the vehicle 10.

[0044] Also, in the manual driving mode, the driver mainly drives the vehicle 10. In the manual driving mode, the automatic control device 12 controls the operations of the vehicle 10 such as steering, driving, and braking based on the operation on the driver's operation unit 5. In the manual driving mode, the automatic control device 12 controls the operation of the vehicle 10 based on the operation of at least one of the steering wheel, brake pedal, or accelerator pedal (not shown) by the driver.

[0045] The automatic control device 12 outputs a control signal for controlling steering to a steering device (not shown) via the in-vehicle network 14. The automatic control device 12 outputs a drive signal for controlling driving to a drive device (not shown) via the in-vehicle network 14. The automatic control device 12 outputs a braking signal for controlling braking to a braking device (not shown) via the in-vehicle network 14. The steering signal for controlling steering, the drive signal for controlling driving, and the braking signal for controlling braking are an example of information representing the operation of the vehicle.

[0046] The determination device 13 executes a generation process, a determination process, a change process, and a decision process. For this purpose, the determination device 13 includes a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the determination device 13 to the in-vehicle network 14.

[0047] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores a computer program of an application used in information processing executed by the processor 23 and various types of data.

[0048] All or part of the functions of the determination device 13 are functional modules realized by, for example, a computer program operating on the processor 23. The processor 23 includes a generation unit 231, a determination unit 232, a change unit 233, and a decision unit 234. Alternatively, the functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The processor 23 includes one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logical arithmetic unit, a numerical arithmetic unit, or a graphic processing unit.

[0049] FIG. 3 is an example of an operation flowchart regarding the change process of the determination device 13 of the present embodiment. With reference to FIG. 3, the change process of the determination device 13 will be described below. The determination device 13 executes a determination process according to the operation flowchart shown in FIG. 3 at a change time having a predetermined period.

[0050] First, the change unit 233 changes the reference of the determination process to the initial setting (step S101). In the present embodiment, the determination unit 232 has an identifier that estimates the entry probability that a pedestrian will enter the road after a predetermined time based on the pedestrian information. The entry probability is represented by, for example, 0.0 to 1.0. The determination unit 232 determines that a pedestrian will enter the road when the entry probability exceeds a predetermined determination reference value. The change unit 233 changes this determination reference value to the initial value. The initial value of the determination reference value is, for example, 0.7. The lower the determination reference value, the easier it is to determine that a pedestrian will enter the road.

[0051] Next, the generation unit 231 generates state information that can be confirmed from the pedestrian and represents the state of the driver driving the vehicle 10 or the state of the vehicle 10 in a predetermined past period (step S102).

[0052] As state information representing the state of the driver who can be confirmed from the pedestrian and drives the vehicle 10, for example, the driver's line of sight is directed at the pedestrian, and the driver's line of sight is not directed at the pedestrian. Before entering the road, the pedestrian observes whether the driver of the approaching vehicle 10 is looking at him / her.

[0053] [[ID=|16]]Also, as state information that can be confirmed from the pedestrian and represents the state of the vehicle 10, for example, the vehicle 10 has reduced its speed, the vehicle 10 has increased its speed, and the vehicle 10 has been steered away from the position of the pedestrian. Before entering the road, the pedestrian observes the speed and the steering direction of the approaching vehicle 10. The process by which the generation unit 231 generates state information will be described later.

[0054] Next, the change unit 233 determines whether the state information satisfies the first change criterion (step S103). The first change criterion includes that the state information indicates that the vehicle 10 has decelerated, or the driver's line of sight is directed at the pedestrian, or the vehicle 10 has been steered away from the position of the pedestrian. When the state information satisfies any of the conditions of the first change criterion, the change unit 233 determines that the state information satisfies the first change criterion.

[0055] When the state information satisfies the first change criterion (step S103-Yes), the change unit 233 changes the criterion for the determination process so that it is more likely to be determined that the pedestrian is about to enter the road (step S104), and ends the series of processes. Specifically, the change unit 233 reduces the determination reference value from the initial value. For example, when the initial value of the determination reference value is 0.7, the change unit 233 changes the determination reference value to 0.5.

[0056] When the pedestrian 60 confirms that the vehicle 10 has decelerated, or the driver's line of sight is directed at the pedestrian 60, or the vehicle 10 has been steered away from the position of the pedestrian 60, it is considered that the pedestrian 60 determines that the vehicle 10 recognizes that the pedestrian 60 is about to cross the road 50. The pedestrian tries to cross the road 50 actively.

[0057] On the other hand, when the state information does not satisfy the first change criterion (step S103-No), the change unit 233 determines whether the state information satisfies the second change criterion (step S105). The second change criterion includes that the state information indicates that the vehicle 10 has accelerated, or the driver's line of sight is not directed at the pedestrian. When the state information satisfies any of the conditions of the second change criterion, the change unit 233 determines that the state information satisfies the second change criterion.

[0058] When the status information meets the second change criterion (step S105 - Yes), the change unit 233 changes the criterion of the determination process so that it is more likely to be determined that the pedestrian does not enter the road (step S106), and ends the series of processes. Specifically, the change unit 233 increases the determination reference value from the initial value. For example, when the initial value of the determination reference value is 0.7, the change unit 233 changes the determination reference value to 0.9.

[0059] When it is confirmed that the vehicle 10 has increased its speed or the driver's line of sight is not directed at the pedestrian 60, it is considered that the pedestrian 60 determines that the vehicle 10 does not recognize that the pedestrian 60 crosses the road 50. The pedestrian becomes passive about crossing the road.

[0060] On the other hand, when the status information does not meet the second change criterion (step S105 - No), the series of processes ends. The determination unit 232 executes the determination process using the determination reference value of the initial value.

[0061] FIG. 4 is an example of an operation flowchart regarding the generation process of the determination device 13 of the present embodiment. The generation process of the determination device 13 will be described below with reference to FIG. 4. The determination device 13 executes the generation process according to the operation flowchart shown in FIG. 4 in step S102 described above.

[0062] First, the generation unit 231 generates status information based on the information representing the driver of the vehicle 10 (step S201). The generation unit 231 estimates the line-of-sight direction of the driver based on the monitoring image. The generation unit 231 includes an identifier that has been learned to detect predetermined parts such as the outer corner of the eye, the inner corner of the eye, and the pupil center as face feature points by inputting the monitoring image. This identifier inputs the monitoring image and detects the types and positions of the face feature points included in the monitoring image.

[0063] The identifier has a deep neural network (DNN) having a plurality of layers connected in series from an input side to an output side. By inputting an image including facial feature points in advance as teacher data into the DNN and performing learning, the DNN operates as an identifier that detects the types and positions of facial feature points. Also, as the identifier, a machine learning model such as a support vector machine or a random forest may be used.

[0064] The generation unit 231 estimates the driver's line of sight direction using the facial feature points. The driver's line of sight direction is represented by an angle of a horizontal direction with respect to the traveling direction of the vehicle 10. The generation unit 231 may estimate the line of sight direction based on either the left or right eye of the driver. Also, the generation unit 231 may estimate the line of sight direction of each of the left and right eyes and use the averaged direction thereof as the line of sight direction.

[0065] The generation unit 231 determines whether the pedestrian is located on the left side or the right side with respect to the traveling direction of the vehicle 10 based on the current position of the vehicle 10 and the position of the pedestrian.

[0066] The generation unit 231 determines whether the driver's line of sight direction is directed toward the pedestrian's direction. When the pedestrian's position is on the left side with respect to the traveling direction of the vehicle 10 and the driver's line of sight direction is directed toward the left side with respect to the traveling direction of the vehicle 10, the generation unit 231 determines that the driver's line of sight direction is directed toward the pedestrian's direction. That the driver's line of sight direction is directed toward the left side means, for example, that when the traveling direction of the vehicle 10 is set to 0 degrees, the driver's line of sight direction is in the range from 0 degrees to 90 degrees to the left side.

[0067] Also, when the pedestrian's position is on the right side with respect to the traveling direction of the vehicle 10 and the driver's line of sight direction is directed toward the right side with respect to the traveling direction of the vehicle 10, the generation unit 231 determines that the driver's line of sight direction is directed toward the pedestrian's direction. That the driver's line of sight direction is directed toward the right side means, for example, that when the traveling direction of the vehicle 10 is set to 0 degrees, the driver's line of sight direction is in the range from 0 degrees to 90 degrees to the right side.

[0068] During a recent predetermined monitoring period, if the driver's line of sight is directed towards the pedestrian for a period equal to or longer than a predetermined reference time, the generation unit 231 determines that the driver's line of sight is directed towards the pedestrian. The generation unit 231 generates state information indicating that the driver's line of sight is directed towards the pedestrian. The monitoring period can be set to, for example, 5 seconds, and the reference time can be set to, for example, 3 seconds.

[0069] On the other hand, during a recent monitoring period, if the driver's line of sight is not directed towards the pedestrian for a period equal to or longer than the reference time, the generation unit 231 determines that the driver's line of sight is not directed towards the pedestrian. The generation unit 231 generates state information indicating that the driver's line of sight is not directed towards the pedestrian.

[0070] Next, the generation unit 231 generates state information based on information representing the operation of the vehicle 10 (step S202) and ends a series of processes. The generation unit 231 determines whether the speed of the vehicle 10 has decreased or increased during a recent predetermined monitoring period based on the speed information. If the speed of the vehicle 10 has decreased beyond the reference speed, the generation unit 231 determines that the speed of the vehicle 10 has decreased. The generation unit 231 generates state information indicating that the speed of the vehicle 10 has decreased. On the other hand, if the speed of the vehicle 10 has increased beyond the reference speed, the generation unit 231 determines that the speed of the vehicle 10 has increased. The generation unit 231 generates state information indicating that the speed of the vehicle 10 has increased. The monitoring period can be set to, for example, 5 seconds, and the reference speed can be set to, for example, 5 km / h.

[0071] The generation unit 231 determines whether the vehicle 10 has turned to the left, or the vehicle 10 has turned to the right, or the vehicle 10 has not turned, based on the steering signal. If the steering angle to the left exceeds a predetermined reference angle during a recent predetermined monitoring period, the generation unit 231 determines that the vehicle 10 has turned to the left. If the steering angle to the right exceeds the reference angle during the recent monitoring period, the generation unit 231 determines that the vehicle 10 has turned to the right. If no steering exceeding the reference angle has been performed during a recent predetermined monitoring period, the generation unit 231 determines that the vehicle 10 has not turned. The monitoring period can be, for example, 5 seconds, and the reference angle can be, for example, 5 degrees.

[0072] Further, the generation unit 231 determines whether the pedestrian is located on the left side or the right side with respect to the traveling direction of the vehicle 10 based on the current position of the vehicle 10 and the position of the pedestrian. When the position of the pedestrian is opposite to the direction in which the vehicle 10 has turned, the generation unit 231 determines that the vehicle 10 has steered away from the position of the pedestrian. The generation unit 231 generates state information indicating that the vehicle 10 has been steered away from the position of the pedestrian.

[0073] When the position of the pedestrian coincides with the direction in which the vehicle 10 has turned, and when the vehicle 10 has not turned, the generation unit 231 generates state information indicating that the vehicle 10 has not been steered away from the position of the pedestrian. The above is the description of the generation process.

[0074] FIG. 5 is an example of an operation flowchart regarding the determination process of the determination device 13 of the present embodiment. With reference to FIG. 5, the determination process of the determination device 13 will be described below. The determination device 13 executes a determination process according to the operation flowchart shown in FIG. 5 at a determination time having a predetermined period.

[0075] First, the determination unit 232 determines whether a pedestrian has been detected (step S301). If pedestrian information has been input between the previous determination time and the current determination time, the determination unit 232 determines that a pedestrian has been detected. On the other hand, if no pedestrian information has been input between the previous determination time and the current determination time, the generation unit 231 determines that no pedestrian has been detected.

[0076] When a pedestrian has been detected (step S301 - Yes), the determination unit 232 determines whether the pedestrian will enter the road (step S302). Specifically, the determination unit 232 determines whether the pedestrian will enter the road based on road information representing a predetermined range of the road in the traveling direction of the vehicle 10 and pedestrian information representing a pedestrian in a predetermined range in the traveling direction of the vehicle 10.

[0077] The determination unit 232 has an identifier that estimates the probability of entry that the pedestrian will enter the road after a predetermined time by inputting the road information and the pedestrian information. The predetermined time can be, for example, 5 seconds. The probability of entry is represented by a numerical value between 0.0 and 1.0. The road information includes at least the position of the lane dividing line representing the road. The road information may further include the position of the traffic signal and the lighting state of the traffic signal. The pedestrian information includes the position of the pedestrian tracked within a recent predetermined time. The identifier has a deep neural network (DNN) that is pre-trained to estimate the probability of entry that the pedestrian will enter the road after a predetermined time from the road information and the pedestrian information.

[0078] Note that a camera image including the pedestrian may be used as the pedestrian information. Further, the identifier may estimate the trajectory of the position of the pedestrian until a predetermined time based on the pedestrian information.

[0079] Furthermore, the pedestrian information may further include the age of the pedestrian and information indicating that the pedestrian is carrying luggage such as a suitcase. If the age of the pedestrian is high or the pedestrian is carrying luggage, the discriminator estimates a lower entry probability than in the case where this is not so. The discriminator may have a deep neural network (DNN) pre-trained to estimate the entry probability that a pedestrian will enter the road after a predetermined time using images representing pedestrians of various ages and images representing pedestrians with luggage from the pedestrian information.

[0080] When the entry probability exceeds the determination reference value, the determination unit 232 determines that the pedestrian will enter the road. On the other hand, when the entry probability does not exceed the determination reference value, the determination unit 232 determines that the pedestrian will not enter the road.

[0081] When it is determined that the pedestrian will enter the road (step S302 - Yes), the determination unit 234 determines the control of the vehicle 10 (step S303) and ends the series of processes.

[0082] If the distance between the current position of the vehicle 10 and the pedestrian is equal to or less than the first reference distance, the determination unit 234 determines to notify the driver of a warning using the UI7. For example, the determination unit 234 uses the display device 7a to display a warning to the driver. In response to the warning, the driver may manually operate the vehicle 10.

[0083] If the distance between the current position of the vehicle 10 and the pedestrian is equal to or less than a second reference distance shorter than the first reference distance, it is determined to stop the vehicle 10. The determination unit 234 notifies the automatic control device 12 of a stop request for stopping the vehicle 10. The automatic control device 12 stops the vehicle 10 in response to the stop request.

[0084] If the distance between the current position of the vehicle 10 and the pedestrian is longer than the first reference distance, the series of processes is ended. Also, when no pedestrian is detected (step S301 - No), the series of processes is ended.

[0085] As described above, the determination device according to the present embodiment can accurately determine whether a pedestrian enters a road based on the state of the pedestrian and the state of the driver or vehicle that can be confirmed by the pedestrian.

[0086] In the present disclosure, the determination device of the above-described embodiment can be appropriately changed as long as it does not deviate from the gist of the present disclosure. Further, the technical scope of the present disclosure is not limited to those embodiments, and extends to the invention described in the claims and equivalents thereof.

[0087] For example, in the above-described embodiment, the state information is shown as an example and is not limited thereto. Further, in the above-described embodiment, changing the determination reference value of the entry probability is an example of changing the reference of the determination process. Changing the reference of the determination process is not limited thereto.

[0088] The estimation process of the identifier of the determination unit in the above-described embodiment is an example, and the estimation process may be performed using other methods.

Description of Reference Numerals

[0089] 2 Front camera 3 Millimeter-wave radar 4 Surveillance camera 5 Operation unit 6 Vehicle speed sensor 7 User interface 7a Display device 10 Vehicle 11 Object detection device 12 Automatic control device 13 Determination device 21 Communication interface 22 Memory 23 Processor 231 Generation unit 232 Determination unit 233 Change unit 234 Decision unit 14 In-vehicle network

Claims

1. A determination unit that executes a determination process on whether a pedestrian enters a road based on road information representing a road in a predetermined range in the traveling direction of a vehicle and pedestrian information representing a pedestrian; A generation unit that generates state information that can be confirmed by a pedestrian and represents the state of a driver driving the vehicle or the vehicle in a predetermined period in the past based on information representing the driver of the vehicle or information representing the operation of the vehicle; A change unit that changes the criterion of the determination process of the determination unit so that it is more likely to be determined that a pedestrian enters the road or it is more likely to be determined that a pedestrian does not enter the road based on the state information generated by the generation unit; characterized by comprising: The determination unit executes the determination process on whether a pedestrian enters the road using the criterion of the determination process changed by the change unit. A determination device characterized by this.

2. The determination unit estimates an entry probability that a pedestrian enters the road after a predetermined time based on the pedestrian information; The determination unit determines that a pedestrian enters the road when the entry probability exceeds a predetermined reference value; The change unit changes the reference value based on the state information generated by the generation unit. The determination device according to claim 1.

3. The pedestrian information includes information representing the age of the pedestrian or information representing the luggage of the pedestrian. The determination device according to claim 2.

4. When the state information generated by the generation unit represents that the vehicle has reduced its speed, the driver's line of sight is directed at the pedestrian, or the vehicle has been steered away from the position of the pedestrian, the change unit changes the criterion of the determination process so that it is more likely to be determined that a pedestrian enters the road. The determination device according to any one of claims 1 to 3.

5. When the state information generated by the generation unit represents that the vehicle has increased its speed or the driver's line of sight is not directed at the pedestrian, the change unit changes the criterion of the determination process so that it is more likely to be determined that a pedestrian does not enter the road. The determination device according to any one of claims 1 to 3.

Images