Vehicle control device

Abstract

To provide a vehicle control device correctly estimating a future state of a detected moving object, and performing control of a vehicle with a timewise margin on the basis of the estimated future state of the moving object.SOLUTION: A vehicle control device includes: a detection part detecting a moving object on the basis of environmental information presenting an environment around a vehicle; a first estimation part estimating a state of the moving object at an estimated time when a first front time has passed, on the basis of the detected information presenting the moving object detected by the detection part; and a determination part determining the control of the vehicle on the basis of the state of the moving object estimated by the first estimation part. The first front time is the sum of: a representative estimated time the first estimation part requires for estimating the state of the moving object; a representative notification time required for notifying the state of the moving object estimated by the first estimation part to the determination part; a representative determination time required for determining the control of the vehicle by the determination part; and a second front time capable of performing the control determined by the determination part.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a vehicle control device. [Background technology]

[0002] A control device installed in the vehicle detects objects around the vehicle and controls the vehicle to maintain a safe distance between the detected objects and the vehicle.

[0003] When a control device detects a moving object, it estimates the future position of the moving object based on the past position, speed, etc. of the moving object (see, for example, Patent Document 1). If the future position of the moving object is ahead of the vehicle, the control device issues a warning to the driver or stops the vehicle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-128614 Summary of the Invention [Problem to be solved by the invention]

[0005] The process of estimating the future position of a moving object takes a relatively long time, and even during this process, if the vehicle and the moving object are close to each other, there may not be enough time to control the vehicle.

[0006] Therefore, it is preferable to set the detection time of an object to a time sufficiently earlier than the estimated time when the future position of the object is estimated. In this case, the question arises as to how far earlier the detection time should be set relative to the estimated time.

[0007] If the time between the detection time and the estimated time is short, there will be insufficient time to control the vehicle, which may result in a lack of safety for the vehicle. On the other hand, if the time between the detection time and the estimated time is long, the accuracy of estimating the future position of the moving object will decrease, which may also result in a lack of safety for the vehicle.

[0008] Therefore, the present disclosure aims to provide a vehicle control device that can correctly estimate the future state of a detected moving object and control the vehicle with ample time based on the estimated future state of the moving object. [Means for solving the problem]

[0009] (1) According to one embodiment, a vehicle control device is provided, which includes: a detection unit that detects a moving object based on environmental information representing an environment around the vehicle; a first estimation unit that estimates a state of the moving object at an estimation time when a first forward time has elapsed based on detection information representing the moving object detected by the detection unit; and a determination unit that determines control of the vehicle based on the state of the moving object estimated by the first estimation unit, wherein the first forward time is the sum of a typical estimation time required for the first estimation unit to estimate the state of the moving object, a typical notification time required for the state of the moving object estimated by the first estimation unit to be notified to the determination unit, a typical decision time required for the determination unit to decide on control of the vehicle, and a second forward time during which the control determined by the decision unit can be executed.

[0010] (2) In the vehicle control device of (1), it is preferable that the detection unit starts detecting a moving object at a detection time having a predetermined detection period, and at least a part of the process of estimating the state of the moving object in the first estimation unit is performed in parallel with the process of the detection unit detecting the moving object at the detection time in the next detection period, and that the vehicle control device has a judgment unit that judges whether the state of the moving object estimated by the first estimation unit based on detection information representing the moving object detected by the detection unit at the detection time in the previous detection period is correct, based on detection information representing the moving object detected by the detection unit at the detection time in the next detection period.

[0011] (3) In the vehicle control device of (2), it is preferable that the device has a second estimation unit that estimates the state of the moving object at the estimation time in a shorter time than the first estimation unit based on detection information representing the moving object detected by the detection unit at the detection time in the next detection cycle, and when the judgment unit determines that the estimated state of the moving object is correct, the determination unit determines control of the vehicle based on the state of the moving object estimated by the first estimation unit, and when the judgment unit determines that the estimated state of the moving object is incorrect, the determination unit determines control of the vehicle based on the state of the moving object estimated by the second estimation unit.

[0012] (4) In the vehicle control device of (3), it is preferable that the first estimation unit estimates the state of the moving object using a trained classifier, and the second estimation unit estimates the state of the moving object using linear prediction.

[0013] (5) In any of the vehicle control devices (1) to (4), it is preferable that the first forward time is obtained by subtracting a typical detection time required for the detection unit to detect a moving object from the sum of a typical estimation time, a typical notification time, and a typical decision time, which is a third forward time between the estimated time at which the state of the moving object is estimated by the first estimation unit based on detection information representing the moving object detected by the detection unit and the time at which the detection unit starts processing to detect the moving object, assuming that the detection unit starts processing to detect the moving object at the time at which the decision unit starts processing to decide control of the vehicle. [Effects of the Invention]

[0014] The vehicle control device according to the present disclosure can estimate the future state of a detected moving object and control the vehicle with ample time based on the estimated future state of the moving object, thereby enabling safe control of the vehicle. [Brief explanation of the drawings]

[0015] [Figure 1]FIG. 1A is a diagram illustrating an outline of the operation of the object detection device according to the first embodiment, and FIG. 1B is a diagram illustrating the second forward time. [Figure 2] 1 is a schematic configuration diagram of a vehicle in which an object detection device according to a first embodiment is implemented. [Figure 3] 4 is an example of an operational flowchart relating to a vehicle control process of the object detection device of the first embodiment. [Figure 4] 10A to 10C are diagrams illustrating the operation of the object detection device according to the second embodiment. [Figure 5] 10 is an example of an operational flowchart relating to a vehicle control process of an object detection device according to a second embodiment. [Figure 6] 10 is an example of an operational flowchart relating to a determination process of an object detection device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Fig. 1(A) is a diagram illustrating an outline of the operation of the object detection device 11 of the first embodiment, and Fig. 1(B) is a diagram illustrating the second forward time. Below, an outline of the operation related to the vehicle control processing of the object detection device 11 of the first embodiment disclosed in this specification will be described with reference to Fig. 1(A) and Fig. 1(B). The object detection device 11 is an example of a vehicle control device.

[0017] As shown in FIG. 1(A), a vehicle 10 is traveling on a road 50. The vehicle 10 has an object detection device 11 and an automatic control device 12. The object detection device 11 detects objects around the vehicle 10 based on environmental information representing the environment around the vehicle 10, such as a camera image. The automatic control device 12 controls the vehicle 10 based on the position of the detected object and information detected using a sensor mounted on the vehicle 10. The vehicle 10 may be an autonomous vehicle.

[0018] When an object detection device 11 detects a moving object such as a pedestrian, it estimates the future position of the moving object. The object detection device 11 has a first CPU 23a (Central Processing Unit) and a second CPU 23b. The first CPU 23a and the second CPU 23b execute information processing for the object detection device 11.

[0019] If a pedestrian is located in front of the vehicle 10, there is a risk of contact between the vehicle 10 and the pedestrian. Therefore, the estimation unit 232 estimates the future position of the pedestrian, and the determination unit 233 determines control of the vehicle 10 based on the position of the pedestrian.

[0020] First, at time t1, the first CPU 23a of the object detection device 11 starts detecting the pedestrian 60 based on environmental information that represents the environment around the vehicle 10. The time required for the detection process by the first CPU 23a is Td.

[0021] The first CPU 23a notifies the second CPU 23b of pedestrian information representing the pedestrian 60. The time required for the pedestrian information representing the pedestrian 60 to be notified from the first CPU 23a to the second CPU 23b is Tc. The pedestrian information is an example of detection information.

[0022] Next, at time t2, the second CPU 23b starts estimating the position of the pedestrian 60 at time t3, which is the first forward time Tf1 since time t2, based on the pedestrian information representing the pedestrian 60. The time required for the estimation process by the second CPU 23b is Tp.

[0023] Next, at time t4, the first CPU 23a starts determining control of the vehicle 10 based on the position of the pedestrian 60 at time t3. The time required for the first CPU 23a to perform the determination process is Tk. For example, if the distance between the pedestrian 60 and the vehicle 10 at time t3 is equal to or less than a first distance, the first CPU 23a determines to issue a warning to the driver of the vehicle 10. Furthermore, if the distance between the pedestrian 60 and the vehicle 10 at time t3 is equal to or less than a second distance that is shorter than the first distance, the first CPU 23a determines to stop the vehicle 10.

[0024] The object detection device 11 repeatedly performs the above-described detection process, estimation process, and detection process. The detection process and determination process are performed by the first CPU 23a, and the estimation process is performed by the second CPU 23b. The first CPU 23a starts detecting a moving object at a detection time having a predetermined detection period Tz.

[0025] At least a part of the process of estimating the position of the pedestrian 60 in the second CPU 23b is performed in parallel with the process of detecting the pedestrian by the first CPU 23a at the detection time t6 in the next detection cycle Tz.

[0026] Here, while the second CPU 23b is performing the estimation process, the first CPU 23a executes the determination process for the previous detection cycle Tz and the detection process for the next detection cycle Tz. By executing the estimation process, which takes a relatively long time, and the determination process and detection process in parallel, it is possible to ensure a long second forward time Tf2 between time t5, when it is decided to control the vehicle 10 toward the pedestrian 60 at time t3, and time t3.

[0027] This allows the automatic control device 12 to safely control the vehicle 10 based on the control of the vehicle 10 determined by the object detection device 11.

[0028] The first forward time Tf1 represents a future time at which the position of the pedestrian 60 is estimated in the estimation process. The first forward time Tf1 can be determined, for example, as follows. The first forward time Tf1 is obtained as the sum of a typical estimation time Tpr required for the second CPU 23b to estimate the position of the pedestrian 60, a typical notification time Tcr required for the position of the pedestrian 60 estimated by the second CPU 23b to be notified to the first CPU 23a, a typical determination time Tkr required for the first CPU 23a to determine control of the vehicle 10, and a second forward time Tf2 during which the automatic control device 12 can execute the control determined by the first CPU 23a. As a result, the estimation process estimates the position of the pedestrian 60 in the relatively near future, making it possible to estimate the position of the pedestrian 60 with high accuracy.

[0029] Here, the second forward time Tf2 is preferably determined to have a length that allows the automatic control device 12 to control the vehicle 10 safely.

[0030] Here, as shown in Figure 1 (B), assuming that the first CPU23a starts the process of detecting a pedestrian 60 at time t4, the second forward time Tf2 may be a value obtained by subtracting a typical detection time Tdr required for the detection process from the third forward time Tf3 between time t3 and time t4 at which the position of the pedestrian 60 is estimated by the second CPU23b based on detection information representing the pedestrian 60 detected by the first CPU23a.

[0031] In this case, Tf1 = Tpr + Tcr + Tkr + Tf2, and Tf3 = Tdr + Tf2, so Tf1 can be expressed as Tpr + Tcr + Tkr + Tkf3 - Tdr. That is, the third forward time Tf3 is the sum of the representative estimation time Tp, the representative notification time Tcr, the representative decision time Tkr, and the third forward time Tf3, minus the representative detection time Tdr.

[0032] 1(B), the detection process, estimation process, and decision process are executed by the same CPU. The time margin Tm between time t7, when it is decided to control the vehicle 10 to the pedestrian 60, and time t3 is short. Therefore, there is a risk that the automatic control device 12 will not be able to safely control the vehicle 10 within the time margin Tm.

[0033] In the object detection device 11, the estimation process, the decision process, and the detection process are performed in parallel in time, thereby ensuring a long second forward time Tf2 between time t5, when control of the vehicle 10 is decided, and time t3.

[0034] As described above, the object detection device 11 can correctly estimate the future state of a detected moving object and control the vehicle 10 with ample time based on the estimated future state of the moving object, thereby enabling safe control of the vehicle 10.

[0035] 2 is a schematic configuration diagram of a vehicle 10 in which an object detection device 11 according to this embodiment is implemented. The vehicle 10 includes a camera 2, a millimeter-wave radar 3, a positioning information receiver 4, a vehicle speed sensor 5, a user interface (UI) 6, the object detection device 11, an automatic control device 12, and the like.

[0036] The camera 2, millimeter wave radar 3, positioning information receiver 4, vehicle speed sensor 5, user interface (UI) 6, object detection device 11, and automatic control device 12 are communicatively connected via an in-vehicle network 13 that complies with a standard such as a controller area network.

[0037] Camera 2 is an example of an imaging unit provided on vehicle 10. Camera 2 is attached to vehicle 10 so as to face forward of vehicle 10. Camera 2 captures camera images that represent the environment of an area within a predetermined field of view ahead of vehicle 10, for example, at camera image capture times that are set at a predetermined cycle. The camera images may represent the road included in the predetermined area ahead of vehicle 10 and road features such as lane markings on the road surface. Camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or C-MOS. Camera 2 also has an imaging optical system that forms an image of the area to be captured on the two-dimensional detector. The camera image is an example of environmental information that represents the environment around the vehicle.

[0038] Every time the camera 2 captures a camera image, it outputs the camera image and the time the camera image was captured to the object detection device 11 or the like via the in-vehicle network 13. The camera image is used by the object detection device 11 in processing to detect objects and road features around the vehicle 10.

[0039] The millimeter-wave radar 3 is attached, for example, to the exterior of the vehicle 10, so as to face forward of the vehicle 10. The millimeter-wave radar 3 emits millimeter waves to scan a predetermined field of view ahead of the vehicle 10 at a reflected wave information acquisition time set at a predetermined cycle. The millimeter-wave radar 3 then receives the reflected waves reflected by a reflecting object. The time required for the reflected waves to return contains distance information between the vehicle 10 and another object located in the direction from which the millimeter waves were emitted. The millimeter-wave radar 3 outputs the reflected wave information, along with the time at which the millimeter waves were emitted and the reflected wave information was acquired, to the object detection device 11 or the like via the in-vehicle network 13. The reflected wave information includes the direction in which the millimeter waves were emitted and the time required for the reflected waves to return. The reflected wave information acquisition time indicates the time at which the millimeter waves were emitted. The reflected wave information is used by the object detection device 11 to detect objects around the vehicle 10. The reflected wave information is an example of environmental information that describes the environment around the vehicle.

[0040] The positioning information receiver 4 outputs positioning information indicating the current position of the vehicle 10. For example, the positioning information receiver 4 can be a GNSS receiver. Every time the positioning information receiver 4 acquires positioning information at a predetermined reception cycle, it outputs the positioning information and the time the positioning information was acquired to the object detection device 11, the automatic control device 12, etc. The positioning information acquisition time indicates the time when the positioning information was acquired.

[0041] The vehicle speed sensor 5 detects speed information indicating the speed of the vehicle 10. The vehicle speed sensor 5 has, for example, a measurement unit that measures the number of rotations of the tires of the vehicle 10. The vehicle speed sensor 5 outputs the speed information to the object detection device 11, the automatic control device 12, etc. via the in-vehicle network 13. The speed information is used in the object detection device 11 and the automatic control device 12 in processing to determine the speed of the vehicle 10.

[0042] The UI 6 is an example of a notification unit. The UI 6 is controlled by the object detection device 11, the automatic control device 12, etc., and notifies the driver of driving information, etc., of the vehicle 10. The driving information of the vehicle 10 includes the current location of the vehicle, notifications to the driver, etc. The UI 6 has a display device 6a, such as a liquid crystal display or a touch panel, for displaying the driving information, etc. The UI 6 may also have an audio output device (not shown) for notifying the driver of the driving information, etc. The UI 6 also generates an operation signal in response to an operation performed by the driver on the vehicle 10. The UI 6 has, for example, a touch panel or an operation button as an input device for inputting operation information from the driver to the vehicle 10. The UI 6 outputs the input operation information to the object detection device 11, the automatic control device 12, etc., via the in-vehicle network 13.

[0043] The object detection device 11 executes a detection process, an estimation process, and a determination process. To this end, the object detection device 11 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the object detection device 11 to the in-vehicle network 13.

[0044] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores computer programs of applications used in information processing executed by the processor 23 and various data.

[0045] All or part of the functions of the object detection device 11 are functional modules implemented by, for example, a computer program running on the processor 23. The processor 23 includes a detection unit 231, an estimation unit 232, and a determination unit 233. Alternatively, the functional modules included in the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The processor 23 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic operation unit, a numerical operation unit, or a graphics processing unit. The processor 23 includes a first CPU 23a and a second CPU 23b. The first CPU 23a executes the detection process of the detection unit 231 and the determination process of the determination unit 233. The second CPU 23b executes the estimation process of the estimation unit 232.

[0046] The detection unit 231 starts detection processing at a detection time having a predetermined detection period Tz. The detection unit 231 detects objects and their types around the vehicle 10 based on camera images. The objects include moving objects such as pedestrians and vehicles. The detection unit 231 also detects road features such as lane markings and traffic lights based on camera images. The detection unit 231 may detect the lighting status of the traffic lights.

[0047] The detection unit 231 has, for example, a classifier that receives a camera image and detects objects and road features shown in the image. As the classifier, for example, a deep neural network (DNN) that has been trained in advance to detect objects and road features shown in the input image can be used. The detection unit 231 may also use a classifier other than a DNN.

[0048] The classifier may also be trained to detect the posture of a pedestrian when the object is a pedestrian.

[0049] The detection unit 231 may also detect objects around the vehicle 10 based on the reflected wave information. The detection unit 231 may also determine the direction of the object relative to the vehicle 10 based on the position of the object in the camera image, and determine the distance between the object and the vehicle 10 based on this direction and the reflected wave information. The detection unit 231 estimates the position of the object, expressed, for example, in the vehicle coordinate system, based on the current position of the vehicle 10 and the distance and direction of the object relative to the vehicle 10. The detection unit 231 may also track the object detected in the latest camera image by associating the object detected in the latest image with the object detected in the previous image according to a tracking process based on optical flow. The tracked object is assigned an object identification number. The detection unit 231 may then determine the trajectory of the object being tracked based on the position of the object in the latest image from the previous image. The detection unit 231 can estimate the speed of the object relative to the vehicle 10 based on changes in the object's position over time. The detection unit 231 can also estimate the acceleration of the object based on changes in the object's speed over time. The detection unit 231 may determine the position of the road feature in the same manner as described above. The position of the road feature is expressed, for example, in the vehicle coordinate system.

[0050] The detection unit 231 notifies the automatic control device 12 and the like of object detection information including information representing an object and road feature information representing a road feature. The object detection information includes information indicating the type of detected object, information indicating its position, speed, acceleration, and information indicating the traveling lane. For tracked objects, the object detection information includes an object identification number. When multiple pedestrians are detected, the detection unit 231 notifies the automatic control device 12 of the object detection information including object identification information identifying each of the multiple pedestrians. The detection unit 231 also generates surrounding information including the positions of lane markings representing the road, the positions of traffic lights, and the lighting status of the traffic lights.

[0051] The automatic control device 12 controls the vehicle 10 based on positioning information, speed information, object detection information, road feature information, etc. The automatic control device 12 generates a driving plan that represents a planned driving trajectory of the vehicle 10 up to a predetermined time (for example, 5 seconds) ahead. The automatic control device 12 controls each part of the vehicle 10 based on the current position of the vehicle 10, the vehicle speed, the driving plan, etc. The automatic control device 12 generates a steering signal that controls a steering device (not shown), a drive signal that controls a drive device (not shown), and a braking signal that controls a braking device (not shown), and outputs these signals to each device.

[0052] When the vehicle 10 is operated by the driver, the automatic control device 12 may generate a steering signal, a drive signal, and a braking signal based on the driver's operation.

[0053] The object detection device 11 and the automatic control device 12 are, for example, electronic control units (ECUs). In Fig. 2, the object detection device 11 and the automatic control device 12 are illustrated as separate devices, but all or part of these devices may be configured as a single device.

[0054] Fig. 3 is an example of an operational flowchart relating to the vehicle control processing of the object detection device 11 of this embodiment. The vehicle control processing of the object detection device 11 will be described below with reference to Fig. 3. The object detection device 11 executes the vehicle control processing in accordance with the operational flowchart shown in Fig. 3 at vehicle control times having a predetermined cycle.

[0055] First, the detection unit 231 determines whether or not a pedestrian has been detected (step S101). A pedestrian is an example of a moving object. As described above, the detection process of the detection unit 231 is executed by the first CPU 23a of the processor 23 at a detection time having a detection period Tz.

[0056] If a pedestrian is detected (step S101-Yes), the detection unit 231 notifies the estimation unit 232 of pedestrian information (step S102). Specifically, the detection unit 231 notifies the estimation unit 232 of pedestrian information including the position, speed, acceleration, and object identification number of the pedestrian tracked within the most recent predetermined time. The pedestrian information includes a combination of the pedestrian's position, speed, and acceleration at each position within the most recent predetermined time. The pedestrian information is an example of detection information.

[0057] The estimation process of the estimation unit 232 is executed by the second CPU 23b. The notification of pedestrian information from the detection unit 231 to the estimation unit 232 is performed based on, for example, the operation clock of the first CPU 23a. The notification of pedestrian information from the detection unit 231 to the estimation unit 232 may be performed in one cycle of the operation clock of the first CPU 23a. Furthermore, the notification of pedestrian information from the first CPU 23a to the second CPU 23b may be performed at a timing delayed from one cycle. The notification of pedestrian information from the first CPU 23a to the second CPU 23b can be changed as appropriate to match the operation timing of the second CPU 23b that executes the estimation process. Furthermore, the detection unit 231 may notify the estimation unit 232 of surrounding information together with the pedestrian information.

[0058] Next, the estimation unit 232 estimates the state of the pedestrian at the time when the first forward time Tf1 has elapsed since the time when the pedestrian information was notified (hereinafter also referred to as the estimated time) (step S103). Specifically, the state of the pedestrian represents the position of the pedestrian, the reliability of the pedestrian's location, and an object identification number that identifies the pedestrian. The first forward time Tf1 can be set to, for example, 1 to 5 seconds.

[0059] If a pedestrian is located in front of the vehicle 10 at the estimated time, there is a risk of contact between the vehicle 10 and the pedestrian. Therefore, the estimation unit 232 estimates the future position of the pedestrian, and the determination unit 233 determines control of the vehicle 10 based on the position of the pedestrian.

[0060] The estimation unit 232 has a classifier that receives pedestrian information and detects the position of the pedestrian at the estimated time. The classifier outputs the position of the pedestrian at the estimated time, a confidence level that the pedestrian is at this position, and an object identification number that identifies the pedestrian. For example, the classifier may be a deep neural network (DNN) that has been trained in advance to detect the position of the pedestrian at the time when the first forward time Tf1 has elapsed from the input pedestrian information. Note that the classifier may also be trained to detect information other than the pedestrian's position. Furthermore, the classifier may also be trained to estimate the state of the pedestrian at the time when the first forward time Tf1 has elapsed, based on pedestrian information including the pedestrian's posture.

[0061] Furthermore, the estimation unit 232 may estimate the state of the pedestrian at the estimated time based on the pedestrian information and the surrounding information. The estimation unit 232 may have a classifier that detects the position of the pedestrian at the time when the first forward time Tf1 has elapsed by inputting the pedestrian information and the surrounding information. The classifier can more accurately estimate the state of the pedestrian by learning to estimate the state of the pedestrian at the estimated time based on the pedestrian information and the surrounding information.

[0062] Next, the estimation unit 232 notifies the determination unit 233 of the state of the pedestrian at the estimated time (step S104). Specifically, the estimation unit 232 notifies the determination unit 233 of the position of the pedestrian at the estimated time and the reliability that the pedestrian is at this position.

[0063] The determination process of the determination unit 233 is executed by the first CPU 23a of the processor 23. The second CPU 23b notifies the first CPU 23a of the state of the pedestrian at the estimated time. The notification of the pedestrian state from the estimation unit 232 to the determination unit 233 may be performed in one cycle of the operation clock of the second CPU 23b. Furthermore, the notification of the pedestrian information from the second CPU 23b to the first CPU 23a may be performed at a timing delayed from one cycle. The notification of the pedestrian information from the first CPU 23a to the second CPU 23a can be changed as appropriate to match the operation timing of the first CPU 23a that executes the estimation process. The time required for the estimation unit 232 to notify the determination unit 233 of the state of the pedestrian corresponds to the notification time Tc in FIG. 1(A).

[0064] Next, the decision unit 233 decides to control the vehicle 10 based on the state of the pedestrian, and ends the series of processes (step S105). Specifically, if the reliability of the pedestrian's position is equal to or higher than a predetermined reference reliability, the decision unit 233 decides to control the vehicle 10 based on the state of the pedestrian. If the reliability of the pedestrian's position is lower than the reference reliability, the series of processes ends. Reliability This is because the vehicle 10 cannot be safely controlled based on the low position of the pedestrian.

[0065] First, the determination unit 233 estimates the position of the vehicle 10 at the estimated time based on the current position of the vehicle 10 and the speed of the vehicle 10. The determination unit 233 obtains the position of the pedestrian at the estimated time and the distance between the position of the vehicle 10 and the position of the vehicle 10.

[0066] If the distance between the pedestrian's position and the vehicle 10 at the estimated time is equal to or shorter than the first reference distance, the decision unit 233 decides to notify a warning to the driver using the UI 6. The decision unit 233 displays the warning to the driver using, for example, the display device 6a. For example, in response to the warning, the driver may manually operate the vehicle 10.

[0067] Furthermore, if the distance between the pedestrian's position and the vehicle 10 at the estimated time is equal to or shorter than a second reference distance that is shorter than the first reference distance, the determination unit 233 determines to stop the vehicle 10. The determination unit 233 notifies the automatic control device 12 of a stop request requesting that the vehicle 10 be stopped. The automatic control device 12 stops the vehicle 10 in response to the stop request.

[0068] If the distance between the pedestrian's position and the vehicle 10 at the estimated time is longer than the first reference distance, the series of processes ends. If no pedestrian is detected (step S101-No), the series of processes ends.

[0069] Next, the first forward time Tf1 in the estimation process described above will be described below with reference to Fig. 1(A). The estimation unit 232 estimates the position of a pedestrian from the time when the pedestrian information, etc. is notified. Specifically, the estimation unit 232 estimates the position of a pedestrian at the time (estimated time) when the first forward time Tf1 has elapsed since the time when the pedestrian information is notified.

[0070] The first forward time Tf1 is obtained as the sum of a typical estimation time Tpr required for the second CPU23b to estimate the state of the pedestrian 60, a typical notification time Tcr required for the state of the pedestrian 60 estimated by the second CPU23b to be notified to the first CPU23a, a typical decision time Tkr required for the first CPU23a to decide on control of the vehicle 10, and a second forward time Tf2 during which the automatic control device 12 can execute the control decided by the first CPU23a.

[0071] The time required to estimate the state of a pedestrian may vary depending on the positional relationship between the pedestrian and the vehicle, the number of pedestrians, etc. The representative estimation time Tpr is an average value required for the first CPU23a to estimate the position of a pedestrian in a typical situation. Alternatively, the representative estimation time Tpr may be a maximum value required for the first CPU23a to estimate the position of a pedestrian in an expected situation.

[0072] The time required for the second CPU 23b to notify the first CPU 23a of information may vary depending on the operating conditions of the first CPU 23a and the second CPU 23b and the amount of information to be notified. The typical notification time Tcr is the average value required for the second CPU 23b to notify the first CPU 23a of typical information. Alternatively, the typical notification time Tcr may be the maximum value required for the second CPU 23b to notify the first CPU 23a of typical information under expected conditions.

[0073] The time required for the first CPU 23a to detect a pedestrian may vary depending on the camera image and the number of pedestrians depicted in the camera image. The representative detection time Tdr is the average value required for the first CPU 23a to detect a pedestrian in a typical situation. Alternatively, the representative detection time Tdr may be the maximum value required for the first CPU 23a to detect a pedestrian in an expected situation.

[0074] The time required for the first CPU 23a to decide on control of the vehicle 10 may vary depending on the state of the vehicle 10, the positional relationship between the vehicle 10 and pedestrians, and the number of pedestrians. The representative decision time Tkr is an average value required for the first CPU 23a to decide on control of the vehicle 10 in a typical situation. Alternatively, the representative detection time Tkr may be a maximum value required for the first CPU 23a to decide on control of the vehicle 10 in an expected situation.

[0075] The estimated time Tpr, the notified time Tcr, the determined time Tkr, and the second forward time Tf2 constitute the first forward time Tf1.

[0076] The estimation time Tpr, notification time Tcr, and decision time Tkr are times required for processing. The second forward time Tf2 can be determined by balancing the ability to execute safe control and the ability to accurately estimate the state of the pedestrian. As a result, the estimation process estimates the position of the pedestrian 60 in the relatively near future, making it possible to estimate the position of the pedestrian 60 with high accuracy.

[0077] The way in which the second forward time Tf2 is determined is not limited, and may be determined, for example, as follows: As shown in Fig. 1(B), it is assumed that the detection unit 231 starts processing to detect the pedestrian 60 at time t4, and the estimation unit 232 estimates the position of the pedestrian 60 at time t3 based on pedestrian information representing the pedestrian 60. The second forward time Tf2 may be a value obtained by subtracting a typical detection time Tdr required for the detection processing from a third forward time Tf3 between time t3 and time t4.

[0078] In the object detection device 11, the estimation process, the determination process, and the detection process are performed in parallel in time, thereby ensuring a long second forward time Tf2 between the time t5 at which it is determined to control the vehicle 10 and the time t3. On the other hand, in the example shown in FIG. 1(B), the time margin Tm is short.

[0079] As described above, the object detection device of this embodiment can correctly estimate the future state of a detected moving object and control the vehicle with ample time based on the estimated future state of the moving object, thereby enabling safe control of the vehicle.

[0080] Next, an object detection device according to a second embodiment will be described below with reference to Figures 4 to 6. For points not described in the second embodiment, the description of the first embodiment above applies as appropriate.

[0081] In this embodiment, the processor 23 has a determination unit 234 that determines the estimation process of the estimation unit 232. The determination unit 234 is a functional module realized by a computer program that runs on the first CPU 23a of the processor 23, for example.

[0082] 4 is a diagram illustrating the operation of object detection device 11 of this embodiment. At time t6, detection unit 231 executes detection processing based on environmental information. Based on pedestrian information representing a pedestrian detected by detection unit 231 at time t6, determination unit 234 determines whether the state of the pedestrian estimated by estimation unit 232 is correct.

[0083] Specifically, the determination unit 234 determines whether the state of the pedestrian estimated by the estimation unit 232 based on the pedestrian information representing the pedestrian detected by the detection unit 231 at detection time t6 in the next detection cycle Tz is correct, based on the pedestrian information representing the pedestrian detected by the detection unit 231 at detection time t1 in the immediately previous detection cycle Tz. The time required for the determination process by the determination unit 234 is Th. The time Th is sufficiently shorter than the time Tp required for the estimation process. Therefore, the automatic control device 12 is fully capable of controlling the vehicle 10 during the second forward time Tf2.

[0084] Alternatively, the second forward time Tf2 may be increased by the determination time Thr required for a typical determination process. In this case, the estimated time t3 is shifted forward by the determination time Thr. The first forward time Tf1 may be the sum of the estimated time Tpr, the notification time Tcr, the decision time Tkr, the determination time Thr, and the second forward time Tf2.

[0085] The time required for the first CPU 23a to make a determination may vary depending on the number of pedestrians shown in the camera image. The representative determination time Thr is the average value required for the first CPU 23a to make a determination in a typical situation. Alternatively, the representative determination time Thr may be the maximum value required for the first CPU 23a to make a determination in an expected situation.

[0086] If the state of the pedestrian estimated by the estimation unit 232 is correct, the determination unit 233 determines the control of the vehicle 10 based on the state of the pedestrian estimated by the estimation unit 232.

[0087] On the other hand, if the state of the pedestrian estimated by the estimation unit 232 is incorrect, the determination unit 233 estimates the state of the pedestrian at time t3 based on the pedestrian information detected by the detection unit 231 at time t6 in a shorter time than the estimation unit 232. The determination unit 233 executed by the first CPU 23a is an example of a second estimation unit. The determination unit 233 determines control of the vehicle 10 based on the estimated state of the pedestrian.

[0088] The detection unit 231 starts the detection process at detection time t6 having the detection cycle Tz. Time t6 is the detection time following time t1, when the previous detection process was performed.

[0089] Fig. 5 is an example of an operational flowchart relating to vehicle control processing of the object detection device 11 of this embodiment. In the operational flowchart shown in Fig. 5, steps S205 and S207 are added to the operational flowchart shown in Fig. 3. The processing of steps S201 to S204 and S206 is the same as steps S101 to S105 described above.

[0090] In step S204, the estimation unit 232 notifies the determination unit 233 and the judgment unit 234 of the state of the pedestrian at the estimated time.

[0091] Next, the determination unit 234 determines whether or not the state of the pedestrian estimated by the estimation unit 232 is correct, based on the pedestrian information representing the pedestrian detected by the detection unit 231 at time t6 (step S205). The determination process by the determination unit 234 will be described later with reference to FIG.

[0092] If the estimated state of the pedestrian is correct (step S205-Yes), the decision unit 233 decides on the control of the vehicle 10 based on the state of the pedestrian estimated by the estimation unit 232 (step S206), and ends the series of processes.

[0093] On the other hand, if the estimated state of the pedestrian is incorrect (step S205-No), the determination unit 233 estimates the state of the pedestrian at time t3, which is the first forward time Tf1 since time t2, in a shorter time than the estimation unit 232, based on pedestrian information representing the pedestrian detected by the detection unit 231 at time t6 (step S207).

[0094] For example, the determination unit 233 estimates the state of the pedestrian using linear prediction. Specifically, the determination unit 233 estimates the position of the pedestrian at time t3 based on the position, orientation, and speed of the pedestrian at time t6. The orientation of the pedestrian at time t6 is determined based on the position of the pedestrian within a predetermined most recent time period, for example. The time required for estimation processing using linear prediction is shorter than that required for estimation processing using a classifier.

[0095] Next, the decision unit 233 decides on the control of the vehicle 10 based on the state of the pedestrian estimated by the decision unit 233 (step S206), and ends the series of processes.

[0096] 6 is an example of an operational flowchart relating to the determination process of the object detection device 11 of this embodiment. In step S206 described above, the determination unit 234 executes the determination process based on the operational flowchart shown in FIG.

[0097] First, the determination unit 234 determines whether or not a pedestrian is located in front of the vehicle 10 based on the pedestrian information notified by the estimation unit 232 (step S301). If the position of the pedestrian is within a predetermined reference distance from a straight line extending from the current position of the vehicle 10 in the traveling direction, the determination unit 234 determines that the pedestrian is located in front of the vehicle 10. For example, the center of gravity can be used as the position of the vehicle 10.

[0098] On the other hand, if the position of the pedestrian is farther away than a predetermined reference distance from a straight line extending from the current position of the vehicle 10 in the traveling direction of the vehicle 10, the determination unit 234 determines that the pedestrian is not located in front of the vehicle 10. The reference distance can be set to, for example, 2 m to 5 m.

[0099] If a pedestrian is located in front of the vehicle 10 (step S301-Yes), the determination unit 234 determines whether or not the object identification number ID (hereinafter also referred to as estimated object identification number ID) included in the pedestrian information notified by the estimation unit 232 is included in the pedestrian information detected by the detection unit 231 at time t6 (hereinafter also referred to as current pedestrian information) (step S302). If a pedestrian is detected by the detection unit 231 at time t6, the pedestrian information includes an object identification number representing this pedestrian.

[0100] If the pedestrian represented by the estimated object identification number ID is included in the current pedestrian information (step S302-Yes), the determination unit 234 determines that the state of the pedestrian estimated by the estimation unit 232 is correct (step S303). Since the pedestrian included in the pedestrian information notified by the estimation unit 232 was detected by the detection unit 231 even at time t6, it is considered that the estimation by the estimation unit 232 is correct.

[0101] On the other hand, if the pedestrian represented by the estimated object identification number ID is not currently included in the pedestrian information (step S302-No), the determination unit 234 determines that the state of the pedestrian estimated by the estimation unit 232 is incorrect (step S304). Since the pedestrian included in the pedestrian information notified by the estimation unit 232 has not been detected by the detection unit 231 at time t6, the estimation by the estimation unit 232 is considered to be incorrect. The case where the pedestrian represented by the estimated object identification number ID is not currently included in the pedestrian information also includes the case where the pedestrian is not detected at time t6.

[0102] Since the estimation unit 232 estimates the future state of the pedestrian, the estimation result may be incorrect. Therefore, in this embodiment, the determination unit 234 determines the validity of the estimation result of the estimation unit 232. If the state of the pedestrian estimated by the estimation unit 232 is incorrect, the vehicle 10 is safely controlled by determining how to control the vehicle 10 based on the most recent detection result.

[0103] As described above, the object detection device of this embodiment determines the validity of the estimated state of a moving object, thereby enabling safer vehicle control. Furthermore, the object detection device of this embodiment achieves the same effects as the first embodiment.

[0104] In the present disclosure, the vehicle control device of the above-described embodiment can be appropriately modified without departing from the spirit of the present disclosure. Furthermore, the technical scope of the present disclosure is not limited to those embodiments, but extends to the inventions set forth in the claims and their equivalents.

[0105] For example, in the above-described embodiment, the estimation process is performed by a CPU different from the CPU that performs the detection process and the determination process. However, if the estimation process and the detection process and the determination process can be performed in parallel in a single CPU, the estimation process and the detection process and the determination process may be performed by the same CPU.

[0106] The estimation process in the above-described embodiment is merely an example, and the estimation process may be performed using other methods. Furthermore, in the above-described embodiment, the state of a pedestrian as a moving object is estimated, but the moving object may be an object other than a pedestrian. For example, the moving object may be a car or a bicycle. [Explanation of symbols]

[0107] 2 Cameras 3 Millimeter wave radar 4. Positioning information receiver 5 Vehicle speed sensor 6 User Interface 6a Display device 10 vehicles 11 Object detection device 21 Communication Interface 22 Memory 23 processors 231 Detector 232 Estimation Department 233 Decision Section 234 Judgment section 12 Automatic control devices 13 In-vehicle network

Claims

1. a detection unit that detects a moving object based on environmental information that represents an environment around the vehicle; a first estimation unit that estimates a state of the moving object at an estimation time when a first forward time has elapsed, based on detection information representing the moving object detected by the detection unit; a determination unit that determines control of the vehicle based on the state of the moving object estimated by the first estimation unit; and A vehicle control device characterized in that the first forward time is the sum of a typical estimation time required for the first estimation unit to estimate the state of a moving object, a typical notification time required for the state of the moving object estimated by the first estimation unit to be notified to the determination unit, a typical decision time required for the determination unit to decide on control of the vehicle, and a second forward time during which the control decided by the determination unit can be executed.

2. the detection unit starts detecting a moving object at a detection time having a predetermined detection cycle, and at least a part of the process of estimating a state of the moving object by the first estimator is performed in parallel with a process of detecting a moving object by the detection unit at the detection time in the next detection cycle; 2. The vehicle control device according to claim 1, further comprising a determination unit that determines whether the state of a moving object estimated by the first estimation unit based on detection information representing a moving object detected by the detection unit at the detection time in the previous detection cycle is correct, based on detection information representing a moving object detected by the detection unit at the detection time in the next detection cycle.

3. a second estimation unit that estimates a state of a moving object at the estimation time in a shorter time than the first estimation unit, based on detection information representing the moving object detected by the detection unit at the detection time in the next detection cycle; When the determination unit determines that the estimated state of the moving object is correct, the decision unit determines control of the vehicle based on the state of the moving object estimated by the first estimation unit, and 3. The vehicle control device according to claim 2, wherein, when the determination unit determines that the estimated state of the moving object is incorrect, the decision unit determines to control the vehicle based on the state of the moving object estimated by the second estimation unit.

4. The vehicle control device according to claim 3 , wherein the first estimation unit estimates the state of the moving object using a trained classifier, and the second estimation unit estimates the state of the moving object using linear prediction.

5. The first forward time is a third forward time between the estimation time at which the first estimation unit estimates a state of a moving object based on detection information representing the moving object detected by the detection unit and the time at which the detection unit starts processing to detect the moving object, assuming that the detection unit starts processing to determine control of the vehicle at the time at which the decision unit starts processing; The vehicle control device according to any one of claims 1 to 4, wherein the representative estimation time, the representative notification time, and the representative determination time are obtained by subtracting a representative detection time required for the detection unit to detect a moving object from the sum of the representative estimation time, the representative notification time, and the representative determination time.

Images