Control device, control method, and control program

Abstract

To achieve early and wide-spread use of an advanced driving assistance vehicle or an automatic driving vehicle.SOLUTION: A control device (2) mounted on a vehicle (V) comprises a processor (21) that executes a program instruction. The processor executes the following processing: on the basis of an image acquired by an image sensor (34) mounted on the vehicle, calculating a mode of a periodical change in a feature parameter of pixels constituting the image; and estimating the distance to a target on the basis of the calculated mode of the periodical change.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a control device, a control method, and a control program to be mounted on a vehicle. [Background technology]

[0002] In recent years, research and development of so-called driving assistance technology or automatic driving technology for vehicles such as automobiles has been actively carried out, and various proposals have been made regarding this (see, for example, Patent Document 1, etc.). Driving assistance technology is technology that reduces the burden on the driver and enables comfortable and safe driving, and includes, for example, following distance control, lane keeping assistance control, lane change assistance control, parking assistance control, obstacle warning, collision avoidance assistance control, etc. Automatic driving technology is technology that allows a vehicle to run automatically, that is, autonomously, without the need for a driver's driving operation. A vehicle equipped with such driving assistance technology or automatic driving technology is equipped with sensing devices such as cameras and radars to grasp the situation of the area ahead and the surrounding area. Then, vehicle control such as steering, driving, and braking is automatically performed based on the road conditions, etc. acquired by the traveling vehicle using sensing devices such as cameras and map information, etc. It is possible to improve safety and reliability by using a high-precision map including road data for each lane. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 202397 Summary of the Invention [Problem to be solved by the invention]

[0004] Research and development of these types of driving assistance technology and autonomous driving technology has been accelerating in recent years from various perspectives, such as improving the accuracy of detection or recognition by sensing devices and improving convenience. In other words, by further improving such performance, it is possible to promote the early and widespread dissemination of so-called advanced driving assistance vehicles and autonomous driving vehicles. The present disclosure has been made in consideration of the circumstances exemplified above. [Means for solving the problem]

[0005] The control device (2) mounted on a vehicle (V) according to claim 1 comprises: A processor (21) for executing program instructions; A memory (22) that is a non-transitory tangible recording medium for storing the program instructions; Equipped with The processor, Calculating a periodic change in a characteristic parameter of pixels constituting an image based on an image acquired by an image sensor (34) mounted on the vehicle; The distance to the target is estimated based on the calculated periodic change pattern. The control method according to claim 14 is executed by a processor (21) provided in a control device (2) mounted on a vehicle (V), Calculating a periodic change in a characteristic parameter of pixels constituting an image based on an image acquired by an image sensor (34) mounted on the vehicle; The distance to the target is estimated based on the calculated periodic change pattern. The control program according to claim 15 is a computer-readable program executed by a processor (21) provided in a control device (2) mounted on a vehicle (V), The process executed by the processor includes: A process of calculating, based on an image acquired by an image sensor (34) mounted on the vehicle, a periodic change pattern of a characteristic parameter of pixels constituting the image; A process of estimating a distance to the target based on the calculated periodic change pattern; Includes.

[0006] In addition, in each section of the application documents, each element may be given a reference number in parentheses. In this case, the reference number merely indicates an example of the correspondence between the element and the specific configuration described in the embodiment described later. Therefore, the present disclosure is not limited in any way by the description of the reference number. [Brief description of the drawings]

[0007] [Figure 1] 1 is a plan view showing a schematic configuration of a vehicle equipped with an in-vehicle system including a driving ECU as a control device according to an embodiment. [Diagram 2] FIG. 2 is a schematic block diagram showing the overall configuration of the in-vehicle system shown in FIG. [Diagram 3] FIG. 3 is a diagram showing an outline of target detection using the sonar sensor shown in FIG. 2. [Figure 4] FIG. 3 is a diagram showing an outline of target detection using the sonar sensor shown in FIG. 2. [Diagram 5] 3 is a diagram showing an outline of a target detection range of each of the plurality of sonar sensors shown in FIG. 2. [Figure 6] 3 is a diagram showing an outline of parking space detection using the sonar sensor shown in FIG. 2. [Figure 7] 3 is a diagram showing an outline of target detection using the radar sensor shown in FIG. 2. [Figure 8] 3 is a diagram showing an outline of target detection using the laser radar sensor shown in FIG. 2. [Figure 9] 9 is a diagram illustrating an outline of the configuration and operation of the laser radar sensor illustrated in FIG. 8. [Figure 10] FIG. 3 is a diagram showing an outline of target detection using the camera shown in FIG. 2. [Figure 11] FIG. 3 is a diagram showing an outline of target detection using the camera shown in FIG. 2. [Figure 12] FIG. 3 is a diagram showing an outline of driver state detection using the driver state monitor shown in FIG. 2. [Figure 13] FIG. 3 is a diagram showing an outline of driver state detection using the driver state monitor shown in FIG. 2. [Figure 14] FIG. 3 is a diagram showing a schematic configuration of the head-up display shown in FIG. 2. [Figure 15] 15 is a schematic diagram showing a road surface ahead of the vehicle and a part of the dashboard seen through the front windshield from the viewpoint of the driver shown in FIG. 14. [Figure 16] 3 is a block diagram showing an example of a schematic functional configuration realized by an operation ECU shown in FIG. 2. [Figure 17] 17 is a block diagram showing an example of a schematic functional configuration realized by a recognition unit shown in FIG. 16. [Figure 18] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 19] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 20] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 21] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 22] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 23] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 24] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Diagram 25] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 26] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 27] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 28] FIG. 4 is a schematic diagram showing an example of a control scene using a target recognition result in the embodiment. [Figure 29]FIG. 11 is a schematic diagram showing another example of a control scene using a target recognition result in the embodiment. [Diagram 30] 18 is a flowchart showing an example of a process for road marking recognition shown in FIG. 17. [Diagram 31] 1 is a schematic diagram showing a state in which a pedestrian crossing, which is one of the recognition objects in an embodiment, has deteriorated with age. [Diagram 32] FIG. 1 is a schematic diagram showing a state in which a pedestrian crossing is blocked by an obstacle; [Diagram 33] FIG. 11 is a schematic diagram showing an example of how a pedestrian crossing is seen from a vehicle when the pedestrian crossing is relatively far from the vehicle. [Diagram 34] 1 is a schematic diagram showing an example of how a pedestrian crossing is seen from a vehicle when the pedestrian crossing is relatively close to the vehicle; [Diagram 35] 1 is a graph showing an outline of a marking pattern shape at a pedestrian crossing. [Diagram 36] 34 is a graph showing how pixel feature parameters change along the imaginary horizontal line indicated by the dashed dotted line in FIG. 33. [Figure 37] 35 is a graph showing the state of change in pixel feature parameters along the imaginary horizontal line indicated by the dashed dotted line in FIG. 34. [Figure 38] 13 is a graph showing the relationship between the distance to a pedestrian crossing and the paint cycle. [Figure 39] 39 is a graph showing a method for obtaining the relationship shown in FIG. 38. [Diagram 40] 11 is a graph showing how pixel feature parameters change in the road width direction when there is blurring or occlusion on the pedestrian crossing. [Diagram 41] 41 is a graph showing the results of Fourier transform of the changes in pixel feature parameters shown in FIG. 40. [Figure 42A] 11 is a graph showing changes in pixel feature parameters in the road width direction of an actual pedestrian crossing when the pedestrian crossing is far from the vehicle. [Figure 42B] 42B is a graph showing the results of Fourier transform of the changes in the pixel feature parameters shown in FIG. 42A. [Figure 43A]13 is a graph showing changes in pixel feature parameters in the road width direction of an actual pedestrian crossing when the distance between the pedestrian crossing and the vehicle is medium. [Figure 43B] 43B is a graph showing the results of Fourier transform of the changes in the pixel feature parameters shown in FIG. 43A. [Figure 44A] 11 is a graph showing changes in pixel feature parameters in the road width direction of an actual pedestrian crossing when the pedestrian crossing is close to the vehicle. [Figure 44B] 44B is a graph showing the results of Fourier transform of the changes in the pixel feature parameters shown in FIG. 44A. [Diagram 45] 4 is a flowchart showing an outline of a distance estimation process in the present embodiment. [Figure 46] 11 is a schematic diagram showing another example of a distance estimation target in the present embodiment. FIG. [Figure 47] 11 is a schematic diagram showing still another example of a distance estimation target in the present embodiment. FIG. [Figure 48] 11 is a schematic diagram showing still another example of a distance estimation target in the present embodiment. FIG. [Figure 49] 11 is a schematic diagram showing still another example of a distance estimation target in the present embodiment. FIG. [Figure 50] 11 is a schematic diagram showing still another example of a distance estimation target in the present embodiment. FIG. [Figure 51] 11 is a schematic diagram showing still another example of a distance estimation target in the present embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] (Embodiment) Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings as appropriate. Note that the configuration, functions, and implementation examples of the embodiments can be modified as appropriate. Therefore, multiple embodiments and variations may be described in the specification. In this case, in the following description, the same reference symbols will be given to parts that are identical or equivalent between one embodiment and another embodiment, between one embodiment and its variation, or between one variation and another variation. And, the preceding description of such parts may be appropriately applied to other embodiments or variations that follow, unless there is a technical contradiction or a special additional description.

[0009] (Overall vehicle configuration) Referring to FIG. 1, the system-equipped vehicle V, which is a vehicle to which the present disclosure is applied, is a so-called four-wheeled vehicle, and includes a vehicle body V1 that is substantially rectangular in plan view. Hereinafter, a virtual straight line that passes through the center of the system-equipped vehicle V in the vehicle width direction and is parallel to the vehicle overall length direction of the system-equipped vehicle V is referred to as a vehicle center line LC. In FIG. 1, the vehicle width direction is the left-right direction in the figure. The vehicle overall length direction is a direction perpendicular to the vehicle width direction and perpendicular to the vehicle height direction. The vehicle height direction is a direction that defines the vehicle height of the system-equipped vehicle V, and is a direction parallel to the direction of gravity when the system-equipped vehicle V is stably placed on a horizontal plane in a drivable state. In addition, for convenience of explanation, the "front", "rear", "left", and "right" of the system-equipped vehicle V are defined as shown by arrows in FIG. 1. That is, the vehicle overall length direction is synonymous with the front-rear direction. In addition, the vehicle width direction is synonymous with the left-right direction. The vehicle height direction may not be parallel to the direction of gravity depending on the placement conditions or running conditions of the system-equipped vehicle V. However, in most cases, the vehicle height direction is aligned with the direction of gravity.

[0010] Inside the vehicle body V1, a vehicle compartment V2 is formed, which constitutes a riding space for occupants including the driver D. Wheels V3 are provided at each of the four corners of the vehicle body V1. That is, a left front wheel V3a is provided at the left front part of the vehicle body V1. A right front wheel V3b is provided at the right front part of the vehicle body V1. A left rear wheel V3c is provided at the left rear part of the vehicle body V1. A right rear wheel V3d is provided at the right rear part of the vehicle body V1. The system-equipped vehicle V is not limited to a four-wheeled vehicle, but may be a three-wheeled vehicle, or a six- or eight-wheeled vehicle such as a cargo truck. There is no particular limitation on the position or number of drive wheels.

[0011] A front bumper V12 is attached to a front portion V11, which is the front end of the vehicle body V1. A rear bumper V14 is attached to a rear portion V13, which is the rear end of the vehicle body V1. The side and upper portions of the vehicle body V1 are covered by a vehicle body panel V15. The vehicle body panel V15 includes a door panel V16. In the specific example shown in FIG. 1, a total of four door panels V16 are provided, two on each side. A door mirror V17 is attached to each of the pair of left and right door panels V16 on the front side. In addition, the front side of the vehicle interior V2 is covered by a front windshield V18. The front windshield V18 is formed in a plate shape from translucent glass or synthetic resin. The front windshield V18 is inclined so that the upper end is located further rearward than the lower end when viewed from the side in a line of sight parallel to the vehicle width direction.

[0012] A dashboard V21 is provided at the front end of the vehicle interior V2. A plurality of seats V22 are arranged behind the dashboard V21. Of the plurality of seats V22, a handle V24 is provided in front of a driver's seat V23 in which the driver D sits, for the driver D to grasp and operate when steering. The handle V24 is typically a so-called steering wheel formed in a substantially circular, elliptical, or polygonal ring shape, but may be rod-shaped or control stick-shaped.

[0013] (Overview of the in-vehicle system) The system-equipped vehicle V is equipped with an in-vehicle system 1. The in-vehicle system 1 is configured to function as a driving automation system in the system-equipped vehicle V by being installed in the system-equipped vehicle V. The system-equipped vehicle V may be referred to as the "own vehicle" hereinafter. The "driving automation system" is a system for realizing a driving automation level corresponding to at least one of levels 1 to 5 specified in the standard "SAE J3016" published by SAE International. SAE stands for Society Of Automotive Engineers. Level X in "SAE J3016" is hereinafter simply referred to as "SAE level X". X is any one of 0 to 5. The larger the value of X in SAE level X, or the more dynamic driving tasks the driving automation system is in charge of, i.e., executes, the higher the driving automation level is expressed as being. In addition, a change in the driving automation level to a higher level is referred to as an "increase" in the driving automation level. In contrast, the smaller the value of X, or the fewer dynamic driving tasks that the driving automation system is responsible for, that is, executes, the lower the level of driving automation is expressed. Also, a change to a lower level of driving automation is referred to as a "degradation" of the level of driving automation.

[0014] (Definition of Driving Assistance and Autonomous Driving) The contents of SAE levels 0 to 5 are specifically described below. In the following description, the driver D is a passenger who is in charge of or performs the dynamic driving task. The "dynamic driving task" refers to all operational and tactical functions that need to be performed in real time when operating a vehicle in road traffic, excluding strategic functions. Driving behavior in general can be classified into three types of functions: strategic, tactical, and operational. The "strategic" functions include itinerary planning, route selection, etc., and specifically include deciding or selecting itinerary plans such as "whether to go or not, when, where, and how to go." The "tactical" functions relate to vehicle operation in traffic situations, such as deciding whether and when to overtake or change lanes during the trip, selecting an appropriate speed, and checking mirrors. The "operational" functions relate to instantaneous reactions, such as making minor corrections to steering, braking, accelerator, and acceleration operations to maintain the position of the road lane or to avoid sudden obstacles or dangerous events in the vehicle's path. "OEDR" is an abbreviation for Object and Event Detection and Response, and is also referred to as "detection and response of objects and events." OEDR includes monitoring of the driving environment. Monitoring of the driving environment includes detection, recognition, and classification of objects and events. Monitoring of the driving environment also includes preparation to respond to objects and events as necessary. A "limited domain" is a specific condition in which a driving automation system or its functions are designed to operate, and is also referred to as an operational design domain or ODD. ODD stands for Operational Design Domain. A limited domain includes at least one of multiple constraints, such as geographical, environmental, speed, and time.

[0015] Level 0: Manual driving…Driver D performs all dynamic driving tasks. Level 1: Driving assistance: The driving automation system continuously performs either the longitudinal vehicle motion control subtask or the lateral vehicle motion control subtask of the dynamic driving task in a specific limited area. The longitudinal vehicle motion control subtask is starting, accelerating / decelerating, and stopping. The lateral vehicle motion control subtask is steering. However, the driving automation system does not perform both the longitudinal vehicle motion control subtask and the lateral vehicle motion control subtask simultaneously. Level 2: Advanced driving assistance: The driving automation system continuously performs the longitudinal vehicle motion control subtask and the lateral vehicle motion control subtask of the dynamic driving task in a specific limited area. The driver D is expected to supervise the driving automation system by performing the OEDR, which is a subtask of the dynamic driving task. Level 3: Conditional Autonomous Driving: The driving automation system continuously performs all dynamic driving tasks in specific limited areas. In principle, driver D is not obligated to perform OEDR such as periphery monitoring. Periphery monitoring is the monitoring of the traffic environment around the vehicle. However, if it becomes difficult to continue this driving automation level, the driving automation system will request driver D to take over driving with ample time to spare. Driver D must respond to the request appropriately. Level 4: Highly automated driving: The driving automation system continuously performs all dynamic driving tasks in specific limited areas. If it becomes difficult to maintain the driving automation level in a limited area, the driving automation system will take action. Level 5: Fully automated driving: The driving automation system performs all dynamic driving tasks continuously, without being limited to specific restricted areas. If it becomes difficult to continue with this level of driving automation, the driving automation system will also respond without being limited to specific restricted areas.

[0016] The in-vehicle system 1 is configured to be capable of executing various driving controls during driving of the vehicle and various associated notification operations. Specifically, the in-vehicle system 1 has a configuration as a driving automation system for realizing or implementing driving assistance or automatic driving in the vehicle. "Autonomous driving" refers to a driving automation level corresponding to SAE levels 3 to 5, in which the driving automation system is responsible for, i.e., executes, all dynamic driving tasks. In contrast, "driving assistance" refers to a driving automation level corresponding to SAE levels 1 to 2, in which the driving automation system is responsible for, i.e., executes, some dynamic driving tasks. Hereinafter, the expression "driving assistance" includes "advanced driving assistance" of SAE level 2, unless it is clearly stated as "SAE level 1 driving assistance" or clearly distinguished from "advanced driving assistance" such as "driving assistance or advanced driving assistance".

[0017] In this embodiment, the in-vehicle system 1 is configured to be capable of performing at least SAE level 3 or higher autonomous driving, SAE level 2 advanced driving assistance, and SAE level 1 driving assistance. Therefore, the in-vehicle system 1 according to this embodiment can be called an "autonomous driving system" during autonomous driving, while it can be called a "driving assistance system" during driving assistance or advanced driving assistance. The advanced driving assistance that can be performed by the in-vehicle system 1 according to this embodiment includes so-called "hands-off driving". In hands-off driving, the in-vehicle system 1 automatically executes control of starting, steering, acceleration / deceleration, lane change, and stopping, on the condition that the driver D responds appropriately to an intervention request from the in-vehicle system 1. In hands-off driving, the driver D is not required to be in a hands-on state, but is required to monitor the road conditions, traffic conditions, and obstacle presence conditions around the vehicle. The "hands-on state" is a state in which the driver D can interfere with the steering of the vehicle, i.e., the lateral vehicle motion control subtask. The hands-on state is typically a state in which the driver D is seated in the driver's seat V23 in a position in which he or she can drive the vehicle and can operate the steering wheel V24 with his or her hands. In the hands-on state, the driver D usually holds the steering wheel V24 with his or her hands, but even if the driver D's hands are touching the steering wheel V24 in a state in which the driver D can immediately hold it, this state can be considered to be a "hands-on state." For example, a state in which the driver D is performing a steering operation, that is, actively operating the steering wheel V24, is considered to be a hands-on state. In addition, a state in which the driver D holds the steering wheel V24 against the steering control by the in-vehicle system 1 is also considered to be a hands-on state.

[0018] (In-vehicle system configuration) The overall configuration of the in-vehicle system 1 will be described below with reference to FIG. 1 and FIG. 2. The in-vehicle system 1 includes a driving ECU 2, a driving information input unit 3, an in-vehicle communication device 4, an HD map 5, a navigation device 6, an HMI device 7, lighting devices 8, and a motion control device 9. ECU stands for Electronic Control Unit. HD stands for High Definition. HMI stands for Human Machine Interface. The in-vehicle system 1 is configured by connecting the driving ECU 2, the driving information input unit 3, the in-vehicle communication device 4, the HD map 5, the navigation device 6, the HMI device 7, lighting devices 8, and the motion control device 9 to each other via an in-vehicle communication line 10. The in-vehicle communication line 10 is configured to comply with a predetermined communication standard such as CAN (international registered trademark: international registration number 1048262A). CAN (international registered trademark) stands for Controller Area Network. The in-vehicle communication line 10 may have a sub-network conforming to LIN, FlexRay, etc., in addition to a main network conforming to CAN (international registered trademark). LIN is an abbreviation for Local Interconnect Network.

[0019] (Drive ECU) The driving ECU 2 as a control device according to the present disclosure is configured to control the overall operation of the in-vehicle system 1 by being mounted in the system-mounted vehicle V. That is, the driving ECU 2 has a configuration as a so-called AD / ADAS ECU, which is a controller for driving assistance or automatic driving. AD stands for automatic driving. ADAS stands for Advanced Driver-Assistance Systems. Specifically, the driving ECU 2 has a configuration as an in-vehicle computer including a processor 21 and a memory 22 connected to the processor 21. More specifically, the driving ECU 2 has at least one processor 21 and at least one memory 22.

[0020] The processor 21 has a configuration as a CPU or MPU. CPU stands for Central Processing Unit. MPU stands for Micro Processing Unit. At least one memory 22 includes at least a ROM and a RAM among various non-transient substantial recording media such as ROM, RAM, non-volatile rewritable memory, etc. ROM stands for Read Only Memory. RAM stands for Random Access Memory. A "recording medium" may also be called a "storage medium." A non-volatile rewritable memory is a storage device that can rewrite information while the power is on, but retains information in an unrewritable manner while the power is off, such as a flash ROM or an EEPROM. EEPROM stands for Electronically Erasable and Programmable ROM. Data and program instructions used by the processor 21 are stored in advance in the ROM or non-volatile rewritable memory included in the at least one memory 22. The driving ECU 2 is configured to execute various processes or operations such as vehicle control operations and information notification to occupants by having the processor 21 read and execute program instructions stored in the memory 22.

[0021] (Driving information input section) The driving information input unit 3 is provided to input information required for various processing operations in the driving ECU 2 to the driving ECU 2. Specifically, the driving information input unit 3 includes a sonar sensor 31, a radar sensor 32, a laser radar sensor 33, a camera 34, an operation sensor 35, a behavior sensor 36, a driver state monitor 37, an operation switch 38, and a locator 39. The sonar sensor 31, the radar sensor 32, the laser radar sensor 33, and the camera 34 may be collectively referred to as a "periphery monitoring sensor" or an "ADAS sensor." Each element provided as the driving information input unit 3 will be described below in order.

[0022] (Sonar sensor) The sonar sensor 31 is an ultrasonic distance measuring sensor and is attached to the vehicle body V1. As shown in FIG. 3, the sonar sensor 31 is a so-called ultrasonic sensor, and is configured to detect the target B by irradiating sonar search waves Wsp in an ultrasonic band toward the space outside the vehicle and receiving sonar reflected waves Wsr that are reflected by the target B. Specifically, the sonar sensor 31 is configured to calculate the distance to the target B based on the TOF and the speed of sound. TOF is an abbreviation for Time of Flight. That is, TOF is the time from the transmission time of the sonar search wave Wsp to the reception time of the sonar reflected wave Wsr in the propagation path Ls of the sonar search wave Wsp, which is an ultrasonic wave, and the sonar reflected wave Wsr, and may also be referred to as the propagation time.

[0023] As shown in FIG. 4, when there are a pair of sonar sensors 31, a first sonar sensor 311 and a second sonar sensor 312, which are arranged at different positions in the horizontal direction, it is possible to calculate the relative position of the target B with respect to the vehicle by so-called triangulation. Here, in FIG. 4, the X-axis is set along a straight line connecting the first sonar sensor 311 and the second sonar sensor 312, and the Y-axis is set in a direction perpendicular to the X-axis. Both the X-axis and the Y-axis are assumed to be horizontal, that is, perpendicular to the vehicle height direction. In addition, the first sonar sensor 311 transmits a sonar search wave Wsp, and the reflected wave by the target B is received by the first sonar sensor 311 and the second sonar sensor 312. In this case, the position of the target B in the XY two-dimensional coordinate system is calculated based on the TOF in the first propagation path Ls1 and the TOF in the second propagation path Ls2. The first propagation path Ls1 is a propagation path of an ultrasonic wave that returns to the first sonar sensor 311 via the target B from the first sonar sensor 311. The ultrasonic waves propagating through the first propagation path Ls1 are referred to as "direct waves." The second propagation path Ls2 is a propagation path of ultrasonic waves that travels from the first sonar sensor 311 through the target B to the second sonar sensor 312. The ultrasonic waves propagating through the second propagation path Ls2 are referred to as "indirect waves."

[0024] As shown in FIG. 1, a vehicle body V1 is equipped with a plurality of sonar sensors 31. Specifically, a first front sonar SF1, a second front sonar SF2, a third front sonar SF3, and a fourth front sonar SF4 are mounted on a front bumper V12 as the sonar sensors 31. Similarly, a first rear sonar SR1, a second rear sonar SR2, a third rear sonar SR3, and a fourth rear sonar SR4 are mounted on a rear bumper V14 as the sonar sensors 31. In addition, a first side sonar SS1, a second side sonar SS2, a third side sonar SS3, and a fourth side sonar SS4 are mounted on a side portion of the vehicle body V1 as the sonar sensors 31. When it is not specified that the sonar sensor is one of the first front sonar SF1 to the fourth side sonar SS4, the singular expression "sonar sensor 31" or the expression "plurality of sonar sensors 31" may be used hereinafter. FIG. 5 shows the detection range of each of the plurality of sonar sensors 31. Each of the multiple sonar sensors 31 will be described in turn below with reference to FIGS.

[0025] The first front sonar SF1 is mounted on the front bumper V12 at a position close to the left end in the vehicle width direction so as to transmit a sonar search wave Wsp to the left front of the vehicle. The second front sonar SF2 is mounted on the front bumper V12 at a position close to the right end in the vehicle width direction so as to transmit a sonar search wave Wsp to the right front of the vehicle. The first front sonar SF1 and the second front sonar SF2 are disposed symmetrically with respect to the vehicle center line LC. The front corner sonar detection range Rsc, which is the detection range of the first front sonar SF1 and the second front sonar SF2, is set to have a detection distance of, for example, about 60 cm. The "detection distance" is the maximum distance that can be measured from the sonar sensor 31.

[0026] The third front sonar SF3 and the fourth front sonar SF4 are arranged in the vehicle width direction at a position closer to the center of the front bumper V12 in the vehicle width direction. The third front sonar SF3 is arranged between the first front sonar SF1 and the vehicle center line LC in the vehicle width direction so as to transmit a sonar search wave Wsp approximately in front of the vehicle. The fourth front sonar SF4 is arranged between the second front sonar SF2 and the vehicle center line LC in the vehicle width direction so as to transmit a sonar search wave Wsp approximately in front of the vehicle. The third front sonar SF3 and the fourth front sonar SF4 are arranged symmetrically with respect to the vehicle center line LC. The front sonar detection range Rsf, which is the detection range of the third front sonar SF3 and the fourth front sonar SF4, is set to a detection distance of, for example, about 1 m.

[0027] The first front sonar SF1 and the third front sonar SF3 mounted on the left side of the vehicle body V1 are arranged at different positions in the horizontal direction. The first front sonar SF1 and the third front sonar SF3, which are adjacent to each other in the vehicle width direction, are arranged in a positional relationship such that the reflected wave of the sonar search wave Wsp emitted by one of them by the target B can be received as a received wave by the other. That is, the first front sonar SF1 is arranged so as to be able to receive both the direct wave corresponding to the sonar search wave Wsp emitted by itself and the indirect wave corresponding to the sonar search wave Wsp emitted by the third front sonar SF3. Similarly, the third front sonar SF3 is arranged so as to be able to receive both the direct wave corresponding to the sonar search wave Wsp emitted by itself and the indirect wave corresponding to the sonar search wave Wsp emitted by the first front sonar SF1.

[0028] Similarly, the third front sonar SF3 and the fourth front sonar SF4, which are mounted toward the center of the vehicle body V1 in the vehicle width direction, are disposed at different positions from each other in the horizontal direction. Moreover, the third front sonar SF3 and the fourth front sonar SF4, which are adjacent to each other in the vehicle width direction, are disposed in a positional relationship such that the one can receive the sonar search wave Wsp emitted by the other sonar and reflected by the target B as a received wave.

[0029] Similarly, the second front sonar SF2 and the fourth front sonar SF4 mounted on the right side of the vehicle body V1 are positioned at different positions from each other in the horizontal direction. The second front sonar SF2 and the fourth front sonar SF4, which are adjacent to each other in the vehicle width direction, are positioned in such a way that the reflected wave of the sonar search wave Wsp emitted by one of them by the target B can be received as a received wave by the other sonar.

[0030] The first rear sonar SR1 is mounted on the rear bumper V14 at a position close to the left end in the vehicle width direction so as to transmit a sonar search wave Wsp to the left rear of the vehicle. The second rear sonar SR2 is mounted on the rear bumper V14 at a position close to the right end in the vehicle width direction so as to transmit a sonar search wave Wsp to the right rear of the vehicle. The first rear sonar SR1 and the second rear sonar SR2 are disposed symmetrically across the vehicle center line LC. The rear corner sonar detection range Rsd, which is the detection range of the first rear sonar SR1 and the second rear sonar SR2, is set to a detection distance of, for example, about 60 cm.

[0031] The third rear sonar SR3 and the fourth rear sonar SR4 are arranged in the vehicle width direction at a position closer to the center of the rear bumper V14 in the vehicle width direction. The third rear sonar SR3 is arranged between the first rear sonar SR1 and the vehicle center line LC in the vehicle width direction so as to transmit a sonar search wave Wsp approximately behind the vehicle. The fourth rear sonar SR4 is arranged between the second rear sonar SR2 and the vehicle center line LC in the vehicle width direction so as to transmit a sonar search wave Wsp approximately behind the vehicle. The third rear sonar SR3 and the fourth rear sonar SR4 are arranged symmetrically with respect to the vehicle center line LC. The rear sonar detection range Rsr, which is the detection range of the third rear sonar SR3 and the fourth rear sonar SR4, is set to a detection distance of, for example, about 1.5 m.

[0032] The first rear sonar SR1 and the third rear sonar SR3 mounted on the left side of the vehicle body V1 are arranged at different positions in the horizontal direction. In addition, the first rear sonar SR1 and the third rear sonar SR3, which are adjacent to each other in the vehicle width direction, are arranged in a positional relationship such that the reflected wave of the sonar search wave Wsp emitted by one of them by the target B can be received as a received wave by the other. That is, the first rear sonar SR1 is arranged so as to be able to receive both the direct wave corresponding to the sonar search wave Wsp emitted by itself and the indirect wave corresponding to the sonar search wave Wsp emitted by the third rear sonar SR3. Similarly, the third rear sonar SR3 is arranged so as to be able to receive both the direct wave corresponding to the sonar search wave Wsp emitted by itself and the indirect wave corresponding to the sonar search wave Wsp emitted by the first rear sonar SR1.

[0033] Similarly, the third rear sonar SR3 and the fourth rear sonar SR4, which are mounted toward the center of the vehicle body V1 in the vehicle width direction, are disposed at different positions from each other in the horizontal direction. In addition, the third rear sonar SR3 and the fourth rear sonar SR4, which are adjacent to each other in the vehicle width direction, are disposed in a positional relationship such that the reflected wave of the sonar search wave Wsp emitted by one of them by the target B can be received as a received wave by the other.

[0034] Similarly, the second rear sonar SR2 and the fourth rear sonar SR4 mounted on the right side of the vehicle body V1 are disposed at different positions from each other in the horizontal direction. The second rear sonar SR2 and the fourth rear sonar SR4, which are adjacent to each other in the vehicle width direction, are disposed in a positional relationship such that the reflected wave of the sonar search wave Wsp emitted by one of them by the target B can be received as a received wave by the other.

[0035] The first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4 are provided so as to transmit sonar search waves Wsp from the side of the vehicle body V1 to the side of the vehicle. The first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4 are provided so as to be able to receive only direct waves. The side sonar detection range Rss, which is the detection range of the first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4, is set to be, for example, a detection distance of about 2 to 3 m.

[0036] The first side sonar SS1 is disposed between the left door mirror V17 and the first front sonar SF1 in the longitudinal direction so as to transmit sonar search waves Wsp to the left of the vehicle. The second side sonar SS2 is disposed between the right door mirror V17 and the second front sonar SF2 in the longitudinal direction so as to transmit sonar search waves Wsp to the right of the vehicle. The first side sonar SS1 and the second side sonar SS2 are provided symmetrically across the vehicle center line LC. The first side sonar SS1 and the second side sonar SS2 may be attached to the vehicle body panel V15, or may be attached to both ends of the front bumper V12 in the vehicle width direction, which are extended rearward along the vehicle overall length direction.

[0037] The third side sonar SS3 is disposed between the left rear door panel V16 and the first rear sonar SR1 in the front-rear direction so as to transmit sonar search waves Wsp to the left of the vehicle. The fourth side sonar SS4 is disposed between the right rear door panel V16 and the second rear sonar SR2 in the front-rear direction so as to transmit sonar search waves Wsp to the right of the vehicle. The third side sonar SS3 and the fourth side sonar SS4 are provided symmetrically across the vehicle center line LC. The third side sonar SS3 and the fourth side sonar SS4 may be attached to the vehicle body panel V15, or may be attached to both ends of the rear bumper V14 in the vehicle width direction, which are extended forward along the vehicle overall length direction.

[0038] FIG. 6 shows an example of detecting a parking space PS for parallel parking on the left side of the vehicle using a first side sonar SS1, which is one of the multiple sonar sensors 31. In the figure, a sonar detection point Psr is a point on the surface of the target B that is estimated to have reflected a sonar search wave Wsp emitted from the sonar sensor 31, and may also be called a "distance measurement point." As shown in FIG. 6, the parking space PS can be detected based on the distribution state of the sonar detection points Psr acquired based on the sonar reflection wave Wsr from the target B, which is a parked vehicle lined up along the traveling direction of the vehicle traveling straight. Specifically, for example, in the case of parallel parking shown in FIG. 6, the parking space PS can be detected at a location where the width of the section where the arrangement of the sonar detection points Psr lined up along the traveling direction of the vehicle is interrupted is equal to or greater than a predetermined width that exceeds the overall width of the vehicle.

[0039] (Radar Sensor) 1 and 2 again, the radar sensor 32 is configured to detect the target B by transmitting and receiving radar waves, which are millimeter waves or submillimeter waves. In this embodiment, the radar sensor 32 is mounted on the center of the front part V11 of the vehicle body V1 in the vehicle width direction. As shown in FIG. 7, the radar sensor 32 is a so-called long-range radar having a detection range of about 10 to 250 m and a radar scanning angle range θr1 centered on the vehicle's forward direction Df, and has a phased array type or beam forming type configuration. The vehicle's forward direction Df is the direction in which a virtual straight line extending the vehicle center line LC forward of the vehicle extends, and corresponds to the traveling direction when the vehicle is traveling forward, that is, when the vehicle is traveling with a shift position other than reverse shift. The vehicle's forward direction Df coincides with the traveling direction when the vehicle is traveling straight, and becomes the tangent direction of the traveling trajectory when the vehicle is traveling on a curve. The radar sensor 32 is configured to detect the target B within the radar detection range Rg1 by emitting radar search waves Wrp while scanning within the radar scan angle range θr1 and receiving radar reflected waves Wrr that are waves reflected by the target B. The radar detection range Rg1 is a sector-shaped range with the radar scan angle range θr1 as its central angle.

[0040] Specifically, the radar sensor 32 is an FMCW radar device equipped with a so-called array antenna, and is configured to be able to detect the distance to the target B, the azimuth angle θ, and the relative speed of the target B with respect to the vehicle, based on the frequency difference and phase difference of the transmitted and received millimeter waves. The azimuth angle θ is an angle between an imaginary line extending forward from the vehicle center line LC, which forms the center line of the radar detection range Rg1, and an imaginary line connecting the radar sensor 32 and the target B. The relative speed is the difference between the target moving speed vb and the vehicle speed vm, which is the traveling speed of the vehicle. More specifically, the radar sensor 32 transmits a radar search wave Wrp generated based on a transmission signal of a predetermined modulation frequency, receives a radar reflected wave Wrr, which is a wave reflected by the target B, and acquires a reception signal indicating the frequency characteristic of the radar reflected wave Wrr. The radar sensor 32 generates a beat signal, which is the deviation between the frequency of the transmission signal and the frequency of the reception signal. The radar sensor 32 performs a fast Fourier transform on the beat signal to obtain a frequency spectrum, analyzes the frequency spectrum to obtain a beat frequency, and calculates the distance and relative speed based on the beat frequency. In addition to the long-range radar sensor 32, so-called medium-range and short-range radar sensors 32 may also be provided in the in-vehicle system 1. The detection distance of the medium-range type is about 1 to 100 m, and the detection distance of the short-range type is about 15 cm to 30 m.

[0041] (Laser radar sensor) As shown in FIG. 8, the laser radar sensor 33 is configured to detect the target B by emitting a detection light Lp, which is a laser light in the infrared band, to the outside of the vehicle and receiving a reflected light Lr by the target B. The laser radar sensor 33 may also be called a LIDAR. LIDAR is an abbreviation for Light Detection and Ranging or Laser Imaging Detection and Ranging. In the vehicle-mounted system 1 according to this embodiment, a long-distance laser radar sensor 33 having a configuration as a so-called scanning LIDAR is mounted in the center of the front part V11 of the vehicle body V1 in the vehicle width direction, as shown in FIG. 1. The laser radar sensor 33 is configured to detect the target B within the LIDAR detection range Rg2 by scanning the detection light Lp in the horizontal direction within the LIDAR scan angle range θr2 centered on the vehicle forward direction Df. The LIDAR detection range Rg2 is a sector-shaped range with a radius of 200 m or more and with the LIDAR scan angle range θr2 as a central angle.

[0042] Fig. 9 shows a schematic configuration of the laser radar sensor 33. As shown in Fig. 9, the laser radar sensor 33 includes a light emitting unit 331, a scanning unit 332, and a light receiving unit 333. The light emitting unit 331 is configured to generate detection light Lp. The scanning unit 332 has a configuration as a so-called MEMS mirror. MEMS is an abbreviation for Micro Electro Mechanical Systems. That is, the scanning unit 332 is configured to scan the detection light Lp along the horizontal scanning direction Ds and the vertical scanning direction Dh by electrically controlling the orientation of a reflecting mirror that reflects the detection light Lp emitted from the light emitting unit 331 by a MEMS mechanism.

[0043] The light receiving unit 333 includes a light receiving sensor 334 that is a two-dimensional image sensor. That is, in the light receiving sensor 334, a plurality of light receiving elements 335 are two-dimensionally arranged in the vertical and horizontal directions, that is, in the directions corresponding to the horizontal scanning direction Ds and the vertical scanning direction Dh. The light receiving elements 335 are configured as light detection elements such as APD, SPAD, and photodiodes. APD stands for avalanche photodiode. SPAD stands for Single Photon Avalanche Diode. The laser radar sensor 33 is configured to generate detection point cloud data based on the reflected light Lr received by the light receiving unit 333, and detect the target B based on the detection point cloud data. The detection point cloud data is a two-dimensional array of a plurality of lidar detection points Prr arranged in an image-like manner in a two-dimensional direction corresponding to the horizontal scanning direction Ds and the vertical scanning direction Dh in FIG. 9. The LIDAR detection point Prr is a point on the target B that is estimated to have reflected the detection light Lp corresponding to the reflected light Lr detected by one light receiving element 335, and includes position information and distance measurement information in the above-mentioned two-dimensional array. Therefore, the detection point cloud data may also be called "distance measurement point cloud data." In this way, the laser radar sensor 33 acquires detection point cloud data in which a plurality of LIDAR detection points Prr are two-dimensionally arranged like an image. Therefore, the laser radar sensor 33 can be said to be a type of image sensor.

[0044] (camera) 1 and 2 again, the camera 34 as an image sensor is mounted on the vehicle so as to move with the movement of the vehicle while capturing images of the surroundings of the vehicle. That is, the camera 34 is configured to generate image information corresponding to the captured images of the surroundings of the vehicle. The image information may also be referred to as image data. The camera 34 is a so-called digital camera device and includes an image sensor such as a CCD or a CMOS. CCD stands for Charge Coupled Device. CMOS stands for Complementary Metal Oxide Semiconductor.

[0045] In this embodiment, the system-equipped vehicle V is equipped with multiple cameras 34, namely, a front camera CF, a rear camera CB, a left camera CL, and a right camera CR. When not indicating a specific one of the front camera CF, the rear camera CB, the left camera CL, and the right camera CR, hereinafter, the singular expression "camera 34" or the expression "multiple cameras 34" may be used.

[0046] The front camera CF is disposed at the upper part of the vehicle interior V2 near the upper end of the front windshield V18, at a central position in the vehicle width direction, i.e., on the vehicle center line Lc in a plan view, so as to obtain image information corresponding to an image in front of the vehicle. The front camera CF may be mounted on the front part V11 of the vehicle body V1. The rear camera CB is disposed at a central position in the vehicle width direction on the rear part V13 of the vehicle body V1 so as to obtain image information corresponding to an image behind the vehicle. The left camera CL is mounted on the left door mirror V17 so as to obtain image information corresponding to an image to the left of the vehicle. The right camera CR is mounted on the right door mirror V17 so as to obtain image information corresponding to an image to the right of the vehicle.

[0047] Based on an image captured by the camera 34, it is possible to recognize the target B, that is, to determine at least the presence and type of the target B. FIG. 10 shows an overview of an operation for recognizing the target B included in the foreground based on an image of the foreground including the road Rd in front of the vehicle captured by the front camera CF, as an example. Specifically, as shown in FIG. 10, a detection area Aw is set in the captured image. The detection area Aw is all or a part of the captured image area, i.e., the angle of view area of ​​the camera 34. Based on pixel feature parameters in the image within the detection area Aw, a feature point image Gp can be acquired. The pixel feature parameters are feature parameters of pixels constituting an image, such as brightness, contrast, color, etc. The "brightness" may also be called "lightness". The "color" may also be called "saturation". The feature point image Gp is ​​an image in which feature points Pt extracted based on the difference in pixel feature parameters between adjacent pixels, the gradient of change, etc. are arranged in an image shape. The feature points Pt are points that characterize the shape of the target B in the captured image. In other words, the feature points Pt are characteristic points, i.e., pixels, in the image within the detection area Aw. Then, by pattern matching between a feature point group Pg, which is a collection of the feature points Pt in the feature point image Gp, and a pre-stored pattern, it is possible to determine the type of the target B corresponding to the feature point group Pg.

[0048] It should be noted that the method of extracting the feature points Pt is well known at the time of filing the present application. Specifically, as the method of extracting the feature points Pt, for example, well-known methods such as a Sobel filter, a Laplacian filter, and a Canny method can be used. Therefore, in this specification, a detailed description of the method of extracting the feature points Pt is omitted. Also, the "extraction" of the feature points Pt can be expressed as "detection."

[0049] Also, it is possible to estimate the relative position or distance of the recognized target B with respect to the vehicle itself based on an image captured by the camera 34. FIG. 11 shows an overview of calculating an estimated point Pb, which is an estimation result of the three-dimensional position of a feature point Pt corresponding to the target B, by a technique called monocular moving stereo or SFM. SFM is an abbreviation for Structure from Motion. Note that the present disclosure is not limited to such monocular moving stereo. That is, the camera 34 may be a so-called compound eye stereo camera.

[0050] 11, it is assumed that the camera 34 moves in the host vehicle movement direction Dv as the host vehicle travels, and the position of the camera 34 at time t1 is a first camera position Pc1, and the position of the camera 34 at time t2 is a second camera position Pc2. The first captured image A1 is assumed to be an image captured by the camera 34 at the first camera position Pc1. The second captured image A2 is assumed to be an image captured by the camera 34 at the second camera position Pc2.

[0051] The first feature point Pt1 is a feature point Pt extracted from the first captured image A1. The second feature point Pt2 is a feature point Pt extracted from the second captured image A2 and estimated to correspond to the first feature point Pt1 at time t1. That is, the second feature point Pt2 corresponds to a point on the target B corresponding to the first feature point Pt1 at time t1, which is estimated to have moved after (t2-t1) seconds. It is possible to determine that the first feature point Pt1 and the second feature point Pt2 are the same, i.e., corresponding points, by a publicly known or well-known method such as optical flow. Then, the intersection of a first straight line L1 passing through the first camera position Pc1 and the first feature point Pt1 and a second straight line L2 passing through the second camera position Pc2 and the second feature point Pt2 is estimated as an estimated point Pb. Such an estimated point Pb is a point on the target B corresponding to the first feature point Pt1 and the second feature point Pt2 in a three-dimensional coordinate system based on the vehicle. If the estimated point Pb is a stationary point, it satisfies the epipolar constraint condition. The epipolar constraint condition is a condition defined by epipolar geometry, which states that the first camera position Pc1, the second camera position Pc2, and the estimated point Pb exist on the same plane Π at both times t1 and t2.

[0052] (Various sensors) 1 and 2 again, the operation sensor 35 is provided to generate outputs corresponding to various quantities related to the driving operation state of the vehicle by the driver D. The "various quantities related to the driving operation state" are, for example, an accelerator operation amount, a brake operation amount, a shift position, a steering angle, and the like. That is, the operation sensor 35 is a collective name for well-known sensors such as an accelerator pedal sensor, a brake pedal sensor, a shift position sensor, a steering angle sensor, a steering torque sensor, and the like, for the sake of simplicity of illustration and explanation. In addition, the operation sensor 35 may be provided with a so-called steering wheel sensor that detects whether the driver D is gripping the steering wheel V24.

[0053] The behavior sensor 36 is provided to generate outputs corresponding to various quantities related to the driving behavior of the vehicle. The "various quantities related to the driving behavior" include physical quantities related to the behavior or motion state of the vehicle, such as vehicle speed, yaw rate, longitudinal acceleration, lateral acceleration, etc. In other words, the behavior sensor 36 is a collective term for well-known sensors such as a vehicle speed sensor, a yaw rate sensor, an acceleration sensor, etc., for the sake of simplicity of illustration and explanation.

[0054] (Driver status monitor) 1 and 2, and also FIG. 12 and FIG. 13, the driver state monitor 37 is configured to sequentially detect the driver state based on the captured image of the driver D. The driver state includes, for example, whether or not the driver is awake. Specifically, the driver state monitor 37 includes a driver monitor camera provided in the vehicle interior V2 so that at least the head D1 of the driver D seated in the driver's seat V23 in a posture that allows the driver to drive the vehicle appropriately can be captured in the field of view so as to capture the face D2 from the front. The driver monitor camera is configured as a near-infrared camera. The driver state monitor 37 detects the driver state by performing image recognition processing on the captured image by the driver monitor camera.

[0055] The detection contents using the driver state monitor 37 include, for example, the direction of the face D2 of the driver D, the degree of opening of the eyes D3, the position of the pupils D4, and the like, as shown in FIG. 12 and FIG. 13. The direction of the face D2 includes a yaw angle θy and a pitch angle θp. The yaw angle θy represents the rotation angle of the face D2 of the driver D about a vertical axis Dx1 extending in the up-down direction. The yaw angle θy is 0° when the face D2 faces forward. The yaw angle θy is a positive value when the face D2 faces leftward, and a negative value when the face D2 faces rightward. The pitch angle θp represents the rotation angle of the face D2 about a horizontal axis Dx2 extending in the left-right direction. The pitch angle θp is 0° when the face D2 faces horizontally. The pitch angle θp is a positive value when the face D2 faces downward, and a negative value when the face D2 faces upward. FIG. 12 shows an example of a captured image when the driver D is in an awake state. Fig. 13 shows an example of a captured image when the driver D is not awake. As shown in Fig. 12 and Fig. 13, it is possible to detect the driver's state, including whether or not the driver is awake, the line of sight D5, etc., based on the direction of the face D2, the degree of opening of the eyes D3, the position of the pupils D4, etc.

[0056] (Operation switch) 1 and 2 again, the operation switches 38 are various switches operated by the driver D when driving the vehicle, and are provided in various locations in the vehicle compartment V2. The operation switches 38 correspond to switches that are not subject to detection of the operation amount or operation state by the operation sensor 35, and include at least an ignition switch 381, a blinker switch 382, ​​and an AD / ADAS switch 383. The ignition switch 381 is a switch for turning on and off the activation of the system-equipped vehicle V, i.e., the in-vehicle system 1, and may also be called a start switch or a power switch. The blinker switch 382 is a switch for detecting the operation state of a blinker lever. The AD / ADAS switch 383 is a switch for the driver D to input instructions such as start, end, level setting, and function selection related to driving assistance or automatic driving.

[0057] (locator) Locator 39 is configured to acquire highly accurate position information of the vehicle. Specifically, locator 39 is a so-called composite positioning system, and includes GNSS receiver 391, inertial acquisition unit 392, and locator ECU 393. GNSS is an abbreviation for Global Navigation Satellite System. The "highly accurate position information" is, for example, position information having a position accuracy of SAE level 2 or higher that can be used for advanced driving assistance or automated driving, specifically, an error of less than 10 cm. Note that, as locator 39, a commercially available system such as "POSLV" manufactured by Applanix, for example, can be used.

[0058] The GNSS receiver 391 is provided so as to be capable of receiving a positioning signal transmitted from at least one positioning satellite, i.e., an artificial satellite. In this embodiment, the GNSS receiver 391 is configured so as to be capable of receiving a positioning signal from a positioning satellite in at least one of satellite positioning systems such as GPS, QZSS, GLONASS, Galileo, IRNSS, and the Beidou satellite navigation system. GPS is an abbreviation for Global Positioning System. QZSS is an abbreviation for Quasi-Zenith Satellite System. GLONASS is an abbreviation for Global Navigation Satellite System. IRNSS is an abbreviation for Indian Regional Navigation Satellite System.

[0059] The inertia acquisition unit 392 is configured to acquire the acceleration and angular velocity acting on the host vehicle. In this embodiment, the inertia acquisition unit 392 is provided as a three-axis gyro sensor and a three-axis acceleration sensor built into a box-shaped housing of the locator 39. The locator ECU 393 has a configuration as a so-called in-vehicle microcomputer including a CPU, a ROM, a RAM, an input / output interface, and the like. The locator ECU 393 is configured to sequentially determine the position, direction, and the like of the host vehicle based on the positioning signal received by the GNSS receiver 391, the acceleration and angular velocity acquired by the inertia acquisition unit 392, and the like.

[0060] (In-vehicle communication device) The in-vehicle communication device 4 is an in-vehicle communication module also called DCM, and is provided so as to be able to communicate information with base stations around the vehicle by wireless communication conforming to a communication standard such as LTE or 5G. DCM stands for Data Communication Module. LTE stands for Long Term Evolution. 5G stands for 5th Generation. Specifically, for example, the in-vehicle communication device 4 acquires traffic information such as congestion information from a probe server on the cloud and / or a predetermined database. The "traffic congestion information" includes the location and length of the congestion section. Specifically, the traffic congestion information includes the head position of the congestion, the tail position of the congestion, the estimated congestion distance, the estimated congestion time, and the like. The traffic information is also called "road traffic information." In addition, the in-vehicle communication device 4 acquires the latest HD map information from the above-mentioned probe server and stores it in the HD map 5.

[0061] (HD Map) The HD map 5 is mainly composed of a non-volatile rewritable memory so that the HD map information can be rewritably stored and the stored contents can be retained even when the power is cut off. The HD map information can also be called high-precision map data. The HD map information includes map information with higher precision than the map information in the SD map 601 used in the navigation device 6, which has a position error of about several meters. SD stands for Standard. Specifically, the HD map 5 stores information that can be used for advanced driving assistance or automatic driving, such as three-dimensional road shape information, lane number information, and regulation information, in accordance with a predetermined standard such as the ADASIS standard. ADASIS stands for Advanced Driver Assistance Systems Interface Specification.

[0062] (Navigation device) The navigation device 6 is configured to calculate a planned driving route from the current position of the vehicle to the destination. In this embodiment, the navigation device 6 is configured to calculate a planned driving route based on the destination set by the driver D or the like, map information stored in the HD map 5 or SD map 601, position information of the vehicle acquired from the locator 39, and the like. The navigation device 6 is also configured to be able to provide various information including route information, which is the calculation result, to each part such as the driving ECU 2 and the HMI device 7 via the in-vehicle communication line 10. That is, the navigation device 6 is configured to be able to display a navigation screen on the HMI device 7 for map display, route display, and the like.

[0063] (HMI device) The HMI device 7 is a so-called vehicle HMI, and has a configuration for realizing information transmission between the vehicle and the occupants including the driver D. Specifically, the HMI device 7 is configured to present various information related to the vehicle to the occupants at least visually, and to receive input operations by the occupants corresponding to the presented contents. The presented information includes, for example, various kinds of guidance, input operation instructions, input operation content notification, warnings, and the like. Typically, the HMI device 7 has a configuration as a so-called "dashboard HMI" mainly composed of devices attached to the dashboard V21 and the steering wheel V24. However, some of the input / output devices provided in the HMI device 7 may be provided in a place other than the dashboard V21, for example, in the ceiling part of the vehicle interior V2, or in the center console between the driver's seat V23 and the seat V22 adjacent thereto, i.e., the passenger seat, or the like.

[0064] The HMI device 7 includes an HCU 701, a meter panel 702, a main display device 703, a head-up display 704, a speaker 705, and an operation device 706. HCU is an abbreviation for HMI Control Unit. The HCU 701 has a configuration as a so-called in-vehicle microcomputer including a CPU, a ROM, a RAM, an input / output interface, and the like, and is configured to control display output and audio output by the HMI device 7. That is, the HCU 701 is provided to control the operations of the meter panel 702, the main display device 703, the head-up display 704, the speaker 705, and the like.

[0065] The meter panel 702 is provided in front of the driver's seat V23 on the dashboard V21. The meter panel 702 is configured to execute meter display of the vehicle speed, coolant temperature, remaining fuel amount, etc., as well as various information display such as date and time, outside air temperature, mileage, radio station, etc.

[0066] The main display device 703 is a display device also called a CID device, and is disposed in the approximate center of the dashboard V21 in the vehicle width direction so that the display screen can be viewed by the occupant. CID stands for Center Information Display. The main display device 703 can display a navigation screen for map display and route display by the navigation device 6. The main display device 703 can also display information and content different from the navigation screen. Specifically, the main display device 703 is configured to be able to execute a display for setting operations related to driving modes such as "comfort", "normal", "sports", and "circuit". The main display device 703 is also configured to be able to execute a display related to a second task that the driver D can use during automatic driving. The second task is a task other than driving operations that the driver D executes, and includes, for example, reading, operating a mobile communication terminal, watching video content, and the like. The "video content" is, for example, a movie, a concert video, a music video, a television broadcast, and the like. The second task is also called a "non-driving task" or a "secondary activity".

[0067] 1 and 2, and also with reference to FIG. 14 and FIG. 15, the head-up display 704 is provided to display a display image M including characters, figures, and / or symbols in the forward field of vision of the driver D. In this embodiment, the head-up display 704 is configured to display the display image M as a virtual image in front of the driver D using AR technology, thereby enabling the display image M to be superimposed on the foreground including the road surface FR ahead of the vehicle. AR is an abbreviation for Augmented Reality. The "superimposed display" refers to displaying related information in a positional correspondence with a target of interest included in the foreground, for example, displaying related information so as to be superimposed on the target of interest or displaying it in the vicinity of the target of interest. The "target of interest" is, for example, a target object to which attention should be paid or attention should be paid during driving, such as a road marking, a road sign, a vehicle ahead, a pedestrian, etc. The "road marking" may also be referred to as a "road marking". For example, various displays such as a route display, a traveling direction display, and a traffic information display on the road surface FR ahead as an object of attention correspond to "superimposed display".

[0068] As shown in FIG. 14, the head-up display 704 is configured so that the depression angle AD is a positive value of 0 degrees or more. In FIG. 14, the depression angle AD is an angle between a virtual horizontal line passing through the viewpoint EP of the driver D and a virtual straight line connecting the viewpoint EP and the upper end of the projection range AP. The depression angle AD is a positive value when the upper end of the projection range AP is looked down from the viewpoint EP, and a negative value when the upper end is looked up. In addition, the head-up display 704 is configured so that the projection range AP is a horizontally elongated approximately rectangular shape as shown in FIG. 15 by making the horizontal angle of view larger than the vertical angle of view AV. The vertical angle of view AV is an angle between a virtual straight line connecting the viewpoint EP and the upper end of the projection range AP and a virtual straight line connecting the viewpoint EP and the lower end of the projection range AP in FIG. That is, the vertical angle of view AV is an angle range in the vertical direction in which the display image M can be viewed from the viewpoint EP, and may also be referred to as a viewing angle in the vertical direction.

[0069] As shown in FIG. 14, the head-up display 704 includes a projector 741 and a magnifying optical system 742. The projector 741 is configured to generate a display image light LV based on a display image signal generated by the HCU 701 and emit the display image light LV toward the magnifying optical system 742. The magnifying optical system 742 includes a plurality of optical elements including a concave mirror and an actuator for controlling the alignment of the optical elements. The magnifying optical system 742 is configured to adjust the projection state of the display image light LV onto the front windshield V18 by controlling the above alignment with the actuator according to the viewpoint EP detected by the driver state monitor 37. The head-up display 704 projects the display image light LV onto a projection range AP on the front windshield V18 and allows the driver D to visually recognize the reflected light, thereby realizing the display of the display image M.

[0070] FIG. 15 shows an example of a display state of the head-up display 704. The head-up display 704 can display an information display M1 and a graphic display M2. The information display M1 is a display content that displays various information such as the maximum speed on the road Rd on which the vehicle is traveling, the current vehicle speed of the vehicle, the distance to the destination, the estimated time of arrival at the destination, building names, intersection names, etc. The information display M1 is typically a display using characters, road sign-like figures, or symbols. The graphic display M2 is a display content using graphics such as lines and arrows, and is used to display, for example, information related to the traveling direction of the vehicle and lane selection.

[0071] The head-up display 704 can display superimposed content and non-superimposed content. The "superimposed content" is a display image content that is associated with or related to a specific target of attention included in the foreground and is superimposed on the target of attention. In contrast, the "non-superimposed content" is a display image content that is not associated with a specific target of attention included in the foreground and is not superimposed on the specific target of attention. In the display example of FIG. 15, a linear graphic display M2 is superimposed as superimposed content on the lane in which the vehicle is currently traveling, and an information display M1 indicating the maximum speed is displayed as non-superimposed content at an appropriate position that does not overlap with the graphic display M2. In this way, the HMI device 7 has a configuration as a notification unit that notifies a passenger such as the driver D of the vehicle of information.

[0072] 1 and 2 again, speaker 705 is provided to output audio corresponding to the display contents on meter panel 702, main display device 703, and head-up display 704. Note that speaker 705 can also output audio that does not correspond to the display contents on meter panel 702, main display device 703, and head-up display 704, such as music, radio audio, and the like.

[0073] The operation device 706 is an input device that is not included in the operation switch 38 and is not subject to detection of the operation amount or operation state by the operation sensor 35. Specifically, for example, switches provided around the display screen of the main display device 703 and a transparent touch panel provided so as to cover such a display screen correspond to the operation device 706. Also, for example, switches provided on the spokes of the handle V24 of the steering wheel structure correspond to the operation device 706. Furthermore, a pointing device such as a touch panel provided on the center console corresponds to the operation device 706.

[0074] (Lighting device) The lighting device 8 includes a body ECU 801, a headlight 802, and a blinker 803. The body ECU 801 has a configuration as a so-called in-vehicle microcomputer equipped with a CPU, a ROM, a RAM, an input / output interface, etc. The body ECU 801 is configured to control the lighting state of the headlights 802 and the blinkers 803 based on information input from the driving ECU 2 and the driving information input unit 3.

[0075] (Motion Control Device) The motion control device 9 is provided to control the motion, i.e., the driving behavior, of the vehicle based on information input from the driving ECU 2 and the driving information input unit 3. Specifically, the motion control device 9 includes a drive device 91, a shift device 92, a braking device 93, and a steering device 94.

[0076] The drive device 91 includes a drive system ECU 911 and a drive mechanism 912. The drive system ECU 911 has a configuration as a so-called in-vehicle microcomputer including a CPU, ROM, RAM, an input / output interface, etc., and is configured to control the operation of the drive mechanism 912. The drive mechanism 912 that generates the driving force for running the system-equipped vehicle V includes an engine and / or an electric motor. That is, the system-equipped vehicle V may be any of a gasoline engine vehicle, a diesel engine vehicle, a biofuel vehicle, a hydrogen engine vehicle, a hybrid vehicle, a BEV, a fuel cell vehicle, etc. BEV is an abbreviation for Battery Electric Vehicle.

[0077] The shift device 92 includes a shift ECU 921 and a shift mechanism 922. The shift ECU 921 has a configuration as a so-called in-vehicle microcomputer including a CPU, a ROM, a RAM, an input / output interface, etc., and is configured to control the operation of the shift mechanism 922. The shift mechanism 922 is provided between the wheels V3 functioning as drive wheels and the drive device 91, and is configured to switch the shift position, that is, switch between forward running and reverse running, and change the gear ratio during forward running. The shift device 92 may have a configuration as a so-called shift-by-wire.

[0078] The braking device 93 includes a brake ECU 931 and a brake mechanism 932. The brake ECU 931 has a configuration as a so-called on-board microcomputer including a CPU, a ROM, a RAM, an input / output interface, etc., and is configured to control the operation of the brake mechanism 932. The brake mechanism 932 that generates a braking force in the system-equipped vehicle V may include a regenerative brake mechanism in addition to a friction brake mechanism. The braking device 93 may have a so-called brake-by-wire configuration.

[0079] The steering device 94 includes a steering ECU 941 and a steering mechanism 942. The steering ECU 941 has a configuration as a so-called on-board microcomputer including a CPU, a ROM, a RAM, an input / output interface, and the like, and is configured to control the operation of the steering mechanism 942. The steering mechanism 942 is configured to be able to change the orientation of the left front wheel V3a and the right front wheel V3b, which are steered wheels. The steering mechanism 942 may also be able to change the orientation of the left rear wheel V3c and the right rear wheel V3d. That is, the host vehicle may be a so-called four-wheel steering vehicle. The steering device 94 may also have a configuration as a so-called steer-by-wire.

[0080] (Driving control based on target recognition results) 16 and 17 show an example of a functional configuration realized by the driving ECU 2 by the processor 21 executing the program instructions stored in the memory 22. Hereinafter, an overview of the driving control of the host vehicle executed by the driving ECU 2 based on the recognition result of the target B around the host vehicle using the ADAS sensor will be described. Hereinafter, in this specification, "recognition" and "detection" are defined as follows. "Recognition" is a concept including "detection", "identification" and "recognition". "Detection" refers to finding the target B based on images and point cloud data. The "detected" target B is only judged to exist, and the shape, meaning, etc. are not judged. "Identification" refers to classifying the shape, meaning, etc. of the detected target B. The "identified" target B is judged not only to exist, but also to be what it is, for example, whether it is a human, a car, or a building. "Recognition" refers to judging the detected and identified target B as something that should be taken into consideration in the driving control of the host vehicle, such as a pedestrian, an oncoming vehicle, etc. It should be noted that "detection" includes "detection."

[0081] 16, the driving ECU 2 has a recognition unit 2001, an operation determination unit 2002, and a control output unit 2003 as functional configurations realized by the processor 21. The recognition unit 2001 is adapted to execute a recognition operation of a target B around the vehicle based on information input from a surrounding monitoring sensor, an operation sensor 35, a behavior sensor 36, and the like. The operation determination unit 2002 is adapted to determine the content of a control operation required for the vehicle at the present time based on the recognition result of the target B by the recognition unit 2001 and information input from the operation sensor 35, the behavior sensor 36, and the like. The content of such a control operation is, for example, a collision avoidance operation, an emergency stop operation, a warning operation to the driver D, and the like. The control output unit 2003 is adapted to generate a control output according to the control operation content determined by the operation determination unit 2002, and output it to each unit in the in-vehicle system 1. The control output is, for example, a steering amount, a braking amount, a warning message code, and the like.

[0082] 17, the recognition unit 2001 has an input information acquisition function 2101, an input information processing function 2102, a target recognition function 2103, a vehicle lane recognition function 2104, an intersection recognition function 2105, and a surrounding environment recognition function 2106. The input information acquisition function 2101 holds a predetermined amount of information input to the recognition unit 2001 in time series. The input information processing function 2102 executes various processes such as noise removal and coordinate conversion on the information acquired, i.e., held, by the input information acquisition function 2101. The target recognition function 2103 executes a recognition operation of the target B based on the information processed by the input information processing function 2102. The target recognition function 2103 has a lane marking recognition function 2131, a road marking recognition function 2132, a roadside structure recognition function 2133, a traffic light recognition function 2134, a road sign recognition function 2135, a lane recognition function 2136, a pedestrian recognition function 2137, a surrounding vehicle recognition function 2138, and an obstacle recognition function 2139.

[0083] The recognition targets by each recognition function in the recognition unit 2001 are illustrated in FIG. 18, FIG. 19, etc. Referring to FIG. 18, the recognition targets B include, broadly speaking, a traffic-related three-dimensional object B1, another vehicle B2, a general three-dimensional object B3, and a road marking B4. Referring to FIG. 19, the recognition targets B include a parking frame line B5, a wheel stopper B6, and a parked vehicle B7 as objects related to the detection of the parking space PS and parking assistance. The traffic-related three-dimensional object B1 is a three-dimensional object for use in road traffic, such as a traffic light B11 and a road sign B12. The other vehicle B2 is a vehicle that may be a following target or an obstacle during the travel of the host vehicle, and does not include the parked vehicle B7 shown in FIG. 19. The general three-dimensional object B3 is a three-dimensional object other than the traffic-related three-dimensional object B1, the other vehicle B2, the wheel stopper B6, and the parked vehicle B7, and mainly constitutes an "obstacle." Furthermore, for example, it is possible to recognize the lane LN based on the recognition result of the target B using an image. That is, the parking space PS and the lane LN are different from the target B as a direct recognition target by the ADAS sensor, but can be said to be indirect recognition targets based on the recognition result of the target B. The lane LN as such an indirect recognition target can include, for example, the own lane LNm in which the own vehicle is currently traveling, the oncoming lane LNc, etc. Further details of each recognition target in the recognition unit 2001 will be described later.

[0084] The own lane recognition function 2104 recognizes the position of the own vehicle in the width direction of the road Rd on which the own vehicle is currently traveling, based on the recognition result of the target B by the target recognition function 2103. If the road Rd includes multiple lanes LN, the own lane recognition function 2104 recognizes which of the multiple lanes LN lined up in the width direction the own lane LNm is. The intersection recognition function 2105 recognizes the intersection Xr around the own vehicle, based on the recognition result of the target B by the target recognition function 2103. Specifically, the intersection recognition function 2105 recognizes, for example, the presence or absence and light color of a traffic light B11, the presence status of road markings B4 such as a stop line B42, the positions of the intersection entrance Xr1 and the intersection exit Xr2, the course division, etc., for the intersection Xr to which the own vehicle is currently approaching. The surrounding environment recognition function 2106 recognizes the driving environment around the vehicle, that is, for example, the presence or absence of obstacles, based on the recognition result of the target B by the target recognition function 2103. These recognition results are used by the operation determination unit 2002 shown in Fig. 16 to determine the content of the control operation currently required for the vehicle.

[0085] FIG. 18 shows an example of a foreground view from the vehicle in a situation where the vehicle is approaching an intersection Xr that forms a crossroads and has a traffic light B11. The intersection Xr that forms a crossroads is hereinafter simply referred to as a "crossroads". The recognition results of the traffic light B11 and the road sign B12 included in the traffic-related three-dimensional object B1 can be used, for example, for information notification or warning to the driver D by the HMI device 7. In addition, such recognition results can be used for the motion control of the vehicle during automatic driving or driving assistance. In addition to the traffic light B11 and the road sign B12, the traffic-related three-dimensional object B1 includes roadside structures B13 such as guardrails and curbs. The roadside structures B13 are recognized as obstacles when controlling the running of the vehicle. In addition, the traffic-related three-dimensional object B1 includes road studs B14 and poles B15 as shown in FIG. 20. Depending on the height and installation situation, the road studs B14 are recognized as obstacles when controlling the running of the vehicle. The pole B15 is recognized as an obstacle during driving control of the host vehicle.

[0086] 18 again, the general three-dimensional object B3 includes a fallen object B31 on the road, a pedestrian B32, a cyclist B33, a structure B34, etc. The fallen object B31 on the road, the pedestrian B32, and the cyclist B33 are recognized as obstacles when controlling the running of the vehicle. The structure B34 is not recognized as an obstacle because it is usually located outside the road Rd, but it can be a target for superimposed display when notifying the driver D of information by the head-up display 704, for example.

[0087] The road markings B4 include a crosswalk B41, a stop line B42, and a road dividing line B43. As shown in FIG. 21, the road markings B4 also include a character marking B44 and a graphic display B45. The character markings B44 include, for example, a number indicating a maximum speed, a vehicle traffic division such as "bus only", and a regulatory character such as "stop". The graphic display B45 includes, for example, an arrow indicating a possible travel direction and a graphic for calling attention to the crosswalk B41 ahead. The recognition result of the road markings B4 can be used, for example, to estimate the positions of an intersection entrance Xr1, an intersection exit Xr2, and an intersection center Xrc. The intersection entrance Xr1 is the edge of the intersection of interest on the side where the host vehicle enters. The intersection of interest is an intersection Xr that the host vehicle is currently approaching and is closest to the host vehicle, and is an intersection where the host vehicle is scheduled to pass or is likely to pass. An intersection Xr that "the vehicle plans to pass through" is an intersection Xr on the planned travel route when the destination and planned travel route of the vehicle are set in the navigation device 6. An intersection Xr that "is likely to be passed through" is an intersection Xr that is estimated with a high probability that the vehicle will pass through, based on the current vehicle speed of the vehicle and the distance to the intersection Xr, when the destination and planned travel route of the vehicle are not set in the navigation device 6. The intersection exit Xr2 is the edge of the intersection of interest on the side from which the vehicle will exit. The intersection center Xrc is the center of the intersection of interest.

[0088] As shown in Figs. 20 to 22, the road division line B43 includes an outer roadway line B431, a center line B432, and a lane boundary line B433. The outer roadway line B431 is provided to indicate the outer edge of the roadway in the width direction of the road Rd. The "roadway" is the part of the road Rd on which vehicles travel. The center line B432 is provided to divide the road Rd, which is a two-way road, i.e., a non-one-way road, by traffic direction. The center line B432 may be a single line, a double line, or a triple line. The lane boundary line B433 is provided to divide multiple lanes LN of the same traffic direction on a road Rd with two or more lanes on one side. The road division line B43 may be painted white or yellow. In the example of Fig. 20, the center line B432 has a triple line pattern with yellow solid lines on both sides of a white solid line, and road studs B14 and poles B15 are provided on the central white solid line. In the example of Fig. 21, the center line B432 has a double line pattern with white solid lines, and the lane boundary line B433 has a yellow solid line portion beyond the white broken line portion. From the recognition results of the roadside structures B13 and road division lines B43, etc., it becomes possible to recognize the driving lane LNd, the oncoming lane LNc, the passing lane LNp, the right-turn lane LNr, the road shoulder LNs, the emergency parking zone EZ, etc.

[0089] The road markings B4 include a guidance strip marking B461, a safety zone marking B462, a no entry sign B463, a no stopping sign B464, and the like, as shown in Figs. 23 to 26. The guidance strip marking B461 shown in Fig. 23 is provided to indicate a guidance strip for guiding vehicles to travel safely and smoothly. The guidance strip is also called a zebra zone. The guidance strip is a marking for guiding vehicles not to pass through, but the entry of vehicles into the guidance strip itself is not prohibited by law. The guidance strip marking B461 is provided, for example, near an intersection Xr, a junction, or a branch. The safety zone marking B462 shown in Fig. 24 is provided to indicate a safety zone. The safety zone is an area provided on the road Rd for the purpose of ensuring the safety of pedestrians B32, etc., and vehicles are not allowed to enter inside the safety strip. The no entry sign B463 shown in FIG. 25 is provided to indicate a no entry area. The no entry area is an area that is not used for vehicle passage, and vehicles are not allowed to enter inside the area. The no stop sign B464 shown in FIG. 26 is provided to indicate a no stop area. The no stop area is an area that a vehicle is not allowed to enter when there is a risk that the vehicle will have to stop due to the situation ahead of the path that the vehicle is about to proceed. In other words, a vehicle is not allowed to stop inside the no stop sign B464 when waiting at a traffic light or in a traffic jam.

[0090] FIG. 27 shows an example of a road sign B12. The road sign B12 is a signboard that is installed beside or above the road Rd and provides necessary information to users through its design. The design includes an external shape, a color, and a symbol. The external shape is a circle, a triangle, an inverted triangle, a square, a diamond, an octagon, etc. The colors are white, black, red, blue, yellow, etc. In FIG. 27, the cross-hatched portion indicates red, the diagonal hatched portion indicates blue, the dot-hatched portion indicates yellow, and the blackened portion indicates black. The symbols include letters, figures, and combinations thereof.

[0091] In FIG. 27, examples of signs in Japan, Europe, and the United States are shown, categorized by meaning. For European examples, examples from Germany and France are used. For example, the design of "No entry" is almost the same in each country. In contrast, the color and symbol of "Stop" are common, but the external shape is almost the same in Europe and the United States, while the Japanese sign is different from Europe and the United States. The design of maximum speed signs is almost the same in Japan and Europe, except for the color of the letters, but the American sign is completely different from Japan and Europe in terms of external shape and color. The design of "Watch out for railroad crossings" is low in common between Japan, Europe, and the United States. The design of "No right turns" is similar in Europe and the United States, but there is no corresponding sign in Japan. However, for example, if a "No going in any direction other than the specified direction" sign, which shows the straight-ahead direction and the left-turn direction with white arrows, is provided at a cross intersection, as shown in FIG. 27, such a sign essentially indicates "No right turns."

[0092] Accurate recognition of the road signs B12 satisfactorily realizes automated driving and advanced driving assistance, but as described above, the design of the road signs B12 may differ greatly from country to country. Therefore, a database for recognizing the meaning indicated by the detected road signs B12 by pattern matching is stored in advance in the memory 22 shown in FIG. 2. Such a database may be country-specific or region-specific, or may be common to the entire world. Note that such a database may be stored in a host computer connected to the vehicle via the in-vehicle communication device 4 in place of or together with the memory 22. V2X is an abbreviation for Vehicle to X.

[0093] FIG. 28 shows an example of a scene in which the recognition result of the target B is used, in which the vehicle is approaching a crossroads. In this scene, for example, an information display M1 such as "10m to stop position" showing the distance between the vehicle and the stop line B42 can be superimposed on the stop line B42. In addition, a voice notification such as "10m to stop position" can be given. This makes it possible to effectively support the driver D using the driving support function to stop at the stop line B42 without fail. Alternatively, by notifying the passengers including the driver D using the automatic driving function of the scheduled occurrence of a stop event at an appropriate timing, it is possible to effectively avoid impairing the comfort of the passengers.

[0094] FIG. 29 shows another example of a scene in which the recognition result of the target B is used, in which the vehicle is approaching a crossroads with a traffic light B11 where the vehicle is scheduled to turn right. In this scene, for example, an information display M1 such as "20m ahead, turn right" indicating the distance to the right turn point can be superimposed on the road surface FR in front of the field of vision of the driver D in the right turn lane LNr, which is the vehicle lane LNm. In addition, a voice notification such as "turn right at the intersection 20m ahead" can be given. Furthermore, an arrow graphic display M2 indicating the planned travel route of the vehicle can be superimposed on the road surface FR so as to pass through the vicinity of the intersection center Xrc from the right turn lane LNr, which is the vehicle lane LNm, and reach the road Rd where the vehicle is to turn right. This allows the vehicle's course guidance and lateral acceleration generation behavior to be properly notified to the driver D and other occupants, thereby making it possible to properly avoid impairing the comfort of the occupants.

[0095] 17 again, the target recognition function 2103 recognizes the type and state of the target B based on the recognition result using the ADAS sensor and the HD map information stored in the HD map 5. The "state" includes the position. The "position" includes the distance from the vehicle, the relative position with respect to the vehicle, the lane position, i.e., one of the multiple lanes LN, and the like.

[0096] The lane marking recognition function 2131 recognizes road markings B43 provided around the vehicle, including on the road surface FR of the road Rd where the vehicle is heading. For example, the lane marking recognition function 2131 recognizes whether the outer roadway line B431, center line B432, lane boundary line B433, etc., as shown in Figs. 20 to 23, are white or yellow, solid or broken, single or multiple lines, etc. Technology for recognizing road markings B43 was well known at the time of filing the present application, and therefore detailed description thereof will be omitted in this specification.

[0097] The road marking recognition function 2132 recognizes the type, meaning, and position of road markings B4 provided around the vehicle, other than the road dividing line B43. Recognition targets by the road marking recognition function 2132 include, for example, a crosswalk B41, a stop line B42, and a graphic marking B45. Details of the road marking recognition according to this embodiment will be described later.

[0098] The roadside structure recognition function 2133 recognizes the type and position of the roadside structure B13 provided around the vehicle. At least one of the image recognition result using the camera 34, the recognition result using the laser radar sensor 33, and HD map information is used to recognize the roadside structure B13. The traffic light recognition function 2134 recognizes at least the position and light color of the traffic light B11 provided at the travel destination of the vehicle. The road sign recognition function 2135 recognizes the road signs B12 provided around the vehicle. The lane recognition function 2136 recognizes the lane, that is, recognizes the number and type of the lanes LN on the road Rd on which the vehicle is traveling. At least one of the image recognition result using the camera 34, i.e., the front camera CF, the recognition result using the laser radar sensor 33, and HD map information is used to recognize the lane. Specifically, in lane recognition, the image recognition results obtained using the front camera CF are usually used, but so-called "sensor fusion" that uses other information as well can be performed as necessary.

[0099] The pedestrian recognition function 2137 recognizes a pedestrian B32 existing around the vehicle. The surrounding vehicle recognition function 2138 recognizes another vehicle B2 existing around the vehicle. The obstacle recognition function 2139 recognizes obstacles such as road debris B31 and pedestrians B32 existing around the vehicle. The recognition methods of the traffic light recognition function 2134, the road sign recognition function 2135, the lane recognition function 2136, the pedestrian recognition function 2137, the surrounding vehicle recognition function 2138, and the obstacle recognition function 2139 are well known at the time of filing of the present application. Therefore, the details of these recognition methods will not be described in this specification.

[0100] (road marking recognition) FIG. 30 shows an overview of the process of recognizing road marking B4 based on an image captured by camera 34. In FIG. 30, "S" is an abbreviation of "step." In the road marking recognition process, processor 21 shown in FIG. 2 executes each process of steps 101 to 107 shown in FIG. 30 in order. Step 101 mainly corresponds to the process by input information acquisition function 2101 shown in FIG. 17. Step 102 mainly corresponds to the process by input information processing function 2102 shown in the same figure. Step 103 and onwards mainly correspond to the process by target recognition function 2103 shown in the same figure.

[0101] In step 101, the processor 21 reads input information necessary for road marking recognition. The input information includes, for example, image information corresponding to an image captured by the front camera CF. In step 102, the processor 21 executes a predetermined pre-processing on the input information read in step 101. The pre-processing includes, for example, at least one of brightness adjustment, noise removal, resolution conversion, lens distortion correction, bird's-eye view transformation, affine transformation, edge enhancement processing, trimming processing, and the like. In step 103, the processor 21 detects an object target, that is, an object B that is a target for road marking recognition. In step 104, the processor 21 estimates the relative position of the object target detected in step 103, that is, the road marking B4, with respect to the vehicle or the distance from the vehicle, using a monocular moving stereo technique or the like. In step 105, the processor 21 executes an identification process for the object target. In step 106, the processor 21 executes a recognition process for the object target identified in step 105. In step 107, the processor 21 outputs the recognition result of the target object to the operation determination unit 2002 shown in FIG.

[0102] 28 and 29, for example, an error may occur in the estimated values ​​of the relative positions of the crosswalk B41, the stop line B42, the intersection entrance Xr1, and the intersection center Xrc with respect to the vehicle and the distances from the vehicle. In this case, an error may occur in the timing of the driving control of the vehicle, or the position of the superimposed display or the timing of the voice notification may deviate from the original case, causing discomfort to the driver D or other passengers. For this reason, it is preferable to achieve higher recognition accuracy in the scene where the recognition result of the target B is used.

[0103] However, as shown in FIG. 31, for example, due to aging caused by long-term use, defects such as fading or partial loss of the pattern may occur in the crosswalk B41. Or, even if there is no such defect, as shown in FIG. 32, a part of the pattern of the crosswalk B41 may be blocked by another vehicle B2 or a pedestrian B32 preceding the vehicle. In such cases, the feature amount for image recognition may decrease, making it difficult to extract the feature point Pt or the feature point group Pg well, or the distance to the blocking object B such as the other vehicle B2 may be erroneously estimated, resulting in a deterioration in recognition accuracy, i.e., distance estimation accuracy. In addition, since the way in which the detection rectangle hits in the detection area Aw (for example, the position and size) is not stable, the features may differ depending on the position where the detection rectangle hits, and the estimated distance may fluctuate over time. The "detection rectangle" is a rectangle set in the detection area Aw, and is a part that detects a recognition target such as the crosswalk B41 and performs semantic judgment, distance estimation, etc.

[0104] In this regard, for example, as shown in Figures 33 and 34, the crosswalk B41 is formed by a pattern having periodicity in the road width direction. For this reason, for example, in a scene of image recognition using the camera 34 as an image sensor, i.e., the front camera CF, pixel feature parameters such as luminance values ​​change periodically in the detection width direction, i.e., the image width direction, which intersects with the vehicle forward direction Df, which is the detection direction. The period changes between the long distance case shown in Figure 33 and the short distance case shown in Figure 34.

[0105] FIG. 35 shows the actual white paint pattern shape of a crosswalk B41 in terms of changes in pixel feature parameters. In FIG. 35, the horizontal axis Dx indicates the position or distance in the road width direction, and the vertical axis FP indicates the pixel feature parameters. The crosswalk B41 has a repeating pattern of two pixel features of widths W1 and W2, with a period W=W1+W2. For example, in Japan, width W1 is a white painted area, width W2 is a non-painted area, and W1=W2=0.45 to 0.5 m. Outside Japan, width W1 may be a yellow painted area, and width W2 may be a black painted area.

[0106] FIG. 36 shows the change in pixel feature parameters on the first virtual line Ly1 shown in FIG. 33. FIG. 37 shows the change in pixel feature parameters on the second virtual line Ly2 shown in FIG. 34. The first virtual line Ly1 and the second virtual line Ly2 are virtual straight lines that pass through the center in the depth direction of the crosswalk B41 in the image captured by the front camera CF and are parallel to the image width direction. In FIG. 36 and FIG. 37, the horizontal axis Px indicates the number of pixels or pixel positions in the detection width direction, i.e., the image width direction, and the vertical axis FP indicates pixel feature parameters. The image width direction is the horizontal direction of pixels arranged two-dimensionally vertically and horizontally, and is also called the row direction. The "row direction" is the arrangement direction of multiple pixels arranged horizontally, i.e., in the horizontal direction. Alternatively, the "row direction" is the direction that specifies the horizontal position, i.e., the column number, of a pixel row arranged vertically, i.e., in the vertical direction. The vertical direction, i.e., the vertical direction, is called the column direction.

[0107] As shown in Fig. 36 and Fig. 37, the change period λ of the pixel feature parameter changes depending on the distance from the camera 34 to the crosswalk B41. Fig. 38 shows the relationship between the distance Dt and the change period λ when the crosswalk B41 is photographed by the front camera CF. The relationship shown in Fig. 38 can be obtained by fitting a plot of the experimental results in which the crosswalk B41 is photographed by the front camera CF while changing the distance Dt in a known photographing environment such as a test course to a function using a method such as the least squares method, as shown in Fig. 39.

[0108] The change in pixel characteristic parameter when a part of the crosswalk B41 is scratched or obscured is shown by a solid line in FIG. 40. In this case, as shown in FIG. 40, the change in pixel characteristic parameter does not appear to have a clear periodicity at first glance. However, even when scratched or obscured, there should be a wave with a period as shown by a dashed line in the figure, which best expresses the periodicity of the crosswalk B41. Therefore, when the solid waveform in FIG. 40 is Fourier transformed, a peak occurs in the intensity spectrum as shown in FIG. 41. In FIG. 41, the horizontal axis λ indicates the period, which is the inverse of the frequency, the vertical axis P indicates the intensity, and the horizontal line parallel to the horizontal axis indicates the threshold for distinguishing from the background. FIG. 41 can be called a frequency spectrum. In the example of FIG. 41, three peaks exceeding the threshold appear at the positions of the horizontal axis λ=λ1, λ2, and λ3. Of these three peaks, the peak of λ1 having the highest intensity is highly likely to correspond to the dashed waveform shown in FIG. 40.

[0109] In this manner, in this embodiment, based on an image acquired by the camera 34, the periodic change pattern of the pixel feature parameters is calculated in the detection width direction intersecting with the detection direction from the camera 34 toward the target B in the field of view. The periodic change pattern of the pixel feature parameters, i.e., the period or frequency, can be calculated, for example, by using a Fourier transform. Then, in this embodiment, the distance to the target B is estimated based on the calculated periodic change pattern. Specifically, in this embodiment, a representative period corresponding to a peak having an intensity equal to or greater than a threshold in the frequency spectrum is acquired, i.e., calculated, as the periodic change pattern of the pixel feature parameters. Typically, such a representative period corresponds to a peak having the highest intensity. Based on the calculated periodic change pattern, i.e., the representative period, and a correspondence relationship between the periodic change pattern and the distance prepared in advance as shown in FIG. 38, the distance can be estimated.

[0110] FIG. 42A shows the change in pixel feature parameters acquired based on an actually captured image when the distance from the front camera CF to the crosswalk B41 is long. FIG. 42B shows the Fourier transform result of FIG. 42A. FIG. 43A shows the change in pixel feature parameters acquired based on an actually captured image when the distance from the front camera CF to the crosswalk B41 is medium. FIG. 43B shows the Fourier transform result of FIG. 43A. FIG. 44A shows the change in pixel feature parameters acquired based on an actually captured image when the distance from the front camera CF to the crosswalk B41 is short. FIG. 44B shows the Fourier transform result of FIG. 44A. As shown in FIG. 42B, FIG. 43B, and FIG. 44B, the strongest peak indicated by the arrow clearly occurs higher than the other peaks, and the value of the period λ corresponding to the peak increases as the distance increases.

[0111] Fig. 45 shows an overview of a process for estimating the distance to the crosswalk B41 based on an image captured by the front camera CF. This process is part of the process executed in step 104 in Fig. 30. In Fig. 45 as well, "S" is an abbreviation of "step." In this distance estimation process, the processor 21 shown in Fig. 2 executes the processes of steps 201 to 206 shown in Fig. 45 in order.

[0112] In step 201, the processor 21 selects a distance estimation range for the captured image. The distance estimation range is a processing target range for image processing in the following step 202, and corresponds to a detection rectangle placed within the detection area Aw shown in Fig. 10. In step 202, the processor 21 executes image processing such as grayscale conversion on image information within the distance estimation range set in step 201.

[0113] In step 203, the processor 21 calculates a frequency spectrum. Specifically, the processor 21 generates a waveform of the change in pixel feature parameter in the image width direction as shown by the solid line in Fig. 40 based on the image processing result in step 202. Then, the processor 21 applies a frequency analysis method such as Fourier transform to the waveform to calculate a frequency spectrum as shown in Fig. 41.

[0114] In step 204, the processor 21 extracts a representative period from the frequency spectrum calculated in step 203. In step 205, the processor 21 reads out a correspondence relationship between the period and the distance, such as that shown in FIG. 38, from the memory 22. Then, in step 206, the processor 21 calculates an estimated distance based on the representative period extracted in step 204 and the correspondence relationship read out in step 205.

[0115] The object of distance estimation according to the present embodiment is not limited to the crosswalk B41. That is, for example, a stop position marking B471 as shown in FIG. 46 may be a road marking B4 having periodicity in the road width direction. The stop position marking B471 is a road marking B4 for indicating a stop position like the stop line B42, but is displayed as a broken line and is provided in a place where it is preferable to stop and check for safety, although traffic regulation for a temporary stop is not required. The stop position marking B471 shown in FIG. 46 is also called a "guidance stop line" in Japan. In addition, as shown in FIG. 47, a guardrail B1301, which is a roadside structure B13 provided in the vehicle forward direction Df along a direction intersecting the vehicle forward direction Df, may also be a target. This is because the installation intervals of the posts B1302 in the guardrail B1301 are also set to a constant value according to the standard. Furthermore, a similar process can be performed on a three-dimensional object such as a road traffic sign that is installed three-dimensionally. The detection direction of the periodicity in this case will be described later.

[0116] Also, as shown in Fig. 47, when the first road Rd1 on which the vehicle is traveling intersects with the second road Rd2 ahead, a broken line portion B4311 may be provided on the outer road line B431 of the second road Rd2 on the near side of the intersection Xr. Such a broken line portion B4311 may also be a target. This situation may occur not only at a T-shaped intersection as shown in Fig. 47, but also at a Y-shaped intersection or a cross intersection.

[0117] Furthermore, a stop position marking B471 as a road marking B4 having periodicity such as a dotted line or a dashed line may be provided at a roundabout intersection Xrr as shown in FIG. 48. The roundabout intersection Xrr has a circular road Rd3, an entrance road Rd4, and an exit road Rd5. The entrance road Rd4 is a road Rd for entering the circular road Rd3. The stop position marking B471 is provided at the boundary between the circular road Rd3 and the entrance road Rd4 so as to also indicate the outer boundary line of the circular road Rd3. In the entrance road Rd4, a crosswalk B41 or a stop line B42 may be provided in front of the stop position marking B471. Note that the installation position, installation mode, and installation purpose of the stop position marking B471 are not limited to the examples shown in FIG. 46 and FIG. 48. Therefore, for example, the shape of the stop position marking B471 is not limited to a simple dotted or dashed line as shown in Figures 46 and 48. That is, the stop position marking B471 may be formed as a double dashed or double dotted line as shown in Figure 49. Alternatively, the stop position marking B471 may be formed as a so-called shark tooth shape in which inverted triangles are arranged in a row as shown in Figures 50 and 51. The stop position marking B471 may also be called a "Yield Line" depending on the installation position, installation mode, or installation purpose.

[0118] Thus, according to this embodiment, it is possible to estimate the distance with good accuracy for the target B having periodic changes in pixel feature parameters such as brightness in image information obtained by using the camera 34, which is an image sensor, by utilizing the periodicity. This makes it possible to execute vehicle control and information notification to the occupants at appropriate positions and timings in autonomous driving and driving assistance. In addition, normal stereo camera techniques such as compound eye stereo and monocular stereo estimate the distance using two images, i.e., two frames, taken at different shooting points and / or shooting timings. In contrast, this embodiment estimates the distance using one image, i.e., one frame, taken by the camera 34, and is therefore advantageous in terms of processing speed and processing load for distance estimation. Furthermore, such a periodic target B is typically a road marking B4 such as a crosswalk B41 provided corresponding to the intersection Xr. Based on the distance estimated for this type of road marking B4, it is possible to estimate the position of the intersection entrance Xr1, the intersection exit Xr2, or the intersection center Xrc. Based on the position estimation result of the intersection entrance Xr1, etc., it becomes possible to very effectively control the motion of the host vehicle or notify the occupants of information by the HMI device 7 when the host vehicle approaches or passes through the intersection Xr. Note that the "passage" of the host vehicle through the intersection Xr here includes not only passing through by going straight ahead, but also passing through by turning right or left, etc.

[0119] (Modification) The present disclosure is not limited to the above embodiment. Therefore, the above embodiment can be modified as appropriate. Representative modified examples will be described below. In the following description of the modified examples, differences from the above embodiment will be mainly described. In addition, the same reference numerals are used for parts that are the same or equivalent to each other in the above embodiment and the modified examples. Therefore, in the following description of the modified examples, the description of the above embodiment can be appropriately used for components having the same reference numerals as the above embodiment, unless there is a technical contradiction or special additional explanation.

[0120] The present disclosure is not limited to the specific device configurations shown in the above embodiments. That is, for example, the shape and structure of the vehicle body V1 in the system-equipped vehicle V are not limited to a box shape, i.e., a substantially rectangular shape in a plan view. In addition, the vehicle body panel V15 does not need to be provided on the upper side of the vehicle interior V2. Alternatively, the vehicle body panel V15 covering the upper side of the vehicle interior V2 may be detachable. There are also no particular limitations on the use of the system-equipped vehicle V, the positions of the driver's seat V23 and the steering wheel V24, the number of occupants, and the like.

[0121] The definition, level, or category of driving automation is not limited to those specified in "SAE J3016". Specifically, the "automated driving" in the above embodiment is a driving automation level corresponding to levels 3 to 5 in "SAE J3016" where the driving automation system is responsible for, i.e., executes, all dynamic driving tasks. For this reason, the definition of "automated driving" in the above embodiment naturally includes the fact that the driver D is not required to monitor the surroundings. However, the present disclosure is not limited to such an aspect. That is, for example, depending on the definition of "automated driving", not only "automated driving without the obligation to monitor the surroundings" but also "automated driving with the obligation to monitor the surroundings" can be conceived. Specifically, for example, the hands-off driving in the above embodiment can be interpreted as "automated driving with the obligation to monitor the surroundings". In this case, "automated driving" is a concept that includes so-called "partially automated driving" in which the driver D is responsible for, i.e., executes, some dynamic driving tasks such as the obligation to monitor the surroundings. "Partially automated driving" can be evaluated as being substantially synonymous with "advanced driving assistance". In addition, in the road traffic system of each country, the type and conditions of automated driving (for example, feasible roads, driving speed range, lane change availability, etc.) may be appropriately considered according to domestic circumstances, etc. For this reason, the present disclosure may be implemented with specifications that are compatible with the road traffic system of each country.

[0122] The configuration of the in-vehicle system 1 is not limited to the above example. For example, there is no particular limitation on the number and mounting positions of so-called ADAS sensors such as the sonar sensor 31, the radar sensor 32, and the camera 34. That is, for example, by reducing the number of radar sensors 32 and laser radar sensors 33, which are relatively expensive at the time of filing the present application, as much as possible, it is possible to promote the early spread of autonomous vehicles. Therefore, for example, the radar sensor 32 or the laser radar sensor 33 may be omitted.

[0123] Locator 39 is not limited to a configuration having inertia acquisition unit 392 built therein. Specifically, inertia acquisition unit 392 may receive output information from an angular velocity sensor and an acceleration sensor provided outside locator 39 as behavior sensor 36. Locator 39 may be integrated with HD map 5.

[0124] The navigation device 6 may be connected to the HMI device 7 so as to be capable of communicating information via a sub-communication line different from the in-vehicle communication line 10. Alternatively, the navigation device 6 may have a display screen dedicated to displaying a navigation screen, separate from the HMI device 7. Alternatively, the navigation device 6 may be provided as a part of the HMI device 7. Specifically, for example, the navigation device 6 may be integrated with the main display device 703.

[0125] The HMI device 7 is not limited to a configuration including a meter panel 702, a main display device 703, and a head-up display 704. That is, for example, the meter panel 702 and the main display device 703 may be integrated into one display device. In this case, the meter panel 702 may be provided as a part of a display area in one display device that is a liquid crystal or organic EL display. That is, the meter panel 702 may be realized by displaying images of bezels, pointers, scales, etc. corresponding to a tachometer, a speedometer, a water temperature gauge, etc. Also, the HMI device 7 does not have to include the head-up display 704.

[0126] In the above embodiment, each ECU has a configuration as a so-called in-vehicle microcomputer including a CPU, an MPU, and the like. However, the present disclosure is not limited to such a configuration. That is, all or a part of the ECU may be configured to include a digital circuit configured to enable the above-mentioned operation, such as an ASIC or an FPGA. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array. That is, in the ECU, an in-vehicle microcomputer portion and a digital circuit portion may coexist.

[0127] The program according to the present disclosure, which enables the execution of various operations, procedures, or processes described in the above embodiment, may be downloaded or upgraded via V2X communication by the in-vehicle communication device 4. Alternatively, such a program may be downloaded or upgraded via a terminal device provided in a vehicle manufacturing plant, a maintenance plant, a dealer, or the like. Such a program may be stored on a memory card, an optical disk, a magnetic disk, or the like.

[0128] In this way, each of the above functional configurations and methods may be realized by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, each of the above functional configurations and methods may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, each of the above functional configurations and methods may be realized by one or more dedicated computers configured by combining a processor and a memory programmed to execute one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transitive tangible recording medium as instructions to be executed by a computer. In other words, each of the above functional configurations and methods may be expressed as a computer program including a procedure for realizing the same, or as a non-transitive tangible recording medium storing the program.

[0129] The present disclosure is not limited to the specific processing mode shown in the above embodiment. That is, for example, the recognition target by the road sign recognition function 2132 may include the character marking B44. The frequency or period analysis method is also not limited to the specific example shown in the above embodiment. For example, the Fourier transform is not limited to a one-dimensional Fourier transform performed on a linear array in the row direction in the detection rectangular plane represented by a two-dimensional array. That is, a two-dimensional Fourier transform may be performed on a two-dimensional array of a part of a surface part belonging to the detection rectangular plane represented by a two-dimensional array, or on the entire area of ​​the detection rectangular plane, to calculate the periodic change mode. In addition, the calculation method of the periodic change mode is not limited to the Fourier transform, and for example, a wavelet transform can also be used, but is not limited to these. In addition, when obtaining an estimated distance from a representative period based on the relationship shown in FIG. 38, a lookup table or an arithmetic expression may be used. That is, the relationship shown in FIG. 38 may be stored in the memory 22 as a lookup table or an arithmetic expression. Furthermore, the detection direction of the periodicity is not limited to the detection width direction of the camera 34, which is an image sensor, as in the example of the crosswalk B41 described above. That is, for example, in measuring the distance to a three-dimensional object, periodicity may be detected in the vertical direction of the screen.

[0130] As shown in FIG. 41, a plurality of peaks exceeding the threshold may appear. In this case, in the above embodiment, the representative period is determined using the period λ1 with the highest intensity among the plurality of peaks, and the estimated distance is calculated using the representative period. However, the present disclosure is not limited to such an embodiment. That is, for example, the distance may be estimated based on a first estimated distance estimated based on a plurality of peaks having an intensity equal to or greater than the threshold, and a second estimated distance estimated by another distance measurement method. Specifically, in the example of FIG. 41, the estimated distance by the period λ1, the estimated distance by the period λ2, and the estimated distance by the period λ3 are calculated individually as the first estimated distance. Meanwhile, the distance measurement result by another ADAS sensor, for example, the laser radar sensor 33, is used as the second estimated distance. Then, the estimated distance closest to the distance measurement result is selected. The selected estimated distance may be used for vehicle control as it is, or may be subjected to statistical processing such as averaging or weighted averaging between the selected estimated distance and the second estimated distance.

[0131] In the above embodiment, the distance estimation of the target B based on the image captured by the front camera CF has been mainly described, but the present disclosure is not limited to such an embodiment. That is, for example, the distance estimation of the target B based on the image captured by the rear camera CB, the left camera CL, or the right camera CR is also possible.

[0132] The detection point cloud data corresponding to the crosswalk B41 obtained by the laser radar sensor 33 may also have periodicity along the lateral direction corresponding to the horizontal scanning direction Ds. For this reason, the distance estimation technique according to the present disclosure is also applicable to image-like detection point cloud data obtained by the laser radar sensor 33 as an image sensor. That is, for example, it is possible to correct the distance measurement result of the crosswalk B41 existing in the traveling direction of the vehicle based on the change in the lateral existence period of the detection point cloud data depending on the distance. Therefore, the application of the present disclosure is not limited to image information from the camera 34.

[0133] Similar expressions such as "obtain," "calculate," "estimate," "detect," "sensing," and "determine" may be substituted for each other as appropriate to the extent that they are not technically inconsistent. Furthermore, "information" and "signal" may be substituted for each other as appropriate to the extent that they are not technically inconsistent. Furthermore, "above the threshold" and "exceed the threshold" may be substituted for each other as appropriate to the extent that they are not technically inconsistent. Similarly, "below the threshold" and "below the threshold" may be substituted for each other as appropriate to the extent that they are not technically inconsistent.

[0134] Needless to say, the elements constituting the above-mentioned embodiments are not necessarily essential, except when expressly stated as essential or when clearly considered essential in principle. In addition, when the numbers, amounts, ranges, etc. of the components are mentioned, the present disclosure is not limited to the specific values, except when expressly stated as essential or when clearly limited to a specific value in principle. Similarly, when the shapes, directions, positional relationships, etc. of the components are mentioned, the present disclosure is not limited to the shapes, directions, positional relationships, etc., except when expressly stated as essential or when clearly limited to a specific shape, direction, positional relationship, etc. in principle.

[0135] The modified examples are not limited to the above examples. For example, multiple modified examples may be combined with each other as long as there is no technical contradiction. Furthermore, all or part of the above embodiment and all or part of the modified examples may be combined with each other as long as there is no technical contradiction.

[0136] (Disclosure Perspective) As is apparent from the above description of the embodiments and modifications, at least the following disclosure items are disclosed in this specification.

[0137] [Point 1-1] A control device (2) mounted on a vehicle (V), A processor (21) for executing program instructions; A memory (22) that is a non-transitory tangible recording medium for storing the program instructions; Equipped with The processor, Calculating a periodic change in a characteristic parameter of pixels constituting an image based on an image acquired by an image sensor (34) mounted on the vehicle; estimating a distance to the target based on the calculated periodic change pattern; Control device. [Point 1-2] The image is a single image captured by the image sensor. A control device as described in aspect 1-1. [Points 1-3] The processor estimates the distance based on the calculated periodic change pattern and a correspondence relationship between the periodic change pattern and the distance that is prepared in advance. A control device according to aspect 1-1 or 1-2. [Points 1-4] The processor calculates the periodic variation pattern using a Fourier transform or a wavelet transform. The control device according to any one of Aspects 1-1 to 1-3. [Points 1-5] the periodic change pattern is a representative period corresponding to a peak in a frequency spectrum calculated by applying a frequency analysis method to a waveform representing a change in the feature parameter; The control device according to any one of Aspects 1-1 to 1-4. [Points 1-6] the processor obtains the representative period by the peak having the greatest intensity in the frequency spectrum. The control device according to aspects 1-5. [Point 1-7] The processor obtains the representative period based on the peak having an intensity equal to or greater than a threshold in the frequency spectrum. A control device according to aspect 1-5 or 1-6. [Point 1-8] The processor estimates the distance based on a first estimated distance estimated based on a plurality of peaks having intensities equal to or greater than the threshold in the frequency spectrum, and a second estimated distance estimated by another distance measuring method. A control device as described in aspects 1-7. [Point 1-9] The processor calculates the periodic change in the characteristic parameter in a detection width direction intersecting a detection direction from the image sensor toward a target (B) in a field of view of the image sensor. The control device according to any one of Aspects 1-1 to 1-8. [Points 1-10] The detection direction is a traveling direction of the vehicle. The control device according to aspects 1-9. [Point 1-11] The target is a road marking (B41) having periodicity in the road width direction. A control device as described in aspects 1-10. [Point 1-12] The road marking is provided corresponding to an intersection (Xr), The processor estimates a location of an entrance (Xr1) or an exit (Xr2) of the intersection based on the estimated distance. The control device according to aspects 1-11. [Point 1-13] The vehicle includes a notification unit (7) that notifies an occupant (D) of information, The processor executes motion control of the vehicle or notifies the occupant of information by the notification unit in response to the approach or passage of the vehicle to the intersection based on the estimated position. The control device according to aspects 1-12.

[0138] [Point 2-1] A control method executed by a processor (21) provided in a control device (2) mounted on a vehicle (V), comprising: Calculating a periodic change in a characteristic parameter of pixels constituting an image based on an image acquired by an image sensor (34) mounted on the vehicle; estimating a distance to the target based on the calculated periodic change pattern; Control methods. [Point 2-2] The image is a single image captured by the image sensor. The control method according to aspect 2-1. [Point 2-3] estimating the distance based on the calculated periodic change pattern and a correspondence relationship between the periodic change pattern and the distance that is prepared in advance; The control method according to aspect 2-1 or 2-2. [Point 2-4] Calculating the periodic variation pattern using Fourier transform or wavelet transform; The control method according to any one of Aspects 2-1 to 2-3. [Point 2-5] the periodic change pattern is a representative period corresponding to a peak in a frequency spectrum calculated by applying a frequency analysis method to a waveform representing a change in the feature parameter; The control method according to any one of Aspects 2-1 to 2-4. [Point 2-6] obtaining the representative period based on the peak having the greatest intensity in the frequency spectrum; The control method according to aspect 2-5. [Point 2-7] obtaining the representative period based on the peak having an intensity equal to or greater than a threshold in the frequency spectrum; The control method according to aspect 2-5 or 2-6. [Point 2-8] estimating the distance based on a first estimated distance estimated based on a plurality of peaks having intensities equal to or greater than the threshold in the frequency spectrum, and a second estimated distance estimated by another distance measuring method; The control method according to aspect 2-7. [Point 2-9] Calculating the periodic change in the characteristic parameter in a detection width direction intersecting with a detection direction from the image sensor toward a target (B) in a field of view of the image sensor; The control method according to any one of Aspects 2-1 to 2-8. [Point 2-10] The detection direction is a traveling direction of the vehicle. The control method according to aspect 2-9. [Point 2-11] The target is a road marking (B41) having periodicity in the road width direction. The control method according to aspect 2-10. [Point 2-12] The road marking is provided corresponding to an intersection (Xr), estimating a location of an entrance (Xr1) or an exit (Xr2) of the intersection based on the estimated distance; The control method according to aspect 2-11. [Point 2-13] The vehicle includes a notification unit (7) that notifies an occupant (D) of information, Based on the estimated position, a motion control of the vehicle or a notification of information to the occupant by the notification unit is performed in association with the approach of the vehicle to or the passage of the vehicle through the intersection. The control method according to aspect 2-12.

[0139] [Point 3-1] A control program executed by a processor (21) provided in a control device (2) mounted on a vehicle (V), The process executed by the processor includes: A process of calculating, based on an image acquired by an image sensor (34) mounted on the vehicle, a periodic change pattern of a characteristic parameter of pixels constituting the image; A process of estimating a distance to the target based on the calculated periodic change pattern; A control program including: [Point 3-2] The image is a single image captured by the image sensor. A control program according to aspect 3-1. [Point 3-3] estimating the distance based on the calculated periodic change pattern and a correspondence relationship between the periodic change pattern and the distance that is prepared in advance; A control program according to aspect 3-1 or 3-2. [Point 3-4] Calculating the periodic variation pattern using Fourier transform or wavelet transform; The control program according to any one of Aspects 3-1 to 3-3. [Point 3-5] the periodic change pattern is a representative period corresponding to a peak in a frequency spectrum calculated by applying a frequency analysis method to a waveform representing a change in the feature parameter; The control program according to any one of Aspects 3-1 to 3-4. obtaining the representative period based on the peak having the greatest intensity in the frequency spectrum; A control program according to aspects 3-5. [Point 3-7] obtaining the representative period based on the peak having an intensity equal to or greater than a threshold in the frequency spectrum; A control program according to aspect 3-5 or 3-6. [Point 3-8] estimating the distance based on a first estimated distance estimated based on a plurality of peaks having intensities equal to or greater than the threshold in the frequency spectrum, and a second estimated distance estimated by another distance measuring method; A control program according to any one of claims 3-7. [Point 3-9] Calculating the periodic change in the characteristic parameter in a detection width direction intersecting with a detection direction from the image sensor toward a target (B) in a field of view of the image sensor; The control program according to any one of Aspects 3-1 to 3-8. [Point 3-10] The detection direction is a traveling direction of the vehicle. A control program according to any one of claims 3-9. [Point 3-11] The target is a road marking (B41) having periodicity in the road width direction. A control program according to aspect 3-10. [Point 3-12] The road marking is provided corresponding to an intersection (Xr), estimating a location of an entrance (Xr1) or an exit (Xr2) of the intersection based on the estimated distance; A control program according to aspect 3-11. [Point 3-13] The vehicle includes a notification unit (7) that notifies an occupant (D) of information, Based on the estimated position, a motion control of the vehicle or a notification of information to the occupant by the notification unit is performed in association with the approach of the vehicle to or the passage of the vehicle through the intersection. A control program according to aspect 3-12.

[0140] [Point 4-1] A computer-readable non-transient tangible recording medium (22) having a control program executed by a processor (21) provided in a control device (2) mounted on a vehicle (V), The process executed by the processor includes: A process of calculating, based on an image acquired by an image sensor (34) mounted on the vehicle, a periodic change pattern of a characteristic parameter of pixels constituting the image; A process of estimating a distance to the target based on the calculated periodic change pattern; Including, Recording medium. [Point 4-2] The image is a single image captured by the image sensor. A recording medium according to aspect 4-1. [Point 4-3] estimating the distance based on the calculated periodic change pattern and a correspondence relationship between the periodic change pattern and the distance that is prepared in advance; A recording medium according to aspect 4-1 or 4-2. [Point 4-4] Calculating the periodic variation pattern using Fourier transform or wavelet transform; The recording medium according to any one of Aspects 4-1 to 4-3. [Point 4-5] the periodic change pattern is a representative period corresponding to a peak in a frequency spectrum calculated by applying a frequency analysis method to a waveform representing a change in the feature parameter; The recording medium according to any one of Aspects 4-1 to 4-4. [Point 4-6] obtaining the representative period based on the peak having the greatest intensity in the frequency spectrum; A recording medium according to aspect 4-5. [Point 4-7] obtaining the representative period based on the peak having an intensity equal to or greater than a threshold in the frequency spectrum; A recording medium according to aspect 4-5 or 4-6. [Point 4-8] estimating the distance based on a first estimated distance estimated based on a plurality of peaks having intensities equal to or greater than the threshold in the frequency spectrum, and a second estimated distance estimated by another distance measuring method; A recording medium according to any one of aspects 4 to 7. [Point 4-9] Calculating the periodic change in the characteristic parameter in a detection width direction intersecting with a detection direction from the image sensor toward a target (B) in a field of view of the image sensor; The recording medium according to any one of Aspects 4-1 to 4-8. [Point 4-10] The detection direction is a traveling direction of the vehicle. A recording medium according to any one of aspects 4 to 9. [Point 4-11] The target is a road marking (B41) having periodicity in the road width direction. A recording medium according to any one of claims 4 to 10. [Point 4-12] The road marking is provided corresponding to an intersection (Xr), estimating a location of an entrance (Xr1) or an exit (Xr2) of the intersection based on the estimated distance; A recording medium according to any one of aspects 4-11. [Point 4-13] The vehicle includes a notification unit (7) that notifies an occupant (D) of information, Based on the estimated position, a motion control of the vehicle or a notification of information to the occupant by the notification unit is performed in association with the approach of the vehicle to or the passage of the vehicle through the intersection. A recording medium according to aspect 4-12.

[0141] [Point 5-1] A control method executed by a processor (21) provided in a control device (2) mounted on a vehicle (V), comprising: The processor, selecting a distance estimation range for an image captured by the imaging device; performing image processing for generating a waveform of a change in pixel feature parameter on image information of the selected distance estimation range; generating a waveform of the change in the pixel feature parameter based on the image processing result, and calculating a frequency spectrum by applying a frequency analysis method to the generated waveform; A representative period is extracted from the calculated frequency spectrum; A correspondence relationship between the extracted representative period and the distance from the vehicle to the target is read from the memory (22) (S205); A method for calculating an estimated distance from the host vehicle to a target object based on the correspondence relationship. [Point 5-2] The captured image is a single image captured by the photographing device. The method according to aspect 5-1. [Point 5-3] The processor extracts the representative period using a Fourier transform or a wavelet transform. The method according to aspect 5-1 or 5-2. [Point 5-4] The representative period is a representative period corresponding to a peak in the frequency spectrum. The method according to any one of Aspects 5-1 to 5-3. [Point 5-5] the processor obtains the representative period by the peak having the greatest intensity in the frequency spectrum. The method according to aspect 5-4. [Points 5-6] The processor obtains the representative period based on the peak having an intensity equal to or greater than a threshold in the frequency spectrum. The method according to aspect 5-4 or 5-5. [Points 5-7] The processor estimates the distance based on a first estimated distance estimated based on a plurality of the peaks having an intensity equal to or greater than the threshold, and a second estimated distance estimated by another distance measuring method. The method according to any one of aspects 5-4 to 5-6. [Point 5-8] The detection direction is a traveling direction of the host vehicle. The method according to any one of Aspects 5-1 to 5-7. [Point 5-9] The processor calculates the representative period in a detection width direction intersecting a detection direction from the image capture device toward a target (B) in a field of view of the image capture device. The method according to any one of Aspects 5-1 to 5-8. [Points 5-10] The target is a road marking (B41) having periodicity in the road width direction. The method according to any one of Aspects 5-1 to 5-9. [Point 5-11] The road marking is provided corresponding to an intersection (Xr), The processor estimates a location of an entrance (Xr1) or an exit (Xr2) of the intersection based on the estimated distance. The method according to aspects 5-10. [Point 5-12] The vehicle includes a notification unit (7) that notifies an occupant (D) of information, The processor executes motion control of the host vehicle or notifies the occupant of information by the notification unit in response to the host vehicle approaching or passing through the intersection based on the estimated position. The method according to aspects 5-11. [Explanation of symbols]

[0142] 1. In-vehicle systems 2. Control device 21 Processors 22 Memory 33 Laser radar sensor (image sensor) 34 Camera (image sensor) B Target B4 Road marking B41 Crosswalk V Vehicle

Claims

1. A control device (2) mounted on a vehicle (V), A processor (21) for executing program instructions; A memory (22) that is a non-transitory tangible recording medium for storing the program instructions; Equipped with The processor, Calculating a periodic change in a characteristic parameter of a pixel constituting an image based on an image acquired by an image sensor (34) mounted on the vehicle; estimating a distance to the target based on the calculated periodic change pattern; Control device.

2. The image is a single image captured by the image sensor. The control device according to claim 1 .

3. The processor estimates the distance based on the calculated periodic change pattern and a correspondence relationship between the periodic change pattern and the distance that is prepared in advance. The control device according to claim 1 .

4. The processor calculates the periodic variation pattern using a Fourier transform or a wavelet transform. The control device according to claim 1 .

5. the periodic change pattern is a representative period corresponding to a peak in a frequency spectrum calculated by applying a frequency analysis method to a waveform representing a change in the feature parameter; The control device according to claim 1 .

6. the processor obtains the representative period by the peak having the greatest intensity in the frequency spectrum. The control device according to claim 5.

7. The processor obtains the representative period based on the peak having an intensity equal to or greater than a threshold in the frequency spectrum. The control device according to claim 5.

8. The processor estimates the distance based on a first estimated distance estimated based on a plurality of peaks having intensities equal to or greater than the threshold in the frequency spectrum, and a second estimated distance estimated by another distance measuring method. The control device according to claim 7.

9. The processor calculates the periodic change in the characteristic parameter in a detection width direction intersecting a detection direction from the image sensor toward a target (B) in a field of view of the image sensor. The control device according to claim 1 .

10. The detection direction is a traveling direction of the vehicle. The control device according to claim 9.

11. The target is a road marking (B41) having periodicity in the road width direction. The control device according to claim 10.

12. The road marking is provided corresponding to an intersection (Xr), The processor estimates a location of an entrance (Xr1) or an exit (Xr2) of the intersection based on the estimated distance. The control device according to claim 11.

13. The vehicle includes a notification unit (7) that notifies an occupant (D) of information, The processor executes motion control of the vehicle or notifies the occupant of information by the notification unit in response to the approach or passage of the vehicle to the intersection based on the estimated position. The control device according to claim 12.

14. A control method executed by a processor (21) provided in a control device (2) mounted on a vehicle (V), comprising: Based on an image acquired by an image sensor (34) mounted on the vehicle, a periodic change in a characteristic parameter of pixels constituting the image is calculated in a detection width direction intersecting with a detection direction from the image sensor toward a target (B) within a field of view of the image sensor; estimating a distance to the target based on the calculated periodic change pattern; Control methods.

15. A control program executed by a processor (21) provided in a control device (2) mounted on a vehicle (V), The process executed by the processor includes: A process of calculating, based on an image acquired by an image sensor (34) mounted on the vehicle, a periodic change in a characteristic parameter of pixels constituting the image in a detection width direction intersecting with a detection direction from the image sensor toward a target (B) within the field of view of the image sensor; A process of estimating a distance to the target based on the calculated periodic change pattern; A control program including:

Images