Distance estimation method and device

Abstract

The invention relates to a distance estimation method and device. A processor of the distance estimation device executes distance estimation program instructions to, respectively, (i) calculate, based on an image captured by an image sensor mounted to the vehicle, information included in a periodic variation in a value of at least one pixel characteristic parameter included in selected pixels of the image, and (ii) estimating a distance from the vehicle to a target object associated with the periodic change based on information included in the calculated periodic change.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based upon and claims the benefit of priority from Japanese patent application No. 2023-142298, filed on September 1, 2023, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to distance estimation methods and devices. Background Art

[0004] Research and development of driving assistance technology and / or autonomous driving technology has been actively carried out. Various types of proposals related to driving assistance technology and / or autonomous driving technology have been implemented, one of which is disclosed in internal patent publication WO 2019 / 202397.

[0005] Driving assistance technologies are intended to reduce the burden on the driver so that the driver can drive the vehicle comfortably and safely. For example, these driving assistance technologies include technologies related to following distance control, lane keeping assist control, lane change assist control, parking assist control, obstacle warning, collision avoidance assist control or other vehicle control technologies. These autonomous driving technologies are intended to enable the vehicle to travel automatically, that is, autonomously, without the need for the driver's driving operation. Various sensing devices such as camera devices and / or radar devices are installed in vehicles with such driving assistance technologies and / or driving assistance technologies. Various sensing devices are configured to detect the surrounding conditions around the vehicle. Autonomous control operations of autonomous vehicles in motion, such as autonomous steering, autonomous driving and / or autonomous braking, are performed based on map information of visual representations of surrounding conditions detected by sensing devices and / or areas around the current position of the autonomous vehicle, for example. In particular, high-accuracy map information including road data for each lane around the autonomous vehicle can be used to perform autonomous control operations of autonomous vehicles in motion, so that the safety and reliability of autonomous control operations of autonomous vehicles in motion can be improved. Summary of the invention

[0006] In view of the improvement in accuracy of object detection and / or object recognition by sensing devices and the improvement in user convenience, research and development of these autonomous driving technologies and driving assistance technologies have been accelerated recently. That is, further improvement of these autonomous driving technologies and driving assistance technologies enables advanced driving assistance vehicles and / or autonomous driving vehicles to be popularized earlier and more widely.

[0007] Based on the above situation, the present disclosure provides a distance estimation method. The distance estimation method includes:

[0008] (i) establishing a distance estimation area in an image captured by an image capture device mounted to the vehicle,

[0009] (ii) applying at least one image processing task to image data included in the distance estimation region,

[0010] (iii) calculating a waveform of periodic changes in the value of at least one pixel characteristic parameter included in the image data based on a result of applying at least one image processing task,

[0011] (iv) applying frequency analysis methods to the periodically varying waveform to calculate the intensity-frequency spectrum accordingly,

[0012] (v) obtaining, according to the intensity-frequency spectrum, a representative period corresponding to a selected local intensity peak in the intensity-frequency spectrum,

[0013] (vi) reading correspondence information from a memory device, the correspondence information indicating a correspondence between a variable of a distance from a vehicle to a target object and a variable of a representative cycle,

[0014] (vii) Estimate the distance from the vehicle to the target object based on the obtained representative period and correspondence information.

[0015] The present disclosure based on the above situation provides a distance estimation device. The distance estimation device includes a memory device storing distance estimation program instructions and a processor, the processor being configured to execute the distance estimation program instructions to accordingly: (i) calculate information included in the periodic changes in the value of at least one pixel feature parameter included in selected pixels of the image based on an image captured by an image capture device installed on a vehicle, and (ii) estimate the distance from the vehicle to a target object associated with the periodic changes based on the calculated information about the periodic changes.

[0016] The present disclosure based on the above situation provides a distance estimation method. The distance estimation method includes: calculating information included in the periodic change of the value of at least one pixel feature parameter included in selected pixels of the image based on an image captured by an image capture device installed on a vehicle, and estimating the distance from the vehicle to a target object associated with the periodic change based on the information included in the calculated periodic change.

[0017] The present disclosure based on the above situation provides a processor program product including a storage medium readable by a processor installed in a vehicle and a distance estimation program instruction. The distance estimation program instruction causes the processor to (i) calculate information included in the periodic variation of the value of at least one pixel feature parameter included in selected pixels of the image based on an image captured by an image capture device installed to the vehicle, and (ii) estimate the distance from the vehicle to a target object associated with the periodic variation based on the information included in the calculated periodic variation.

[0018] Note that each parenthesized reference numeral assigned to a corresponding element in the present disclosure merely represents an example of a relationship between the corresponding element and a corresponding specific element described in an exemplary embodiment described later, and thus the present disclosure is not limited to the parenthesized reference numerals. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Other aspects of the present disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 is a plan view schematically showing a vehicle in which a vehicle system includes a driving ECU serving as a control device according to an exemplary embodiment;

[0021] Figure 2 It is shown Figure 1 A schematic block diagram of the overall structure of the vehicle system shown in ;

[0022] Figure 3 is a schematic diagram showing the use of Figure 2 A plan view of one of the sonar sensors shown in FIG. 1 performing target object detection;

[0023] Figure 4 is a schematic diagram showing the use of Figure 2 A plan view of a pair of sonar sensors performing target object detection as shown in FIG.

[0024] Figure 5 is a schematic diagram showing the use of Figure 2 A plan view of each sonar sensor performing target object detection as shown in FIG.

[0025] Figure 6 is a schematic diagram showing the use of Figure 2 A plan view of a first sonar sensor performing parking space detection as shown in FIG.

[0026] Figure 7 is a schematic diagram showing the use of Figure 2 A plan view of a radar sensor performing target object detection as shown in FIG.

[0027] Figure 8 is a schematic diagram showing the use of Figure 2 A plan view of a lidar sensor performing target object detection as shown in FIG.

[0028] Fig. 9 It is schematically shown Figure 8 A perspective view of the configuration and operation of a lidar sensor as shown in;

[0029] Fig.10 is a schematic diagram showing the use of Figure 2 FIG. 1 shows a camera device performing target object detection;

[0030] Fig.11 is a schematic diagram showing the use of Figure 2 FIG. 1 shows a camera device performing target object detection;

[0031] Fig.12 is a schematic diagram showing the use of Figure 2 A diagram showing a driver state monitor performing driver state detection;

[0032] Fig.13 is a schematic diagram showing the use of Figure 2 A diagram showing a driver state monitor performing driver state detection;

[0033] Fig.14 It is shown Figure 2 A diagram of a schematic configuration of a head-up display shown in FIG.

[0034] Fig.15 is a schematic diagram showing the following: Fig.14 The driver's perspective shown in FIG. 1 is viewing (i) the road surface in front of the own vehicle through the front windshield and (ii) a portion of the instrument panel;

[0035] Fig.16 It is indicated by Figure 2 A block diagram of a schematic functional configuration implemented by the driving ECU shown in FIG.

[0036] Fig.17 It is indicated by Fig.16 A block diagram of a schematic functional configuration implemented by the identification unit shown in ;

[0037] Fig.18 is a schematic diagram illustrating a first example of a target object to be identified by an exemplary embodiment;

[0038] Fig.19 is a schematic diagram illustrating a second example of a target object to be identified by an exemplary embodiment;

[0039] Fig. 20is a schematic diagram illustrating a third example of a target object to be identified by the exemplary embodiment;

[0040] Fig.21 is a schematic diagram illustrating a fourth example of a target object to be identified by the exemplary embodiment;

[0041] Fig. 22 is a schematic diagram illustrating a fifth example of a target object to be identified by the exemplary embodiment;

[0042] Fig.23 is a schematic diagram illustrating a sixth example of a target object to be identified by the exemplary embodiment;

[0043] Fig.24 is a schematic diagram illustrating a fifth example of a target object to be identified by the exemplary embodiment;

[0044] Fig.25 is a schematic diagram illustrating a sixth example of a target object to be identified by the exemplary embodiment;

[0045] Fig.26 is a schematic diagram illustrating a seventh example of a target object to be identified by the exemplary embodiment;

[0046] Fig. 27 is a diagram schematically showing eight examples of traffic signs as target objects;

[0047] Fig.28 is a schematic diagram showing a first example of indicating how to use a recognition result of a target object;

[0048] Fig.29 is a schematic diagram showing a second example of indicating how to use the recognition result of the target object;

[0049] Fig.30 is a schematic diagram showing that Fig.17 A flowchart of a road surface marking recognition task performed by a road surface marking recognition module shown in;

[0050] Fig.31 is a schematic diagram showing an aged crosswalk marking as one of the identification targets of the exemplary embodiment;

[0051] Fig.32 is a schematic diagram showing a situation where a portion of a crosswalk marking is blocked by an blocking object;

[0052] Fig.33 is a schematic diagram showing an example of how a crosswalk marking located relatively far from the own vehicle is viewed from the own vehicle;

[0053] Fig.34is a schematic diagram showing an example of how a crosswalk marking located relatively close to the own vehicle is viewed from the own vehicle;

[0054] Fig.35 is a graph schematically showing a marking pattern configuration of a crosswalk marking;

[0055] Fig.36 is schematically shown at Fig.33 A graph showing how the value of at least one pixel characteristic parameter on the dotted line on the pedestrian crossing mark shown in FIG. 1 changes;

[0056] Fig.37 is schematically shown at Fig.34 A graph showing how the value of at least one pixel characteristic parameter on the dotted line on the pedestrian crossing mark shown in FIG. 1 changes;

[0057] Fig.38 is a graph schematically showing a relationship between each changed value of the distance to a pedestrian crossing mark and a corresponding value of a change period;

[0058] Fig.39 is a schematic diagram showing how to calculate Fig.38 A graph of the relationship shown in ;

[0059] Fig.40 is a graph schematically showing how a value of at least one pixel feature parameter included in a specific pixel corresponding to a corresponding white painted line of a crosswalk marking changes in a road width direction, some portions of the crosswalk marking being missing and / or blocked;

[0060] Fig.41 It is schematically shown Fig.40 a graph of a result of a Fourier transform of a change in a value of at least one pixel characteristic parameter shown in;

[0061] Fig.42A is a graph schematically showing how a value of at least one pixel feature parameter included in a specific pixel corresponding to a corresponding white painted line of a crosswalk marking located at a relatively long distance from an own vehicle changes in a road width direction;

[0062] Fig.42B It is schematically shown Fig.42A a graph of a result of a Fourier transform of a change in a value of at least one pixel characteristic parameter shown in;

[0063] Fig.43A is a graph schematically showing how a value of at least one pixel feature parameter included in a specific pixel corresponding to a corresponding white painted line of a crosswalk marking located at a relatively middle distance from an own vehicle changes in a road width direction;

[0064] Fig.43B It is schematically shown Fig.43A a graph of a result of a Fourier transform of a change in a value of at least one pixel characteristic parameter shown in;

[0065] Fig.44A is a graph schematically showing how a value of at least one pixel feature parameter included in a specific pixel corresponding to a corresponding white painted line of a crosswalk marking located at a relatively close distance from an own vehicle changes in a road width direction;

[0066] Fig.44B It is schematically shown Fig.44A a graph of a result of a Fourier transform of a change in a value of at least one pixel characteristic parameter shown in;

[0067] Fig.45 is a flow chart schematically illustrating a distance estimation routine according to an exemplary embodiment;

[0068] Fig.46 are examples of targets whose distances are to be estimated by the exemplary embodiments;

[0069] Fig.47 is another example of a target whose distance is to be estimated by the exemplary embodiment;

[0070] Fig.48 is yet another example of a target whose distance is to be estimated by the exemplary embodiments;

[0071] Fig.49 is yet another example of a target whose distance is to be estimated by the exemplary embodiments;

[0072] Fig.50 is yet another example of a target whose distance is to be estimated by an exemplary embodiment; and

[0073] Fig.51 is yet another example of a target whose distance is to be estimated by the exemplary embodiments. DETAILED DESCRIPTION

[0074] The following describes exemplary embodiments of the present disclosure and specific examples thereof with reference to the accompanying drawings. The configurations, functions and / or examples described in the following exemplary embodiments and specific examples thereof may be freely modified. In the exemplary embodiments, specific examples and modifications thereof, the same reference numerals are assigned to equivalent or identical components in the exemplary embodiments, specific examples and modifications thereof. Among equivalent or identical components, the description of the previous component may be directly used to describe the latter component or subsequent multiple components, as long as (i) no technical contradiction occurs and / or there is no additional description.

[0075] Overall configuration of the vehicle

[0076] Figure 1 A vehicle V to which the present disclosure is applied according to an exemplary embodiment is shown. The vehicle V to which the system is installed is a four-wheeled motor vehicle. The vehicle V to which the system is installed has a vehicle body V1 having a substantially rectangular shape in a plan view. The vehicle body, i.e., the vehicle body V1, has a center line LC defined as a virtual line passing through a center point VC of the vehicle V in the width direction of the vehicle V and extending in the front-rear direction, i.e., the longitudinal direction, of the vehicle V to which the system is installed. The center line LC will be referred to as a vehicle center line LC.

[0077] exist Figure 1 , the width direction of the vehicle V on which the system is mounted (which will be referred to as the vehicle width direction) corresponds to the horizontal direction therein, and the vehicle height direction is defined as the height direction of the vehicle V on which the system is mounted, and is parallel to the gravity direction of the vehicle V on which the system is mounted in the case where the vehicle V on which the system is mounted is movably and stably mounted on a horizontal surface. The longitudinal direction of the vehicle V on which the system is mounted (which will be referred to as the vehicle longitudinal direction) is defined as being perpendicular to both the vehicle width direction and the vehicle height direction.

[0078] In a vehicle V in which the system is installed, if Figure 1 The front direction, rear direction, left direction, and right direction are defined as shown in . That is, the longitudinal direction of the vehicle is synonymous with the front-rear direction, and the width direction of the vehicle is synonymous with the left-right direction. Depending on the situation where the vehicle V on which the system is installed is installed on any road and / or the condition under which the vehicle V on which the system is installed is traveling, the vehicle height direction may not be parallel to the gravity direction of the vehicle V on which the system is installed. The vehicle height direction is generally parallel to the gravity direction of the vehicle V on which the system is installed.

[0079] The vehicle body V1 defines therein an interior V2 serving as a compartment for one or more occupants including a driver of the vehicle V in which the system is installed.

[0080] The vehicle body V1 has a front end portion, a rear end portion, a left side portion, a right side portion, a top portion, and four corners including a left front corner, a right front corner, a left rear corner, and a right rear corner.

[0081] The vehicle V includes four wheels V3, i.e., four wheels V3a, V3b, V3c, and V3d mounted to the respective four corners of the vehicle body V1. Specifically, the wheel V3a, i.e., the left front wheel V3a, is mounted to the left front corner of the vehicle body V1, the wheel V3b, i.e., the right front wheel V3b, is mounted to the right front corner of the vehicle body V1, the wheel V3c, i.e., the left rear wheel V3c, is mounted to the left rear corner of the vehicle body V1, and the wheel V3d, i.e., the right rear wheel V3d, is mounted to the right rear corner of the vehicle body V1. The vehicle V on which the system is installed is not limited to such a four-wheeled motor vehicle, and a three-wheeled motor vehicle or a six-wheeled or eight-wheeled vehicle such as a truck can be used as the vehicle V on which the system is installed. The number of drive wheels among the wheels of the vehicle V on which the system is installed can be freely determined, and each drive wheel can be freely positioned to the vehicle body V1 of the vehicle V on which the system is installed.

[0082] The vehicle V on which the system is installed includes a front bumper V12 installed to the front end portion, i.e., the front side, of the vehicle body V1. The vehicle V on which the system is installed includes a rear bumper V14 installed to the rear end portion, i.e., the rear side, of the vehicle body V1. The vehicle body V1 of the vehicle V on which the system is installed includes a vehicle body panel V15 arranged to constitute the left and right sides and the top of the vehicle body V1. The vehicle body panel V15 includes a door panel V16. Figure 1 In the exemplary embodiment shown in , the vehicle body panel V15 includes a front pair of left and right door panels V16 and a rear pair of left and right door panels V16; the front pair and the rear pair of left door panels V16 are located at the left side portion of the vehicle body V1, and the front pair and the rear pair of right door panels V16 are located at the right side portion of the vehicle body V1. The vehicle V in which the system is installed includes a pair of door mirrors V17 mounted to the front pair of corresponding left and right doors V16. The vehicle body V1 includes a front windshield V18 covering the front side of the interior V2.

[0083] The front windshield V18 has a generally plate-like shape and is made of translucent glass or translucent synthetic resin. The front windshield V18 is attached to the vehicle body panel V15 and is inclined so that when viewed in a side view parallel to the vehicle width direction, the bottom of the front windshield V18 is positioned closer to the front of the vehicle body V1 than the top of the front windshield V18.

[0084] The vehicle V on which the system is installed includes an instrument panel V21 and a plurality of seats V22 including a driver's seat V23 on which a driver D sits. The instrument panel V21 is disposed in the front portion of the interior V2, and the seat V22 is located behind the instrument panel V21. The vehicle V on which the system is installed includes a steering wheel V24 located in front of the driver's seat V23. The driver D holds the steering wheel V24 and performs steering manipulation on the steering wheel V24 to thereby control the steering manipulation of the vehicle V on which the system is installed. The steering wheel V24 generally has a substantially annular shape, a substantially elliptical shape, or a substantially polygonal annular shape, but may have a rod-like shape or a joystick shape.

[0085] Overall configuration of vehicle systems

[0086] The vehicle V in which the system is installed includes a vehicle system 1 installed therein. The vehicle system 1 installed in the vehicle V in which the system is installed is configured to be used as a driving automation system or an autonomous driving system in the vehicle V in which the system is installed. The vehicle V in which the system is installed will also be referred to as its own vehicle V. The autonomous driving system is configured to implement one of the levels 1 to 5 included in all six autonomous driving levels (levels 0 to 5) defined in the SAEJ2016 standard issued by SAE International; SAE is the abbreviation of the "Society of Automotive Engineers". Any level X in the autonomous driving levels 0 to 6 will also be referred to as SOE level X. That is, the variable X can take any one of 0, 1, 2, 3, 4 and 5. The higher the SOE level X, the higher the driving automation level. In other words, the more dynamic driving tasks performed by the driving system, the higher the autonomous driving level. When the autonomous driving level is changed to a higher level, the autonomous driving level is improved. In contrast, the lower the SOE level X, the lower the autonomous driving level. In other words, the fewer the number of dynamic driving tasks performed by the autonomous driving system, the lower the autonomous driving level. When the autonomous driving level is changed to a lower level, the autonomous driving level is reduced.

[0087] Definition of driver assistance and autonomous driving

[0088] SAE levels 0 to 5 will be described in detail below.

[0089] In the following, the driver D is an occupant who processes and performs dynamic driving tasks. The dynamic driving tasks show all operating functions and all maneuvering functions that need to be performed in real time by the driver D when the driver D drives the own vehicle V on a traffic road, except for the strategic function. The overall driving action can be classified into operating functions, maneuvering functions, and strategic functions.

[0090] The strategic function may include a function of planning a travel itinerary and selecting one or more locations through the planned travel itinerary. Specifically, the strategic function may include a function of determining or selecting a travel itinerary that shows (i) whether to go to a destination, (ii) when to go to a destination, and (iii) how to go to a destination.

[0091] The maneuvering function may include the function of determining various maneuvers in various traffic situations, and the various maneuvers may include, during a predetermined driving period, (i) determining whether and when to overtake during a predetermined driving period, (ii) determining whether and when to change lanes, (iii) selectively setting an appropriate speed of the own vehicle V, and (iv) checking the rearview mirror.

[0092] The operational function may include the driver's immediate reaction, such as steering operation, braking operation, acceleration operation and / or minor adjustments to these operations to maintain the position of the own vehicle V in the corresponding lane of the road and / or avoid at least one obstacle and / or at least one dangerous event on the path of the moving own vehicle V. OEDR is an abbreviation for "object and event detection and response" and can be referred to as "object and event detection and response". OEDR includes monitoring of the driving environment around the own vehicle V. Monitoring of the driving environment around the own vehicle V may include detection, identification and classification of one or more objects and / or events. Monitoring of the driving environment around the own vehicle V may additionally include preparation for response to one or more objects and / or events. Operational design domain (ODD) conditions (which will also be referred to as "domain-specific conditions") represent specific conditions under which a given autonomous driving system or its features are designed to function. ODD conditions may include, for example, at least one of a plurality of restrictive conditions, the plurality of restrictive conditions including, for example, geographical conditions, environmental conditions, speed conditions and time conditions.

[0093] Level 0 means no autonomous driving, which means that the driver D performs all dynamic driving tasks.

[0094] Level 1 indicates driving assistance, which means that the autonomous driving system can continuously perform the lateral vehicle motion control subtask or the longitudinal vehicle motion control subtask in the dynamic driving task under specific ODD conditions. The longitudinal vehicle motion control subtask includes, for example, forward / reverse operation, acceleration / deceleration operation, and stop operation. The lateral vehicle motion control task includes, for example, steering operation. In particular, the autonomous driving system is configured not to perform both the lateral vehicle motion control subtask and the longitudinal vehicle motion control subtask.

[0095] Level 2 partial autonomous driving or advanced driving assistance, which means that the autonomous driving system can sustainably perform both the lateral vehicle motion control subtask and the longitudinal vehicle motion control subtask in the dynamic driving task under specific ODD conditions with the expectation that the driver D completes the OEDR subtask and supervises the autonomous driving system.

[0096] Level 3 indicates conditional autonomous driving, which means that the driving automation system can continuously perform all dynamic driving tasks under specific ODD conditions. Under specific ODD conditions, the driver D is not required to perform one of the OEDR subtasks of monitoring the traffic environment around the own vehicle V, but in the case where it is difficult for the driving automation system to continue at level 3, the driving automation system requests the driver D to control the own vehicle V with sufficient time, and the driver D needs to respond to the request smoothly.

[0097] Level 4 indicates high automation, which means that the driving automation system can continuously perform all dynamic driving tasks under specific ODD conditions. In the case where the driving automation system has difficulty continuing at level 4, the driving automation system will resolve the difficulty.

[0098] Level 5 indicates full automation, which means that the driving automation system on the own vehicle V can continuously perform all dynamic driving tasks without restriction under all ODD conditions. In the event that the driving automation system has difficulty continuing at level 5, the driving automation system resolves the difficulty without restriction under all ODD conditions.

[0099] The vehicle system 1 is configured to perform various driving control tasks and various notification tasks based on various driving tasks during driving of the own vehicle V. Specifically, the vehicle system 1 is configured as an autonomous driving system that implements driving assistance for the own vehicle V and / or autonomous driving of the own vehicle V. Autonomous driving corresponds to each of SAE levels 3 to 5. That is, autonomous driving in each of SAE levels 3 to 5 means that the vehicle system 1 is used as an autonomous driving system to perform all dynamic driving tasks in the corresponding level of SAE levels 3 to 5. In contrast, driving assistance corresponds to each of SAE levels 1 and 2. That is, autonomous driving in each of SAE levels 1 and 2 means that the vehicle system 1 is used as an autonomous driving system to perform a part of the dynamic driving tasks in the corresponding level of SAE levels 1 and 2. That is, driving assistance may include both SAE level 1 "driver assistance" and SAE level 2 "partial autonomous driving" or "advanced driving assistance", unless the expression "driving assistance of SAE level 1" is used or driving assistance is used to distinguish from partial autonomous driving of SAE level 2.

[0100] The vehicle system 1 of the exemplary embodiment may be configured to perform: (i) autonomous driving at each level of SAE Levels 3 to 5, (ii) partially autonomous driving at SAE Level 2, i.e., advanced driving assistance, and driving assistance at SAE Level 1. Driving assistance that may be performed by the vehicle system 1 of the exemplary embodiment may include hands-free driving. Hands-free driving enables the vehicle system 1 to automatically move the own vehicle V forward or backward, steer the own vehicle V, accelerate or decelerate the own vehicle, change lanes of the own vehicle V, and / or stop the own vehicle V, as long as the driver D appropriately handles the intervention request issued from the vehicle system 1.

[0101] Hands-off driving requests the driver D to monitor the road conditions around the own vehicle V, the traffic conditions around the own vehicle V, and information about whether there are one or more obstacles around the own vehicle V, without requesting the driver D to be in a hands-on state. The hands-on state indicates a state of the driver D in which the driver D is ready to perform steering manipulation on the own vehicle V, that is, ready to intervene in the lateral vehicle motion control subtask. The hands-on state generally indicates a state of the driver D in which the driver D sits on the driver's seat V23 in a posture capable of driving the own vehicle V and is ready to operate the steering wheel V24 with his / her hands. The driver D in the hands-on state holds the steering wheel V24 with his / her hands. The state in which the driver D touches the steering wheel V24 with his / her hands while being ready to hold the steering wheel V24 is applicable to the hands-on state. For example, the state in which the driver D is operating the steering wheel V24, that is, the state in which the driver D is actively operating the steering wheel V24 is applicable to the hands-on state. The state in which the driver D holds the steering wheel V24 relative to the control steering manipulation of the steering wheel V24 by the vehicle system 1 is applicable to the hands-on state.

[0102] Vehicle system configuration

[0103] The following reference Figure 1 as well as Figure 2 The overall configuration of the vehicle system 1 is described.

[0104] The vehicle system 1 includes: a driving electronic control unit (ECU) 2, a driving information input unit 3, a vehicle communication module (in other words, a data communication module (DCM)) 4, a high-definition (HD) map database 5, a navigation system 6, a human-machine interface (HMI) system 7, a lighting system 8 and a motion control system 9.

[0105] The vehicle system 1 is configured such that a driving ECU 2, a driving information input unit 3, a vehicle communication module 4, an HD map database 5, a navigation system 6, an HMI system 7, a lighting system 8, and a motion control system 9 are communicatively connected to each other via a vehicle communication network 10. The vehicle communication network 10 includes a main network conforming to one of various communication standards, such as a controller area network. (International Registration No. 1048262A). In addition to the main network of the vehicle communication network 10, the vehicle communication network 10 may also include a sub-network compliant with a local interconnect network (LIN) or FlexRay.

[0106] Driving ECU

[0107] The driving ECU 2 used as a control device according to the present disclosure (which is installed in a vehicle V in which the system is installed) is configured to control the overall operation in the vehicle system 1. The driving ECU 2 is configured as an autonomous driving / advanced driver assistance system (AD / ADAS) ECU used as both a driving assistance controller and an autonomous driving controller. Specifically, the driving ECU 2 includes at least one processor 21 and at least one memory device 22 connected in communication with the at least one processor 21.

[0108] At least one processor 21, which will be simply referred to as the processor 21, includes a central processing unit (CPU) or a microprocessing unit (MPU). At least one memory device 22 may include, for example, at least a read-only memory (ROM) and a random access memory (RAM) selected from various nonvolatile memory devices (such as ROM, RAM, and nonvolatile rewritable recording media). Such a recording medium may be referred to as a storage medium. These nonvolatile rewritable recording media, such as flash ROM or EEPROM, make the information stored therein rewritable when power is on, and keep the information unwritable when power is off. EEPROM is an abbreviation of electronically erasable and programmable ROM. At least one nonvolatile rewritable memory device in the ROM or nonvolatile rewritable memory device included in the memory device 22 pre-stores data and program instructions for the processor 21. The driving ECU 2 is configured to read the program instructions stored in the memory device 22, and execute the read program instructions to perform various tasks and operations accordingly, including the own vehicle control operation and the notification operation to the occupant.

[0109] Driving information input unit

[0110] The driving information input unit 3 is configured to input information required for the driving ECU 2 to perform various operations and tasks. Specifically, the driving information input unit 3 may include at least one sonar sensor 31, a radar sensor 32, a laser radar sensor 33, at least one camera 34, an operation sensor 35, a behavior sensor 36, a driver state monitor 37, an operation switch 38, and a locator 39. The sonar sensor 31, the radar sensor 32, the laser radar 33, and the camera 34 are collectively referred to as a surrounding monitor sensor or an ADAS sensor. The components of the driving information input unit 3 are described in turn below.

[0111] Sonar Sensor

[0112] At least one sonar sensor (which will be simply referred to as a sonar sensor) 31 is an ultrasonic distance sensor mounted to the vehicle body V1. The sonar sensor 31 is an ultrasonic sensor, and as Figure 3The sonar sensor 31 is configured to transmit a sonar detection wave Wsp in an ultrasonic frequency band toward an external space outside the own vehicle V. The sonar sensor 31 is configured to receive a sonar echo Wsr generated by reflection of the sonar detection wave Wsp by the target object B to detect various information about the target object B accordingly. In this specification, for the sake of convenience, a reference numeral B is assigned to any target object, but target objects assigned the same reference numeral B may not necessarily represent the same target object.

[0113] Specifically, the sonar sensor 31 is configured to calculate the distance of the object B from the sonar sensor 31 based on the time of flight (TOF) and the speed of sound. TOF represents the time defined from the time when the sonar detection wave (i.e., pulse) Wsp is emitted to the time when the sonar echo Wsr is received through the propagation path Ls of the pulses Wsp and Wsr; therefore, TOF will also be referred to as the propagation time.

[0114] If the driving information input unit 3 includes a pair of sonar sensors 31, that is, a pair of first sonar sensor 311 and second sonar sensor 312 (see Figure 4 ), then the known triangulation method is used to measure the relative distance of the target object B relative to the own vehicle V. Figure 4 In FIG. 1 , an X-axis is defined along a virtual line connecting between the first sonar 311 and the second sonar 312, and a Y-axis is defined to be perpendicular to the X-axis. The X-axis and the Y-axis extend along a reference horizontal plane perpendicular to the vehicle height direction.

[0115] The first sonar sensor 311 is configured to transmit a sonar detection wave Wsp, and each of the first sonar sensor 311 and the second sonar sensor 312 is configured to receive a sonar echo Wsr generated by reflecting the sonar detection wave Wsp from the target object B. The position of the target object B in a two-dimensional XY coordinate system composed of an X-axis and a Y-axis can be calculated based on a first TOF through a first propagation path Ls1 and a second TOF through a second propagation path Ls2. The first propagation path Ls1 is defined as a propagation path through which an ultrasonic wave (pulse) emitted from the first sonar 311 as a sonar detection wave Wsp propagates through the target object B to return to the first sonar 311 as a sonar echo Wsr. The ultrasonic wave (pulse) propagated through the propagation path Ls1 will also be referred to as a direct wave (pulse). The second propagation path Ls2 is defined as a propagation path through which an ultrasonic wave (pulse) emitted from the first sonar 311 as a sonar detection wave Wsp propagates through the target object B to reach the second sonar 312 as a sonar echo Wsr. The ultrasonic wave (pulse) propagating through the propagation path Ls2 will also be referred to as an indirect wave (pulse).

[0116] For example, the driving information input unit 3 according to the exemplary embodiment includes a plurality of sonar sensors 31 (see Figure 1 ). Specifically, the sonar sensor 31 includes a first front sonar SF1, a second front sonar SF2, a third front sonar SF3, and a fourth front sonar SF4 mounted to the front bumper V12. Similarly, the sonar sensor 31 includes a first rear sonar SR1, a second rear sonar SR2, a third rear sonar SR3, and a fourth rear sonar SR4 mounted to the rear bumper V14. In addition, the sonar sensor 31 includes a first side sonar SS1, a second side sonar SS2, a third side sonar SS3, and a fourth side sonar SS4. The first side sonar SS1 and the third side sonar SS3 are mounted to the left side portion of the vehicle body V1, and the second side sonar SS2 and the fourth side sonar SS4 are mounted to the right side portion of the vehicle body V1.

[0117] If there is no need to identify any one of the sonars SF1 to SS4 , the first to fourth front sonars SF1 to SF4 , the first to fourth rear sonars SR1 to SR4 , and the first to fourth side sonars SS1 to SS4 will also be simply collectively referred to as the sonar sensor 31 or multiple sonar sensors 31 .

[0118] Figure 5 A predetermined detection area, that is, a predetermined sensing area, of each of the sonar sensors 31 is shown.

[0119] The following reference Figure 1 and Figure 5 The sonar sensor 31 will be described in turn.

[0120] The first front sonar SF1 is mounted to a portion of the front bumper V12 that is closer to its left edge than the right edge of the front bumper V12 in the vehicle width direction, and is configured to emit a sonar detection wave Wsp obliquely to the left front. The second front sonar SF2 is mounted to a portion of the front bumper V12 that is closer to its right edge than the left edge of the front bumper V12 in the vehicle width direction, and is configured to emit a sonar detection wave Wsp obliquely to the right front. The first front sonar SF1 and the second front sonar SF2 are arranged symmetrically with respect to the vehicle center line LC. Each of the first front sonar SF1 and the second front sonar SF2 has a predetermined detection area Rsc, and the detection area Rsc of each of the first front sonar SF1 and the second front sonar SF2 (also referred to as the front corner detection area Rsc) is designed so that the detection range of the front corner detection area Rsc is set to, for example, 60 cm or thereabouts. The detection range of the sonar sensor 31 represents the maximum measurable range (distance) of the sonar sensor 31.

[0121] The third front sonar SF3 and the fourth front sonar SF4 are mounted to the middle portion of the front bumper V12 to be aligned in the vehicle width direction. Specifically, the third front sonar SF3 is arranged between the first front sonar SF1 and the vehicle center line LC, and is configured to emit a sonar detection wave Wsp substantially forward, and the fourth front sonar SF4 is arranged between the second front sonar SF2 and the vehicle center line LC, and is configured to emit a sonar detection wave Wsp substantially forward. The third front sonar SF3 and the fourth front sonar SF4 are arranged symmetrically with respect to the vehicle center line LC. Each of the third front sonar SF3 and the fourth front sonar SF4 has a predetermined detection area Rsf, and the detection area Rsf of each of the third front sonar SF3 and the fourth front sonar SF4 (also referred to as the front detection area Rsf) is designed so that the detection range of the front detection area Rsf is set to, for example, 1 m or about 1 m thereof.

[0122] The first front sonar SF1 and the third front sonar SF3 mounted to the front bumper V12 on the left side relative to the vehicle center line LC are arranged at different positions in the vehicle width direction, that is, in the horizontal direction. The first front sonar SF1 and the third front sonar SF3 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the first front sonar SF1 and the third front sonar SF3 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the first front sonar SF1 and the third front sonar SF3 by a target object as a reception echo.

[0123] Specifically, the first front sonar SF1 is arranged to receive (i) a direct echo generated by the target object reflecting the sonar detection wave Wsp emitted from the first front sonar SF1 and (ii) an indirect echo generated by the target object reflecting the sonar detection wave Wsp emitted from the third front sonar SF3. Similarly, the third front sonar SF3 is arranged to receive (i) a direct echo generated by the target object reflecting the sonar detection wave Wsp emitted from the third front sonar SF3 and (ii) an indirect echo generated by the target object reflecting the sonar detection wave Wsp emitted from the first front sonar SF1.

[0124] Similarly, the third front sonar SF3 and the fourth front sonar SF4 mounted to the middle portion of the front bumper V12 in the vehicle width direction are arranged at different positions in the vehicle width direction, i.e., in the horizontal direction. The third front sonar SF3 and the fourth front sonar SF4 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the third front sonar SF3 and the fourth front sonar SF4 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the third front sonar SF3 and the fourth front sonar SF4 by a target object as a reception echo.

[0125] The second front sonar SF2 and the fourth front sonar SF4 mounted to the front bumper V12 on the right side with respect to the vehicle center line LC are arranged at different positions in the vehicle width direction, i.e., in the horizontal direction. The second front sonar SF2 and the fourth front sonar SF4 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the second front sonar SF2 and the fourth front sonar SF4 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the second front sonar SF2 and the fourth front sonar SF4 by a target object as a reception echo.

[0126] The first rear sonar SR1 is mounted to a portion of the rear bumper V14 that is closer to its left edge than the right edge of the rear bumper V14 in the vehicle width direction, and is configured to emit a sonar detection wave Wsp obliquely to the left rear. The second rear sonar SR2 is mounted to a portion of the rear bumper V14 that is closer to its right edge than the left edge of the rear bumper V14 in the vehicle width direction, and is configured to emit a sonar detection wave Wsp obliquely to the right rear. The first rear sonar SR1 and the second rear sonar SR2 are arranged symmetrically with respect to the vehicle center line LC. Each of the first rear sonar SR1 and the second rear sonar SR2 has a predetermined detection area Rsd, and the detection area Rsd of each of the first rear sonar SR1 and the second rear sonar SR2 (also referred to as the rear corner detection area Rsd) is designed so that the detection range of the rear corner detection area Rsd is set to, for example, 60 cm or about 60 cm.

[0127] The third rear sonar SR3 and the fourth rear sonar SR4 are mounted to the middle portion of the rear bumper V14 to be aligned in the vehicle width direction. Specifically, the third rear sonar SR3 is arranged between the first rear sonar SR1 and the vehicle center line LC, and is configured to emit a sonar detection wave Wsp substantially to the rear, and the fourth rear sonar SR4 is arranged between the second rear sonar SR2 and the vehicle center line LC, and is configured to emit a sonar detection wave Wsp substantially to the rear. The third rear sonar SR3 and the fourth rear sonar SR4 are arranged symmetrically with respect to the vehicle center line LC. Each of the third rear sonar SR3 and the fourth rear sonar SR4 has a predetermined detection area Rsr, and the detection area Rsr of each of the third rear sonar SR3 and the fourth rear sonar SR4 (also referred to as the rear detection area Rsr) is designed so that the detection distance of the rear detection area Rsr is set to, for example, 1.5 m or about 1.5 m.

[0128] The first rear sonar SR1 and the third rear sonar SR3 mounted to the rear bumper V14 on the left side with respect to the vehicle center line LC are arranged at different positions in the vehicle width direction, i.e., the horizontal direction. The first rear sonar SR1 and the third rear sonar SR3 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the first rear sonar SR1 and the third rear sonar SR3 to receive a sonar echo generated by a target object reflecting a sonar detection wave Wsp emitted from the other of the first rear sonar SR1 and the third rear sonar SR3 as a reception echo.

[0129] Specifically, the first rear sonar SR1 is arranged to receive (i) a direct echo generated by the target object reflecting the sonar detection wave Wsp emitted from the first rear sonar SR1 and (ii) an indirect echo generated by the target object reflecting the sonar detection wave Wsp emitted from the third rear sonar SR3. Similarly, the third rear sonar SR3 is arranged to receive (i) a direct echo generated by the target object reflecting the sonar detection wave Wsp emitted from the third rear sonar SR3 and (ii) an indirect echo generated by the target object reflecting the sonar detection wave Wsp emitted from the first rear sonar SR1.

[0130] Similarly, the third rear sonar SR3 and the fourth rear sonar SR4 mounted to the middle portion of the rear bumper V14 in the vehicle width direction are arranged at different positions in the vehicle width direction, i.e., in the horizontal direction. The third rear sonar SR3 and the fourth rear sonar SR4 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the third rear sonar SR3 and the fourth rear sonar SR4 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the third rear sonar SR3 and the fourth rear sonar SR4 by a target object as a reception echo.

[0131] The second rear sonar SR2 and the fourth rear sonar SR4 mounted to the rear bumper V14 on the right side with respect to the vehicle center line LC are arranged at different positions in the vehicle width direction, i.e., in the horizontal direction. The second rear sonar SR2 and the fourth rear sonar SR4 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the second rear sonar SR2 and the fourth rear sonar SR4 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the second rear sonar SR2 and the fourth rear sonar SR4 by a target object as a reception echo.

[0132] Each of the first side sonar SS1 and the third side sonar SS3 is mounted to a portion of the left side portion of the vehicle body V1, and is configured to transmit the sonar detection wave Wsp to the left side relative to the own vehicle V. Similarly, each of the second side sonar SS2 and the fourth side sonar SS4 is mounted to a portion of the right side portion of the vehicle body V1, and is configured to transmit the sonar detection wave Wsp to the right side relative to the own vehicle V. Each of the first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4 is arranged to receive only a direct echo generated by reflection of the sonar detection wave Wsp emitted from a corresponding one of the first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4. Each of the first side sonar SS1, the second side sonar SS2, the third side sonar SS3 and the fourth side sonar SS4 has a predetermined detection area Rss, and the detection area Rss of each of the first side sonar SS1 to the fourth side sonar SS4 (also referred to as the side detection area Rss) is designed so that the detection distance of the side detection area Rss is set to, for example, within the range from 2m to 3m, both inclusive.

[0133] The first side sonar SS1 is arranged between the first front sonar SF1 and the door mirror V17 mounted to the front left door panel V16, which will also be referred to as the left door mirror V17. The first side sonar SS1 is configured to emit a sonar detection wave Wsp to the left relative to the own vehicle V. The second side sonar SS2 is arranged between the second front sonar SF2 and the door mirror V17 mounted to the front right door panel V16, which will also be referred to as the right door mirror V17. The second side sonar SS2 is configured to emit a sonar detection wave Wsp to the right relative to the own vehicle V. The first side sonar SS1 and the second side sonar SS2 are arranged symmetrically relative to the vehicle center line LC. The first side sonar SS1 and the second side sonar SS2 may be mounted to the vehicle body panel V15, or to portions of the corresponding left and right edges of the front bumper V12 in the vehicle width direction; a corresponding one of the first side sonar SS1 and the second side sonar SS2 extends rearward in the vehicle longitudinal direction to a portion of each of the left and right edges of the front bumper V12.

[0134] The third side sonar SS3 is arranged between the first rear sonar SR1 and the rear left door panel V16. The third side sonar SS3 is configured to emit sonar detection waves Wsp to the left relative to the own vehicle V. The fourth side sonar SS2 is arranged between the second rear sonar SR2 and the rear right door panel V16. The fourth side sonar SS4 is configured to emit sonar detection waves Wsp to the right relative to the own vehicle V. The third side sonar SS3 and the fourth side sonar SS4 are arranged symmetrically with respect to the vehicle center line LC. The third side sonar SS3 and the fourth side sonar SS4 can be mounted to the body panel V15, or to the corresponding left and right edge portions of the rear bumper V14 in the vehicle width direction; the corresponding one of the third side sonar SS3 and the fourth side sonar SS4 extends forward in the vehicle longitudinal direction to each of the left and right edges of the rear bumper V14.

[0135] Figure 6 An example is schematically shown in which the driving ECU 2 uses the first side sonar SS1 to detect a parking space or parking spot PS for parallel parking on the left side relative to the own vehicle V. The parking space PS is a rectangular frame-shaped area in a plan view and indicates a space where a vehicle can be parked. Figure 6 1 shows a sonar detection point Psr, which is a point where it is estimated that the sonar detection wave Wsp emitted from any sonar sensor 31 will be reflected; the sonar detection point Psr may also be referred to as a ranging point Psr. Figure 6 As shown, the driving ECU 2 can detect the parking space PS based on the distribution of sonar detection points Psr acquired based on sonar echoes Wsr from target objects B that are parked vehicles arranged side by side in the straight forward moving direction of the own vehicle V.

[0136] Specifically, when Figure 6 When acquiring the sonar detection points Psr of adjacent rows arranged in the traveling direction of the own vehicle V in the case of performing parallel parking of the own vehicle V as shown, the driving ECU 2 can determine whether the length of the space between each adjacent row pair of the sonar detection points Psr exceeds the entire width of the own vehicle V, and in response to determining that the length of the space between at least one adjacent row pair of the sonar detection points Psr exceeds the entire width of the own vehicle V, detect the space between at least one adjacent row pair of the sonar detection points Psr as the position of the parking space PS.

[0137] Radar Sensor

[0138] Reference Figure 1 and Figure 2The radar sensor 32 is configured to transmit millimeter radar waves or submillimeter radar waves and receive reflected radar waves to detect the target object B accordingly. The radar sensor 32 of the exemplary embodiment is mounted to the middle of the front portion V11 of the vehicle body V1 in the vehicle width direction. Figure 7 The radar sensor 32 shown is a long-range radar having a substantially fan-shaped radar detection area Rg1 in a plan view. The fan-shaped radar detection area Rg1 has a predetermined detection range from 10m to 250m (inclusive), and has a radar scanning angle θr1 around the forward moving direction Df of the own vehicle V. The radar sensor 32 is configured as a phased array radar sensor or a beamforming radar sensor. The forward moving direction Df indicates the direction of a virtual line extending forward from the vehicle center line LC, and indicates the traveling direction of the own vehicle V when the current shift position of the shift lever is set to any position other than the reverse position. The forward moving direction Df of the own vehicle V matches the traveling direction of the own vehicle V when the own vehicle V is moving straight forward, and becomes a tangent direction to the traveling path of the own vehicle V when the own vehicle V is traveling around a curve.

[0139] The radar sensor 32 is configured to transmit a radar detection wave Wrp, scan the transmitted radar detection wave Wrp within a radar scanning angle θr1 , and receive a radar wave Wrr generated by reflection of the radar detection wave Wrp by a target object B located in the radar detection area Rg1 .

[0140] Specifically, the radar sensor 32 includes an FMCW radar device equipped with an array antenna, and is configured to detect (i) the distance from it to the target object B, (ii) the azimuth angle θa of the target object B, and (iii) the relative speed of the target object B with respect to the own vehicle V, based on the frequency difference between the transmitted millimeter wave and the received millimeter wave and / or the phase difference between the transmitted millimeter wave and the received millimeter wave. The azimuth angle θa of the target object B is defined as the angle formed by a first virtual line generated by extending the vehicle center line LC toward the front of the own object V and a second virtual line connecting the radar sensor 32 and the target object B; the first center line CL1 represents the center line of the radar detection range Rg1.

[0141] The relative speed of the target object B with respect to the own vehicle V is defined as the difference between the moving speed vb of the target object B and the traveling speed vm of the own vehicle V.

[0142] Specifically, the radar sensor 32 is configured to transmit a radar detection wave Wrp generated based on a transmission signal having a predetermined modulation frequency, and receive a radar wave Wrr generated by reflecting the radar detection wave Wrp from the target object B, so as to detect a received signal representing the frequency characteristics of the received radar wave Wrr based on the received radar wave Wrr. Then, the radar sensor 32 is configured to calculate the deviation between the modulation frequency of the transmission signal and the frequency of the received signal, so as to generate a beat signal based on the corresponding frequency deviation. The radar sensor 32 is configured to perform a first Fourier transform on the beat signal, so as to calculate the frequency-power spectrum of each beat signal in the beat signal. Then, the radar sensor 32 is configured to analyze the frequency-power spectrum of each beat signal, so as to obtain the beat frequency, and calculate the distance of the target object B from the radar sensor 32 and the relative speed of the target object B relative to the own vehicle V based on the beat frequency.

[0143] In addition to the long-range radar sensor 32, a medium-range radar sensor or a short-range radar sensor may be installed in the vehicle system 1. Such a medium-range radar sensor has, for example, a predetermined detection range from 1 m to 100 m (inclusive), and such a short-range radar sensor has, for example, a predetermined detection range from 15 cm to 30 m (inclusive).

[0144] LiDAR Sensor

[0145] like Figure 8 As shown, the laser radar sensor 33 is configured to illuminate the exterior of the own vehicle V with detection light Lp, i.e., laser light in the infrared band, and receive reflected light Lr generated by the target object B reflecting the detection light Lp, so as to detect the target object B accordingly. The laser radar sensor 33 may also be referred to as LIDAR, which is an abbreviation of light detection and ranging or laser imaging detection and ranging. The vehicle system 1 of the exemplary embodiment includes a long-range laser radar sensor 33 including a scanning LIDAR as the laser radar sensor 33. As shown in FIG. Figure 1 As shown, the laser radar sensor 33 is mounted to the center of the front portion V11 of the vehicle body V1 in the vehicle width direction.

[0146] The laser radar sensor 33 has a substantially fan-shaped LIDAR detection area Rg2 in a plan view. The fan-shaped radar detection area Rg2 has a predetermined radius of, for example, 200 m or more, and has a LIDAR scanning angle θr2 around the forward moving direction Df of the own vehicle V. The laser radar sensor 33 is configured to horizontally scan the laser detection light Lp within the LIDAR scanning angle θr2 to detect the target object B located in the LIDAR detection area Rg2 accordingly.

[0147] Fig. 9 2 shows a schematic configuration of the laser radar sensor 33. The laser radar 33 includes a light emitting unit 331, a scanning unit 332, and a light receiving unit 333.

[0148] The light emitting unit 331 is configured to emit the detection light Lp. The scanning unit 332 includes a MEMS mirror unit and a MEMS mechanism, wherein the MEMS mirror unit includes at least one mirror located on the optical path of the detection light Lp emitted by the light emitting unit 331, and the MEMS mechanism, for example, rotates at least one mirror to correspondingly change the direction of the light reflected by the at least one mirror; MEMS is the abbreviation of micro-electromechanical system. Specifically, the scanning unit 332 is configured to electrically control the MEMS mechanism so that the MEMS mechanism rotates at least one mirror, thereby scanning the detection light Lp emitted by the light emitting unit 331 in both the horizontal scanning direction Ds and the vertical scanning direction Dh.

[0149] The light receiving unit 333 includes a light receiving sensor 334 as a two-dimensional image sensor. Specifically, the light receiving sensor 334 includes a plurality of light receiving elements 335 arranged two-dimensionally in a horizontal direction corresponding to the horizontal scanning direction Ds and in a vertical direction corresponding to the vertical scanning direction Dh. Each of the light receiving elements 335 including an avalanche photodiode (APD) or a single photon avalanche diode (SPAD) is configured to detect a corresponding portion of the reflected light Lr generated by at least one target object B reflecting the detection light Lp.

[0150] The laser radar sensor 33 is configured to generate at least one detection point data cloud based on the reflected light Lr received by the light receiving unit 333, and detect at least one target object B based on the at least one detection point data cloud.

[0151] At least one detection point data cloud represents a two-dimensional array of a plurality of LIDAR detection points Prr, such as an image (i.e., a frame image), which are close to each other and arranged two-dimensionally in the horizontal scanning direction Ds and the vertical scanning direction Dh (see Fig. 9 ). Each LIDAR detection point Prr arranged to the corresponding light receiving element 335 represents a point of at least one target object B estimated to be a part of the reflected detection light Lp; a part of the reflected light Lr received by the corresponding light receiving element 335 corresponds to a part of the detection light Lp. Therefore, this results in the LIDAR detection points Prr arranged two-dimensionally in the horizontal scanning direction Ds and the vertical scanning direction Dh including position information and range information on a two-dimensional array. Therefore, this enables at least one detection point data cloud to also be referred to as at least one range point data cloud.

[0152] That is, the laser radar sensor 33 is configured to detect at least one detection point data cloud including LIDAR detection points Prr arranged two-dimensionally like a frame image in the horizontal scanning direction Ds and the vertical scanning direction Dh. Therefore, this enables the laser radar sensor 33 to be also called an image sensor.

[0153] That is, if there are target objects B, the laser radar sensor 33 is configured to detect, for each of the target objects B, a detection point data cloud including LIDAR detection points Prr arranged two-dimensionally like a frame image in the horizontal scanning direction Ds and the vertical scanning direction Dh.

[0154] Camera device

[0155] Reference Figure 1 and Figure 2 , at least one camera device 34 serving as an image sensor is mounted to the own vehicle V. The at least one camera device 34 is configured to capture images of a peripheral field of view around the own vehicle V while moving together with the own vehicle V. That is, the at least one camera device 34 is configured to generate image information about each captured image around the own vehicle V, which will also be referred to as image data.

[0156] The at least one camera device 34 is configured as a digital camera device including an image sensor such as a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. The image sensor of the at least one camera device 34 includes a plurality of photosensitive elements such as photodiodes, which correspond to a plurality of pixels arranged two-dimensionally in both the vertical direction (i.e., vehicle height direction) and the horizontal direction (i.e., vehicle width direction) of the own vehicle V, respectively.

[0157] The driving information input unit 3 of the exemplary embodiment includes a plurality of cameras, namely, a front camera CF, a rear camera CB, a left camera CL, and a right camera CR mounted to the own vehicle V. If it is not necessary to identify any of the cameras CF, CR, CL, and CR, the front camera CF, the rear camera CR, the left camera CL, and the right camera CR will also be simply collectively referred to as one or more cameras 34.

[0158] The front camera device CF is mounted to the upper end of the front windshield V18 approximately in the middle of the vehicle width direction at the top of the interior V2, and has a forward field of view in front of the own vehicle V. That is, the front camera device CF is located on the vehicle center line LC in a plan view. The front camera device CF can be mounted to the front portion V11 of the vehicle body V1. The front camera device CF is configured to capture an image of the forward field of view to obtain information about the captured image of the forward field of view accordingly.

[0159] The rear camera CB is mounted to the substantially middle of the rear end V13 of the vehicle body V1 in the vehicle width direction and has a rear view field behind the own vehicle V. The rear camera CR is configured to capture an image of the rear view field to accordingly acquire information about the captured image of the rear view field.

[0160] The left camera CL is mounted to the left door mirror V17 and has a left field of view at the left side of the own vehicle V. The left camera CL is configured to capture an image of the left field of view to correspondingly acquire information about the captured image of the left field of view.

[0161] The right camera CR is mounted to the right door mirror V17 and has a right field of view at the right side of the own vehicle V. The right camera CR is configured to capture an image of the right field of view to correspondingly acquire information about the captured image of the right field of view.

[0162] The image captured by each camera 34 includes two-dimensionally arranged pixels, which respectively correspond to the two-dimensionally arranged photosensitive elements of the corresponding camera 34 .

[0163] The driving ECU 2 can recognize the target object B based on the image captured by any of the cameras 34 , that is, determine at least the position of the target object B and the type of the target object B based on the image captured by any of the cameras 34 .

[0164] Fig.10 An example is shown of how the driving ECU 2 recognizes a target object B included in a front field of view based on an image of the front field of view captured by the front camera CF; the front field of view includes a road Rd in front of the own vehicle V. In the specification, for convenience, reference numeral Rd is assigned to any road, but roads assigned the same reference numeral Rd may not necessarily represent the same road.

[0165] Specifically, the driving ECU 2 sets a detection area Aw in the captured image. The detection area Aw represents a part or all of the entire area of ​​the captured image, that is, the entire viewing angle area of ​​the front camera device CF. Then, the driving ECU 2 obtains the feature point image Gp based on the pixel feature parameters of the image data included in the detection area Aw. The pixel feature parameters of the image data represent the feature parameters of each pixel constituting the image data based on the corresponding received light, and may include, for example, the brightness, contrast, and hue of each pixel of the image data. The brightness of each pixel may be referred to as the brightness of the corresponding pixel, and the hue of each pixel may be referred to as the chromaticity of the corresponding pixel.

[0166] The feature point image Gp includes feature points Pt arranged two-dimensionally in the horizontal and vertical directions like the frame image; each of the feature points Pt is extracted from the image data based on, for example, the difference and / or change gradient between corresponding adjacent pixel pairs of the image data included in the detection area Aw. The feature point Pt characterizes the shape of the target object B included in the detection area Aw of the captured image. In other words, the feature point Pt is a characteristic point of the image data included in the detection area Aw, that is, a characteristic pixel.

[0167] The driving ECU 2 performs a pattern matching task of matching one or more feature point clouds Pg (each feature point cloud Pg is a set of corresponding feature points Pt) with a predetermined pattern stored therein, so as to identify, for each of the feature point clouds Pg, the type of a corresponding one of the target objects B based on the corresponding one of the feature point clouds Pg.

[0168] Various methods for extracting feature points Pt from the image data included in the detection area Aw are known. For example, an extraction method using a Sobel filter, an extraction method using a Laplacian filter, an extraction method using a Canny algorithm can be used to extract feature points Pt from the image data included in the detection area Aw. Therefore, a detailed description of one of the known extraction methods used in the specification is omitted. Extracting feature points Pt from the image data included in the detection area Aw can be expressed as detecting feature points Pt from the image data included in the detection area Aw.

[0169] The driving ECU 2 may estimate the relative position of the recognized target object B with respect to the own vehicle V and / or the distance of the recognized target object B with respect to the own vehicle V, ie, the range.

[0170] Fig.11 An example is shown in which the driving ECU 2 calculates the estimated point Pb as the estimated result of the three-dimensional position of each feature point Pt corresponding to the target object B using, for example, a monocular motion stereo method, in other words, a structure from motion (SFM) method. The present disclosure may calculate the estimated point Pb as the estimated result of the three-dimensional position of each feature point Pt corresponding to the target object B using one of the other calculation methods selected. That is, each camera 34 may be configured as a compound eye stereo camera.

[0171] Assume that Fig.11As shown, the camera 34 moves together with the own vehicle V toward the own vehicle moving direction Dv of the own vehicle V. The position of the camera 34 at time t1 is referred to as a first camera position Pc1, and the position of the camera 34 at time t2 after time t1 is referred to as a second camera position Pc2. The first captured image A1 is an image captured by the camera 34 at the first camera position Pc1, and the second captured image A2 is an image captured by the camera 34 at the second camera position Pc2.

[0172] The first feature point Pt1 is a feature point Pt extracted from the first captured image A1, and the second feature point Pt2 is extracted from the second captured image A2 and is estimated to correspond to the first feature point Pt1 at time t1. That is, the second feature point Pt2 is a point on the target object B to which the point corresponding to the first feature point Pt1 is estimated to have moved within the elapsed time (t2-t1) from time t1 to time t2. The driving ECU 2 can use one of the well-known methods such as the optical flow method to determine whether the first feature point Pt1 and the second feature point Pt1 are based on the same point on the target object B, that is, whether the first feature point Pt1 on the target object B corresponds to the second feature point Pt2 thereon. Then, the driving ECU 2 defines a first line L1 passing through the first camera position Pc1 and the first feature point Pt1 and a second line L2 passing through the second camera position Pc2 and the second feature point Pt2, and calculates the intersection of the first line L1 and the second line L2 as the estimated point Pb. The estimated point Pb represents a point on the target object B corresponding to both the first feature point Pt1 and the second feature point Pt2 in a three-dimensional coordinate system defined relative to the own vehicle V. If the estimated point Pb is a stationary point, the estimated point Pb satisfies the epipolar constraint. The epipolar constraint is an epipolar geometric constraint, that is, the first camera position Pc1, the second camera position Pc2, and the estimated point Pb are located on the same plane Π at any time t1 or t2.

[0173] Various sensors

[0174] Reference Figure 1 and Figure 2, the operation sensors 35 are each arranged in the own vehicle V, and are used to output parameters indicating the corresponding driver operation state of the own vehicle V. The parameters to be output from each operation sensor 35 (each indicating the corresponding driver operation state of the own vehicle V) may include, for example, (i) the driver operation amount of the accelerator pedal of the own vehicle V, (ii) the driver operation amount of the brake pedal of the own vehicle V, (iii) the current gear position set by the driver of the gear lever, (iv) the driver operation steering angle of the steering wheel V24, and (v) the steering torque applied by the driver of the steering wheel V24. That is, the operation sensor 35 includes known sensors, including, for example, an accelerator pedal sensor, a brake pedal sensor, a gear position sensor, a steering angle sensor, and a steering torque sensor. That is, for the sake of simple explanation and simple description, these known sensors are collectively referred to as the operation sensor 35. The operation sensor 35 may include a steering wheel sensor, which is used to detect information indicating whether the driver D holds the steering wheel V24.

[0175] The behavior sensors 36 are each provided in the own vehicle V, and are used to output parameters indicating the corresponding driving behavior of the own vehicle V. The parameters to be output from the respective behavior sensors 36 (each indicating the corresponding behavior of the own vehicle V) may include, for example, (i) the speed of the own vehicle V, (ii) the yaw rate of the own vehicle V, (iii) the acceleration of the own vehicle V in the longitudinal direction, and (iv) the acceleration of the own vehicle V in the vehicle width direction. That is, the behavior sensors 36 include known sensors, including, for example, a vehicle speed sensor, a yaw rate sensor, and an acceleration sensor. That is, for the sake of simple explanation and simple description, these known sensors are collectively referred to as behavior sensors 36.

[0176] Driver Status Monitor

[0177] Reference Figure 1 , Figure 2 , Fig.12 and Fig.13 The driver state monitor 37 is configured to sequentially capture images of the driver D and sequentially detect driver state parameters representing the driver's state based on the captured images of the driver D. The driver's condition includes, for example, information indicating whether the driver D is in a sober state.

[0178] Specifically, the driver state monitor 37 includes a driver monitor camera having a predetermined field of view; the driver monitor camera is located in the interior V2 so that at least the head D1 of the driver D2 sitting on the driver seat V23 is located within the field of view of the driver monitor camera. This enables the driver monitor camera to capture an image of the face D2 of the driver D from the front. The driver monitor camera may be configured as a near infrared camera.

[0179] The driver state monitor 37 comprises an image processing unit configured to perform image processing tasks on images captured by the driver monitor camera to detect the driver's state parameters accordingly.

[0180] like Fig.12 and Fig.13 As shown, the state parameters of the driver to be detected by the driver state monitor 37 include, for example, (i) the direction of the face D2 of the driver D, (ii) the degree of opening of each driver's eye D3, and (iii) the position of each pupil D4 of the driver D. The direction of the face D2 of the driver D is defined by the deflection angle θy and the pitch angle θp of the face D2 of the driver D. The deflection angle θy represents the rotation angle of the face D2 of the driver D around the vertical axis Dx1, which extends vertically through the head D1 of the driver D. When the face D2 of the driver D faces forward, the deflection angle θy is 0°. That is, when the face D2 of the driver D faces the left side relative to the front, the deflection angle θy becomes a corresponding positive angle, and when the face D2 of the driver D faces the right side relative to the front, the deflection angle θy becomes a corresponding negative angle.

[0181] The pitch angle θp represents the rotation angle of the face D2 of the driver D around the horizontal axis Dx2, which extends horizontally through the face D2 of the driver D. When the face D2 of the driver D faces forward, the pitch angle θp is 0°. That is, when the face D2 of the driver D is upward relative to the front, the pitch angle θp becomes a corresponding positive angle, and when the face D2 of the driver D is downward relative to the front, the pitch angle θp becomes a corresponding negative angle.

[0182] Fig.12 is an example of an image captured by the driver state monitor 37 in which the driver D is awake, and Fig.13 is an example of an image captured by the driver state monitor 37 in which the driver D is not in a wakeful state. Fig.12 and Fig.13The driver status parameters, (i) the direction of the driver D's face D2, (ii) the degree of openness of each driver's eye D3, and (iii) the position of each pupil D4 of the driver D are shown, so that the driving ECU 2 can detect the driver's status, including, for example, (i) information about whether the driver D is in a waking state and (ii) information about the driver's line of sight D5.

[0183] Operation switch

[0184] Reference Figure 1 and Figure 2 The operation switches 38 are various switches that can be operated by the driver E when the driver E drives the own vehicle E, and are provided at predetermined positions in the interior V2. Each of the operation switches 38 is a switch whose operation amount and operation state are exempted from detection by the operation sensor 35. Specifically, the operation switch 38 includes, for example, an ignition switch 381, a flasher switch 382, ​​and one or more AD / ADAS switches 383. The ignition switch 381, which may be referred to as a start switch or a power switch, is a switch for activating or deactivating the system-mounted vehicle V (i.e., the vehicle system 1). The flasher switch 382 is configured to detect the operation state of the flasher lever of the own vehicle V by the driver. The AD / ADAS switch 383 is a switch that enables the driver E to input various instructions related to the driving assistance or autonomous driving of the own vehicle V to the driving ECU 2 when operating them; the various instructions related to the driving assistance or autonomous driving of the own vehicle V include, for example, a start instruction, a stop instruction, a level setting instruction, and a function selection instruction.

[0185] The locator 39 is configured to obtain high-accuracy position information about the own vehicle V, which position information will also be referred to as complex position information. Specifically, the locator 39 is configured as a complex positioning system for obtaining complex position information about the own vehicle V, and includes a GNSS receiver 391, an inertial detector 392 and a locator ECU 393.

[0186] GNSS is an abbreviation for Global Navigation Satellite System, and the high-accuracy position information about the own vehicle V is position information about the own vehicle V that has at least a position accuracy that can be used by advanced driving assistance of SAL level 2 or higher, in other words, a position accuracy with an error of less than or equal to 10 cm. As the positioner 39, a commercially available positioning system such as the POSLV system for land vehicles, in other words, a positioning orientation system for land vehicles manufactured by Applanix Corporation can be used.

[0187] The GNSS receiver 391 may be configured to receive a navigation signal transmitted from at least one positioning satellite (i.e., at least one artificial satellite). Specifically, the GNSS receiver 391 is configured to be able to receive a navigation signal from a positioning satellite included in at least one GNSS selected from GPS, QZSS, GLONASS, GLONASS, Galileo, IRNSS, and BeiDou navigation satellite systems. GPS is an abbreviation for Global Positioning System, QZSS is an abbreviation for Quasi-Zenith Satellite System, GLONASS is an abbreviation for Global Navigation Satellite System, and IRNSS is an abbreviation for Indian Regional Navigation Satellite System.

[0188] The inertial detector 392 is configured to detect (i) linear accelerations acting on the own vehicle V in respective three axes corresponding to the vehicle longitudinal direction, the vehicle width direction, and the vehicle height direction, and (ii) angular velocities acting on the own vehicle V around the respective three axes. For example, the positioner 39 has a substantially box-shaped housing, and the inertial detector 392 includes a three-axis accelerometer and a three-axis gyro sensor installed in the housing.

[0189] The locator ECU 393 includes a vehicle microcomputer including a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The locator ECU 393 is configured to sequentially determine the current position and / or current orientation of the own vehicle V based on the navigation signal received by the GNSS receiver 391 and the linear acceleration and angular velocity detected by the inertial detector 392.

[0190] Vehicle Communication Module

[0191] The vehicle communication module 4 , which will also be referred to as DCM 4 , may be configured to communicate information with base stations located around the own vehicle V using wireless communications conforming to a predetermined communication standard such as Long Term Evolution (LTE) or fifth generation (5G).

[0192] Specifically, the vehicle communication module 4 is configured to obtain traffic information, such as traffic jam information, from a detection server and / or a predetermined database in a cloud computing environment. The traffic jam information includes, for example, the location and length of at least one traffic jam section. Specifically, the traffic jam information includes, for example, various information items about at least one traffic jam section, such as the head of at least one traffic jam section, the tail of at least one traffic jam section, the estimated length of at least one traffic jam section, and the estimated time of at least one traffic jam section. The traffic information will also be referred to as road traffic information.

[0193] Additionally, the vehicle communication module 4 is configured to retrieve the latest HD map information from at least one of the probe servers and store the HD map information in the HD map database 5 .

[0194] HD Map Database

[0195] The HD map database 5 mainly includes one or more nonvolatile rewritable memories, and is configured to store rewritable HD map information while maintaining the stored HD map information even if power is cut off to the HD map database 5. The HD map information will also be referred to as HD map data.

[0196] The HD map information includes higher definition map information than the map information stored in the standard (SD) map database 601 of the navigation system 6. That is, the higher definition map information has a position error lower than or equal to an error of about several meters of the map information stored in the SD map database 601.

[0197] Specifically, the HD map information database 5 stores, as HD map information, map information usable for example for advanced driving assistance or autonomous driving, including, for example, (i) information on three-dimensional road shapes, (ii) information on the number of lanes in each road, and (iii) information on road traffic regulations. The HD map information is stored in the HD map information database 5 to comply with a predetermined standard, such as ADASIS.

[0198] Navigation system

[0199] The navigation system 6 is configured to calculate a planned route from the current position of the own vehicle V to the destination. The navigation system 6 of the exemplary embodiment is configured to calculate the planned route based on (i) a destination input by the driver D through, for example, the HMI system 7, (ii) HD map information stored in the HD map database 5 or SD map information stored in the SD map database 601, and (iii) position information about the own vehicle V, such as the current position and current orientation of the own vehicle V. The navigation system 6 is also configured to provide various information including the planned route to one or more selected components of the vehicle system 1, such as the driving ECU 2 and / or the HMI system 7, through the vehicle communication network 10. That is, the navigation system 6 is capable of instructing the HMI system 7 to sequentially display navigation images showing, for example, a map on which the current position of the own vehicle V and the planned route appear, respectively.

[0200] HMI System

[0201] The HMI system 7 is designed as a vehicle HMI system, and is configured to enable information communication between the own vehicle V and one or more occupants including a driver D of the own vehicle V.

[0202] Specifically, the HMI system 7 is configured to at least visually provide (i.e., display) various information items to one or more passengers, and enable the passengers to input information input related to the provided information items. The various information items to be provided to one or more passengers include, for example, various guide information items, information items about input operation guidance, notifications of input information, and / or warnings.

[0203] The HMI system 7 typically includes I / O components mounted to the steering wheel V24 or mounted in the dashboard V21 (the so-called "dashboard HMI"). At least one of the I / O components of the HMI system 7 may be mounted to at least one portion of the interior V2 other than the dashboard V21, such as a unit in the interior V2 or a center console between the driver's seat V23 and a passenger seat V22 adjacent to the driver's seat V23.

[0204] The HMI system 7 includes an HMI control unit (HCU) 701 , an instrument panel 702 , a main display device 703 , a head-up display 704 , a speaker 705 , and an operating device 706 .

[0205] The HCU 701 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The HCU 701 is configured to perform overall control of display output and / or auditory output through the HMI system 7. That is, the HCU 701 is configured to control the operation of each of the instrument panel 702, the main display device 703, the head-up display 704, and the speaker 705.

[0206] The instrument panel 702 is installed in the instrument panel V21 so as to be arranged to face the driver's seat V23. The instrument panel 702 is configured to display metering values ​​including, for example, the speed of the own vehicle V, the temperature of the coolant, and the fuel level. The instrument panel 702 is also configured to display various information items including, for example, the current date and time, the outside temperature, and the receivable radio broadcasts.

[0207] The main display device 703, also known as a central information display (CID) device, is installed in the middle of the instrument panel V23 in the vehicle width direction, which enables one or more passengers to visually recognize information displayed thereon.

[0208] The main display device 703 has a housing and a screen mounted on the housing, and can be configured to continuously display a navigation image generated by the navigation system 6 on the screen, the navigation image showing, for example, a map on which the current position and the planned travel route of the own vehicle V appear respectively. The main display device 703 can also be configured to display various information and content different from the navigation image on the screen. For example, the main display device 703 can be configured to display a driving mode setting image on which icons of multiple driving modes are selectively displayed; the multiple driving modes include a comfort driving mode, a normal driving mode, a sports driving mode, and a circuit driving mode. The main display device 703 is also configured to display a second task image on the screen on which icons of multiple second tasks are selectively displayed; the second task other than the driving task of the own vehicle V can be used by the driver D during the autonomous driving of the own vehicle V. For example, the second task includes (i) a task of reading a digital book, (ii) a task of operating a mobile communication terminal, and (iii) a task of watching video content (e.g., a movie, a concert video, a music video, or a television broadcast). The second task can be referred to as a secondary activity or other task.

[0209] Reference Figure 1 , Figure 2 , Fig.14 and Fig.15 , the head-up display 704 is configured to display a virtual image M including letters, numbers and / or characters in the driver's front field of view. The head-up display 704 of the exemplary embodiment is configured to project the virtual image M in front of the driver D using AR technology to superimpose the virtual image M on the front scene including the road surface FR located at the traveling progress of the own vehicle V; AR is an abbreviation for augmented reality. The task of superimposing the virtual image M on the front scene can display information included in the virtual image M, which is related to at least one focus target included in the front scene to be linked to the at least one focus target in position.

[0210] For example, the task of superimposing the virtual image M on the foreground scene may display the information included in the virtual image M while superimposing the information on at least one focus target, or display the information included in the virtual image M while being adjacent to at least one focus target. The at least one focus target is, for example, at least one target object that the driver D driving the own vehicle V should focus on, that is, at least one target object that the driver D driving the own vehicle V should pay attention to. The at least one focus target includes, for example, road surface markings (road markings), road signs, preceding vehicles and / or pedestrians. For example, the head-up display 704 may be configured to superimpose the planned route, the direction of travel of the own vehicle V, traffic information, and other similar information on the front road surface FR as the focus target.

[0211] The area on the front windshield V18 on which the virtual image M is projected will be referred to as a projection area AP.

[0212] like Fig.14 As shown, the head-up display 704 has a depression angle AD, for example, a positive angle greater than 0°. Fig.14 In the side view shown, the depression angle AD of the head-up display 704 is defined as an angle formed between a virtual horizontal plane passing through the viewpoint EP of the driver D and a virtual plane passing through the viewpoint EP of the driver D and the top edge of the projection area AP. The depression angle AD becomes a positive value when the viewpoint EP looks down at the top edge of the projection area AP, or becomes a negative value when the viewpoint EP looks up at the top edge of the projection area AP.

[0213] The head-up display 704 has a vertical viewing angle AV and a horizontal viewing angle, and the vertical viewing angle AV defines the vertical width of the projection area AP, while the horizontal viewing angle defines the horizontal width of the projection area AP. The horizontal viewing angle is set to be larger than the vertical viewing angle AV so that the projection area AP has a substantially rectangular shape. In the example Fig.14 In the left side view shown, the vertical viewing angle AV may be defined as the angle formed between a virtual line passing through the left viewpoint EP of the driver D and the top edge of the projection area AP and a virtual line passing through the left viewpoint EP of the driver D and the bottom edge of the projection area AP. That is, the virtual viewing angle AV represents an angle range in the vertical direction within which the left viewpoint EP can view the virtual image M, which may be referred to as a vertical viewing angle in the vertical direction.

[0214] like Fig.14 As shown, the head-up display 704 includes a projector 741 and a magnifying optical system 742. The projector 741 is configured to generate a light-based image LV based on a display image signal generated by the HCU 701, and transmit the light-based image to the magnifying optical system 742. The magnifying optical system 742 includes a plurality of optical components, the plurality of optical components including a concave mirror and an actuator configured to control the alignment of the optical components. The magnifying optical system 742 is configured to project an enlarged image of the light-based image LV onto the front windshield V18 using the optical components, while controlling the alignment of the optical components by the actuator based on the position of the viewpoint EP detected by the driver state monitor 37 to adjust the projection state of the enlarged image on the front windshield V18 accordingly. This causes the enlarged image based on the light-based image LV to be projected in the projection area AP of the front windshield V18. This enables the driver D to visually recognize the virtual image M based on the reflected light of the enlarged image projected in the projection area AP of the front windshield V18.

[0215] Fig.15An example of a virtual image M is shown. The virtual image M includes, for example, a transparent base image, one or more information contents M1 contained in the transparent base image, and one or more graphic contents M2 contained in the transparent base image. The one or more information contents M1 are information contents showing, for example, the maximum speed limit of the road Rd on which the own vehicle V is traveling, the current speed of the own vehicle V, the distance to the destination, the estimated arrival time to the destination, and the name of at least one building appearing in the foreground scene. The one or more graphic contents M2 are graphic contents of indicating lines and / or arrows, which are used to display information indicating (i) the traveling direction of the own vehicle V and (ii) lane selection.

[0216] The head-up display 704 may be configured to display a virtual image M including overlay content and non-overlay content. Overlay content is image content that is linked to and overlays one or more specific focus targets included in the foreground scene. In contrast, non-overlay content is image content that is not linked to and overlays one or more specific focus targets included in the foreground scene.

[0217] exist Fig.15 In the virtual image M shown as an example, linear graphic content as an example of graphic content M2 is superimposed on the lane in which the own vehicle V is traveling, and information content indicating the maximum speed limit of the current road of the own vehicle V as an example of information content M1 is located at a predetermined position of the virtual image M and is not superimposed on the graphic content M2.

[0218] The above-described HMI system 7 functions as a notification unit for notifying one or more passengers including the driver D in the own vehicle V of information.

[0219] Reference Figure 1 and Figure 2 , the speaker 705 is configured to output a voice related to the information indicated by each of the instrument panel 702, the main display device 703, and the head-up display 704. The speaker 705 may be configured to output a voice unrelated to the information indicated by each of the instrument panel 702, the main display device 703, and the head-up display 704, such as music and / or radio sound.

[0220] The operation device 706 is an input device that is not included in the operation switch 38, and the operation amount and operation state of the operation device 706 are exempted from detection by the operation sensor 35. Specifically, the operation device 706 includes, for example, switches mounted to the periphery of the screen of the housing of the main display device 703, and a transparent touch panel mounted to cover the screen of the main display device 703. The operation device 703 may include switches mounted to the spokes of the steering wheel V24 and a pointing device such as a touch panel mounted to the center console.

[0221] The switches, pointing device, and touch panel of the operating device 706 enable one or more passengers who operate them to input various information items corresponding to the switches, pointing device, and touch panel, respectively.

[0222] The lighting system 8 includes a body ECU 801 , a headlight 802 , and a flasher 803 .

[0223] The body ECU 801 includes a vehicle microcomputer composed of a CPU, ROM, RAM, an input / output (I / O) interface, and other peripheral devices. The body ECU 801 is configured to control how the headlight 802 is lit according to information input from the driving ECU 2 and / or the driving information input unit 3, and to control how the flasher 803 is lit according to information input from the driving ECU 2 and / or the driving information input unit 3, particularly information input from the flasher switch 382.

[0224] The motion control system 9 is configured to control the motion of the own vehicle V, that is, the running behavior of the own vehicle V, according to information input by the driving ECU 2 and / or the driving information input unit 3 .

[0225] Specifically, the motion control system 9 includes, for example, a driving system 91 , a gear shifting system 92 , a braking system 93 , and a steering system 94 .

[0226] The driving system 91 includes a driving ECU 911 and a driving mechanism 912. The driving ECU 911 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface and other peripheral devices. The driving ECU 911 is configured to receive an accelerator signal indicating an acceleration request from an accelerator pedal sensor or a driving ECU 2, and controls the operation of the driving mechanism 912 according to the acceleration request. The driving mechanism 912 is configured to generate a driving power for the vehicle V equipped with the system to travel. Specifically, the driving mechanism 912 includes an engine, i.e., an internal combustion engine, and one or more motors. That is, the vehicle V equipped with the system is any one of a gasoline fuel vehicle, a diesel engine vehicle, a biofuel vehicle, a hydrogen engine vehicle, a hybrid vehicle, a battery electric vehicle (BEV), a fuel cell vehicle or other vehicles.

[0227] The shift system 92 includes a shift ECU 921 and a shift mechanism 922. The shift ECU 921 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The shift ECU 921 is configured to receive a shift position signal indicating a current shift position set by a shift lever or a driving ECU 2 from a shift position sensor, and control the operation of the shift mechanism 922 according to the current shift position of the shift lever. The shift mechanism 922 is provided between the drive wheel of the wheel V3 and the driving mechanism 912, and includes an automatic transmission. Specifically, the shift ECU 921 is configured to control the shift mechanism 922 to perform, based on a shift position signal indicating a current shift position set by the shift lever or the driving ECU 2, (i) a first task: to transfer the forward driving power generated by the driving mechanism 912 to the drive wheels so that the own vehicle V moves forward, (ii) a second task: to transfer the reverse driving power generated by the driving mechanism 912 to the drive wheels so that the own vehicle V moves backward, (iii) a third task: to cut off the driving power of the drive wheels to stop the own vehicle V accordingly, and / or (iv) a fourth task: to change the speed ratio between the input speed from the driving mechanism 912 to the shift mechanism 922 and the output speed from the shift mechanism 922 to the drive wheels when the own vehicle V moves forward. The shift system 92 may be configured as a so-called shift-by-wire configuration.

[0228] The braking system 93 includes a braking ECU 931 and a braking mechanism 932. The braking ECU 931 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The braking ECU 931 is configured to receive a braking signal indicating a braking request from a brake pedal sensor or a driving ECU 2, and control the operation of the braking mechanism 932 according to the braking request. The braking mechanism 932 includes a friction mechanism of each wheel in the wheels V3. That is, the braking ECU 931 is configured to control the friction mechanism of each wheel V3 according to the braking request to apply friction force to each wheel V3 accordingly, thereby decelerating the own vehicle V. The braking mechanism 932 may include: a regenerative braking mechanism, which is configured to rotate the drive wheel by the kinetic energy of the own vehicle V, to decelerate the own vehicle V accordingly due to the load of the drive wheel rotation, and convert the kinetic energy of the own vehicle V into electricity. The braking system 93 may be configured as a so-called wire control brake configuration.

[0229] The steering system 94 includes a steering ECU 941 and a steering mechanism 942. The steering ECU 941 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface and other peripheral devices. The steering ECU 941 is configured to receive a steering signal indicating a steering request from a steering angle sensor or a driving ECU 2, and control the operation of the steering mechanism 942 according to the steering request. That is, the steering ECU 941 is configured to control the steering mechanism 942 to change each steering wheel, such as the direction of each front wheel V3a, V3b, according to the steering request, so as to change the direction of travel of the own vehicle V accordingly. The steering mechanism 942 can be configured to change the direction of each of the front and rear wheels V3a, V3b, V3c and V3d. That is, the own vehicle V can be configured as a four-wheel steering vehicle. The steering system 94 can be configured as a so-called wire control steering configuration.

[0230] Driving control based on the identification result of at least one of the ADAS sensors 31 to 34

[0231] Fig.16 and Fig.17 The example of the functions implemented by the driving ECU 2 is schematically shown. That is, the processor 21 executes the program instructions stored in the memory device 22 to implement the corresponding Fig.16 and Fig.17 Functions shown.

[0232] An outline of autonomous driving of the own vehicle V performed by the driving ECU 2 based on the identification result of at least one of the ADAS sensors 31 to 34 around the own vehicle V is described below.

[0233] Note that in the exemplary embodiment, "identification" conceptually includes "detection," "classification," and "recognition" or "perception." Detection is to find the target object B based on at least one detection point data cloud and / or an image captured by the camera device 34. Detection of the target object B is to determine the existence of the target object B without classifying the shape and / or attributes of the target object B. Classification of the target object B is to classify the shape and / or attributes of the detected target object B into various object types, such as "human," "vehicle," "building," etc. In other words, a classified target object is a target object that has been detected and classified into one of the various object types. Cognition or perception of the target object B is to determine whether the detected and classified target object B should be considered when controlling the driving of the own vehicle V.

[0234] Detection of an object may include sensing the object. Detection may conceptually include classification and / or cognition (perception), classification may conceptually include detection and / or cognition (perception), and cognition may conceptually include detection and classification.

[0235] Reference Fig.16 , the driving ECU 2 includes an identification module 2001 , an operation determiner 2002 , and a control signal output module 2003 as functional components implemented by the processor 21 .

[0236] The identification module 2001 is operable to perform an identification task of one or more target objects B around the own vehicle V according to information items input from the surrounding monitoring sensors 31 to 34, the operation sensor 35, and the behavior sensor 36. The operation determiner 2002 is operable to determine one or more control tasks currently required to control the own vehicle V based on the identification result of the identification module 2001 and the information items input from the operation sensor 35 and the behavior sensor 36. The one or more control tasks include, for example, a collision avoidance task, an emergency stop task, and a warning task for the driver D.

[0237] The control signal output module 2003 is operable to output control signals based on the determined control tasks to selected components of the vehicle system 1. The control signals include, for example, a steering manipulation signal indicating a steering manipulation amount, a brake signal indicating a brake amount, and a message code signal indicating a warning message.

[0238] Reference Fig.17 The identification unit 2001 includes an input information acquisition module 2101, an input information processing module 2102, a target object recognition module 2103, a own lane recognition module 2104, an intersection recognition module 2105 and a surrounding environment recognition module 2106.

[0239] The information acquisition module 2101 acquires information items input from the surrounding monitoring sensors 31 to 34, the operation sensor 35, and the behavior sensor 36, and stores the acquired information items in sequence. The input information processing module 2102 applies one or more predetermined tasks, such as a noise removal task and / or a coordinate conversion task, to the information items stored in the information acquisition module 2101.

[0240] The target object recognition module 2103 is operable to perform a target object recognition task according to the information item subject to a predetermined task.

[0241] Specifically, the target object recognition module 2103 includes, for example, a marking line recognition module 2131, a road surface marking recognition module 2132, a roadside structure recognition module 2133, a traffic light recognition module 2134, a traffic sign recognition module 2135, a lane recognition module 2136, a pedestrian recognition module 2137, a surrounding vehicle recognition module 2138 and an obstacle recognition module 2139.

[0242] exist Fig.18 or Fig.19 , a target object B to be identified by the identification module 2001 is shown as an example.

[0243] Reference Fig.18 , the target object B broadly includes, for example, traffic-related entity objects B1, other vehicles B2, general entity objects B3, and road surface markings B4. The target object B to be identified by the identification module 2001 further includes, for example, Fig.19 The parking space segmentation line B5, wheel stopper B6 and parked vehicles B7 shown in relation to the detection of the parking space PS and / or the execution of the parking assistance. The parking space segmentation line B5 is, for example, a painted line for defining the shape and area of ​​each parking space PS. The wheel stopper B6 is a block-shaped solid installed in each parking space PS for stopping the wheels of the corresponding parked vehicle B7. Each of the parked vehicles B7 is a vehicle parked in the corresponding parking space PS.

[0244] The traffic-related entity objects B1 are entity objects for road traffic safety, such as traffic lights B11 and traffic signs B12. Each of the other vehicles B2 may become a target vehicle tracked by the traveling own vehicle V, or an obstacle to the traveling own vehicle V, so the stopped vehicle B7 is excluded from the other vehicles B2. The general entity objects B3 are entity objects other than the traffic-related entity objects B1, other vehicles B2, wheel stoppers B6, and stopped vehicles V7, and may mainly constitute obstacles.

[0245] The lane LN may be identified based on the recognition result of the target object B acquired based on the captured image. Specifically, although the parking space PS or the lane LN is different from the target object B that is the direct recognition target of the ADAS sensors 31 to 34, these parking spaces PS or the lane LN may be used as indirect recognition targets based on the recognition result of the target object B. The lane LN that is the indirect target includes, for example, the own lane LNm in which the own vehicle V is traveling and the oncoming lane LNc. The recognition target to be recognized by the identification module 2001 will be described in detail later.

[0246] The own lane recognition module 2104 is operable to recognize the position of the own vehicle V in the road in the width direction of the road based on the recognition result of the target object B by the target object recognition module 2103; the road is the road on which the own vehicle V is traveling. The width direction of the road is referred to as the road width direction, and the direction perpendicular to the road width direction is referred to as the road extension direction. The road extension direction is the direction extending along the road, and may be referred to as the road extension direction or the road elongation direction. If the road Rd includes a plurality of lanes LN, the own lane recognition module 2104 is operable to recognize any one of the plurality of lanes LN arranged in the road width direction as the own lane LNm.

[0247] The intersection recognition module 2105 is operable to recognize an intersection Xr around the own vehicle V based on the recognition result of the target object B by the target object recognition module 2103 .

[0248] Specifically, the intersection identification module 2105 is operable to identify the intersection Xr that the own vehicle V is approaching based on (i) whether there is a traffic light B11, (ii) if it is determined that there is a traffic light B11, which color signal light is output from the traffic light B11, (iii) whether there is a stop line B42 which is one of the road surface markings B4, (iv) the position of the intersection entrance Xr1, (v) the position of the intersection exit Xr2, and (vi) each traffic sign or mark indicating the corresponding direction of travel.

[0249] The surrounding environment recognition module 2106 is operable to recognize the surrounding environment around the own vehicle V, such as how one or more obstacles are located around the own vehicle V, based on the recognition result of the target object B by the target object recognition module 2103.

[0250] These recognition results of the recognition modules 2103 to 2106 are used for the operation determiner 2002, which determines one or more control tasks required to currently control the own vehicle V.

[0251] Fig.18This is an example of a foreground scene from the own vehicle V in a situation where the own vehicle V is approaching an intersection Xr constituting an intersection and the intersection Xr includes a traffic light B11. The intersection Xr constituting an intersection is simply referred to as a cross intersection Xr. For the HMI system 7, the recognition results of the traffic light B11 and the traffic sign B12 included in the traffic-related entity object B1 can be used to notify the driver D of information and / or output a warning to the driver D. For the driving ECU 2, the recognition results of the traffic light B11 and the traffic sign B12 included in the traffic-related entity object B1 can also be used in the autonomous driving of the own vehicle V and / or the driving assistance of the own vehicle V to perform motion control of the own vehicle V. In addition to the traffic light B11 and the traffic sign B12, the traffic-related entity object B1 also includes roadside structures B13 such as guardrails and curbs. The traffic-related entity object B11 also includes such Fig. 20 The road spikes B14 and the poles B15 are shown. The road spikes may be recognized as obstacles depending on (i) the height of the road spikes and / or (ii) how the road spikes are arranged around the own vehicle V. When the driving ECU 2 performs travel control of the own vehicle V, the poles B15 are recognized as obstacles.

[0252] Reference Fig.18 , general entity objects B3 include, for example, fallen objects on the road surface B31, pedestrians B32, cyclists B33, and buildings B34. When the driving ECU 2 performs driving control of the own vehicle V, the fallen objects B31, pedestrians B32, and cyclists B33 on the road surface are recognized as obstacles. The building B34 is not recognized as an obstacle because it is located outside the road Rd, but when the head-up display 704 displays the virtual image M, it may become content superimposed on the foreground scene.

[0253] The road surface markings B4 include, for example, a pedestrian crossing marking B41, a stop line B42, and a road dividing line B43. The road surface markings B41 also include, for example, Fig.21 The letter mark B44 and the symbol mark B45 are shown. The letter mark B44 includes, for example, a number indicating the maximum speed limit, a vehicle traffic area such as "bus only" or a restriction letter such as "STOP". The symbol mark B45 includes, for example, an arrow indicating the corresponding mandatory direction and a warning symbol for a pedestrian crossing ahead.

[0254] The recognition result of the road traffic mark B4 is used to estimate the position of each of the intersection entrance Xr1 and the intersection exit Xr2 and the position of the intersection center Xrc. The intersection entrance Xr1 represents the edge of the focused intersection that the own vehicle V is about to enter. The focused intersection is the nearest intersection Xr that (i) the own vehicle V is approaching and (ii) the own vehicle V plans to pass through or may pass through. The intersection that the own vehicle V may pass through is an intersection that the own vehicle V is estimated to pass with a high probability based on (i) the distance from the own vehicle V to the intersection and (ii) the speed of the own vehicle V when the navigation system 6 has not determined the planned driving route and destination of the own vehicle V. The intersection exit Xr2 represents the edge of the focused intersection that the own vehicle V is about to leave. The intersection center Xrc is the center of the focused intersection.

[0255] like Fig. 22 and Fig.23 As shown, the road marking line B43 includes, for example, a vehicle-road edge line B431, a center line B432, and a lane line B433. Vehicle-road edge lines B431 are provided for each vehicle-road, and they respectively show the left edge and the right edge of the corresponding vehicle-road in the road width direction. The vehicle-road of the road Rd represents a portion of the road Rd where vehicles can travel. Each center line B432 represents a line running downward along the center of the two-way road, and divides the two-way road into two parts in corresponding opposite directions of travel. Each center line B432 is composed of a single straight line pattern or a single dotted line pattern, a double line pattern, or a triple line pattern. Lane lines B433 are provided for a road Rd having two or more lanes in each direction, and they divide the road Rd into two or more lanes in each direction. Each of the lane lines B433 is a white painted line or a yellow painted line. As an example, Fig. 20 The center line B432 is shown to be composed of a three-line pattern consisting of a solid white line and a pair of yellow lines on both sides of the white line. The road studs B14 and the rods B15 are installed on Fig. 20 The white line of the three-line pattern of the center line B432 is shown. As another example, Fig.21 It is shown that the center line B432 is composed of a double line pattern composed of a white solid line and the lane line B433 is composed of a white dashed line and a yellow solid line extending from the white dashed line. Based on the recognition results of the roadside structure B13 and the road division line B43, it is possible to recognize (i) the driving lane LNd or the own lane LNm in which the own vehicle V is traveling (see, for example Fig. 20 and Fig.21 ), (ii) the oncoming lane LNc (see e.g. Fig. 20 and Fig.21 ), (iii) through lane LNp (see e.g. Fig. 20), (iv) right turn lane LNr (see Fig.21 ), (v) shoulder LNs (see Fig. 20 and Fig.21 ), and (vi) an emergency stop zone EZ (not shown).

[0256] The road traffic signs B4 further include, for example, diversion area signs, namely, a zebra crossing area sign B461 , a safety area sign B462 , a no-entry area sign B463 , and a no-stop area sign B464 .

[0257] The diversion area mark B461 is painted on the road and indicates a diversion area where the safe and smooth operation of the vehicle needs to be guided. As described above, the diversion area mark B461 is a mark indicating a diversion area for guiding the safe and smooth operation of the vehicle, and the vehicle is not legally prohibited from entering the diversion area. The diversion area mark B461 can be set adjacent to (i) an intersection Xr, (ii) a junction of roads, or (iii) a fork in the road.

[0258] Fig.24 The safety zone mark B462 shown is used to indicate a safety zone. A safety zone refers to an area set up on the road to ensure the safety of, for example, pedestrians B32, so vehicles cannot legally enter the safety zone.

[0259] Fig.25 The prohibited entry area mark B463 shown is used to indicate a prohibited entry area. The prohibited entry area indicates an area on the road that cannot be used for vehicle driving, so vehicles cannot enter the prohibited entry area.

[0260] Fig.26 The shown no-stopping area mark B464 is used to indicate a no-stopping area. The no-stopping area represents an area on the road that a vehicle should not enter when the vehicle may have to stop according to the conditions in front of the vehicle. That is, when any vehicle is waiting for a traffic light or is in a traffic jam, the vehicle should not stop within the no-stopping area mark B464.

[0261] Fig. 27 An example of a traffic sign B12 is shown. Each of the traffic signs B12 is installed adjacent to a corresponding one or more roads Rd, or installed above a corresponding one or more roads Rd. Each of the traffic signs B12 is composed of a display panel designed to provide corresponding necessary information to a user facing a corresponding one of the traffic signs B12. The display panel of each traffic sign B12 has (i) a designed shape, (ii) one or more colors painted on at least one main surface thereof, and (iii) one or more symbols. The one or more colors include, for example, white, black, red, blue, and yellow. Fig. 27In the traffic sign B12, cross hatching indicates red, oblique hatching indicates blue, dot hatching indicates yellow, and shaded portions indicate black. Each symbol used in the traffic sign B12 includes, for example, one or more letters, one or more numbers, and a combination of one or more letters and one or more numbers.

[0262] Fig. 27 Examples of traffic signs B12 in Japan (JP), European Union (EU), and the United States (US) are shown, classified by meaning. As typical examples of traffic signs B12 in EP, examples of traffic signs B12 in Germany and / or France are used.

[0263] For example, the corresponding traffic signs of JP, EU, and US meaning "Do Not Enter" or "No Entry" have substantially the same design. In contrast, the corresponding traffic signs of JP, EU, and US meaning "Stop" have substantially the same color and symbol, and the shape of the plate of the traffic sign of EU is substantially the same as that of US, but the shape of the plate of the traffic sign of JP is different from that of EU and US.

[0264] The corresponding JP and EU traffic signs using the letters "50" to mean "maximum speed limit" have basically the same design except for the color of the letters, but the US traffic sign using the letters "50" to mean "speed limit" is different from that of JP and EU in the shape and color of the plate.

[0265] The corresponding traffic signs of JP, EU, and US meaning "Railway Crossing Caution" have a low level of commonality in design.

[0266] The corresponding traffic signs of EU and US meaning "No Right Turn" have basically the same design, but JP has no corresponding traffic sign.

[0267] JP's traffic sign means "drive only in the direction of the arrow" and uses white arrows to indicate the forward direction and left turn direction, such as Fig. 27 As shown, it similarly means "no right turn".

[0268] The accurate recognition of the traffic sign B12 enables smooth and safe implementation of autonomous driving and / or advanced driving assistance. As described above, the design of some of the traffic signs B12 varies greatly between JP, EU and US. For this reason, the memory device 22 pre-stores at least one database, which stores information indicating patterns of traffic signs used in, for example, each of all countries in the world or each of all regions in the world. Pattern matching between the identified traffic sign B12 and the information stored in at least one database enables detection of the meaning of the identified traffic sign B12. The at least one database may be configured as a public database for the whole world, or may be composed of a plurality of databases provided for all countries in the world, each of which stores information indicating patterns of traffic signs used in a corresponding one of the countries in the world. At least one database may be stored in a host computer, instead of or in addition to the memory device 22; the driving ECU 2 is communicatively connected to the host computer through the vehicle communication module 4 based on vehicle-to-everything technology (V2X). This enables the driving ECU 2 to freely access at least one database stored in the host computer.

[0269] Fig.28 A first example is schematically shown indicating how to use the recognition result of the target object B in a situation where the own vehicle V is approaching an intersection. In this situation, the head-up display 704 can superimpose the information content M1 indicating the message "10m from the stop position" indicating the distance 10m from the stop line B42 on the stop line B42. In addition, the HMI system 7 can output a voice message indicating "10m from the stop line" using the speaker 705. This effectively assists the driver D who uses the advanced driving assistance to reliably stop the own vehicle V before the stop line B42. Alternatively, the voice message output from the HMI system 7 enables one or more passengers of the own vehicle V to reliably grasp the occurrence of the temporary stop event of the own vehicle V in advance, thereby avoiding a reduction in the comfort of one or more passengers of the own vehicle V.

[0270] Fig.29A second example is schematically shown indicating how to use the recognition result of the target object B in a situation where the own vehicle V is approaching an intersection with a traffic light B11 and the own vehicle V is going to turn right at the intersection. In this situation, the head-up display 704 can superimpose information content M1 indicating "turn right at the intersection 20m ahead" on the own lane LN, that is, the right-turn lane LNr of the road surface FR in front of the own vehicle V. The HMI system 7 can also output a voice message indicating "turn right at the intersection 20m ahead" using the speaker 705. In addition, the HMI system 7 can superimpose graphic content M2 of an arrow extending from the right-turn lane LNr through the center Xrc of the intersection or its vicinity to the target road Rd on the road surface FR. This effectively provides one or more passengers including the driver D of the own vehicle V with advance notice of the travel direction guidance of the own vehicle V and / or the behavior of the own vehicle V undergoing lateral acceleration during the right turn. This therefore enables one or more passengers of the own vehicle V to reliably grasp the occurrence of the behavior of the own vehicle V in advance, thereby avoiding a reduction in the comfort of one or more passengers of the own vehicle V.

[0271] Reference Fig.17 The target object recognition module 2103 is operable to recognize the type and state of at least one target object B based on the recognition result of the at least one target object B and the HD map information stored in the HD map information database 5. The state of the at least one target object B includes, for example, position information about the at least one target object B. The position information about the at least one target object B includes, for example, the distance of the at least one target object B relative to the own vehicle V, the relative position of the at least one target object B relative to the own vehicle V, and the position of one of the lanes LN in the case where the at least one target object B is one of the multiple lanes LN.

[0272] Specifically, the marking line recognition module 2131 of the target object recognition module 2103 is operable to recognize a road marking line B43 in a peripheral area around the own vehicle V, the peripheral area including a road surface FR of a road Rd located at a travel route of the own vehicle V. For example, the marking line recognition module 2131 is operable to recognize, for example Figure 20 to Figure 23 Each of the vehicle-road edge line B431, the center line B432 and / or the lane line B433 shown is (i) white or yellow, (ii) solid or dashed, (iii) single-line or multi-line. Because the recognition technology of the road marking line B43 is well known at the time of filing this application, a detailed description of the recognition technology of the road marking line B43 is omitted in the present disclosure.

[0273] The road surface marking recognition module 2132 of the target object recognition module 2103 is operable to recognize the type, meaning and position of each of the road surface markings B4 except the road marking line B43; the recognition targets of the road surface marking recognition module 2132 include, for example, a crosswalk marking B41, a stop line B42 and a symbol marking B45.

[0274] The roadside structure identification module 2133 is operable to identify the type and location of at least one roadside structure B13 located around the own vehicle V based on, for example, (i) a recognition result based on an image captured by the camera device 34, (ii) a recognition result based on at least one detection point data cloud detected by the lidar sensor 33, and (iii) at least one of the HD map information stored in the HD map information database 5.

[0275] The traffic light identification module 2134 is operable to identify (i) whether the traffic light B11 is located at the driving route of the own vehicle V, and (ii) if it is identified that the traffic light B11 is located at the driving route of the own vehicle V, identify the location of the traffic light B11 and which of the color signal lights is output from the traffic light B11.

[0276] The traffic sign recognition module 2135 is operable to recognize a traffic sign B12 located around the own vehicle V.

[0277] The lane recognition module 2136 is operable to perform a lane recognition task: identifying the number of lanes LN in the road Rd on which the own vehicle V is traveling and the type of each lane LN in the road Rd. That is, the lane recognition module 2136 is operable to perform the lane recognition task based on, for example, (i) a recognition result based on an image captured by the front camera CF of the camera 34, (ii) a recognition result based on at least one detection point data cloud detected by the laser radar sensor 33, and (iii) the HD map information stored in the HD map information database 5.

[0278] Specifically, the lane recognition module 2136 can operate normally to perform a lane recognition task based on a recognition result based on an image captured by the camera device 34, or, if necessary, perform a sensor fusion lane recognition task based on a combination of (i) a recognition result based on an image captured by the front camera CF of the camera device 34 and (ii) a recognition result based on at least one detection point data cloud detected by the lidar sensor 33 and (iii) at least one of the HD map information stored in the HD map information database 5.

[0279] The pedestrian recognition module 2137 is operable to recognize one or more pedestrians B32 located around the own vehicle V. The surrounding vehicle recognition module 2138 is operable to recognize one or more other vehicles B2 located around the own vehicle V. The obstacle recognition module 2139 is operable to recognize one or more obstacles, such as one or more fallen objects B31 on the road surface and / or one or more pedestrians B32 located around the own vehicle V. How each of the traffic light recognition module 2134, the traffic sign recognition module 2135, the lane recognition module 2136, the pedestrian recognition module 2137, the surrounding vehicle recognition module 2138 and the obstacle recognition module 2139 recognizes the corresponding one or more target objects is well known at the time of filing this application, and therefore a detailed description of how each of the modules 2134 to 2139 recognizes the corresponding one or more target objects is omitted in this disclosure.

[0280] Road surface marking recognition

[0281] Fig.30 FIG. 2 is a flow chart schematically showing a road surface marking recognition task of recognizing a road surface marking B4 based on an image captured by the camera device 34, the road surface marking recognition task being programmed to be executed by the processor 21. Fig.30 In the figure, the reference symbol S is an abbreviation for step.

[0282] When the road surface marking recognition task is started, the processor 21 sequentially executes Fig.30 The operations in steps S101 to S107 shown in FIG.

[0283] The processor 21 is mainly used for Fig.17 The input information acquisition module 2101 shown in FIG. 1 is used to perform the operation in step S101 and is mainly used as Fig.17 The input information processing module 2102 shown in FIG. 1 is used to perform the operation in step S102 , and is mainly used as a target object recognition module 2103 to perform the following operations in steps S103 to S107 .

[0284] In step S101, the processor 21 reads input information required to perform the road surface marking recognition task. The input information read by the processor 21 in step S101 includes, for example, at least one of the information items input from the surrounding monitoring sensors 31 to 34, the operation sensor 35, and the behavior sensor 36. For example, the processor 21 according to the exemplary embodiment reads input information including image information based on a frame image captured by the front camera device CF; the frame image includes pixels arranged two-dimensionally in the vertical direction and the horizontal direction. Each pixel of the image data of the frame image includes a corresponding feature parameter, and the corresponding feature parameter includes, for example, brightness, contrast, and hue of the pixel based on the corresponding received light.

[0285] Next, in step S102, the processor 21 applies one or more well-known pre-processing tasks to the input information. For example, well-known pre-processing tasks include (i) brightness adjustment, (ii) noise removal, (iii) resolution conversion, (iv) lens distortion correction, (v) bird's-eye view conversion, (vi) affine transformation, (vii) edge enhancement, and (viii) cropping.

[0286] After the operation in step S102, the processor 21 detects the detection area Aw (see FIG. 1 ) of the frame image that has been subjected to the preprocessing task in step S103 based on the feature parameters of each pixel of the frame image. Fig.10 ) detects at least one target object B in the frame image; the at least one target object B is at least one object of the road surface marker recognition task. For example, the processor 21 sets at least one bounding box in the detection area Aw of the frame image; the at least one bounding box is a geometric shape, such as a rectangular shape, which surrounds or surrounds at least one target object B, such as at least one crosswalk B41, in the frame image for detection and / or classification of at least one object of the road surface marker recognition task.

[0287] Then, as described above, in step S104, the processor 21 executes a monocular motion stereo method, in other words, an SFM method based on at least one target object B, to accordingly estimate the relative position of at least one target object B relative to the own vehicle V and / or the distance from the own vehicle V to at least one target object B.

[0288] After the operation in step S104. In step S105, the processor 21 performs a classification task of classifying the shape and / or posture of at least one target object B into one of various object types. Next, in step S106, the processor 21 performs a recognition task of recognizing at least one target object, and outputs the recognition result of at least one target object B to the operation determiner 2002 in step S107.

[0289] For example, in Fig.28or Fig.29 In the scenario shown in , the estimated value of the distance from the own vehicle V to the crosswalk mark B41, the estimated value of the distance from the own vehicle V to the stop line B42, the estimated value of the distance from the own vehicle V to the intersection entrance Xr1, and the estimated value of the distance from the own vehicle V to the intersection exit Xr2 may include errors. Such errors included in each of the target objects B, such as the target objects B41, B42, Xr1, and Xr2, may cause a deviation between the actual time of executing the driving control of the own vehicle V and the optimal time of executing the driving control of the own vehicle V, a deviation between the actual position of the superimposed content and the optimal position of the superimposed content, and / or a deviation between the actual time of outputting the sound message and the optimal time of outputting the sound message. This may cause one or more passengers including the driver D to feel uncomfortable. From this point of view, when the driving ECU 2 uses the recognition result of at least one target object B, the recognition result of at least one target object B preferably has a higher recognition accuracy.

[0290] Unfortunately, due to deterioration caused by age, part of the pedestrian crossing marking B41 may be missing or blurred (see Fig.31 ). In addition, there may be a situation where a portion of the pedestrian crossing marking B41 is blocked by, for example, the preceding vehicle B2 (see Fig.32 ) or pedestrian B32 is blocked from the front field of view of the front camera device CF. In these cases, which will be referred to as low-visibility cases, the reduction of the features of at least one target object B, where a portion is difficult to see, may result in the feature points Pt and / or the detection point data cloud Pg not being able to be smoothly found out from the captured image. In addition, the erroneous estimation of the distance from the own vehicle V to the at least one target object B may result in the recognition accuracy of the at least one target object B, that is, the distance estimation accuracy of the at least one target object B being deteriorated.

[0291] In addition, in the case of low visibility of the object, how to set at least one bounding box in the detection area Aw of each of the sequentially captured frame images, for example, how to set the position and / or size of at least one bounding box in each of the sequentially captured frame images, may be unstable. This may cause the features included in at least one bounding box in any frame image in the sequentially captured frame images to be different from the features included in at least one bounding box in another frame image in the sequentially captured frame images. Therefore, this may cause the sequential estimated distance from the own vehicle V to at least one target object B based on the sequentially captured frame images to fluctuate.

[0292] For example, Fig.33 and Fig.34 Each of them schematically shows a corresponding frame image captured by the front camera device CF, including the pedestrian crossing mark B41. Fig.33The pedestrian crossing marking B41 shown in FIG. Fig.34 The pedestrian crossing marking shown in is farther away from the own vehicle V. In other words, Fig.34 The crosswalk markings shown in the figure are positioned Fig.33 The pedestrian crossing marking shown in FIG. 1 is closer to the own vehicle V.

[0293] Fig.33 and Fig.34 The crosswalk marking B41 shown in each of the above includes a pattern of equal-width lines periodically arranged in the road width direction. For this reason, the value of at least one pixel characteristic parameter, such as brightness, included in each pixel of the frame image captured by, for example, the front camera device CF has a periodic change; a specific pixel corresponds to the crosswalk marking B41 arranged in the horizontal direction of the frame image perpendicular to the forward direction Df of the own vehicle V, that is, in the detection width direction. In addition, Fig.33 The variation period of the value of at least one pixel characteristic parameter shown in is different from Fig.34 The variation period of the value of at least one pixel characteristic parameter shown in Fig.33 The pedestrian crossing mark B41 is positioned further away from the own vehicle V, Fig.34 The pedestrian crossing mark B41 is positioned closer to the own vehicle V.

[0294] Fig.35 is a graph schematically showing the relationship between (i) the position in the road width direction, in other words, the value of the distance (m) relative to, for example, the left edge of the detection area, the field of view of the detection area, such as the front camera CF, and (ii) the value of at least one pixel feature parameter included in a specific pixel of the frame image; the specific pixels arranged in the horizontal direction of the frame image correspond to the corresponding positions of the white drawn line. That is, Fig.35 The graph has a horizontal axis Dx which represents the position in the road width direction, in other words, the value of the distance (m) relative to, for example, the left edge of the detection area, the detection area, for example, the field of view of the front camera CF. Fig.35 The graph of has a vertical axis FP, which represents the value of at least one pixel characteristic parameter included in a specific pixel.

[0295] The pedestrian crossing marking B41 includes a first pattern of white drawn lines, each of which has a first width W1 corresponding to a predetermined value P1 of at least one pixel characteristic parameter, and a second pattern of non-drawn portions, wherein the non-drawn portions are arranged between the white drawn lines; and each of the non-drawn portions has a second width W2 corresponding to a predetermined value P2 of at least one pixel characteristic parameter lower than the predetermined value P1. That is, Fig.35 Shows Fig.35The length W of the repetition period of the crosswalk marking B41 shown in is the sum of the first width W1 and the second width W2. In Japan, the first width W1 of each of the various crosswalk markings B41 is in the range of 0.45 m to 0.5 m, and the first width W1 is equal to the second width W2. In other countries other than Japan, the first pattern of the crosswalk marking B41 may include yellow drawn lines, each of which has the first width W1, or the second pattern of the crosswalk marking B41 may include black drawn portions, each of which has the second width W2.

[0296] Fig.36 is schematically shown at Fig.33 A graph showing how the value of at least one pixel characteristic parameter on the first virtual line Ly1 included in the frame image shown in FIG. 1 changes, and Fig.37 It is shown that the Fig.34 A graph showing how the value of at least one pixel feature parameter on the second virtual line Ly2 included in the frame image shown in FIG. 1 changes. Each of the first virtual line Ly1 and the second virtual line Ly2 is a virtual line that passes through the center of a corresponding one of the crosswalk marks B41 in its transverse direction perpendicular to its longitudinal direction and extends parallel to the horizontal direction of the corresponding one of the frame images in the frame image, i.e., the detection width direction.

[0297] Fig.36 and Fig.37 Each of the graphs has a horizontal axis Px representing the pixel position in the detection width direction of a corresponding frame image in the frame image, that is, in the horizontal direction relative to the left edge of the detection area such as the field of view of the front camera CF. In other words, Fig.36 and Fig.37 The horizontal axis Px of each of represents the pixel distance of each horizontal pixel position from the left edge of the detection area, such as the field of view of the front camera CF. Fig.36 and Fig.37 Each of the graphs has a vertical axis FP, which represents the value of at least one pixel feature parameter included in the corresponding specific pixel. The horizontal direction of each frame image corresponds to the horizontal direction of the pixels arranged two-dimensionally in the vertical direction and the horizontal direction. The horizontal direction of each frame image will also be referred to as the row direction. That is, the row direction represents the horizontal direction in which the pixels are arranged horizontally. In other words, the two-dimensionally arranged pixels of each frame image include pixel columns; the pixels of each column are arranged in the vertical direction corresponding to the column direction. The pixel columns of the frame image are arranged in the row direction, so that the horizontal position of each pixel column of the frame image, that is, the column number, can be defined.

[0298] Fig.36 and Fig.37The change in the value of at least one pixel characteristic parameter corresponding to the white drawn line shown in each of the above has a period λ, which will be referred to as the change period λ. This indicates that the change period λ of the change in the value of at least one pixel characteristic parameter corresponding to the white drawn line of the crosswalk mark B41 varies according to the distance from one camera device selected from the camera devices 34, i.e., the front camera device CF, to the crosswalk mark B41.

[0299] Fig.38 It is a graph showing the relationship between each change value of the distance D1 to the crosswalk mark B41 and the corresponding value of the change period λ when the front camera CF captures the crosswalk mark B41 while changing the distance D1 from the own vehicle V, i.e., the front camera CF, to the crosswalk mark B41. For example, while changing the distance D1 from the own vehicle V, i.e., the front camera CF, to the crosswalk mark B41, the front camera CF captures the crosswalk mark B41 on a predetermined route in a known imaging environment, such as a test route, thereby obtaining the value of the change period λ for each value of the distance Dt. Then, the value of the change period λ for each value of the distance Dt is plotted in a graph in which the horizontal axis is the distance Dt and the vertical axis is the change period λ. For example, as Fig.39 As shown in , fitting is performed based on, for example, the least square method so that the plotted values ​​of the variation period λ are fitted to a smooth function curve.

[0300] Fig.40 is a graph showing, by a solid line, how a value of at least one pixel feature parameter included in a specific pixel corresponding to a corresponding white drawn line of a crosswalk mark B41 changes, the crosswalk mark B41 being partially missing and / or blocked. Fig.40 The change in the value of at least one pixel characteristic parameter shown in does not seem to have obvious periodic characteristics. However, even if part of the pedestrian crossing mark B41 is missing and / or blocked, the periodic wave information (see Fig.39 The dotted wave in the figure) should also be included in the change in the value of at least one pixel characteristic parameter.

[0301] From this perspective, Fig.40 The intensity spectrum is obtained by Fourier transforming the solid line waveform of the change in the value of at least one pixel characteristic parameter included in the pixel corresponding to the corresponding white drawn line of the pedestrian crossing mark B41 shown in . The intensity spectrum represents the intensity of each frequency component included in the change in the value of at least one pixel characteristic parameter, such as the corresponding frequency value or the corresponding period value, and contains a local intensity peak. In other words, the intensity spectrum represents the contribution level of each component included in the change in the value of at least one pixel characteristic parameter, such as the corresponding frequency value or the corresponding period value.

[0302] That is, the value of at least one pixel characteristic parameter included in the pixel is Fourier transformed; the pixel corresponds to the corresponding white drawn line of the crosswalk mark B41 and is aligned in the horizontal direction, ie, the row direction, included in the bounding box. Fig.41 The horizontal axis has a variation period λ which is the inverse of the frequency, and the vertical axis has an intensity. The solid line extending parallel to the horizontal axis represents an intensity threshold value for distinguishing the intensity value of the pixel corresponding to the crosswalk mark B41 from the intensity value in the background component of the frame image. That is, Fig.41 The intensity spectrum shown in will also be referred to as intensity-frequency spectrum.

[0303] Fig.41 The intensity-frequency spectrum shown as an example in FIG shows that three local intensity peaks appear at corresponding values ​​of the period λ1, λ2 and λ3 in the horizontal direction. Among the three local intensity peaks, the local intensity peak at the value of the period λ1 may correspond to Fig.40 The dotted waveform shown in .

[0304] As described above, the driving ECU 2 of the exemplary embodiment is configured to calculate information included in the periodic change in the value of at least one pixel characteristic parameter included in selected pixels aligned in the detection width direction intersecting the detection direction, which is related to the target object B, based on the frame image captured by the at least one camera 34; the detection direction extends from the at least one camera toward the target object B located in the field of view of the at least one camera 34. The information included in the periodic change in the at least one pixel characteristic parameter may include a frequency-related parameter of the periodic change in the at least one pixel characteristic parameter, such as a period and / or a frequency and / or a wavelength, and the frequency-related parameter of the periodic change in the at least one pixel characteristic parameter, such as a period and / or a frequency and / or a wavelength, may be calculated based on, for example, Fourier transform of the periodic change in the at least one pixel characteristic parameter.

[0305] The driving ECU 2 of the exemplary embodiment is also configured to estimate the distance from the own vehicle V to the target object B based on information included in the calculated periodic change of at least one pixel characteristic parameter.

[0306] Specifically, the driving ECU 2 of the exemplary embodiment obtains, i.e., calculates, a representative period corresponding to a selected one of the local peaks of the intensity-frequency spectrum of the change of at least one pixel characteristic parameter as information included in the periodic change of at least one pixel characteristic parameter; the peak of the intensity-frequency spectrum is greater than or equal to a predetermined intensity. The representative period generally corresponds to one of the local peaks with the highest intensity.

[0307] The driving ECU 2 stores, in, for example, the memory device 22, correspondence information indicating the correspondence between each value of the variable of the distance from the own vehicle V to the target object and the corresponding value of the variable of the variation cycle λ (see, for example, Fig.38 ). Therefore, the driving ECU 2 uses the calculated representative period to refer to the correspondence information stored in the memory device 22 to accordingly estimate the distance from the own vehicle V to the target object.

[0308] Fig.42A is a graph showing changes in the value of at least one pixel characteristic parameter included in a frame image captured by the front camera device CF; the changes in the value of at least one pixel characteristic parameter correspond to a crosswalk mark B41 located at a relatively long distance from the own vehicle V. Fig.42B It is shown Fig.42A A graph of a Fourier transform result of a change in a value of at least one pixel characteristic parameter shown in FIG.

[0309] Fig.43A is a graph showing changes in the value of at least one pixel feature parameter included in a frame image captured by the front camera CF; a crosswalk mark B41 included in the frame image is located at a relatively middle distance from the own vehicle V. The middle distance is shorter than the long distance. Fig.43B It is shown Fig.43A A graph of a Fourier transform result of a change in a value of at least one pixel characteristic parameter shown in FIG.

[0310] Fig.44A is a graph showing changes in the value of at least one pixel feature parameter included in a frame image captured by the front camera CF; a crosswalk mark B41 included in the frame image is located at a relatively close distance from the own vehicle. The close distance is shorter than the middle distance. Fig.44B It is shown Fig.44A A graph of a Fourier transform result of a change in a value of at least one pixel characteristic parameter shown in FIG.

[0311] Fig.42B , Fig.43B and Fig.44B Each of shows that the highest intensity local peak indicated by the arrow clearly appears in the corresponding one of the results of Fourier transform, and the value of the period λ increases with increasing distance from the own vehicle V to the crosswalk mark B41.

[0312] Fig.45 A routine for estimating the distance from the own vehicle V to the crosswalk mark B41 based on the frame image captured by the front camera CF according to the exemplary embodiment is schematically shown. Fig.45 In the figure, the reference symbol S is the abbreviation of Step. For example, Figure 2 The processor 21 shown in FIG. 1 is programmed to sequentially execute Fig.45 The operations in steps S201 to S206 shown in FIG.

[0313] In step S201, the processor 21 establishes a distance estimation area in a frame image captured by and transmitted from the front camera CF. The distance estimation area is a target area to be subjected to image processing in step S202 and corresponds to a bounding box set in the detection area Aw of the frame image.

[0314] Next, in step S202 , the processor 21 applies one or more predetermined image processing tasks, such as a grayscale conversion task, to the image data included in the distance estimation area established in step S201 .

[0315] For example, the processor 21 applies a grayscale conversion task to image information included in a distance estimation area of ​​a red, blue, green (RGB) frame image as a frame image to correspondingly generate grayscale image data including pixels, each of which has one of a predetermined number of grayscale brightness levels between white and black.

[0316] After the operation in step S202 , in step S203 , the processor 21 calculates an intensity-frequency spectrum of image data included in the distance estimation region of the frame image based on the results of one or more image processing tasks.

[0317] Specifically, the processor 21 generates a waveform of a change in the value of at least one pixel characteristic parameter included in the pixels of the image data included in the distance estimation area of ​​the frame image based on the grayscale brightness level of each pixel of the image data included in the distance estimation area of ​​the frame image, the at least one pixel characteristic parameter being, for example, the brightness level itself or the contrast of the brightness level relative to the reference contrast; the pixels are aligned in the horizontal direction of the frame image, i.e., in the detection width direction (see Fig.40 Then, the processor 21 applies at least one of various frequency analysis methods, such as Fourier transform, to the changed waveform to calculate the intensity-frequency spectrum accordingly, such as Fig.40 as shown in the example.

[0318] After the operation in step S203, in step S204, the processor 21 extracts a representative period from the intensity-frequency spectrum calculated in step S203, the representative period corresponding to a selected one of the local peaks of the intensity-frequency spectrum, for example, the highest local peak.

[0319] Next, in step S205, the processor 21 reads the correspondence information from the memory device 22, the correspondence information indicating the correspondence between the variable of the distance from the own vehicle V to the target object and the variable of the variation period λ (see, for example, Fig.38 Then, in step S206, the processor 21 uses the calculated representative period reference correspondence information to estimate the distance from the own vehicle V to the target object accordingly.

[0320] The target object whose distance is to be estimated by the driving ECU 2 of the exemplary embodiment is not limited to the crosswalk mark B41.

[0321] For example, the stop position mark B471 may be included in the target object whose distance is to be estimated by the driving ECU 2 of the exemplary embodiment. Fig.46 An example of the stop position mark B471 is shown in . The stop position mark B471 is located at a position on the road where, although it is not required by law and traffic regulations requiring the driver to stop are not necessary, it is recommended to stop and check for safety. Fig.46 The stop position mark B471 shown in FIG. 1 may also be referred to as a guide stop line in Japan.

[0322] In addition, the guardrail B1301 included in the roadside structure B13 positioned to face the forward direction of the own vehicle V and aligned in the direction intersecting the forward direction of the own vehicle V may be included in the target object whose distance is to be estimated by the driving ECU 2 of the exemplary embodiment. This is because the guardrail B1301 includes poles B1302 aligned at regular intervals in the direction intersecting the forward direction Df of the own vehicle V; the regular intervals are determined by a predetermined traffic standard. Three-dimensional objects such as standing signs, traffic light devices, etc. may be included in the target objects whose distance is to be estimated by the driving ECU 2 of the exemplary embodiment. How the driving ECU 2 detects information included in the periodic change of each three-dimensional object will be described later.

[0323] Fig.47 A T-shaped intersection Xr is shown, where the first road Rd1 on which the own vehicle V travels intersects with the second road Rd2 in the forward direction of the own vehicle V. In the T-shaped intersection Xr, a dotted line B4311 may be marked at a portion of one of the vehicle road edge lines B431, which portion is closer to the first road Rd1 than other portions; a portion of one of the vehicle road edge lines B431 is directly connected to the first road Rd1. The dotted line B4311 may be included in a target object whose distance is to be estimated by the driving ECU 2 of the exemplary embodiment. Such a dotted line B431 may be set in a Y-shaped intersection or a cross intersection.

[0324] The stop position mark B471 with a periodically changing pattern included in the road traffic mark B4, such as a dotted line or a dashed line, can be set at Fig.48 The roundabout Xr shown in FIG. 4 includes a roundabout road Rd3, a plurality of entry roads Rd4 entering the roundabout road Rd3, and a plurality of exit roads Rd5 exiting the roundabout road Rd3. Stop position marks B471 are each installed on a boundary between the roundabout road Rd3 and a corresponding one of the entry roads Rd4 to serve as an outer boundary line of the roundabout road Rd3, such as a roundabout road edge line.

[0325] In each of the entry roads Rd4, a crosswalk mark B41 or a stop line B42 may be installed just before the corresponding stop position mark B471. The installation position, installation size and / or shape of such a stop position mark B471 and the installation purpose are not limited to Fig.46 and Fig.48 The installation position, installation size and / or shape of the stop position mark B471 shown in the figure and the installation purpose. For example, the stop position mark B471 is not limited to having a simple dot or dashed line pattern. Specifically, the stop position mark B471 may have a double dashed line or a double dot-dash line pattern (see Fig.49 ), or may have a shark tooth line comprising inverted triangle markings aligned in rows (see Fig.50 or Fig.51 ). Depending on its installation position, installation size and / or shape and installation purpose, this stop position mark B471 can be called a "give way line" or a "give way line".

[0326] As described above, the driving ECU 2 according to the exemplary embodiment is configured to calculate information included in the periodic change of the value of at least one pixel characteristic parameter such as the brightness level included in the target object B based on the frame image captured by at least one camera 34 as an image sensor, and to perform estimation with higher accuracy based on the information included in the periodic change of the value of the calculated at least one pixel characteristic parameter. Therefore, this configuration makes it possible to perform driving control of the own vehicle V based on the estimated distance of the target object B at an appropriate time, and output information related to the estimated distance of the target object B to one or more passengers at an appropriate time.

[0327] Typical stereo camera techniques such as compound eye stereo technique or monocular stereo technique perform distance estimation based on two frame images captured at different positions or at different times. In contrast, the driving ECU 2 according to the exemplary embodiment is configured to perform distance estimation using a single frame image captured by at least one camera 34, so that the driving ECU 2 according to the exemplary embodiment is superior to the typical stereo technique in terms of processing speed and processing load required to perform distance estimation.

[0328] A typical example of such a target object B having a periodic pattern is a road surface mark B4 set in the intersection Xr, such as a crosswalk mark B41. The estimated distance to the road surface mark B4 set in the intersection Xr makes it possible to estimate the position of the intersection entrance Xr1, the intersection exit Xr2 and / or the center of the intersection Xr. Therefore, the driving ECU 2 makes it possible to perform (i) driving control of the own vehicle V and (ii) output information related to the estimated distance of the target object B to one or more passengers at an extremely appropriate time based on the estimated distance to, for example, the intersection entrance Xr1 when the own vehicle V is approaching the intersection Xr or has passed the intersection Xr. Note that the own vehicle V passing through the intersection Xr may include (i) the own vehicle V going straight through the intersection Xr, and turning right or left at the intersection Xr.

[0329] Revise

[0330] Although the exemplary embodiment of the present disclosure has been described above, the present disclosure is not limited to the exemplary embodiment. Specifically, the present disclosure includes various modifications and / or substitutions of the exemplary embodiment within the scope of the present disclosure.

[0331] Typical modifications of the exemplary embodiment are described below. In the typical modifications, the same reference numerals are assigned to the same or equivalent parts of the exemplary embodiment, so that the description of each of the same or equivalent parts of the typical modifications can adopt the description of the corresponding part of the same or equivalent parts of the exemplary embodiment, unless there is a technical contradiction or otherwise specified.

[0332] The present disclosure is not limited to the specific structures described in the exemplary embodiments. For example, the shape and / or configuration of the body V1 of the system-mounted vehicle V is not limited to a box shape, i.e., a roughly rectangular shape in plan view. The body panel V15 may be configured not to cover the top of the interior V2, or a portion of the body panel V15 covering the top of the interior V2 may be removable. The various applications of the system-mounted vehicle V, the position of each of the driver's seat V23 and the steering wheel V24, and the number of occupants in the system-mounted vehicle V may not be limited.

[0333] The definition of autonomous driving according to an exemplary embodiment, driving levels of autonomous driving, and various categories of autonomous driving are defined in the SAE J2016 standard, but the present disclosure is not limited thereto.

[0334] Specifically, autonomous driving in each of SAE levels 3 to 5 means that the vehicle system 1 functions as an autonomous driving system to perform all dynamic driving tasks in a corresponding one of SAE levels 3 to 5. For this reason, the above definition of autonomous driving according to the exemplary embodiment naturally does not include a requirement for the driver to monitor the traffic environment around the own vehicle V. The present disclosure is not limited to the above definition.

[0335] Specifically, the definition of autonomous driving may include not only autonomous driving without the requirement for the driver to monitor the traffic environment around the own vehicle V, but also autonomous driving with the requirement for the driver to monitor the traffic environment around the own vehicle V. For example, hands-free driving according to an exemplary embodiment may be interpreted as autonomous driving with the requirement for the driver to monitor the traffic environment around the own vehicle V. The concept of autonomous driving with the requirement for the driver to monitor the traffic environment around the own vehicle V may include partially autonomous driving, in which the driver D performs a part of the dynamic driving task, such as the task of monitoring the traffic environment around the own vehicle V. Partial autonomous driving may be evaluated as being essentially synonymous with advanced driving assistance. The road traffic system of each country may have restrictions on the type and / or conditions of autonomous driving, such as autonomous driving executable roads, the allowable driving speed range of autonomous driving, and lane change enablement / disablement. For this reason, the specifications of the present disclosure may be modified to comply with the road traffic system of each country.

[0336] The configuration of the vehicle system 1 according to the present disclosure is not limited to the vehicle system 1 of the above-described exemplary embodiment.

[0337] For example, the number of ADAS sensors 31, 32, 33, and 34 and the position of each of the ADAS sensors 31, 32, 33, and 34 can be freely determined. For example, the number of relatively expensive radar sensors 32 and lidar sensors 34 can be reduced as much as possible, thereby contributing to the earlier popularization of autonomous driving vehicles. The radar sensor 32 and the lidar sensor 34 can be omitted.

[0338] The locator 39 according to the present disclosure is not limited to the above-mentioned configuration including the inertial detector 392. Specifically, the locator 38 according to the present disclosure may be configured not to include the inertial detector 392, and may be configured to receive (i) a linear acceleration measured by an acceleration sensor provided outside the locator 39 as one of the behavior sensors 36 and (ii) an angular velocity measured by an angular velocity sensor provided outside the locator 39 as one of the behavior sensors 36. The locator 39 may be integrated with the HD map database 5.

[0339] The navigation system 6 may be communicatively connected to the HMI system 7 via a sub-network different from the vehicle communication network 10. The navigation system 6 may include a screen for displaying only a navigation image, which is a separate component from the HMI system 7. Alternatively, the navigation system 6 may constitute a part of the HMI system 7. For example, the navigation system 6 may be integrated with the main display device 703.

[0340] The HMI system 7 according to the present disclosure is not limited to the above-mentioned configuration including the instrument panel 702, the main display device 703, and the head-up display 704. Specifically, the HMI system 7 may be configured to include a single display device, such as a liquid crystal display device or an organic EL display device, used as both the instrument panel 702 and the main display device 703. In this modification, the instrument panel 702 may be designed as a part of the display area of ​​the single display device. Specifically, the instrument panel 702 may include a graphic tachometer, a graphic tachometer, a graphic speedometer, and a graphic water temperature meter, each of which includes a frame image, a scale image on the frame image, and an image of an indicator needle on the frame image.

[0341] The HMI system 7 may be configured to not include a heads-up display 704 .

[0342] Each ECU according to the exemplary embodiment is configured as, for example, a vehicle microcomputer including a CPU and / or an MPU, but the present disclosure is not limited thereto.

[0343] Specifically, part or all of each ECU may be configured as one or more digital circuits, such as one or more application specific integrated circuits (ASICs) or one or more field programmable gate array (FPGA) processors. That is, each ECU may include one or more vehicle microcomputers and one or more digital circuits at the same time.

[0344] The computer program (i.e., computer program instructions) described in the exemplary embodiment that enables the processor 21 to perform the various operations, tasks, and / or processes set forth above can be downloaded to the memory device 22 or upgraded through the vehicle communication module 4 using vehicle-to-everything (V2X) communication. The computer program instructions can be downloaded and / or upgraded through a terminal; the terminal is provided, for example, at a manufacturing plant, garage, or authorized distributor of the own vehicle V. The computer program can be stored in a memory card, optical disk, or disk that can be accessed by the processor 21 that can be read out. That is, a memory card, optical disk, or disk can be used as the memory device 22.

[0345] The functional configurations and methods described in the present disclosure may be implemented by a dedicated computer including a memory and a processor, the processor being programmed to execute one or more functions embodied by one or more computer programs.

[0346] The functional configurations and methods described in the present disclosure may also be implemented by a special-purpose computer including a processor composed of one or more special-purpose hardware logic circuits.

[0347] The functional configurations and methods described in the present disclosure may also be implemented by a processor system including a memory, a processor programmed to execute one or more functions embodied by one or more computer programs, and one or more hardware logic circuits.

[0348] One or more computer programs may be stored in a non-transitory storage medium as instructions to be executed by a computer or processor. The functional configurations and methods described in the present disclosure may be implemented as one or more computer programs or a non-transitory storage medium storing these one or more computer programs.

[0349] The present disclosure is not limited to the specific configurations described in the exemplary embodiments.

[0350] Specifically, the recognition target of the road surface marking recognition module 2132 may include the letter mark B44.

[0351] The exemplary embodiment uses the frequency analysis method explained above, but the present disclosure is not limited thereto.

[0352] For example, the Fourier transform according to the exemplary embodiment is applied to the value of at least one pixel feature parameter included in the pixels aligned one-dimensionally in the horizontal direction (i.e., the row direction) included in the bounding box. However, the Fourier transform according to the contents of the present disclosure is not limited to the one-dimensional Fourier transform applied to such one-dimensionally aligned pixels. Specifically, the two-dimensional Fourier transform can be applied to the two-dimensionally arranged pixels in at least a part or all of the two-dimensionally arranged bounding box included in the frame image to calculate the periodic change pattern of the value of at least one pixel feature parameter included in the two-dimensionally arranged pixels accordingly.

[0353] The present disclosure may use wavelet transform instead of Fourier transform.

[0354] Fig.38 The correspondence information shown may be stored as a lookup table or one or more relational expressions, and the processor 21 may use the calculated representative period to refer to the lookup table or relational expression to accordingly estimate the distance from the own vehicle V to the target object.

[0355] An exemplary embodiment calculates a periodic variation pattern of values ​​of at least one pixel feature parameter included in pixels of the captured frame image based on the captured frame image; the pixels are aligned in the horizontal direction of the captured frame image (i.e., the detection width direction). However, the present disclosure is not limited thereto. Specifically, if the target object is a three-dimensional object, the present disclosure may calculate a periodic variation pattern of values ​​of at least one pixel feature parameter included in pixels of the captured frame image based on the captured frame image; the pixels are aligned in the vertical direction of the captured frame image (i.e., the height direction of the three-dimensional object).

[0356] like Fig.41 As shown in the example in , the intensity-frequency spectrum shows that a plurality of local intensity peaks for the target object B in the horizontal direction appear at corresponding values ​​λ1, λ2, and λ3 in the cycle. In this case, the exemplary embodiment selects the value λ1 corresponding to the highest local peak in the cycle as a representative cycle, and calculates the distance from the own vehicle V to the target object B based on the representative cycle. However, the present disclosure is not limited to the above process.

[0357] Specifically, the present disclosure may be configured to estimate multiple distances (to be referred to as first estimated distances) from the own vehicle V to the target object B based on two or more selected values ​​corresponding to two or more local peaks having intensities higher than or equal to a predetermined threshold intensity in a cycle, and to use an alternative metric to estimate a second estimated distance from the own vehicle V to the target object B. Then, the present disclosure may be configured to finally estimate the distance from the own vehicle V to the target object B based on the first estimated distances and the second estimated distances.

[0358] For example, in Fig.41 In the example shown, the present disclosure may be configured to calculate the distance from the own vehicle V to the target object B based on the value λ1 in the cycle, the distance from the own vehicle V to the target object B based on the value λ2 in the cycle, and the distance from the own vehicle V to the target object B based on the value λ3 in the cycle as a first estimated distance. In addition, the present disclosure may be configured to obtain the distance from the own vehicle V to the target object B measured by an alternative ADAS sensor such as a lidar sensor 33. Then, the present disclosure may be configured to select one of the distances that is closest to the distance measured by the alternative ADAS sensor based on the value λ1, the value λ2, and the value λ3 in the cycle. The present disclosure may be configured to use the selected distance to perform vehicle control or calculate an average or weighted average between the selected distance and the distance measured by the alternative ADAS sensor.

[0359] The exemplary embodiment is configured to estimate the distance from the own vehicle V to the target object B mainly using the frame image captured by the front camera CF, but the present disclosure is not limited thereto. Specifically, the present disclosure may be configured to estimate the distance from the own vehicle V to the target object B using the frame image captured by the rear camera CB, the left camera CL, or the right camera CR.

[0360] The detection point data cloud including the LIDAR detection points Prr corresponding to the crosswalk mark B41 detected by the laser radar sensor 33 may have a periodic change pattern in the horizontal scanning direction Ds. Therefore, the distance estimation method according to the present disclosure may be applied to a detection point data cloud similar to a frame image detected by the laser radar sensor 33. Specifically, the present disclosure may be configured to: obtain information about how the change period of the change of the LIDAR detection point Prr of the detection point data cloud corresponding to the white drawn line of the crosswalk mark B41 changes according to the distance from the own vehicle V to the crosswalk mark B41; and based on the obtained information, correct the measured distance from the own vehicle V to the crosswalk mark B41. Therefore, the present disclosure may be configured to estimate the distance from the own vehicle V to the target object B using various information items measured by the ADAS sensor without limiting the image information.

[0361] Similar expressions such as "obtain", "calculate", "estimate", "detect" and "determine" can be substituted for each other unless the substitution produces technical inconsistency. The expression A is more than (greater than or other similar expressions) or equal to B and the expression A is more than B can be substituted for each other unless the substitution produces technical inconsistency. Similarly, the expression A is less than (less than or other similar expressions) or equal to B and the expression A is less than B can be substituted for each other unless the substitution produces technical inconsistency.

[0362] One or more components in the exemplary embodiments are not necessarily essential components, except for the following cases: (i) one or more components are described as one or more essential components; or (ii) one or more components are essential in principle.

[0363] Specific values ​​disclosed in the exemplary embodiments, each of which represents the number of components, physical quantities and / or ranges of physical parameters, are not limited thereto, except in the following cases: (i) the specific value is obviously necessary; or (ii) the specific value is necessary in principle.

[0364] The specific structure and orientation of each component described in the exemplary embodiment are not limited thereto, except in the following cases: (1) the specific structure and orientation are described as essential; or (2) the specific structure and orientation are required in principle. In addition, the specific structure or functional relationship between the components described in the exemplary embodiment is not limited thereto, except in the following cases: (1) the specific structure or functional relationship is described as essential; or (2) the specific structure or functional relationship is required in principle.

[0365] Modifications of the present disclosure are not limited to those described above. For example, the specific examples described above can be combined with each other unless the combination produces a technical inconsistency, and similarly the modifications set forth above can be combined with each other unless the combination produces a technical inconsistency. At least a portion of the exemplary embodiments can be combined with at least a portion of the modifications set forth above unless the combination produces a technical inconsistency.

[0366] The present disclosure includes the following first to twenty-first technical concepts.

[0367] The present disclosure includes the following first to fifth technical concept groups.

[0368] First Technical Concept Group (I)

[0369] [I-1]

[0370] A first aspect of the first technical concept group is a distance estimation device. The distance estimation device includes a memory device (22) storing distance estimation program instructions and a processor (21), the processor being configured to execute the distance estimation program instructions to perform the following operations accordingly: (i) based on an image captured by an image sensor mounted to a vehicle, calculating information included in the periodic variation of the value of at least one pixel feature parameter included in selected pixels of the image; and (ii) based on the information included in the calculated periodic variation, estimating the distance from the vehicle to a target object associated with the periodic variation.

[0371] [I-2]

[0372] A second aspect of the first technical concept group according to the first aspect resides in that the image captured by the image sensor is a single frame image.

[0373] [I-3]

[0374] The third aspect of the first technical concept group according to the first aspect or the second aspect is that the processor is configured to estimate the distance from the vehicle to the target object based on (i) information included in the calculated periodic changes and (ii) correspondence information indicating the correspondence between the variable indicating the distance from the vehicle to the target object and the variable of periodic changes.

[0375] [I-4]

[0376] A fourth aspect of the first technical concept group according to any one of the first to third aspects is that the processor is configured to perform Fourier transform or wavelet transform of a value of at least one pixel characteristic parameter to correspondingly calculate information included in the periodic change.

[0377] [I-5]

[0378] The fifth aspect of the first technical concept group according to any one of the first aspect to the fourth aspect is that the processor is configured to perform the following operations: (i) applying a frequency analysis method to a waveform of a periodic change of at least one pixel characteristic parameter to calculate an intensity-frequency spectrum accordingly; and (ii) obtaining a representative period corresponding to a local intensity peak selected in the intensity-frequency spectrum based on the intensity-frequency spectrum.

[0379] [I-6]

[0380] The sixth aspect of the first technical concept group according to the fifth aspect is that the intensity-frequency spectrum includes multiple local intensity peaks, and the processor is configured to obtain a representative period corresponding to a selected local intensity peak among the multiple local intensity peaks, and the one among the multiple local intensity peaks has the highest intensity peak among the multiple local intensity peaks.

[0381] [I-7]

[0382] The seventh aspect of the first technical concept group according to the fifth aspect or the sixth aspect is that the intensity-frequency spectrum includes multiple local intensity peaks, and the processor is configured to obtain a representative period based on a plurality of local intensity peaks, each of which has an intensity higher than or equal to a predetermined threshold intensity, which are selected as local intensity peaks among the multiple local intensity peaks.

[0383] [I-8]

[0384] The eighth aspect of the first technical concept group according to the eighth aspect is that the processor is configured to perform the following operations: (i) extracting a value of a period of a periodically changing waveform from an intensity-frequency spectrum, the value of the period corresponding to a corresponding plurality of local intensity peaks; (ii) estimating a first estimated distance based on the value of the period; (iii) estimating a second estimated distance using an alternative metric; and (iv) finally estimating the distance from the vehicle to the target object based on the first estimated distance and the second estimated distance.

[0385] [I-9]

[0386] The ninth aspect of the first technical concept group according to any one of the first aspect to the eighth aspect is that the processor is configured to calculate information included in the periodic change of the value of at least one pixel characteristic parameter in selected pixels of the image, and the selected pixels are aligned in a detection width direction intersecting with the detection direction, and the detection direction extends from the image sensor toward a target object (B) located in the field of view of the image sensor.

[0387] [I-10]

[0388] A tenth aspect of the first technical concept group according to any one of the first aspect to the ninth aspect resides in that the detection direction is a traveling direction of the vehicle.

[0389] [I-11]

[0390] An eleventh aspect of the first technical concept group according to any one of the first to tenth aspects resides in that the target object is a road surface marking (B41) installed on a road on which a vehicle travels, the road surface marking having a periodic pattern in a width direction of the road.

[0391] [I-12]

[0392] The twelfth aspect of the first technical concept group according to any one of the first aspect to the eleventh aspect is that a road surface marking (B41) is set in an intersection (Xr) having an entrance (Xr1) and an exit (Xr2), and the processor is configured to estimate the position of at least one of the entrance and the exit based on the distance from the vehicle to the road surface marking.

[0393] [I-13]

[0394] The thirteenth aspect of the first technical concept group according to the twelfth aspect is that the vehicle includes a notification unit (7) for notifying one or more passengers in the vehicle of information, and the processor is configured to perform at least one of (i) driving control of the vehicle and (ii) outputting information to one or more passengers through the notification unit based on the estimated position of at least one of the entrance and exit when the vehicle approaches an intersection or has passed the intersection.

[0395] Second technical concept group (II)

[0396] [II-1]

[0397] The first aspect of the second technical concept group is a distance estimation method. The distance estimation method includes: (i) based on an image captured by an image sensor installed on a vehicle, calculating information included in the periodic change of the value of at least one pixel feature parameter included in selected pixels of the image; and (ii) based on the information included in the calculated periodic change, estimating the distance from the vehicle to a target object associated with the periodic change.

[0398] [II-2]

[0399] A second aspect of the second technical concept group according to the first aspect resides in that the image captured by the image sensor is a single frame image.

[0400] [II-3]

[0401] The third aspect of the second technical concept group according to the first aspect or the second aspect is that the estimation estimates the distance from the vehicle to the target object based on (i) information included in the calculated periodic changes and (ii) correspondence information indicating the correspondence between the variable of the distance from the vehicle to the target object and the variable of the periodic changes.

[0402] [II-4]

[0403] A fourth aspect of the second technical concept group according to any one of the first to third aspects resides in that the calculation performs Fourier transform or wavelet transform of a value of at least one pixel characteristic parameter to correspondingly calculate information included in the periodic change.

[0404] [II-5]

[0405] The fifth aspect of the second technical concept group according to any one of the first to fourth aspects is that the estimation applies a frequency analysis method to a waveform of a periodic change of at least one pixel characteristic parameter to calculate an intensity-frequency spectrum accordingly. The estimation obtains a representative period corresponding to a local intensity peak selected in the intensity-frequency spectrum based on the intensity-frequency spectrum.

[0406] [II-6]

[0407] The sixth aspect of the second technical concept group according to the fifth aspect is that the intensity-frequency spectrum includes multiple local intensity peaks, and the estimation obtains a representative period corresponding to one of the multiple local intensity peaks selected as a local intensity peak, and the one of the multiple local intensity peaks has the highest intensity peak among the multiple local intensity peaks.

[0408] [II-7]

[0409] The seventh aspect of the second technical concept group according to the fifth aspect or the sixth aspect is that the intensity-frequency spectrum includes multiple local intensity peaks, and the estimation is based on obtaining a representative period based on a plurality of local intensity peaks, each of which has an intensity higher than or equal to a predetermined threshold intensity, which are selected as local intensity peaks among the multiple local intensity peaks.

[0410] [II-8]

[0411] The eighth aspect of the second technical concept group according to the eighth aspect is that the estimation extracts the value of the period of the periodically changing waveform from the intensity-frequency spectrum, the value of the period corresponds to the corresponding plurality of local intensity peaks, and estimates the first estimated distance based on the value of the period. Then, the estimation estimates the second estimated distance using an alternative metric, and finally estimates the distance from the vehicle to the target object based on the first estimated distance and the second estimated distance.

[0412] [II-9]

[0413] The ninth aspect of the second technical concept group according to any one of the first aspect to the eighth aspect is that the calculation calculates information included in the periodic change of the value of at least one pixel characteristic parameter in selected pixels of the image, and the selected pixels are aligned in a detection width direction intersecting with the detection direction, and the detection direction extends from the image sensor toward a target object (B) located in the field of view of the image sensor.

[0414] [II-10]

[0415] A tenth aspect of the second technical concept group according to any one of the first aspect to the ninth aspect resides in that the detection direction is a traveling direction of the vehicle.

[0416] [II-11]

[0417] An eleventh aspect of the second technical concept group according to any one of the first aspect to the tenth aspect resides in that the target object is a road surface marking (B41) installed on a road on which a vehicle travels, the road surface marking having a periodic pattern in a width direction of the road.

[0418] [II-12]

[0419] The twelfth aspect of the second technical concept group according to any one of the first aspect to the eleventh aspect is that a road surface marking (B41) is set in an intersection (Xr) having an entrance (Xr1) and an exit (Xr2), and the estimation estimates the position of at least one of the entrance and the exit based on the distance from the vehicle to the road surface marking.

[0420] [II-13]

[0421] The thirteenth aspect of the second technical concept group according to the twelfth aspect is that the vehicle includes a notification unit (7) for notifying one or more passengers in the vehicle of information, and when the vehicle approaches an intersection or has passed the intersection, the estimation is based on the estimated position of at least one of the entrance and exit to perform at least one of (i) driving control of the vehicle and (ii) outputting information to one or more passengers through the notification unit.

[0422] The third technical concept group (III)

[0423] [III-1]

[0424] A first aspect of the third technical concept group is a processor program product comprising a storage medium (22) readable by a processor (21) installed in a vehicle (V) and a distance estimation program instruction. The distance estimation program instruction causes the processor to perform the following operations: (i) based on an image captured by an image sensor installed to the vehicle, calculate information included in the periodic change of the value of at least one pixel feature parameter included in selected pixels of the image; and (ii) based on the information included in the calculated periodic change, estimate the distance from the vehicle to a target object associated with the periodic change.

[0425] [III-2]

[0426] A second aspect of the third technical concept group according to the first aspect resides in that the image captured by the image sensor is a single frame image.

[0427] [III-3]

[0428] The third aspect of the first technical concept group according to the first aspect or the second aspect is that the distance estimation program instructions enable the processor to estimate the distance from the vehicle to the target object based on (i) information included in the calculated periodic changes and (ii) correspondence information indicating the correspondence between the variable indicating the distance from the vehicle to the target object and the variable of periodic changes.

[0429] [III-4]

[0430] A fourth aspect of the third technical concept group according to any one of the first to third aspects is that the distance estimation program instructions cause the processor to perform Fourier transform or wavelet transform of the value of at least one pixel feature parameter to correspondingly calculate information included in the periodic change.

[0431] [III-5]

[0432] The fifth aspect of the third technical concept group according to any one of the first aspect to the fourth aspect is that the distance estimation program instructions cause the processor to perform the following operations: (i) apply a frequency analysis method to a waveform of a periodic change of at least one pixel characteristic parameter to calculate an intensity-frequency spectrum accordingly; and (ii) obtain a representative period corresponding to a local intensity peak selected in the intensity-frequency spectrum based on the intensity-frequency spectrum.

[0433] [III-6]

[0434] The sixth aspect of the third technical concept group according to the fifth aspect is that the intensity-frequency spectrum includes multiple local intensity peaks, and the distance estimation program instructions enable the processor to obtain a representative period corresponding to a selected local intensity peak among the multiple local intensity peaks, and the one of the multiple local intensity peaks has the highest intensity peak among the multiple local intensity peaks.

[0435] [III-7]

[0436] The seventh aspect of the third technical concept group according to the fifth aspect or the sixth aspect is that the intensity-frequency spectrum includes a plurality of local intensity peaks. The distance estimation program instructions cause the processor to obtain a representative cycle based on a plurality of local intensity peaks, each of which has an intensity higher than or equal to a predetermined threshold intensity, which are selected as local intensity peaks among the plurality of local intensity peaks.

[0437] [III-8]

[0438] The eighth aspect of the third technical concept group according to the eighth aspect is that the distance estimation program instructions enable the processor to perform the following operations: (i) extract the value of the period of the periodically changing waveform from the intensity-frequency spectrum, the value of the period corresponding to the corresponding multiple local intensity peaks; (ii) estimate a first estimated distance based on the value of the period; (iii) estimate a second estimated distance using an alternative metric; and (iv) finally estimate the distance from the vehicle to the target object based on the first estimated distance and the second estimated distance.

[0439] [III-9]

[0440] The ninth aspect of the third technical concept group according to any one of the first aspect to the eighth aspect is that the distance estimation program instructions cause the processor to calculate information included in the periodic change of the value of at least one pixel feature parameter in selected pixels of the image, the selected pixels are aligned in a detection width direction intersecting with the detection direction, and the detection direction extends from the image sensor toward a target object (B) located in the field of view of the image sensor.

[0441] [III-10]

[0442] A tenth aspect of the third technical concept group according to any one of the first aspect to the ninth aspect is that the detection direction is a traveling direction of the vehicle.

[0443] [III-11]

[0444] An eleventh aspect of the third technical concept group according to any one of the first to tenth aspects resides in that the target object is a road surface marking (B41) installed on a road on which a vehicle travels, the road surface marking having a periodic pattern in a width direction of the road.

[0445] [III-12]

[0446] The twelfth aspect of the third technical concept group according to any one of the first to eleventh aspects is that a road surface mark (B41) is provided in an intersection (Xr) having an entrance (Xr1) and an exit (Xr2). The distance estimation program instructions cause the processor to estimate the position of at least one of the entrance and the exit based on the distance from the vehicle to the road surface mark.

[0447] [III-13]

[0448] The thirteenth aspect of the third technical concept group according to the twelfth aspect is that the vehicle includes a notification unit (7) for notifying one or more passengers in the vehicle of information. The distance estimation program instructions enable the processor to perform at least one of (i) driving control of the vehicle and (ii) outputting information to one or more passengers through the notification unit based on the estimated position of at least one of the entrance and exit when the vehicle approaches an intersection or has passed through an intersection. The first aspect of the twelfth technical concept group is a non-transitory storage medium (22) that can be read by a processor (21) installed in the own vehicle (V). The non-transitory storage medium stores road shape estimation program instructions. The road shape estimation program instructions enable the processor

[0449] Fourth Technical Concept Group (IV)

[0450] [IV-1]

[0451] A first aspect of the fourth technical concept group is a storage medium (22) that can be read by a processor (21) installed in a vehicle (V). The storage medium stores distance estimation program instructions. The distance estimation program instructions cause the processor to perform the following operations: (i) based on an image captured by an image sensor installed to the vehicle, calculate information included in the periodic change of the value of at least one pixel feature parameter included in selected pixels of the image; and (ii) based on the information included in the calculated periodic change, estimate the distance from the vehicle to a target object associated with the periodic change.

[0452] [IV-2]

[0453] A second aspect of the fourth technical concept group according to the first aspect resides in that the image captured by the image sensor is a single frame image.

[0454] [IV-3]

[0455] The third aspect of the fourth technical concept group according to the first aspect or the second aspect is that the distance estimation program instructions enable the processor to estimate the distance from the vehicle to the target object based on (i) information included in the calculated periodic changes and (ii) correspondence information indicating the correspondence between the variable indicating the distance from the vehicle to the target object and the variable of periodic changes.

[0456] [IV-4]

[0457] A fourth aspect of the fourth technical concept group according to any one of the first to third aspects is that the distance estimation program instructions cause the processor to perform Fourier transform or wavelet transform of the value of at least one pixel feature parameter to correspondingly calculate information included in the periodic change.

[0458] [IV-5]

[0459] The fifth aspect of the fourth technical concept group according to any one of the first aspect to the fourth aspect is that the distance estimation program instructions enable the processor to perform the following operations: (i) apply a frequency analysis method to a waveform of a periodic change of at least one pixel characteristic parameter to calculate an intensity-frequency spectrum accordingly; and (ii) obtain a representative period corresponding to a local intensity peak selected in the intensity-frequency spectrum based on the intensity-frequency spectrum.

[0460] [IV-6]

[0461] The sixth aspect of the fourth technical concept group according to the fifth aspect is that the intensity-frequency spectrum includes multiple local intensity peaks, and the distance estimation program instructions enable the processor to obtain a representative period corresponding to a selected local intensity peak among the multiple local intensity peaks, and the one of the multiple local intensity peaks has the highest intensity peak among the multiple local intensity peaks.

[0462] [IV-7]

[0463] The seventh aspect of the fourth technical concept group according to the fifth aspect or the sixth aspect is that the intensity-frequency spectrum includes a plurality of local intensity peaks. The distance estimation program instructions cause the processor to obtain a representative cycle based on a plurality of local intensity peaks, each of which has an intensity higher than or equal to a predetermined threshold intensity, which are selected as local intensity peaks among the plurality of local intensity peaks.

[0464] [IV-8]

[0465] The eighth aspect of the fourth technical concept group according to the eighth aspect is that the distance estimation program instructions enable the processor to perform the following operations: (i) extract the value of the period of the periodically changing waveform from the intensity-frequency spectrum, the value of the period corresponding to the corresponding multiple local intensity peaks; (ii) estimate a first estimated distance based on the value of the period; (iii) estimate a second estimated distance using an alternative metric; and (iv) finally estimate the distance from the vehicle to the target object based on the first estimated distance and the second estimated distance.

[0466] [IV-9]

[0467] The ninth aspect of the fourth technical concept group according to any one of the first aspect to the eighth aspect is that the distance estimation program instructions cause the processor to calculate information included in the periodic change of the value of at least one pixel feature parameter in selected pixels of the image, the selected pixels are aligned in a detection width direction intersecting with the detection direction, and the detection direction extends from the image sensor toward a target object (B) located in the field of view of the image sensor.

[0468] [IV-10]

[0469] A tenth aspect of the fourth technical concept group according to any one of the first aspect to the ninth aspect resides in that the detection direction is a traveling direction of the vehicle.

[0470] [IV-11]

[0471] An eleventh aspect of the fourth technical concept group according to any one of the first to tenth aspects resides in that the target object is a road surface marking (B41) installed on a road on which a vehicle travels, the road surface marking having a periodic pattern in a width direction of the road.

[0472] [IV-12]

[0473] The twelfth aspect of the fourth technical concept group according to any one of the first to eleventh aspects is that a road surface mark (B41) is provided in an intersection (Xr) having an entrance (Xr1) and an exit (Xr2). The distance estimation program instructions cause the processor to estimate the position of at least one of the entrance and the exit based on the distance from the vehicle to the road surface mark.

[0474] [IV-13]

[0475] The thirteenth aspect of the fourth technical concept group according to the twelfth aspect is that the vehicle includes a notification unit (7) for notifying one or more passengers in the vehicle of information. The distance estimation program instructions cause the processor to perform at least one of (i) driving control of the vehicle and (ii) outputting information to one or more passengers through the notification unit based on the estimated position of at least one of an entrance and an exit when the vehicle approaches an intersection or has passed through the intersection.

[0476] Fifth Technical Concept Group (V)

[0477] [V-1]

[0478] The first aspect of the fifth technical concept group is a distance estimation method. The distance estimation method includes:

[0479] (i) establishing a distance estimation region in an image captured by an image sensor mounted to the vehicle,

[0480] (ii) applying at least one image processing task to image data included in the distance estimation region,

[0481] (iii) calculating a waveform of periodic changes in the value of at least one pixel characteristic parameter included in the image data based on a result of applying at least one image processing task,

[0482] (iv) applying a frequency analysis method to the periodically varying waveform to calculate the intensity-frequency spectrum accordingly,

[0483] (v) obtaining, according to the intensity-frequency spectrum, a representative period corresponding to a local intensity peak selected in the intensity-frequency spectrum,

[0484] (vi) reading, from the memory device, correspondence information indicating a correspondence between a variable of a distance from the vehicle to the target object and a variable of a representative period,

[0485] (vii) Estimate the distance from the vehicle to the target object based on the representative period obtained in (i) and (ii) the correspondence information.

[0486] [V-2]

[0487] A second aspect of the fifth technical concept group according to the first aspect resides in that the image captured by the image sensor is a single frame image.

[0488] [V-3]

[0489] A third aspect of the fifth technical concept group according to the first aspect or the second aspect resides in that the application applies Fourier transform or wavelet transform as a frequency analysis method to a periodically varying waveform to calculate an intensity-frequency spectrum accordingly.

[0490] [V-4]

[0491] A fourth aspect of the fifth technical concept group according to any one of the first to third aspects resides in that the representative period corresponds to a selected local intensity peak in the intensity-frequency spectrum.

[0492] [V-5]

[0493] The fifth aspect of the fifth technical concept group according to any one of the first aspect to the fourth aspect is that the intensity-frequency spectrum includes multiple local intensity peaks, and the representative period corresponding to one of the multiple local intensity peaks selected is obtained, and the one of the multiple local intensity peaks has the highest intensity peak among the multiple local intensity peaks.

[0494] [V-6]

[0495] The sixth aspect of the fifth technical concept group according to the fourth aspect or the fifth aspect is that the intensity-frequency spectrum includes a plurality of local intensity peaks. The representative period is obtained based on a plurality of local intensity peaks, each of which has an intensity higher than or equal to a predetermined threshold intensity, which are selected from the plurality of local intensity peaks.

[0496] [V-7]

[0497] The seventh aspect of the fifth technical concept group according to the fourth aspect or the fifth aspect is that the estimation extracts the value of the period of the periodically changing waveform from the intensity-frequency spectrum, the value of the period corresponding to the corresponding multiple local intensity peaks; estimates a first estimated distance based on the value of the period; estimates a second estimated distance using an alternative metric; and finally estimates the distance from the vehicle to the target object based on the first estimated distance and the second estimated distance.

[0498] [V-8]

[0499] An eighth aspect of the fifth technical concept group according to any one of the first aspect to the eighth aspect resides in that the detection direction is a traveling direction of the vehicle.

[0500] [V-9]

[0501] The ninth aspect of the fifth technical concept group according to any one of the first aspect to the eighth aspect is that the representative period in the detection width direction intersecting with the detection direction is obtained and calculated, and the detection direction extends from the image sensor toward the target object (B) located in the field of view of the image sensor.

[0502] [V-10]

[0503] A tenth aspect of the fifth technical concept group according to any one of the first aspect to the ninth aspect resides in that the target object is a road surface marking (B41) installed on a road on which a vehicle travels, the road surface marking having a periodic pattern in a width direction of the road.

[0504] [V-11]

[0505] The eleventh aspect of the fifth technical concept group according to any one of the first to eleventh aspects is that a road surface mark (B41) is provided in an intersection (Xr) having an entrance (Xr1) and an exit (Xr2). The estimation estimates a position of at least one of the entrance and the exit based on a distance from the vehicle to the road surface mark.

[0506] [V-12]

[0507] According to the twelfth aspect of the fifth technical concept group of the eleventh aspect, the vehicle includes a notification unit (7) for notifying one or more passengers in the vehicle of information. When the vehicle approaches an intersection or has passed through the intersection, the estimation is based on the estimated position of at least one of the entrance and the exit to perform at least one of (i) driving control of the vehicle and (ii) outputting information to one or more passengers through the notification unit.

Claims

1. A distance estimation method, comprising: establishing a distance estimation region in an image captured by an image capture device mounted to the vehicle; applying at least one image processing task to image data included in the distance estimation region; calculating a waveform of periodic variation of a value of at least one pixel characteristic parameter included in the image data based on a result of applying the at least one image processing task; applying a frequency analysis method to the periodically varying waveform to calculate an intensity-frequency spectrum accordingly; Obtaining, according to the intensity-frequency spectrum, a representative period corresponding to a local intensity peak selected in the intensity-frequency spectrum; reading, from a memory device, correspondence information indicating a correspondence between a variable of a distance from the vehicle to a target object and a variable of the representative cycle; as well as The distance from the vehicle to the target object is estimated based on (i) the obtained representative period and (ii) the correspondence information.

2. The distance estimation method according to claim 1, wherein: The image captured by the image capture device is a single-frame image.

3. The distance estimation method according to claim 1 or 2, wherein: The applying applies a Fourier transform or a wavelet transform of the value of the at least one pixel characteristic parameter to calculate the representative period accordingly.

4. The distance estimation method according to any one of claims 1 to 3, wherein: The representative period corresponds to a selected local intensity peak in the intensity-frequency spectrum.

5. The distance estimation method according to claim 4, wherein: The intensity-frequency spectrum comprises a plurality of local intensity peaks; and The obtaining obtains a representative period corresponding to one of the plurality of local intensity peaks as the selected local intensity peak, the one of the plurality of local intensity peaks having a highest intensity peak among the plurality of local intensity peaks.

6. The distance estimation method according to claim 4 or 5, wherein: The intensity-frequency spectrum comprises a plurality of local intensity peaks; and The obtaining obtains the representative period based on a plurality of local intensity peaks each having an intensity higher than or equal to a predetermined threshold intensity, included in the plurality of local intensity peaks as the selected local intensity peak.

7. The distance estimation method according to any one of claims 4 to 6, wherein: The estimation comprises extracting a value of a period of the periodically changing waveform from the intensity-frequency spectrum, wherein the value of the period corresponds to a plurality of corresponding local intensity peaks; and estimating a first estimated distance based on the value of the period; estimating a second estimated distance using the alternative metric; And finally estimating the distance from the vehicle to the target object based on the first estimated distance and the second estimated distance.

8. The distance estimation method according to any one of claims 1 to 7, wherein: The obtaining calculates a representative period in a detection width direction intersecting a detection direction extending from an image sensor toward the target object located in a field of view of the image capture device.

9. The distance estimation method according to claim 8, wherein: The detection direction is a traveling direction of the vehicle.

10. The distance estimation method according to any one of claims 1 to 9, wherein: The target object is a road surface marking installed on the road on which the vehicle travels, the road surface marking having a periodic pattern in the width direction of the road.

11. The distance estimation method according to claim 10, wherein: The road surface marking is provided in an intersection having an entrance and an exit; and The estimating estimates a location of at least one of the entrance and the exit based on a distance from the vehicle to the road surface marking.

12. The distance estimation method according to claim 11, wherein: The vehicle includes a notification unit for notifying one or more occupants of the vehicle of information; as well as When the vehicle approaches the intersection or has passed the intersection, the estimation performs at least one of the following based on the estimated position of at least one of the entrance and the exit: (i) driving control of the vehicle and (ii) outputting information to the one or more passengers through the notification unit.

13. A storage medium capable of being read by a processor installed in a host vehicle, the storage medium storing distance estimation program instructions, the distance estimation program instructions causing the processor to: calculating, based on an image captured by an image sensor mounted to a vehicle, information included in a periodic variation in a value of at least one pixel characteristic parameter included in selected pixels of the image; and A distance from the vehicle to a target object associated with the periodic change is estimated based on information included in the calculated periodic change.

14. A distance estimation device, comprising: a memory device storing distance estimation program instructions; as well as a processor configured to execute the distance estimation program instructions to accordingly: (i) calculating, based on an image captured by an image sensor mounted to a vehicle, information included in a periodic variation in the value of at least one pixel characteristic parameter included in selected pixels of the image; and (ii) estimating a distance from the vehicle to a target object associated with the periodic change based on information included in the calculated periodic change.

Images