Parking space learning method and device

Abstract

A processor of a device for learning a recognition rule for at least one parking space around a vehicle is configured to execute learning program instructions to objectify at least one parking space, which is at least partially included in a training image, as at least one objectified parking space. This objectified parking space consists of (i) several vertices (BP) of the at least one parking space, (ii) a midpoint (BPC) of the at least one parking space, and (iii) attribute information about the at least one parking space. The processor is configured to execute learning program instructions to generate annotation data for the objectified parking space and, based on the training image, to learn a recognition rule for the at least one parking space based on the generated annotation data.

Description

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application is based on and claims priority from Japanese Patent Application No. 2023-168680, filed on September 28, 2023, the disclosure of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present disclosure relates to parking space learning methods and apparatus. BACKGROUND

[0003] Research and development for driving assistance technologies and / or autonomous driving technologies have continued actively. Various types of proposals have been made regarding driving assistance technologies and / or autonomous driving technologies, one of which is disclosed in International Patent Publication WO 2019 / 202397.

[0004] Driving assistance technologies aim to reduce driver burden and enable the driver to drive vehicles comfortably and safely. These driving assistance technologies include, for example, technologies related to following distance control, lane keeping assistance control, lane change assistance control, parking assistance control, obstacle warning, collision avoidance assistance control, or other vehicle control technologies. These autonomous driving technologies aim to make vehicles drive automatically, i.e., autonomously, without requiring driver intervention. Various sensory devices, such as cameras and / or radar devices, are installed in a vehicle with such driving assistance technologies and / or driving support technologies.The various sensory devices are configured to sense environmental situations surrounding the vehicle. Autonomous control operations, such as autonomous steering, autonomous driving, and / or autonomous braking, of a traveling autonomous vehicle are performed based on the environmental situations sensed by the sensory devices and / or map information providing a visual representation of a region surrounding the current location of the autonomous vehicle. In particular, the autonomous control operations of a traveling autonomous vehicle can be performed using highly accurate map information including road data of each lane surrounding the autonomous vehicle, making it possible to improve the safety and reliability of the traveling autonomous vehicle's autonomous control operations. OVERVIEW

[0005] Research and development for these autonomous driving and driver assistance technologies have recently accelerated with a view to improving the accuracy of object detection and / or recognition by sensory devices and improving user-friendliness. Therefore, further improvement of these autonomous driving and driver assistance technologies will enable the earlier and wider adoption of advanced driver assistance and / or autonomous vehicles.

[0006] In view of the above circumstances, the present disclosure provides an apparatus for learning a recognition rule of at least one parking space around a vehicle. The apparatus includes a storage device storing learning program instructions and a processor configured to execute the learning program instructions to correspondingly objectify at least one parking space, at least partially included in a learning image, as at least one objectified parking space consisting of (i) a plurality of vertices (BP) of the at least one parking space, (ii) a center point (BPC) of the at least one parking space, and (iii) attribute information about the at least one parking space.The processor (21) is configured to execute the learning program instructions to generate annotation data for the objectified at least one parking space and, starting from the learning image, to execute the learning of a recognition rule of the at least one parking space based on the generated annotation data for the objectified at least one parking space.

[0007] The present disclosure provides a method for learning a recognition rule of at least one parking space around a vehicle (V), the method comprising (1) Objectifying at least one parking space, which is at least partially included in a learning image, as at least one objectified parking space, which consists of (i) several corner points (BP) of the at least one parking space, (ii) a center point (BPC) of the at least one parking space and (iii) attribute information about the at least one parking space, (2) Generating annotation data for the objectified at least one parking space, (3) Starting from the learning image, learning a recognition rule of the at least one parking space based on the generated annotation data for the objectified at least one parking space.

[0008] The present disclosure provides a processor program product readable by a processor (21) installed in a vehicle (V). The processor program product includes learning program instructions that cause the processor (1) to objectify at least one parking space, which is at least partially included in a learning image, as at least one objectified parking space, which consists of (i) several corner points (BP) of the at least one parking space, (ii) a center point (BPC) of the at least one parking space and (iii) attribute information about the at least one parking space, (2) to generate annotation data for the objectified at least one parking space, (3) starting from the learning image, to carry out the learning of a recognition rule of the at least one parking space based on the generated annotation data for the objectified at least one parking space.

[0009] It should be noted that each reference numeral in parentheses assigned to a corresponding element in the present disclosure merely represents an example of a relationship between the corresponding element and a corresponding specific element described in the following embodiments described below, and therefore the present disclosure is not limited to the reference numerals in parentheses. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Further aspects of the present disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings, in which: Fig. 1 is a plan view schematically illustrating a vehicle in which a vehicle system including a driving ECU serving as a control device according to a basic embodiment; Fig. 2 is a schematic block diagram showing an overall structure of the Fig. 1 illustrated vehicle system; Fig. 3 is a plan view schematically illustrating target object detection using one of the Fig. 2 illustrated sonar sensors; Fig. 4 is a plan view schematically illustrating target object detection using a pair of Fig. 2 illustrated sonar sensors; Fig. 5 is a plan view schematically illustrating target object detection using each of Fig. 2 illustrated sonar sensor; Fig. 6 is a plan view schematically showing a parking space detection using a Fig. 2 illustrated first sonar sensor; Fig. 7 is a plan view schematically showing target object detection using a Fig. 2 illustrated radar sensor; Fig. 8 is a plan view schematically showing target object detection using a Fig. 2 illustrated laser radar sensor; Fig. 9 is a perspective view schematically showing a configuration and operations of the Fig. 8 illustrated laser radar sensor; Fig. 10 is a diagram schematically illustrating target object detection using a Fig. 2 illustrated camera illustrated; Fig. 11 is a diagram schematically illustrating target object detection using a Fig. 2 illustrated camera illustrated; Fig. 12 is a diagram schematically illustrating driver condition detection using a Fig. 2 illustrated driver condition monitor; Fig. 13 is a diagram schematically illustrating driver condition detection using a Fig. 2 illustrated driver condition monitor; Fig. 14 is a diagram showing a schematic configuration of a Fig. 2 illustrated field of view display; Fig. 15 is a schematic diagram showing (i) a road surface in front of a vehicle through a front windscreen and (ii) a part of a dashboard viewed from a Fig. 14 illustrated drivers from illustrated; Fig. 16 is a block diagram illustrating a schematic functional configuration provided by the Fig. 2 illustrated driving ECU is to be implemented; Fig. 17 is a block diagram illustrating a schematic functional configuration provided by a Fig. 16 illustrated identification unit is to be implemented; Fig. 18 is a schematic view illustrating a first example of target objects to be identified by the basic embodiment; Fig. 19 is a schematic view illustrating a second example of target objects to be identified by the basic embodiment; Fig. 20 is a schematic view illustrating a third example of target objects to be identified by the basic embodiment; Fig. 21 is a schematic view illustrating a fourth example of target objects to be identified by the basic embodiment; Fig. 22 is a schematic view illustrating a fifth example of target objects to be identified by the basic embodiment; Fig. 23 is a schematic view illustrating a sixth example of target objects to be identified by the basic embodiment; Fig. 24 is a schematic view illustrating a fifth example of a target object to be identified by the basic embodiment; Fig. 25 is a schematic view illustrating a sixth example of a target object to be identified by the basic embodiment; Fig. 26 is a schematic view illustrating a seventh example of a target object to be identified by the basic embodiment; Fig. 27 is a diagram schematically illustrating traffic signs as an eighth example of target objects; Fig. 28 is a schematic diagram illustrating a first example indicating how to use detection results of target objects; Fig. 29 is a schematic diagram illustrating a second example indicating how to use detection results of target objects; Fig. 30 is a schematic block diagram of a parking space detecting apparatus according to the first embodiment; Fig. 31 is a schematic view illustrating annotation information displayed by a Fig. 30 illustrated parking space annotation generator; Fig. 32 is an example of a learning image including the host vehicle having entered a parking space and several parking spaces arranged around the host vehicle; Fig. 33 is a view showing on a large scale part of a Fig. 32 illustrated target images; Fig. 34 is a diagram illustrating a coding of the existence of a parking space center point in a Fig. 33 illustrated grids; Fig. 35 is a view showing a horizontal position of the parking space center in a Fig. 33 illustrated grids; Fig. 36 is a diagram showing a coding of the horizontal position of the Fig. 35 illustrated parking space center point; Fig. 37 is a view showing a vertical position of the parking space center in a Fig. 33 illustrated grids; Fig. 38 is a diagram showing a first coding of the vertical position of the Fig. 37 illustrated parking space center point; Fig. 39 is a diagram illustrating the first coding of the existence of each parking space center point in the entire target image shown in Fig. 32 is illustrated; Fig. 40 is a diagram illustrating the first encoding of the horizontal position of each parking space center point in the entire target image shown in Fig. 32 is illustrated; Fig. 41 is a diagram illustrating the first encoding of the vertical position of each parking space center point in the entire target image shown in Fig. 32 is illustrated; Fig. 42 is a diagram illustrating a parking space annotation code including first coded data containing the Fig. 39 to Fig. 41 illustrated coded results; Fig. 43 is a view illustrating on a large scale a selected 3 x 3 grid consisting of nine cells separated by a square with thick lines in Fig. 42 selected cells and eight cells surrounding the selected cell; Fig. 44 is a diagram schematically illustrating a connection number table used to generate a second encoding of a Fig. 43 illustrated connection vector; Fig. 45 is a diagram schematically showing a coded result of the Fig. 43 illustrated connection vector using the Fig. 44 illustrated connection number table; Fig. 46 is a diagram illustrating a parking space annotation code including the first coded data and second coded data representing the Fig. 45 illustrated coded results include; Fig. 47 is a view illustrating, on a large scale, a selected 3 x 3 grid and illustrating information about segment lines as part of attribute information; Fig. 48 is a view illustrating a third encoding of the information about a left-sided segment line shown in Fig. 47 is illustrated; Fig. 49 is a view illustrating the third coding of the information on an anterior segment line shown in Fig. 47 is illustrated; Fig. Figure 50 is a diagram illustrating the third coding of the information about the left-sided segment lines in the entire target image shown in Fig. 32 is illustrated; Fig. 51 is a diagram illustrating the third coding of the information about the right-hand segment lines in the entire target image shown in Fig. 32 is illustrated; Fig. Figure 52 is a diagram illustrating the third coding of the information about the left-sided segment lines in the entire target image shown in Fig. 32 is illustrated; Fig. Figure 53 is a diagram illustrating the third coding of the information about the anterior segment lines in the entire target image shown in Fig. 32 is illustrated; Fig. 54 is a diagram illustrating a parking space annotation code including the first coded data, the second coded data and the third coded data containing the Fig. 50 to Fig. 53 illustrated coded results; Fig. 55 is a flowchart schematically illustrating, as an example of a task after detecting the parking spaces from an input image, a position correction routine according to the first embodiment; Fig. 56 is a view schematically illustrating a calculation task of a circumscribed rectangle shown in Fig. 55 is illustrated; Fig. 57 is a view schematically illustrating an edge line generation task and an edge line calculation task performed in Fig. 55 are illustrated; Fig. 58 is a perspective view illustrating a situation in which a portion of a parking space segment and / or at least one parking space vertex may be blocked by at least one parked vehicle; Fig. 59 is a flowchart schematically illustrating part of a training routine according to the second embodiment; Fig. 60 is a view according to the third embodiment, schematically illustrating an image indicating a situation in which it is difficult to determine whether a target parking space that is empty is empty because a parked vehicle occupies a parking space adjacent to the target parking space, blocking the target parking space; Fig. 61 is a flowchart schematically illustrating preprocessing before training the parameters of a machine learning model according to the third embodiment; Fig. 62 is a plan view schematically illustrating a training routine according to the third embodiment in a situation where a left side-by-side parking space train and a right side-by-side parking space train are respectively arranged on both sides of a part of a common parking corridor in which the subject vehicle travels, according to the third embodiment; Fig. 63 is a view schematically illustrating an image indicating a situation in which parking space vertices of the same type from different parking spaces are located in a common cell of a target image according to the fourth embodiment; Fig. 64A and Fig. 64B are each a diagram schematically illustrating a definition of typical orientations of a parking space according to the fourth embodiment; Fig. 65 is a flowchart schematically illustrating a coding routine according to the fourth embodiment; Fig. 66 is a diagram showing schematically an example of the Fig. 65 illustrated coding routine; Fig. 67 is a diagram showing schematically another example of the Fig. 65 illustrated coding routine; Fig. 68 is a diagram illustrating a parking space annotation code including a first coded data arrangement additionally including fourth coded data including the coded results, according to the fourth embodiment; Fig. 69 is a view according to the fifth embodiment, schematically illustrating an image indicating a situation in which at least one of parking space vertices of a parking space may be located outside a target image, and an offset task in the situation; Fig. 70 to Fig. 73 each illustrates a corresponding example of the offset task for offsetting one of four parking space vertices located outside the target image according to the fifth embodiment; Fig. 74 and Fig. 75 illustrates a corresponding example of the offset task for offsetting two of the four parking space vertices of the parking space located outside the target image according to the fifth embodiment; Fig. 76A is a plan view schematically illustrating a determination of whether there is a parking space environment around the subject vehicle according to the sixth embodiment; Fig. 76B is an enlarged view schematically illustrating the determination of whether there is a parking space environment around the subject vehicle according to the sixth embodiment; Fig. 77A is a flowchart schematically illustrating a parking space surroundings annotation information generation routine according to the sixth embodiment; Fig. 77B is a flowchart schematically illustrating part of a training routine according to the sixth embodiment; Fig. 78 is a diagram illustrating an example of fourth coded data representing that the host vehicle is located at an unidentified place or the host vehicle is located outside a parking space according to the sixth embodiment; Fig. 79 is a diagram illustrating an example of fourth coded data representing that the host vehicle is located in a parking space according to the sixth embodiment; Fig. 80 is a diagram schematically illustrating a parking space annotation code for each parking space included in a target image, which additionally includes the Fig. 78 or Fig. 79 illustrated fourth coded data; Fig. 81 is a flowchart schematically illustrating a parking space detection routine, ie, a parking space derivation routine, executed by the processor according to the sixth embodiment; Fig. 82 is a diagram schematically illustrating an example of fifth coded data as a result of fifth coding according to the seventh embodiment; Fig. 83 is a diagram schematically illustrating a parking space annotation code for each parking space included in a target image, which additionally includes the Fig. 82 illustrated fifth coded data; Fig. 84 is a diagram schematically illustrating an example of fifth coded data as a result of sixth coding according to the eighth embodiment; Fig. 85 is a diagram schematically illustrating a parking space annotation code for each parking space included in a target image, which additionally includes the Fig. 84 illustrated sixth coded data; Fig. 86 is a flowchart schematically illustrating a parking space environment determination routine according to the ninth embodiment; Fig. 87 is a plan view schematically illustrating an example of a parking space train; Fig. 88 is a plan view schematically illustrating another example of a parking space train; Fig. 89 is a plan view schematically illustrating another example of a parking space train; Fig. 90 is a plan view schematically illustrating an example of a side-by-side parking space train; Fig. 91 is a plan view schematically illustrating an example of a diagonal side-by-side parking space train; Fig. 92 is a plan view schematically illustrating an example of a parallel parking space train; Fig. 93 is a plan view used to describe a parking space acquisition routine included in the parking space environment determination routine shown in Fig. 86 is illustrated; Fig. 94 is a plan view used to describe the parking space acquisition routine included in the parking space environment determination routine shown in Fig. 86 is illustrated; Fig. 95 is a plan view used to describe the parking space acquisition routine included in the parking space environment determination routine shown in Fig. 86 is illustrated; Fig. 96 is a plan view used to describe the parking space acquisition routine included in the parking space environment determination routine shown in Fig. 86 is illustrated; Fig. 97 is a plan view used to describe the parking space acquisition routine included in the parking space environment determination routine shown in Fig. 86 is illustrated; Fig. 98A and Fig. 98B is a common flowchart schematically illustrating the parking space acquisition routine used in Fig. 86 is illustrated; Fig. 99 and Fig. 100 are plan views, each illustrating a first example showing how the processor expands a typical parking region formed by a corridor and parking space trains located on either side of the corridor; Fig. 101 to Fig. 103 are plan views each illustrating a second example showing how the processor expands a typical parking region formed by a corridor and parking space trains located on either side of the corridor; Fig. 104 to Fig. 113 are plan views each showing an operation of determining whether the host vehicle is in a parking space from step S803 in Fig. 86 illustrate; Fig. 114 is a flowchart schematically illustrating an accelerator pedal misoperation addressing routine according to the ninth embodiment; Fig. 115 is a flowchart schematically illustrating a drive power reduction control subroutine executed in step S1003 of Fig. 114 is illustrated; Fig. 116 is a flowchart schematically illustrating a routine for assisting the driving of the host vehicle to a selected target parking space during parking assistance according to the ninth embodiment; Fig. 117 is a plan view schematically illustrating a parking space search route executed in step S1203 of Fig. 116 is illustrated; and Fig. 118 is a plan view schematically illustrating a parking route generation operation performed in step S1205 of Fig. 116 is illustrated. DETAILED DESCRIPTION OF EMBODIMENTS

[0011] Hereinafter, a basic embodiment and first to ninth embodiments of the present disclosure will be described with reference to the accompanying drawings. Configurations, functions, and / or examples described in the following embodiments can be freely modified. In the embodiments and their modifications, like reference numerals are assigned to equivalent or same components between the embodiments, between each embodiment and its modifications, and between the modifications. Among the equivalent or same components, descriptions of the former component can be directly used to describe the later component(s) as long as there are no technological contradictions and / or no additional descriptions. Overall configuration of vehicle

[0012] Fig. 1 illustrates a system-installed vehicle or system vehicle V according to the basic embodiment to which the present disclosure is applied. The system vehicle V is a four-wheeled motor vehicle. The system vehicle V has a body V1 having a substantially rectangular shape in plan view. The body, i.e., the vehicle body, V1 has a first center line LC1 defined as a virtual line passing through a center point VC of the vehicle V and extending in the front-rear direction, i.e., the longitudinal direction, of the system vehicle V. The system vehicle V has a second center line LC2 defined as a virtual line passing through the vehicle center point VC and extending in the left-right direction, i.e., the width direction, of the system vehicle V. The center point VC of the system vehicle V is defined as a three-dimensional center point of the vehicle body V1.

[0013] In Fig. 1, the width direction of the system vehicle V, referred to as a vehicle width direction, corresponds to the horizontal direction therein. A vehicle height direction is defined as a height direction of the system vehicle V and is parallel to the direction of gravity of the system vehicle V, while the system vehicle V is movably and stably mounted on the horizon plane. The longitudinal direction of the system vehicle V, referred to as a vehicle longitudinal direction, is defined as perpendicular to both the vehicle width direction and the vehicle height direction.

[0014] In the system vehicle V, a forward direction, a backward direction, a left direction and a right direction are shown in Fig. 1. That is, the vehicle longitudinal direction is synonymous with the front-to-rear direction, and the vehicle width direction is synonymous with the left-to-right direction. The vehicle height direction does not need to be parallel to the direction of gravity of the system vehicle V, depending on the situation in which the system vehicle V is mounted on any road and / or the conditions under which the system vehicle V is traveling. The vehicle height direction is usually parallel to the direction of gravity of the system vehicle V.

[0015] The vehicle body V1 has defined therein an interior V2, which serves as a compartment for one or more occupants, including a driver of the system vehicle V.

[0016] The V1 body has a front end section, a rear end section, a left side section, a right side section, a top section, and four corners, which include a front left corner, a front right corner, a rear left corner, and a rear right corner.

[0017] The vehicle V includes four wheels V3, i.e., four wheels V3a, V3b, V3c, and V3d, which are mounted on the respective four corners of the body V1. Specifically, the wheel V3a, i.e., the front left wheel V3a, is mounted on the front left corner of the body V1, the wheel V3b, i.e., the front right wheel V3b, is mounted on the front right corner of the body V1, the wheel V3c, i.e., the rear left wheel V3c, is mounted on the rear left corner of the body V1, and the wheel V3d, i.e., the rear right wheel V3d, is mounted on the rear right corner of the body V1. The system vehicle V is not limited to such a four-wheeled motor vehicle, and a three-wheeled motor vehicle or a six-wheeled or eight-wheeled vehicle such as a truck can be used as the system vehicle V.The number of drive wheels in the wheels of the system vehicle V can be freely determined and each drive wheel can be freely located on the body V1 of the system vehicle V.

[0018] The system vehicle V includes a front bumper V12 mounted on the front end portion V11, i.e., a front side, of the body V1. The system vehicle V includes a rear bumper V14 mounted on the rear end portion V13, i.e., a rear side, of the body V1. The body V1 of the system vehicle V includes a body panel V15 arranged to form the left-side and right-side portions and the upper portion of the body V1. The body panel V15 includes door panels V16. In the Fig. In the basic embodiment illustrated in Figure 1, the body panel V15 includes a front pair of left and right door panels V16 and a rear pair of left and right door panels V16; the left door panels V16 of the front and right pair are located on the left side portion of the body V1, and the right door panels V16 of the front and right pair are located on the right side portion of the body V1. The system vehicle V includes a pair of door mirrors V17 mounted on the respective left and right doors V16 of the front pair. The body V1 includes a front windshield V18 covering a front side of the interior V2.

[0019] The front windshield V18 has a substantially plate-like shape and is made of transparent glass or transparent synthetic resin. The front windshield V18 is attached to the body panel V15 and is inclined such that, when viewed in a side view parallel to the vehicle width direction, a lower surface of the front windshield V18 is closer to the front of the body V1 than a top surface of the front windshield V18.

[0020] The system vehicle V includes a dashboard V21 and a plurality of seats V22, including a driver's seat V23 on which a driver D sits. The dashboard V21 is provided in a front portion of the interior V2, and the seats V22 are located at the rear of the dashboard V21. The system vehicle V includes a steering wheel V24 located in front of the driver's seat V23. The driver D grips the steering wheel V24 and steers the steering wheel V24 to thereby control the steering of the system vehicle V. The steering wheel V24 typically has a substantially annular shape, a substantially ellipsoidal shape, or a substantially polygonal ring shape, but may have a rod-like shape or a joystick shape. Overall configuration of vehicle system

[0021] The system vehicle V includes a vehicle system 1 installed therein. The vehicle system 1 installed in the system vehicle V is configured to serve as a driving automation system or an autonomous driving system in the system vehicle V. The system vehicle V is also referred to as a self-driven vehicle V. The autonomous driving system is configured to implement one of levels 1 to 5 included in all six autonomous driving levels (0 to 5 levels) defined in the SAE J2016 standard published by SAE International; SAE is an abbreviation for "Society of Automotive Engineers". Each level X in the autonomous driving levels 0 to 5 is also referred to as an SOE level X. That is, the variable X can take any number from 0, 1, 2, 3, 4, and 5. The higher the SOE level X, the higher the driving automation level.In other words, the greater the number of dynamic driving tasks performed by the driving system, the higher the autonomous driving level. When the autonomous driving level is changed to a higher level, the autonomous driving level increases. Conversely, the lower the SOE level X, the lower the autonomous driving level. In other words, the smaller the number of dynamic driving tasks performed by the autonomous driving system, the lower the autonomous driving level. When the autonomous driving level is changed to a lower level, the autonomous driving level decreases. Definition of driving assistance and autonomous driving

[0022] SAE levels 0 to 5 are described specifically below.

[0023] In the following, driver D is an occupant who manages and executes dynamic driving tasks. Dynamic driving tasks represent all operational functions and all maneuver functions that driver D must perform in real time when driving the host vehicle V on public roads, excluding strategic functions. Overall driving actions can be categorized into operational functions, maneuver functions, and strategic functions.

[0024] The strategic functions may include functions for planning a travel plan and selecting one or more locations through the planned travel plan. In particular, the strategic functions may include functions for determining or selecting a travel plan that (i) indicates whether to travel to a destination, (ii) indicates when to travel to the destination, and (iii) indicates how to travel to the destination.

[0025] The maneuver functions may include functions for determining, in different traffic situations, different maneuvers, which may include (i) determining, during the planned journey, whether and when to overtake, (ii) determining whether and when to change lanes, (iii) selectively setting a proper speed of the host vehicle V and (iv) checking the mirrors.

[0026] The operational functions may include instantaneous reactions of the driver, such as steering operations, braking operations, acceleration operations or accelerator pedal operations and / or minor adjustments to these operations in order to maintain a position of the host vehicle V in a corresponding lane of a road and / or to avoid at least one obstacle and / or at least one hazard in the path of the moving host vehicle V. OEDR is an abbreviation for “Object and Event Detection and Response” and may be referred to as “detection and response of objects and events.” OEDR involves monitoring a driving environment around the host vehicle V. Monitoring the driving environment around the host vehicle V may involve detecting, recognizing, and classifying one or more objects and / or events.Monitoring the driving environment around the host vehicle V may additionally include preparing a response to the one or more objects and / or events. Operational design domain (ODD) conditions, also referred to as "specific domain conditions," represent specific conditions under which a given autonomous driving system or feature thereof is designed to function. The ODD conditions may, for example, include at least one of several boundary conditions, including, for example, geographical conditions, environmental conditions, speed conditions, and time conditions.

[0027] Level 0 represents no autonomous driving, which means that driver D performs all dynamic driving tasks.

[0028] Level 1 represents driver assistance, which means that an autonomous driving system sustainably executes either the lateral control of vehicle movement subtasks or the longitudinal control of vehicle movement subtasks of the dynamic driving tasks under specific ODD conditions. The longitudinal control of vehicle movement subtasks include, for example, forward / backward operations, acceleration / deceleration operations, and stopping operations. The lateral control of vehicle movement tasks include, for example, a steering operation. In particular, the autonomous driving system is configured not to execute both the lateral control of vehicle movement subtasks and the longitudinal control of vehicle movement subtasks.

[0029] Level 2 represents Partial Autonomous Driving or Advanced Driving Assistance, which represents that an autonomous driving system sustainably executes both the lateral vehicle motion control subtasks and the longitudinal vehicle motion control subtasks of the dynamic driving tasks under specific ODD conditions, with the expectation that the driver completes D OEDR subtasks and monitors the autonomous driving system.

[0030] Level 3 represents Conditional Autonomous Driving, which means that an automated driving system sustainably executes all dynamic driving tasks under specific ODD conditions. Under specific ODD conditions, the driver D is not required to perform any of the OEDR subtasks for monitoring the traffic environment around the self-vehicle V. However, if the automated driving system has difficulty continuing at Level 3, the automated driving system will request the driver D to control the self-vehicle V with plenty of time, requiring the driver D to respond smoothly to the request.

[0031] Level 4 represents High Automation, which means that an automated driving system sustainably performs all dynamic driving tasks under specific ODD conditions. If the automated driving system encounters difficulty continuing at Level 4, the automated driving system addresses the difficulties.

[0032] Level 5 represents Full Automation, which means that an automated driving system on the host vehicle V sustainably executes all dynamic driving tasks without restriction under all ODD conditions. If the automated driving system encounters difficulties continuing at Level 5, the automated driving system addresses the difficulties under all ODD conditions without restriction.

[0033] The vehicle system 1 is configured to perform different driving control tasks and different notification tasks based on the different driving tasks during the driving of the host vehicle V. In particular, the vehicle system 1 is configured as an autonomous driving system that implements driving assistance for the host vehicle V and / or autonomous driving of the host vehicle V. The autonomous driving corresponds to each of the SAE levels 3 to 5. That is, the autonomous driving in each of the SAE levels 3 to 5 represents that the vehicle system 1 serves as the autonomous driving system to perform all dynamic driving tasks in the corresponding one of the SAE levels 3 to 5. In contrast, the driving assistance corresponds to each of the SAE levels 1 and 2.That is, autonomous driving in each of SAE Levels 1 and 2 represents that vehicle system 1 serves as the autonomous driving system to perform part of the dynamic driving tasks in the corresponding one of SAE Levels 1 and 2. That is, driving assistance may include both SAE Level 1 "Driver Assistance" and SAE Level 2 "Partial Autonomous Driving" or "Advanced Driving Assistance," except that the term "SAE Level 1 Driving Assistance" is used or driving assistance is used to distinguish it from SAE Level 2 Partial Autonomous Driving.

[0034] The vehicle system 1 of the basic embodiment may be configured to perform (i) autonomous driving at each of SAE Levels 3 to 5, (ii) partially autonomous driving, i.e., advanced driving assistance at SAE Level 2 and driving assistance at SAE Level 1. The driving assistance that can be performed by the vehicle system 1 of the basic embodiment may include hands-free driving. Hands-free driving enables the vehicle system 1 to automatically move the host vehicle V forward or backward, steer the host vehicle V, accelerate or decelerate the host vehicle V, perform lane changes of the host vehicle V, and / or stop the host vehicle V, as long as the driver D appropriately addresses an intervention request issued by the vehicle system 1.

[0035] Hands-free driving requires driver D to monitor road conditions around the host vehicle V, traffic situations around the host vehicle V, and information about whether there are one or more obstacles around the host vehicle V, without requiring driver D to be in a standby state. The standby state represents a state of driver D in which driver D is ready to steer the host vehicle V, i.e., ready to intervene in the subtasks for lateral control of the vehicle's movement. The standby state typically represents a state of driver D in which driver D is sitting in the driver's seat V23 with a posture that enables driving the host vehicle V and is ready to operate the steering wheel V24 with his hands. Driver D, who is in the standby state, grips the steering wheel V24 with his hands.The state in which driver D touches the steering wheel V24 with his hands while ready to grasp the steering wheel V24 is considered the ready state. For example, the state in which driver D operates the steering wheel V24, i.e., driver D actively operates the steering wheel V24, is considered the ready state. The state in which driver D holds the steering wheel V24 against the controlled steering of the steering wheel V24 by vehicle system 1 is considered the ready state. Configuration of vehicle system

[0036] In the following, an overall configuration of the vehicle system 1 is described with reference to Fig. 1 and Fig. 2 described.

[0037] The vehicle system 1 includes a driving ECU (ECU stands for electronic control unit) 2, a driving information input unit 3, a vehicle communication module, in other words, a data communication module (DCM) 4, a high-definition (HD) map database 5, a navigation system 6, a human machine interface (HMI) system 7, a lighting system 8, and a motion control system 9.

[0038] The vehicle system 1 is configured such that the driving ECU 2, the driving information input unit 3, the vehicle communication module 4, the HD map database 5, the navigation system 6, the HMI system 7, the lighting system 8, and the motion control system 9 are communicatively connected to each other via a vehicle communication network 10. The vehicle communication network 10 includes a main network compliant with one of various communication standards, such as the Controller Area Network (INTERNATIONAL REGISTRATION NUMBER 1048262A). In addition to the main network compliant with CAN, the vehicle communication network 10 may include a subnetwork compliant with Local Interconnect Network (LIN) or FlexRay. Driving ECU

[0039] The driving ECU 2 serves as a control device according to the present disclosure, which is installed in the system vehicle V and is configured to control overall operations in the vehicle system 1. The driving ECU 2 is configured as an autonomous driving / advanced driver-assistance system (AD / ADAS) ECU, which serves as both a driving assistance controller and an autonomous driving controller. In particular, the driving ECU 2 includes at least one processor 21 and at least one storage device 22 communicatively connected to the at least one processor 21.

[0040] The at least one processor 21, referred to simply as a processor 21, consists of a central processing unit (CPU) or a micro processing unit (MPU). The at least one storage device 22 may, for example, include at least one read-only memory (ROM) and one random access memory (RAM), selected from various non-volatile storage devices, such as ROMs, RAMs, and non-volatile rewritable recording media. Such recording media may be referred to as storage media. These non-volatile rewritable recording media, such as Flash ROMs or EEPROMs, allow information stored therein to be rewritable while powered on and keep information unwritable while powered off.EEPROM is an abbreviation for Electronically Erasable and Programmable ROM. The ROM, or at least one of the non-volatile, rewritable storage devices included in the storage device 22, stores data and program instructions used for the processor 21 in advance. The driving ECU 2 is configured to read the program instructions stored in the storage device 22 and execute the read program instructions to perform various tasks and operations, including operations for controlling the host vehicle and operations for notifying the occupants. Driving information input unit

[0041] The driving information input unit 3 is configured to input information required by the driving ECU 2 to perform various operations and tasks into the driving ECU 2. Specifically, the driving information input unit 3 may include at least one sonar sensor 31, a radar sensor 32, a laser radar (LIDAR) sensor 33, at least one camera 34, operation sensors 35, behavior sensors 36, a driver state monitor 37, operation switches 38, and a tracking device 39. The sonar sensor 31, the radar sensor 32, the LIDAR 33, and the camera 34 are collectively referred to as environmental monitoring sensors or ADAS sensors. The components of the driving information input unit 3 are described sequentially below. Sonar sensor

[0042] The at least one sonar sensor, simply referred to as a sonar sensor 31, is an ultrasonic distance sensor mounted on the body V1. The sonar sensor 31 is an ultrasonic sensor and is configured to, as shown in Fig. 3 illustrates emitting sonar sounding waves Wsp within an ultrasonic frequency band toward an external space outside the host vehicle V. The sonar sensor 31 is configured to receive sonar echoes Wsr resulting from the reflection of the sonar sounding waves Wsp by a target object B, to accordingly acquire different information about the target object B. In the description, the reference symbol B is assigned to each target object for the sake of simplicity, but target objects assigned the same reference symbol B may not necessarily represent the same target object.

[0043] Specifically, the sonar sensor 31 is configured to calculate a distance of the object B from the sonar sensor 31 based on the time of flight (TOF) and the speed of sound. The TOF represents the time defined from a time of emitting a sonar sounding wave Wsp, i.e., a pulse, to a time of receiving a sonar echo Wsr through a propagation path Ls of the pulses Wsp and Wsr; the TOF is therefore also referred to as propagation time.

[0044] When the driving information input unit 3 includes a pair of sonar sensors 31, that is, a pair of a first sonar sensor 311 and a second sonar sensor 312 (see Fig. 4), the use of a known triangulation method enables the measurement of a relative distance of the target object B relative to the own vehicle V. In Fig. 4, an X-axis is defined along a virtual line connecting the first and second sonar 311 and 312, and a Y-axis is defined as perpendicular to the X-axis. The X- and Y-axes extend along a horizontal reference plane perpendicular to the vehicle height direction.

[0045] The first sonar sensor 311 is configured to emit the sonar probing waves Wsp, and each of the first and second sonar sensors 311 and 312 is configured to receive the sonar echoes Wsr resulting from the reflection of the sonar probing waves Wsp by the target object B. It is possible to calculate a position of the target object B in a two-dimensional XY coordinate system formed by the X and Y axes based on a first TOF through a first propagation path Ls1 and a second TOF through a second propagation path Ls2. The first propagation path Ls1 is defined as a propagation path through which an ultrasonic wave (a pulse) emitted as a sonar probing wave Wsp from the first sonar 311 propagates through the target object B to be returned to the first sonar 311 as a sonar echo Wsr.Ultrasonic waves (pulses) propagating through the propagation path Ls1 are also called direct waves (pulses). The second propagation path Ls2 is defined as a propagation path through which an ultrasonic wave (pulse) emitted as a sonar sounding wave Wsp from the first sonar 311 propagates through the target object B to reach the second sonar 312 as a sonar echo Wsr. Ultrasonic waves (pulses) propagating through the propagation path Ls2 are also called indirect waves (pulses).

[0046] For example, the driving information input unit 3 according to the basic embodiment includes a plurality of sonar sensors 31 mounted on the body V1 (see Fig. 1). Specifically, the sonar sensors 31 include first, second, third, and fourth front sonar sensors SF1, SF2, SF3, and SF4 mounted on the front bumper V12. Likewise, the sonar sensors 31 include first, second, third, and fourth rear sonar sensors SR1, SR2, SR3, and SR4 mounted on the rear bumper V14. Additionally, the sonar sensors 31 include first, second, third, and fourth side sonar sensors SS1, SS2, SS3, and SS4. The first and third side sonar sensors SS1 and SS3 are mounted on the left-side portion of the body V1, and the second and fourth side sonar sensors SS2 and SS4 are mounted on the right-side portion of the body V1.

[0047] The first to fourth front sonar SF1 to SF4, the first to fourth stern sonar SR1 to SR4 and the first to fourth side sonar SS1 to SS4 are also collectively referred to simply as a sonar sensor 31 or sonar sensors 31 when it is not necessary to identify one of the sonars SF1 to SS4.

[0048] Fig. 5 illustrates a predetermined detection region, ie, a predetermined measurement region, of each of the sonar sensors 31.

[0049] In the following, the sonar sensors 31 are described with reference to Fig. 1 and Fig. 5 described sequentially.

[0050] The first front sonar SF1 is mounted on a portion of the front bumper V12 that is closer to the left edge of the front bumper V12 than its right edge in the vehicle width direction, and is configured to emit the sonar probing waves Wsp diagonally to the front left. The second front sonar SF2 is mounted on a portion of the front bumper V12 that is closer to the right edge of the front bumper V12 than its left edge in the vehicle width direction, and is configured to emit the sonar probing waves Wsp diagonally to the front right. The first and second front sonar SF1 and SF2 are arranged symmetrically with respect to the first center line LC1.Each of the first and second front sonar units SF1 and SF2 has a predetermined detection region Rsc. The detection region Rsc, also referred to as a front corner detection region Rsc, is designed such that a detection range of the front corner detection region Rsc is set to, for example, 60 cm or approximately 60 cm. The detection range of a sonar sensor 31 represents a maximum measurable range (distance) of the sonar sensor 31.

[0051] The third and fourth front sonar SF3 and SF4 are mounted on a central portion of the front bumper V12 so as to be aligned in the vehicle width direction. Specifically, the third front sonar SF3 is arranged between the first front sonar SF1 and the first center line LC1 and is configured to emit the sonar probing waves Wsp substantially forward, and the fourth front sonar SF4 is arranged between the second front sonar SF2 and the first center line LC1 and is configured to emit the sonar probing waves Wsp substantially forward. The third and fourth front sonar SF3 and SF4 are arranged symmetrically with respect to the first center line LC1.Each of the third and fourth front sonars SF3 and SF4 has a predetermined detection region Rsf, and the detection region Rsf, which is also referred to as a front detection region Rsf, of each of the third and fourth front sonars SF3 and SF4 is designed such that the detection range of the front detection region Rsf is set to, for example, 1 m or about 1 m.

[0052] The first and third front sonars SF1 and SF3, mounted on the left side of the front bumper V12 relative to the first center line LC1, are arranged at different positions in the vehicle width direction, that is, the horizontal direction. The first and third front sonars SF1 and SF3, which are adjacent to each other in the vehicle width direction, are arranged to have a predetermined positional relationship, allowing one of the first and third front sonars SF1 and SF3 to receive, as received echoes, sonar echoes resulting from the reflection of the sonar probing waves Wsp emitted by the other of the first and third front sonars SF1 and SF3 by a target object.

[0053] In particular, the first front sonar SF1 is arranged to receive both (i) direct echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the first front sonar SF1 by a target object, and (ii) indirect echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the third front sonar SF3 by the target object. Similarly, the third front sonar SF3 is arranged to receive both (i) direct echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the third front sonar SF3 by a target object, and (ii) indirect echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the first front sonar SF1 by the target object.

[0054] Similarly, the third and fourth front sonars SF3 and SF4, mounted on the central portion of the front bumper V12 in the vehicle width direction, are arranged at different positions in the vehicle width direction, that is, the horizontal direction. The third and fourth front sonars SF3 and SF4, which are adjacent to each other in the vehicle width direction, are arranged to have a predetermined positional relationship, allowing one of the third and fourth front sonars SF3 and SF4 to receive, as received echoes, sonar echoes resulting from the reflection of the sonar probing waves Wsp emitted by the other of the third and fourth front sonars SF3 and SF4 by a target object.

[0055] The second and fourth front sonars SF2 and SF4, mounted on the right side of the front bumper V12 relative to the first center line LC1, are arranged at different positions in the vehicle width direction, that is, the horizontal direction. The second and fourth front sonars SF2 and SF4, which are adjacent to each other in the vehicle width direction, are arranged to have a predetermined positional relationship, allowing one of the second and fourth front sonars SF2 and SF4 to receive, as received echoes, sonar echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the other of the second and fourth front sonars SF2 and SF4 by a target object.

[0056] The first rear sonar SR1 is mounted on a portion of the rear bumper V14 that is closer to the left edge of the rear bumper V14 than its right edge in the vehicle width direction, and is configured to emit the sonar probing waves Wsp diagonally rearward to the left. The second rear sonar SR2 is mounted on a portion of the rear bumper V14 that is closer to the right edge of the rear bumper V14 than its left edge in the vehicle width direction, and is configured to emit the sonar probing waves Wsp diagonally rearward to the right. The first and second rear sonar SR1 and SR2 are arranged symmetrically with respect to the first center line LC1.Each of the first and second stern sonar SR1 and SR2 has a predetermined detection region Rsd, and the detection region Rsd, which is also referred to as a stern corner detection region Rsd, of each of the first and second stern sonar SR1 and SR2 is designed such that the detection range of the stern corner detection region Rsd is set to, for example, 60 cm or about 60 cm.

[0057] The third and fourth rear sonar sensors SR3 and SR4 are mounted on a central portion of the rear bumper V14 so as to be aligned in the vehicle width direction. Specifically, the third rear sonar SR3 is arranged between the first rear sonar SR1 and the first centerline LC1 and is configured to emit the sonar sounding waves Wsp substantially rearward. The fourth rear sonar SR4 is arranged between the second rear sonar SR2 and the first centerline LC1 and is configured to emit the sonar sounding waves Wsp substantially rearward. The third and fourth rear sonar sensors SR3 and SR4 are arranged symmetrically with respect to the first centerline LC1.Each of the third and fourth stern sonar SR3 and SR4 has a predetermined detection region Rsr, and the detection region Rsr, which is also referred to as a stern detection region Rsr, of each of the third and fourth stern sonar SR3 and SR4 is designed such that the detection distance of the stern detection region Rsr is set to, for example, 1.5 m or about 1.5 m.

[0058] The first and third rear sonar sensors SR1 and SR3, mounted on the left side of the rear bumper V14 relative to the first center line LC1, are arranged at different positions in the vehicle width direction, that is, the horizontal direction. The first and third rear sonar sensors SR1 and SR3, which are adjacent to each other in the vehicle width direction, are arranged to have a predetermined positional relationship, allowing one of the first and third rear sonar sensors SR1 and SR3 to receive, as received echoes, sonar echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the other of the first and third rear sonar sensors SR1 and SR3 by a target object.

[0059] In particular, the first stern sonar SR1 is arranged to receive both (i) direct echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the first stern sonar SR1 by a target object, and (ii) indirect echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the third stern sonar SR3 by the target object. Similarly, the third stern sonar SR3 is arranged to receive both (i) direct echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the third stern sonar SR3 by a target object, and (ii) indirect echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the first stern sonar SR1 by the target object.

[0060] Similarly, the third and fourth rear sonar units SR3 and SR4, mounted on the central portion of the rear bumper V14 in the vehicle width direction, are arranged at different positions in the vehicle width direction, that is, the horizontal direction. The third and fourth rear sonar units SR3 and SR4, which are adjacent to each other in the vehicle width direction, are arranged to have a predetermined positional relationship, allowing one of the third and fourth rear sonar units SR3 and SR4 to receive, as received echoes, sonar echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the other of the third and fourth rear sonar units SR3 and SR4 by a target object.

[0061] The second and fourth rear sonar units SR2 and SR4, mounted on the right side of the rear bumper V14 relative to the first center line LC1, are arranged at different positions in the vehicle width direction, that is, the horizontal direction. The second and fourth rear sonar units SR2 and SR4, which are adjacent to each other in the vehicle width direction, are arranged to have a predetermined positional relationship, allowing one of the second and fourth rear sonar units SR2 and SR4 to receive, as received echoes, sonar echoes resulting from the reflection of the sonar sounding waves Wsp emitted by the other of the second and fourth rear sonar units SR2 and SR4 by a target object.

[0062] The first side sonar SS1 and the third sonar SS3 are each mounted on a portion of the left-side portion of the body V1 and are configured to emit the sonar probing waves Wsp to the left relative to the host vehicle V. Similarly, the second side sonar SS2 and the fourth sonar SS4 are each mounted on a portion of the right-side portion of the body V1 and are configured to emit the sonar probing waves Wsp to the right relative to the host vehicle V. The first, second, third, and fourth side sonar SS1, SS2, SS3, and SS4 are each arranged to receive only direct echoes resulting from the reflection of the sonar probing waves Wsp emitted by the corresponding one of the first, second, third, and fourth side sonar SS1, SS2, SS3, and SS4.Each of the first, second, third, and fourth side sonars SS1, SS2, SS3, and SS4 has a predetermined detection region Rss, and the detection region Rss, also referred to as a side detection region Rss, of each of the first to fourth side sonars SS1 to SS4 is designed such that the detection distance of the side detection region Rss is set to, for example, within a range of 2 to 3 m inclusive.

[0063] The first side sonar SS1 is arranged between the first front sonar SF1 and the door mirror V17, which is mounted on the left door panel V16 of the front pair and is also referred to as a left door mirror V17. The first side sonar SS1 is configured to emit the sonar probing waves Wsp to the left relative to the host vehicle V. The second side sonar SS2 is arranged between the second front sonar SF2 and the door mirror V17, which is mounted on the right door panel V16 of the front pair and is also referred to as a right door mirror V17. The second side sonar SS2 is configured to emit the sonar probing waves Wsp to the right relative to the host vehicle V. The first and second side sonar SS1 and SS2 are arranged symmetrically with respect to the first center line LC1.The first and second side sonar SS1 and SS2 may be mounted on the body panel V15 or mounted on portions of the respective left and right edges of the front bumper V12 in the vehicle width direction; the portion of each of the left and right edges of the front bumper V12 to which the corresponding one of the first and second side sonar SS1 and SS2 extends rearward in the vehicle longitudinal direction.

[0064] The third side sonar SS3 is arranged between the first rear sonar SR1 and the left door panel V16 of the rear pair. The third side sonar SS3 is configured to emit the sonar probing waves Wsp to the left relative to the host vehicle V. The fourth side sonar SS2 is arranged between the second rear sonar SR2 and the right door panel V16 of the rear pair. The fourth side sonar SS4 is configured to emit the sonar probing waves Wsp to the right relative to the host vehicle V. The third and fourth side sonar SS3 and SS4 are arranged symmetrically with respect to the first center line LC1.The third and fourth side sonar SS3 and SS4 may be mounted on the body panel V15 or mounted on portions of the respective left and right edges of the rear bumper V14 in the vehicle width direction; the portion of each of the left and right edges of the rear bumper V14 to which the corresponding one of the third and fourth side sonar SS3 and SS4 extends forward in the vehicle longitudinal direction.

[0065] Fig. 6 schematically illustrates an example in which the driving ECU 2 uses the first side sonar SS1 to detect a parking space PS located to the left of the host vehicle V to perform side-by-side parking. The parking space PS is a rectangular frame-like area in plan view and represents a space in which a vehicle can be parked. The parking space PS may also be referred to as a parking space or a parking lot. Fig. 6 illustrates sonar detection points Psr, which are points through which the sonar sounding waves Wsp emitted by any sonar sensor 31 are estimated to be reflected; the sonar detection points Psr may also be referred to as ranging points Psr. As shown in Fig. 6, the driving ECU 2 can detect the parking space PS based on a distribution of the sonar detection points Psr obtained based on sonar echoes Wsr from target objects B, which are parked vehicles arranged side-by-side in the direction of straight forward movement of the host vehicle V.

[0066] In particular, when adjacent rows of the sonar detection points Psr are obtained which are arranged in the traveling direction of the host vehicle V when side-by-side parking of the host vehicle V is carried out as shown in Fig. 6, the driving ECU 2 may determine whether the length of a space between each adjacent pair of rows of the sonar detection points Psr exceeds the entire width of the host vehicle V, and in response to a determination that the length of the space between at least one adjacent pair of rows of the sonar detection points Psr exceeds the entire width of the host vehicle V, detect the space between at least one adjacent pair of rows of the sonar detection points Psr as a location of a parking space PS. radar sensor

[0067] According to Fig. 1 and Fig. 2, the radar sensor 32 is configured to transmit millimetric radar waves or submillimetric radar waves and receive reflected radar waves to detect a target object B accordingly. The radar sensor 32 of the basic embodiment is mounted at the center of a front portion V11 of the body V1 in the vehicle width direction. The radar sensor 32 is, as shown in Fig. 7 illustrates a long-range radar having a substantially fan-shaped radar detection region Rg1 in plan view. The fan-shaped radar detection region Rg1 has a predetermined detection range of 10 to 250 m inclusive and has a radar scanning angle θr1 around a forward movement direction Df of the host vehicle V. The radar sensor 32 is configured as a phased array radar sensor or a beamforming radar sensor. The forward movement direction Df represents the direction of a virtual line extending forward from the first center line LC1 and represents the traveling direction of the host vehicle V, with a current shift position of a shift lever set to any position except the reverse position.The forward movement direction Df of the self-vehicle V is consistent with the traveling direction of the self-vehicle V when the self-vehicle V is moving straight forward, and becomes the direction of a tangent to a traveling path of the self-vehicle V when the self-vehicle V is turning.

[0068] The radar sensor 32 is configured to emit radar probing waves Wrp, scan with the emitted radar probing waves Wrp within the radar scanning angle θr1, and receive radar waves Wrr resulting from the reflection of the radar probing waves Wrp by a target object B located in the radar detection region Rg1.

[0069] Specifically, the radar sensor 32 is comprised of an FMCW radar device equipped with an array antenna and is configured to detect, based on differences in frequency between the emitted millimeter waves and received millimeter waves and / or differences in phase between the emitted millimeter waves and received millimeter waves, (i) a distance to the target object B thereof, (ii) an azimuth angle θa of the target object B, and (iii) a relative speed of the target object B relative to the self-vehicle V. The azimuth angle θa of the target object B is defined as an angle formed by a first virtual line generated by extending the first center line LC1 in front of the self-object V and a second virtual line connecting between the radar sensor 32 and the target object B; the first center line CL1 represents the center line of the radar detection range Rg1.

[0070] The relative speed of the target object B relative to the own vehicle V is defined as a difference between a moving speed vb of the target object B and a traveling speed vm of the own vehicle V.

[0071] Specifically, the radar sensor 32 is configured to transmit radar probing waves Wrp generated based on a transmission signal having a predetermined modulated frequency and receive radar waves Wrr resulting from the reflection of the radar probing waves Wrp by the target object B, to accordingly detect received signals based on the received radar waves Wrr, which represent frequency characteristics of the received radar waves Wrr. Then, the radar sensor 32 is configured to calculate deviations between the modulated frequency of the transmission signal and the frequencies of the received signals, to accordingly generate beat signals based on the respective frequency deviations. The radar sensor 32 is configured to perform a first Fourier transform on the beat signals to accordingly calculate a frequency power spectrum of each of the beat signals.Then, the radar sensor 32 is configured to analyze the frequency power spectrum of each beat signal to obtain beat frequencies accordingly, and to calculate a distance of the target object B from the radar sensor 32 and the relative speed of the target object B relative to the host vehicle V based on the beat frequencies.

[0072] In addition to the long-range radar sensor 32, a medium-range radar sensor or a short-range radar sensor may be installed in the vehicle system 1. Such a medium-range radar sensor has, for example, a predetermined detection range of 1 to 100 m, and such a short-range radar sensor has, for example, a predetermined detection range of 15 cm to 30 m. Laser radar sensor

[0073] The laser radar sensor 33 is configured to, as in Fig. 8 illustrates irradiating the exterior of the host vehicle V with detection light Lp, i.e., laser light within an infrared frequency band, and receiving reflected light Lr resulting from the reflection of the detection light Lp by a target object B, to detect the target object B accordingly. The laser radar sensor 33 may also be referred to as LIDAR, which is an abbreviation for Light Detection and Ranging or Laser Imaging Detection and Ranging. The vehicle system 1 of the basic embodiment includes, as the laser radar sensor 33, a long-range laser radar sensor 33 comprising a scanning LIDAR. The laser radar sensor 33 is, as shown in Fig. 1, mounted at the center of the front section V11 of the body V1 in the vehicle width direction.

[0074] The laser radar sensor 33 has a substantially fan-shaped LIDAR detection region Rg2 in plan view. The fan-shaped radar detection region Rg2 has a predetermined radius of, for example, 200 m or more and has a LIDAR scanning angle θr2 around the forward movement direction Df of the host vehicle V. The laser radar sensor 33 is configured to horizontally scan the laser detection light Lp within the LIDAR scanning angle θr2 to accordingly detect a target object B located in the LIDAR detection region Rg2.

[0075] Fig. 9 illustrates a schematic configuration of the laser radar sensor 33. The laser radar 33 includes a light emitting unit 331, a scanning unit 332, and a light receiving unit 333.

[0076] The light-emitting unit 331 is configured to emit the detection light Lp. The scanning unit 332 is composed of a MEMS mirror unit including at least one reflection mirror located on a light path of the detection light Lp emitted from the light-emitting unit 331, and a MEMS mechanism that rotates, for example, the at least one reflection mirror to accordingly change a direction of the light reflected by the at least one reflection mirror; MEMS is an abbreviation for Micro Electro Mechanical Systems. Specifically, the scanning unit 332 is configured to electrically control the MEMS mechanism so that the MEMS mechanism rotates the at least one reflection mirror, thereby scanning with the detection light Lp emitted from the light-emitting unit 331 in both a horizontal scanning direction Ds and a vertical scanning direction Dh.

[0077] The light receiving unit 333 includes a light receiving sensor 334, which is a two-dimensional image sensor. Specifically, the light receiving sensor 334 is composed of a plurality of light receiving elements 335 arranged two-dimensionally in a horizontal direction corresponding to the horizontal scanning direction Ds and a vertical direction corresponding to the vertical scanning direction Dh. Each of the light receiving elements 335, which is composed of an avalanche photodiode (APD) or a single-photon avalanche diode (SPAD), is configured to detect a corresponding portion of the reflected light Lr resulting from the reflection of the detection light Lp by at least one target object B.

[0078] The laser radar sensor 33 is configured to generate at least one detection point data cloud based on the reflected light Lr received by the light receiving unit 333 and to detect the at least one target object B based on the at least one detection point data cloud.

[0079] The at least one detection point data cloud represents, like an image, ie a single image, a two-dimensional arrangement of a plurality of LIDAR detection points Prr that are close to each other, arranged two-dimensionally in the horizontal scanning direction Ds and the vertical scanning direction Dh (see Fig. 9). Each LIDAR detection point Prr arranged at a corresponding light receiving element 335 represents a point of the at least one target object B that is estimated to reflect a portion of the detection light Lp; a portion of the reflected light Lr received by the corresponding light receiving element 335 corresponds to the portion of the detection light Lp. Therefore, the LIDAR detection points Prr arranged two-dimensionally in the horizontal scanning direction Ds and the vertical scanning direction Dh include position information and range information on the two-dimensional array. This allows the at least one detection point data cloud to also be referred to as at least one range point data cloud.

[0080] That is, the laser radar sensor 33 is configured to acquire at least one detection point data cloud consisting of the LIDAR detection points Prr arranged two-dimensionally, like a single image, in the horizontal scanning direction Ds and the vertical scanning direction Dh. This allows the laser radar sensor 33 to also be referred to as a type of image sensor.

[0081] That is, when there are target objects B, the laser radar sensor 33 is configured to acquire, for each of the target objects B, the detection point data cloud consisting of the LIDAR detection points Prr arranged two-dimensionally, like a single image, in the horizontal scanning direction Ds and the vertical scanning direction Dh. camera

[0082] According to Fig. 1 and Fig. 2, the at least one camera 34 serving as an image sensor is mounted on the host vehicle V. The at least one camera 34 is configured to capture images of a peripheral field of view around the host vehicle V while moving together with the host vehicle V. That is, the at least one camera 34 is configured to generate image information, also referred to as image data, on each captured image around the host vehicle V.

[0083] The at least one camera 34 is configured as a digital camera device consisting of an image sensor, such as a CCD (charge-coupled device) image sensor or a CMOS (complementary metal oxide semiconductor) image sensor. The image sensor of the at least one camera 34 is composed of a plurality of light-sensitive elements, such as photodiodes, each corresponding to a plurality of pixels arranged two-dimensionally in both the vertical direction (i.e., the vehicle height direction) and the horizontal direction (i.e., the vehicle width direction) of the host vehicle V.

[0084] The driving information input unit 3 of the basic embodiment includes a plurality of cameras, ie, a front camera CF, a rear camera CB, a left-side camera CL, and a right-side camera CR, mounted on the host vehicle V. The front camera CF, the rear camera CR, the left-side camera CL, and the right-side camera CR are also collectively referred to simply as a camera 34 or cameras 34 when it is not necessary to identify any of the cameras CF, CR, CL, and CR.

[0085] The front camera CF is mounted at a substantially center of an upper end of the front windshield V18 in the vehicle width direction in the upper part of the passenger compartment V2 and has a front field of view in front of the host vehicle V. That is, the front camera CF is located on the first center line LC1 in plan view. The front camera CF may be mounted on the front portion V11 of the vehicle body V1. The front camera CF is configured to capture an image of the front field of view to accordingly acquire information about the captured image of the front field of view.

[0086] The rear camera CB is mounted at a substantially center of a rear end V13 of the body V1 in the vehicle width direction and has a rear field of view located at the rear of the host vehicle V. The rear camera CR is configured to capture an image of the rear field of view to accordingly obtain information about the captured image of the rear field of view.

[0087] The left-side camera CL is mounted on the left door mirror V17 and has a left-side field of view located on the left side of the host vehicle V. The left-side camera CL is configured to capture an image of the left-side field of view in order to obtain information about the captured image of the left-side field of view accordingly.

[0088] The right-side camera CR is mounted on the right door mirror V17 and has a right-side field of view located on the right side of the host vehicle V. The right-side camera CR is configured to capture an image of the right-side field of view in order to obtain information about the captured image of the right-side field of view accordingly.

[0089] The image captured by each camera 34 consists of two-dimensionally arranged pixels, each corresponding to the two-dimensionally arranged light-sensitive elements of the corresponding camera 34.

[0090] The driving ECU 2 can recognize a target object B based on the images taken by any camera 34, that is, determine at least a location of a target object B and a type of the target object B based on the images taken by any camera 34.

[0091] Fig. 10 illustrates an example of how the driving ECU 2 recognizes target objects B included in the front field of view according to an image of the front field of view captured by the front camera CF; the front field of view includes a road Rd in front of the host vehicle V. In the description, reference symbol Rd is assigned to each road for convenience, but roads assigned the same reference symbol Rd may not necessarily represent the same road.

[0092] Specifically, the driving ECU 2 sets a detection region Aw in a captured image. The detection region Aw represents a part or all of an entire region of the captured image, that is, an entire view angle region of the front camera CF. Then, the driving ECU 2 acquires a feature point image Gp based on pixel feature parameters of the image data included in the detection region Aw. The pixel feature parameters of the image data represent feature parameters of each pixel constituting the image data based on corresponding received light, and may include, for example, a luminance, a contrast, and a hue of each pixel of the image data. The luminance of each pixel may be referred to as the brightness of the corresponding pixel, and the hue of each pixel may be referred to as the chroma of the corresponding pixel.

[0093] The feature point image Gp consists of feature points Pt arranged two-dimensionally in the horizontal and vertical directions like a single image. Each of the feature points Pt is extracted from the image data included in the detection region Aw, for example, based on the difference and / or the change gradient between a corresponding adjacent pair of pixels of the image data. The feature points Pt indicate the shape of a target object B included in the detection region Aw of a captured image. In other words, the feature points Pt are characteristic points, i.e., characteristic pixels, of the image data included in the detection region Aw.

[0094] The driving ECU 2 performs a pattern matching task of matching one or more feature point clouds Pg, each of which is an array of the corresponding feature points Pt, with predetermined patterns stored therein to identify, for each of the feature point clouds Pg, the type of a corresponding one of the target objects B based on the corresponding one of the feature point clouds Pg.

[0095] Various methods for extracting the feature points Pt from the image data included in the detection region Aw are well known. For example, an extraction method using a Sobel filter, an extraction method using a Laplacian filter, and an extraction method using a Canny algorithm can be used to extract the feature points Pt from the image data included in the detection region Aw. Therefore, detailed descriptions of one of the well-known extraction methods used will be omitted from the description. The extraction of the feature points Pt from the image data included in the detection region Aw can be expressed as detecting the feature points Pt from the image data included in the detection region Aw.

[0096] The driving ECU 2 can estimate a relative position of the detected target object B relative to the host vehicle V and / or a distance, ie a range, of the detected target object B relative to the host vehicle V.

[0097] Fig. 11 illustrates an example in which the driving ECU 2 calculates an estimated point Pb as an estimation result of three-dimensional positions of each feature point Pt corresponding to a target object B using, for example, a monocular motion stereo method, in other words, a so-called structure-from-motion (SFM) method. The present disclosure may calculate the estimated point Pb as the estimation result of the three-dimensional positions of each feature point Pt corresponding to a target object B using a selected one of the other calculation methods. That is, each camera 34 may be configured as a compound-eye stereo camera.

[0098] It is assumed that, as in Fig. 11, the camera 34 moves along with the host vehicle V in the direction of the host vehicle movement direction Dv of the host vehicle V. The position of the camera 34 at time t1 is referred to as a first camera position Pc1, and the position of the camera 34 at time t2 after time t1 is referred to as a second camera position Pc2. A first captured image A1 is an image captured by the camera 34 at the first camera position Pc1, and a second captured image A2 is an image captured by the camera 34 at the second camera position Pc2.

[0099] A first feature point Pt1 is a feature point Pt extracted from the first captured image A1, and a second feature point Pt2 extracted from the second captured image A2 is estimated to correspond to the first feature point Pt1 at time t1. That is, the second feature point Pt2 is a point for which a point on the target object B corresponding to the first feature point Pt1 is estimated to have moved to it over an elapsed time (t2 - t1) from time t1 to time t2. The driving ECU 2 can determine whether the first feature point Pt1 and the second feature point Pt1 are based on the same point on the target object B, that is, the first feature point Pt1 on the target object B corresponds to the second feature point Pt2 thereon, using one of the well-known methods such as an optical flow method.Then, the driving ECU 2 defines a first line L1 passing through the first camera position Pc1 and the first feature point Pt1, and a second line L2 passing through the second camera position Pc2 and the second feature point Pt2, and calculates an intersection of the first and second lines L1 and L2 as the estimated point Pb. The estimated point Pb represents, in a three-dimensional coordinate system defined relative to the own vehicle V, a point on the target object B that corresponds to both the first feature point Pt1 and the second feature point Pt2. When the estimated point Pb is a stationary point, the estimated point Pb satisfies an epipolar constraint. The epipolar constraint is an epipolar geometric constraint that the first camera position Pc1, the second camera position Pc2, and the estimated point Pb lie on the same plane Π at any time t1 or t2. Different sensors

[0100] According to Fig. 1 and Fig. 2, the operation sensors 35 are each provided in the host vehicle V for outputting a parameter indicating a corresponding driver operation state of the host vehicle V. The parameters to be output by the respective operation sensors 35, each of which represents the corresponding driver operation state of the host vehicle V, may include, for example, (i) a driver operation amount of an accelerator pedal of the host vehicle V, (ii) a driver operation amount of a brake pedal of the host vehicle V, (iii) a current shift position of the shift lever set by the driver, (iv) a steering angle of the steering wheel V24 operated by the driver, and (v) a steering torque of the steering wheel V24 applied by the driver. That is, the operation sensors 35 include known sensors, including, for example, an accelerator pedal sensor, a brake pedal sensor, a shift position sensor, a steering angle sensor, and a steering torque sensor.That is, for convenience, these known sensors are collectively referred to as the operational sensors 35. The operational sensors 35 may include a steering wheel sensor for detecting information indicating whether the driver D is gripping the steering wheel V24.

[0101] The behavior sensors 36 are each provided in the host vehicle V for outputting a parameter indicative of a corresponding driving behavior of the host vehicle V. The parameters to be output by the respective behavior sensors 36, each of which represents the corresponding behavior of the host vehicle V, may include, for example, (i) a speed of the host vehicle V, (ii) a yaw rate of the host vehicle V, (iii) an acceleration of the host vehicle V in the longitudinal direction, and (iv) an acceleration of the host vehicle V in the vehicle width direction. That is, the behavior sensors 36 include known sensors, including, for example, a vehicle speed sensor, a yaw rate sensor, and acceleration sensors. That is, these known sensors are collectively referred to as the behavior sensors 36 for convenience. Driver condition monitor

[0102] With reference to Fig. 1, Fig. 2, Fig. 12 and Fig. 13, the driver condition monitor 37 is configured to sequentially capture images of the driver D and sequentially acquire driver condition parameters representing the driver conditions based on the captured images of the driver D. The driver conditions include, for example, information indicating whether the driver D is in an awake state.

[0103] In particular, the driver condition monitor 37 includes a driver monitor camera with a predetermined field of view. The driver monitor camera is located in the passenger compartment V2 such that at least the head D1 of the driver D2 sitting in the driver seat V23 is within the field of view of the driver monitor camera. This allows the driver monitor camera to capture images of the face D2 of the driver D from the front. The driver monitor camera can be configured as a near-infrared camera.

[0104] The driver condition monitor 37 includes an image processing unit configured to perform image processing tasks on the images captured by the driver monitor camera to accordingly detect the driver condition parameters.

[0105] The driver condition parameters to be detected by the driver condition monitor 37 include, for example, as shown in Fig. 12 and Fig. 13 illustrates (i) the direction of the face D2 of the driver D, (ii) the opening degree of each eye D3 of the driver D, and (iii) the position of each pupil D4 of the driver D. The direction of the face D2 of the driver D is defined by a yaw angle θy and a pitch angle θp of the face D2 of the driver D. The yaw angle θy represents a rotation angle of the face D2 of the driver D about a vertical axis Dx1 extending vertically through the head D1 of the driver D. When the face D2 of the driver D faces forward, the yaw angle θy is 0°. That is, the yaw angle θy becomes a corresponding positive degree when the face D2 of the driver D faces leftward relative to the front, and becomes a corresponding negative degree when the face D2 of the driver D faces rightward relative to the front.

[0106] The pitch angle θp represents a rotation angle of the face D2 of the driver D about a horizontal axis Dx2 extending horizontally through the face D2 of the driver D. When the face D2 of the driver D is facing forward, the pitch angle θp is 0°. That is, the pitch angle θp becomes a corresponding positive degree when the face D2 of the driver D is facing upward relative to the front, and becomes a corresponding negative degree when the face D2 of the driver D is facing downward relative to the front.

[0107] Fig. 12 is an example of an image taken by the driver state monitor 37 in which the driver D is in the awake state, and Fig. 13 is an example of an image taken by the driver state monitor 37 in which the driver D is not awake. Fig. 12 and Fig. 13 show that the driver state parameters of (i) the direction of the face D2 of the driver D, (ii) the opening degree of each eye D3 of the driver D, and (iii) the position of each pupil D4 of the driver D enable the driving ECU 2 to detect the driver states, including, for example, (i) information about whether the driver D is in the awake state and (ii) information about the driver's line of sight D5. Operation switch

[0108] With reference to Fig. 1 and Fig. 2, the operation switches 38 are various switches that can be operated by the driver E when the driver E drives the host vehicle E, and are provided at predetermined positions in the interior V2. Each of the operation switches 38 is a switch whose operation amount and operation state are exempt from being detected by the operation sensors 35. Specifically, the operation switches 38 include, for example, an ignition switch 381, a turn signal switch 382, ​​and one or more AD / ADAS switches 383. The ignition switch 381, which may be referred to as a start switch or a power switch, is a switch for activating or deactivating the system vehicle V, that is, the vehicle system 1. The turn signal switch 382 is configured to detect a driver operation state of a turn signal lever of the host vehicle V.The AD / ADAS switches 383 are switches that, when operated, allow the driver E to input various instructions related to the driving assistance or autonomous driving of the host vehicle V into the driving ECU 2; the various instructions related to the driving assistance or autonomous driving of the host vehicle V include, for example, a start instruction, a stop instruction, a stage setting instruction, and a function selection instruction.

[0109] The positioning device 39 is configured to obtain highly accurate position information about the host vehicle V. Specifically, the positioning device 39 is configured as a complex positioning system and consists of a GNSS receiver 391, an inertial detector 392, and a positioning device ECU 393.

[0110] GNSS is an abbreviation for Global Navigation Satellite System, and the high-accuracy position information about the host vehicle V is position information about the host vehicle V that has at least a position accuracy usable by the advanced driving assistance at SAL level 2 or more, in other words, a position accuracy with an error of less than or equal to 10 cm. As the positioning device 39, a commercially available positioning system such as a POSLV system for land vehicles, in other words, a positioning azimuth system for land vehicles manufactured by Applanix Corporation, can be used.

[0111] The GNSS receiver 391 may be configured to receive navigation signals transmitted from at least one positioning satellite, i.e., at least one artificial satellite. Specifically, the GNSS receiver 391 is configured to be capable of receiving navigation signals from a positioning satellite included in at least one GNSS selected from GPS, QZSS, GLONASS, Galileo, IRNSS, and the Beidou Navigation Satellite System. GPS is an abbreviation for Global Positioning System, QZSS is an abbreviation for Quasi-Zenith Satellite System, GLONASS is an abbreviation for Global Navigation Satellite System, and IRNSS is an abbreviation for Indian Regional Navigation Satellite System.

[0112] The inertial detector 392 is configured to detect (i) linear accelerations acting on the host vehicle V in respective three axes corresponding to the vehicle longitudinal direction, the vehicle width direction, and the vehicle height direction, and (ii) angular velocities acting on the host vehicle V about the respective three axes. For example, the positioning device 39 has a substantially box-shaped housing, and the inertial detector 392 consists of a three-axis accelerometer and a three-axis gyro sensor installed in the housing.

[0113] The positioning device ECU 393 includes an in-vehicle microcomputer consisting of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The positioning device ECU 393 is configured to sequentially determine the current position and / or the current azimuth of the host vehicle V according to the navigation signals received by the GNSS receiver 391 and the linear accelerations and angular velocities detected by the inertial detector 392. Vehicle communication module

[0114] The vehicle communication module 4, also referred to as DCM 4, may be configured to communicate information with base stations located around the host vehicle V using wireless communications compliant with a predetermined communication standard, such as Long Term Evolution (LTE) or 5th Generation (5G).

[0115] In particular, the vehicle communication module 4 is configured to obtain traffic information, such as traffic congestion information, from probing servers and / or predetermined databases in a cloud computing environment. The traffic congestion information includes, for example, the location and length of at least one traffic congestion segment. In particular, the traffic congestion information includes, for example, various information elements about at least one traffic congestion segment, such as the head of the at least one traffic congestion segment, the end of the at least one traffic congestion segment, an estimated length of the at least one traffic congestion segment, and an estimated time for the at least one traffic congestion segment. The traffic information is also referred to as road traffic information.

[0116] In addition, the vehicle communication module 4 is configured to retrieve the latest HD map information from at least one of the sounding servers and to store the HD map information in the HD map database 5. HD map database

[0117] The HD map database 5 is primarily composed of one or more non-volatile rewritable memories and is configured to store the HD map information in a rewritable manner, while retaining the stored HD map information even when the power supplied to the HD map database 5 is turned off. The HD map information is also referred to as HD map data.

[0118] The HD map information includes higher-resolution map information than map information stored in a standard (SD) map database 601 of the navigation system 6. That is, the higher-resolution map information has a position error less than or equal to an error of approximately several meters of the map information stored in the SD map database 601.

[0119] Specifically, the HD map information database 5 stores, as the HD map information, map information available through advanced driving assistance or autonomous driving, including, for example, (i) information about three-dimensional road shapes, (ii) information about the number of lanes on each road, and (iii) information about traffic regulations. The HD map information is stored in the HD map information database 5 to conform to a predetermined standard, such as ADASIS. navigation system

[0120] The navigation system 6 is configured to calculate a planned route from the current position of the host vehicle V to a destination. The navigation system 6 of the basic embodiment is configured to calculate the planned route based on (i) the destination input by the driver D via, for example, the HMI system 7, (ii) the HD map information stored in the HD map database 5 or the SD map information stored in the SD map database 601, and (iii) the position information about the host vehicle V, such as the current position and the current azimuth of the host vehicle V.The navigation system 6 is additionally configured to provide various information, including the planned route, to one or more selected components of the vehicle system 1, such as the drive ECU 2 and / or the HMI system 7, via the vehicle communication network 10. That is, the navigation system 6 is capable of instructing the HMI system 7 to sequentially display navigation images, for example, showing maps on which the current position of the host vehicle V or the planned route appears. HMI system

[0121] The HMI system 7 is designed as a vehicle HMI system and is configured to implement information communications between the host vehicle V and one or more occupants, including the driver D of the host vehicle V.

[0122] In particular, the HMI system 7 is configured to provide, i.e., display, various information elements at least visibly to the one or more occupants, and to enable the occupant's input during information entry relative to the provided information elements. The various information elements to be provided to the one or more occupants include, for example, various guidance information elements, information elements for guiding the input process, notification of entered information, and / or warnings.

[0123] The HMI system 7 typically consists of I / O components mounted on the steering wheel V24 or installed in the dashboard V21, which is the so-called "dashboard HMI." At least one of the I / O components of the HMI system 7 can be mounted in at least one section of the interior V2, excluding the dashboard V21, such as the cell in the interior V2 or a center console located between the driver's seat V23 and the passenger seat V22, which is adjacent to the driver's seat V23.

[0124] The HMI system 7 includes an HMI control unit (HCU) 701, an instrument panel 702, a main display device 703, a head-up display 704, a speaker 705, and operating devices 706.

[0125] The HCU 701 includes a vehicle microcomputer consisting of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The HCU 701 is configured to perform overall control of the display output and / or audible output by the HMI system 7. That is, the HCU 701 is configured to control the operation of the instrument panel 702, the main display device 703, the head-up display 704, and the speaker 705.

[0126] The instrument panel 702 is installed in the dashboard V21 so as to be arranged to face the driver's seat V23. The instrument panel 702 is configured to display measured values, including, for example, the speed of the host vehicle V, the engine coolant temperature, and the fuel level. The instrument panel 702 is additionally configured to display various information items, including, for example, the current date and time, the outside temperature, and the receivable radio broadcasts.

[0127] The main display device 703, also referred to as a central information display (CID) device, is installed in the center of the instrument panel V21 in the vehicle width direction, allowing the one or more occupants to visibly recognize information displayed thereon.

[0128] The main display device 703 includes a housing and a screen installed in the housing, and can be configured to successively display on the screen the navigation images generated by the navigation system 6, for example, maps showing the current position of the host vehicle V and the planned route. The main display device 703 can also be configured to display on the screen various information and content that are different from the navigation images. For example, the main display device 703 can be configured to display a drive mode setting image on which icons of a plurality of drive modes are selectably displayed; the plurality of drive modes include a comfort drive mode, a normal drive mode, a sports drive mode, and a circuit drive mode.The main display mode 703 is further configured to display on the screen an image of a second task on which icons of a plurality of second tasks are selectably displayed. The second tasks, which are different from the driving tasks of the self-vehicle V, are usable by the driver D during autonomous driving of the self-vehicle V. For example, the secondary tasks include (i) a task of reading digital books, (ii) a task of operating a mobile communication terminal, and (iii) a task of watching video content, such as movies, concert videos, music videos, or television programs. The second tasks may be referred to as secondary activities or other tasks.

[0129] According to Fig. 1, Fig. 2, Fig. 14 and Fig. 15, the front-view display 704 is configured to display a virtual image M including letters, figures, and / or characters in the driver's forward view. The front-view display 704 of the basic embodiment is configured to project the virtual image M in front of the driver D using AR technologies to superimpose the virtual image M on the forward scenery, including the road surface FR located on the travel path of the host vehicle V; AR is an abbreviation for augmented reality. The task of superimposing the virtual image M on the forward scenery may display information included in the virtual image M relating to at least one focus target included in the forward scenery to be positionally linked to the at least one focus target.

[0130] For example, the task of overlaying the virtual image M on the forward scenery may display the information contained in the virtual image M while overlaying the information on the at least one focus target, or display the information contained in the virtual image M while adjacent to the at least one focus target. The at least one focus target is, for example, at least one target object that the driver D driving the host vehicle V should focus on, i.e., that the driver D driving the host vehicle V should pay attention to. The at least one focus target includes, for example, a road surface marking (a road marking), a road sign, a forward vehicle, and / or a pedestrian.For example, the view field display 704 may be configured to superimpose the planned travel route, a traveling direction of the host vehicle V, traffic information, and other similar information on a forward road surface FR as the focusing target.

[0131] An area on the front windshield V18 onto which the virtual image M is projected is called a projection area AP.

[0132] The field of view display 704 has, as shown in Fig. 14, a depression angle AD of, for example, a positive degree greater than 0°. The depression angle AD of the field of view display 704 is, for example, in the Fig. 14, is defined as an angle formed between a virtual horizontal plane passing through the driver D's eye points EP and a virtual plane passing through the driver D's eye points EP and an upper edge of the projection area AP. The depression angle AD becomes a positive value when the driver D's eye points EP look down the upper edge of the projection area AP, or becomes a negative value when the driver D's eye points EP look up the upper edge of the projection area AP.

[0133] The field of view display 704 has a vertical viewing angle AV and a horizontal viewing angle, and the vertical viewing angle AV defines a vertical width of the projection area AP, and the horizontal viewing angle defines a horizontal width of the projection area AP. The horizontal viewing angle, which is set larger than the vertical viewing angle AV, results in the projection area AP having a substantially rectangular shape. The vertical viewing angle AV can, for example, be Fig. 14, the virtual viewing angle AV can be defined as an angle formed between the virtual line passing through the left eye point EP of the driver D and the upper edge of the projection area AP and a virtual line passing through the left eye point EP of the driver D and a lower edge of the projection area AP. That is, the virtual viewing angle AV represents an angular range in the vertical direction within which the left eye point EP can view the virtual image M, which can be referred to as a vertical viewing angle in the vertical direction.

[0134] The field of view display 704 includes, as in Fig. 14 illustrates a projector 741 and a magnifying optical system 742. The projector 741 is configured to generate a light-based image LV based on a display image signal generated by the HCU 701 and emit the light-based image to the magnifying optical system 742. The magnifying optical system 742 includes a plurality of optical components, including a concave mirror and an actuator configured to control the alignment of the optical components.The magnifying optical system 742 is configured to project a magnified image of the light-based image LV onto the front windshield V18 using the optical components. The actuator controls the orientation of the optical components based on the location of the eye points EP detected by the driver state monitor 37 to accordingly adjust the projected state of the magnified image onto the front windshield V18. This results in the magnified image based on the light-based image LV being projected into the projection area AP of the front windshield V18. This allows the driver D to visibly perceive the virtual image M based on reflected light of the magnified image projected into the projection area AP of the front windshield V18.

[0135] Fig. 15 illustrates an example of the virtual image M. The virtual image M consists of, for example, a transparent base image, one or more pieces of information M1 included in the transparent base image, and one or more pieces of graphic content M2 included in the transparent base image. The one or more pieces of information M1 are pieces of information indicating, for example, the maximum speed limit for a road Rd on which the host vehicle V is traveling, the current speed of the host vehicle V, the distance to the destination, an estimated arrival time to the destination, and the name of at least one building appearing in the forward scenery. The one or more pieces of graphic content M2 are pieces of graphic content indicating lines and / or arrows used to display information indicating (i) the traveling direction of the host vehicle V and (ii) selection of lanes.

[0136] The field of view display 704 may be configured to display the virtual image M that includes overlaid content and non-overlaid content. The overlaid content is image content associated with one or more specific focus targets included in the forward scenery and superimposed on the one or more specific focus targets. In contrast, the non-overlaid content is image content that is not associated with the one or more specific focus targets included in the forward scenery and is not superimposed on the one or more specific focus targets.

[0137] In the virtual image M, which is shown as an example in Fig. 15, linear graphic contents, which are an example of the graphic contents M2, are superimposed on a lane on which the host vehicle V is traveling, and information content indicating the maximum speed limit for the current road of the host vehicle V, which is an example of the information contents M1, is located at a predetermined position of the virtual image M while not being superimposed on the graphic contents M2.

[0138] The HMI system 7 described above serves as a notification unit to notify the one or more occupants including the driver D in the host vehicle V with information.

[0139] According to Fig. 1 and Fig. 2, the speaker 705 is configured to output speech related to information indicated by each of the instrument panel 702, the main display device 703, and the head-up display 704. The speaker 705 may be configured to output speech, such as music and / or radio, that is not related to the information indicated by each of the instrument panel 702, the main display device 703, and the head-up display 704.

[0140] The operation devices 706 are input devices not included in the operation switches 38, and the operation amounts and operation states of the operation devices 706 are exempt from being detected by the operation sensors 35. Specifically, the operation devices 706 include, for example, switches mounted on the casing of the main display device 703 around the screen, and a transparent touch panel mounted to cover the screen of the main display device 703. The operation devices 703 may include switches mounted on a spoke of the steering wheel V24 and pointing devices, such as a touch panel, mounted on the center console.

[0141] The switches, pointing devices, and touch panel of the operating devices 706 allow the one or more occupants who operate them to input different items of information corresponding to the switches, pointing devices, and touch panel, respectively.

[0142] The lighting system 8 includes a body ECU 801, headlights 802 and turn signals 803.

[0143] The body ECU 801 includes a vehicle microcomputer consisting of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The body ECU 801 is configured to control how the headlights 802 light up according to information input from the driving ECU 2 and / or the driving information input unit 3, and to control how the turn signals 803 light up according to information input from the driving ECU 2 and / or the driving information input unit 3, specifically, information input from the turn signal switch 382.

[0144] The movement control system 9 is configured to control movements of the host vehicle V, ie, driving behavior of the host vehicle V, according to information input from the driving ECU 2 and / or the driving information input unit 3.

[0145] In particular, the motion control system 9 includes, for example, a driving system 91, a switching system 92, a braking system 93 and a steering system 94.

[0146] The driving system 91 includes a driving ECU 911 and a driving mechanism 912. The driving ECU 911 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The driving ECU 911 is configured to receive an accelerator pedal signal indicating an acceleration request from the accelerator pedal sensor or the driving ECU 2 and control operations of the driving mechanism 912 according to the acceleration request. The driving mechanism 912 is configured to generate drive energy, or driving energy, that causes the system vehicle V to drive. Specifically, the driving mechanism 912 includes an internal combustion engine, i.e., an internal combustion engine, and one or more motors.That is, the system vehicle V is any one of a gasoline-powered vehicle, a diesel-powered vehicle, a biofuel vehicle, a hydrogen-powered vehicle, a hybrid vehicle, a battery electric vehicle (BEV), a fuel cell vehicle, or other vehicles.

[0147] The shift system 92 includes a shift ECU 921 and a shift mechanism 922. The shift ECU 921 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The shift ECU 921 is configured to receive a shift position signal from the shift position sensor indicating the current shift position set by the shift lever or the drive ECU 2 and to control operations of the shift mechanism 922 according to the current shift position of the shift lever. The shift mechanism 922 is provided between the drive wheels V3 and the drive mechanism 912 and includes an automatic transmission. Specifically, the shift ECU 921 is configured to control the shift mechanism 922 according to the shift position signal indicating the current shift position set by the shift lever or the drive ECU 2.to perform (i) a first task of causing forward drive power generated by the drive mechanism 912 to be transmitted to the drive wheels for forward travel of the host vehicle V, (ii) a second task of causing reverse drive power generated by the drive mechanism 912 to be transmitted to the drive wheels for reverse travel of the host vehicle V, (iii) a third task of turning off the drive power to the drive wheels to stop the host vehicle V accordingly, and / or (iv) a fourth task of changing a speed ratio between an input speed from the drive mechanism 912 to the switching mechanism 922 and an output speed output by the switching mechanism 922 to the drive wheels during forward movement of the host vehicle V. The switching system 92 may be configured as a so-called shift-by-wire configuration.

[0148] The braking system 93 includes a brake ECU 931 and a brake mechanism 932. The brake ECU 931 includes an in-vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The brake ECU 931 is configured to receive a braking signal indicating a braking request from the brake pedal sensor or the driving ECU 2 and control operations of the braking mechanism 932 according to the braking request. The braking mechanism 932 includes a friction mechanism for each of the wheels V3. That is, the brake ECU 931 is configured to control the friction mechanism for each wheel V3 according to the braking request to apply friction to each wheel V3 accordingly, resulting in the host vehicle V being decelerated.The braking mechanism 932 may include a regenerative braking mechanism configured to rotate the drive wheels using the kinetic energy of the host vehicle V, to decelerate the host vehicle V according to the load of the rotation of the drive wheels, and to convert the kinetic energy of the host vehicle V into electrical energy. The braking system 93 may be configured as a so-called brake-by-wire configuration.

[0149] The steering system 94 includes a steering ECU 941 and a steering mechanism 942. The steering ECU 941 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The steering ECU 941 is configured to receive a steering signal indicating a steering request from the steering angle sensor or the driving ECU 2 and control operations of the steering mechanism 942 according to the steering request. That is, the steering ECU 941 is configured to control the steering mechanism 942 according to the steering request to change the direction of each steered wheel, for example, each front wheel V3a, V3b, to correspondingly change the traveling direction of the host vehicle V. The steering mechanism 942 may be configured to change the direction of each of the front and rear wheels V3a, V3b, V3c, and V3d. This means that the vehicle V can be configured as a four-wheel steering vehicle.The steering system 94 can be configured as a so-called steer-by-wire configuration. Driving control based on identification results from at least one of the ADAS sensors 31 to 34

[0150] Fig. 16 and Fig. 17 schematically illustrate an example of functions implemented by the driving ECU 2. That is, the processor 21 executes the program instructions stored in the storage device 22 to correspondingly Fig. 16 and Fig. 17 illustrated functions to be implemented.

[0151] The following describes a summary of autonomous driving of the own vehicle V performed by the driving ECU 2 based on identification results from at least one of the ADAS sensors 31 to 34 around the own vehicle V.

[0152] It should be noted that in the basic embodiment, "identification" conceptually includes "detection," "classification," and "recognition." Detection is to find a target object B based on at least one detection point data cloud and / or images captured by the cameras 34. Detection of a target object B is to determine that there is a target object B, not to identify the shape and / or attribute of the target object B. Classification of a target object B is to classify the shape and / or attribute of the detected target object B into one of different object types, such as "people," "vehicles," "buildings," and so on. In other words, a classified target object is a target object that has been detected and classified into one of the different object types.Detection of a target object B is to determine whether the detected and classified target object B should be taken into account in the driving control of the own vehicle V.

[0153] Detecting an object may involve measuring an object. Conceptually, detecting may involve classification and / or detection; conceptually, classification may involve detecting and / or detection; and detection may conceptually involve both detecting and classification.

[0154] According to Fig. 16 includes the driving ECU 2 as functional components implemented by the processor 21, an identification module 2001, an operation determiner 2002, and a control signal output module 2003.

[0155] The identification module 2001 is operative to perform an identification task for one or more target objects B around the host vehicle V according to pieces of information input from the environmental monitoring sensors 31 to 34, the operation sensors 35, and the behavior sensors 36. The operation determiner 2002 is operative to determine, based on an identified result of the identification module 2001 and the pieces of information input from the operation sensors 35 and the behavior sensors 36, one or more control tasks currently required to control the host vehicle V. The one or more control tasks include, for example, a collision avoidance task, an emergency stop task, and a warning task for the driver D.

[0156] The control signal output module 2003 is operative to output control signals based on the determined control tasks to selected components of the vehicle system 1. The control signals include, for example, a signal indicating the steering signal indicating a steering amount, the braking signal indicating the braking amount, and a message code signal indicating a warning message.

[0157] According to Fig. 17, the identification unit 120 includes an input information acquisition module 2101, an input information processing module 2102, a target object detection module 2103, a self-lane detection module 2104, an intersection detection module 2105, and an environment detection module 2106.

[0158] The information acquisition module 2101 acquires the information items input from the environmental monitoring sensors 31 to 34, the operation sensors 35, and the behavior sensors 36, and maintains the acquired information items in a sequential order. The input information processing module 2102 applies one or more predetermined tasks, such as a noise removal task and / or a coordinate conversion task, to the information items maintained in the information acquisition module 2101.

[0159] The target object detection module 2103 is operative to perform a target object detection task according to the information items subjected to the predetermined tasks.

[0160] Specifically, the target object detection module 2103 includes, for example, a marking line detection module 2131, a road surface marking detection module 2132, a roadside structure detection module 2133, a traffic light detection module 2134, a traffic sign detection module 2135, a lane detection module 2136, a pedestrian detection module 2137, a surrounding vehicle detection module 2138, and an obstacle detection module 2139.

[0161] Target objects B, which are to be recognized by the identification module 2001, are shown as examples in Fig. 18 or Fig. 19 illustrated.

[0162] According to Fig. 18, target objects B generally include, for example, traffic-related three-dimensional objects B1, other vehicles B2, general three-dimensional objects B3, and road surface markings B4. The target objects B to be recognized by the identification module 2001 additionally include, as in Fig. 19 illustrates parking space segment lines B5, wheel stoppers B6, and parked vehicles B7 related to the detection of parking spaces PS and / or the execution of parking assistance. The parking space segment lines B5 are, for example, painted lines to define the shape and area of ​​each parking space PS. Each of the wheel stoppers B6 is a block-like solid installed in each parking space PS to stop the wheels of a corresponding parked vehicle B7. Each of the parked vehicles B7 is a vehicle parked in the corresponding parking space PS.

[0163] The traffic-related three-dimensional objects B1 are three-dimensional objects, such as traffic lights B11 and traffic signs B12, used for road safety. Each of the other vehicles B2 can become a target vehicle tracking the moving host vehicle V or an obstacle to the moving host vehicle V, so the parked vehicles B7 are excluded from the other vehicles B2. The general three-dimensional objects B3 are solid objects except for the traffic-related three-dimensional objects B1, the other vehicles B2, the wheel stoppers B6, and the parked vehicles V7, and can mainly represent obstacles.

[0164] It is possible to recognize lanes LN according to the detection results of the target objects B obtained based on captured images. Specifically, although the parking spaces PS or the lanes LN are different from the target objects B that are direct detection targets by the ADAS sensors 31 to 34, these parking spaces PS and the lanes LN can be indirect detection targets based on the detection results of the target objects B. The lanes LN as the indirect targets include, for example, a self-lane LNm on which the self-vehicle V travels and oncoming traffic lanes LNc. The detection targets to be recognized by the identification module 2001 will be described in detail later.

[0165] The self-lane detection module 2104 is operative to detect the location of the self-vehicle V on a road in a road width direction based on the detection results of the target objects B by the target object detection module 2103; the road is a road on which the self-vehicle V is traveling. The road width direction is referred to as a road width direction, and a direction perpendicular to the road width direction is referred to as a road extension direction. The road extension direction is a direction extending along the road and may be referred to as a road extension direction or a road extension direction. When the road Rd includes multiple lanes LN, the self-lane detection module 2104 is operative to detect, as the self-lane LNm, one of the multiple lanes LN arranged in the road width direction.

[0166] The intersection detection module 2105 is operative to detect an intersection Xr around the host vehicle V based on the detection results of the target objects B by the target object detection module 2103.

[0167] Specifically, the intersection detection module 2105 is operative to detect an intersection Xr to which the host vehicle V is approaching, according to (i) whether there is a traffic light B11, (ii) which of the color signal lights are outputted from the traffic light B11 when it is determined that there is the traffic light B11, (iii) whether there is a stop line B42 as one of the road surface markings B4, according to (iv) a location of an intersection entrance Xr1, (v) a location of an intersection exit Xr2, and (vi) traffic signs or markings, each of which indicates a corresponding traveling direction.

[0168] The environment detection module 2106 is operative to detect an environment around the host vehicle V, for example, how one or more obstacles are located around the host vehicle V, based on the detected results of the target objects B by the target object detection module 2103.

[0169] These recognition results by the recognition modules 2103 to 2106 are used for the operation determiner 2002, which determines one or more control tasks currently required to control the host vehicle V.

[0170] Fig. 18 is an example of a forward scene from the host vehicle V in a situation where the host vehicle V is approaching an intersection Xr that forms a road intersection and includes a signal light B11 therein. An intersection Xr that forms a road intersection is referred to simply as an intersection Xr. The detection results of the signal light B11 and the traffic signs B12 included in the traffic-related three-dimensional objects B1 can be used for the HMI system 7 to notify the driver D of information and / or issue warnings to the driver D. The detection results of the signal light B11 and the traffic signs B12The traffic lights B11 and the traffic signs B12 included in the traffic-related three-dimensional objects B1 can also be used by the driving ECU 2 to perform motion control of the host vehicle V during autonomous driving of the host vehicle V and / or during driving assistance of the host vehicle V. The traffic-related three-dimensional objects B1 include, in addition to the signal lights B11 and the traffic signs B12, roadside structures B13, such as guardrails and curbs. The traffic-related solid objects B11 also include, as shown in . Fig. 20 illustrates road posts B14 and poles B15. The road posts may be detected as obstacles depending on (i) the heights of the road posts and / or (ii) how the road posts are arranged around the host vehicle V. The poles B15 are detected as obstacles when the drive ECU 2 executes drive control of the host vehicle V.

[0171] According to Fig. 18, the general three-dimensional objects B3 include, for example, fallen objects on the road surface B31, pedestrians B32, cyclists B33, and buildings B34. The fallen objects on the road surface B31, the pedestrians B32, and the cyclists B33 are recognized as obstacles when the driving ECU 2 executes driving control of the host vehicle V. The buildings B34 are not recognized as obstacles because the buildings B34 are located outside a road Rd, but can become content to be superimposed on the forward scenery when the view display 704 displays the virtual image M.

[0172] The road surface markings B4 include, for example, pedestrian crossing markings B41, stop lines B42 and road finishing lines B43. The road surface markings B4 also include, as in Fig. 21 illustrates letter markings B44 and symbol markings B45. Letter markings B44 include, for example, a number indicating the maximum speed limit, vehicle traffic zones such as "BUS ONLY," or restriction letters such as "STOP." Symbol markings B45 include, for example, arrows indicating mandatory directions and attention symbols for forward pedestrian crossings.

[0173] The detection results of the road traffic markings B4 are used to estimate the location of each of the intersection entrance Xr1 and the intersection exit Xr2, and the location of the intersection center Xrc. The intersection entrance Xr1 represents an edge of a focal intersection into which the host vehicle V will enter. The focal intersection is the nearest intersection Xr that (i) the host vehicle V is approaching and (ii) that the host vehicle V is scheduled to pass through or is likely to pass through. An intersection that the host vehicle V is likely to pass through is when no planned travel route and destination of the host vehicle V are determined by the navigation system 6, and an intersection that is estimated to be passed through by the host vehicle V with a high probability based on (i) a distance to the intersection from the host vehicle V and (ii) the speed of the host vehicle V.The intersection exit Xr2 represents an edge of the intersection in focus that the host vehicle V will leave or exit. The intersection center Xrc is the center of the intersection in focus.

[0174] As in Fig. 22 and Fig. As illustrated in Figure 23, the road marking lines B43 include, for example, vehicular road edge lines B431, center lines B432, and lane lines B433. The vehicular road edge lines B431 are provided for each vehicular road, and they respectively indicate the left and right edges of the corresponding vehicular road in the road width direction. The vehicular road of a road Rd represents a section of the road Rd in which vehicles can travel. Each center line B432 represents a line along the center of a two-way road and divides the two-way road into two sections in the respective opposite travel directions. Each center line B432 consists of a single straight line pattern, a single dashed line pattern, a double line pattern, or a triple line pattern. The lane lines B433 are provided for a road Rd with two or more lanes in each direction, and they divide the road Rd into two or more lanes in each direction.Each of the B433 lane lines is a painted white line or a painted yellow line. Fig. Figure 20 illustrates, as an example, that a center line B432 consists of a triple line pattern consisting of a white solid line and a pair of yellow lines located on either side of the white line. The road posts B14 and the posts B15 are located on the white line of the triple line pattern shown in Fig. 20 illustrated center line B432 mounted. Fig. Figure 21 illustrates, as another example, that a center line B432 consists of a double-line pattern consisting of white solid lines, and the lane line B433 consists of a white dashed line and a yellow solid line extending from the white dashed line. Based on the detection results of the roadside structures B13 and the road marking lines B43, it is possible to (i) identify a lane LNd or an own lane LNm on which the own vehicle V travels (see, for example, Fig. 20 and Fig. 21), (ii) oncoming traffic lanes LNc (see for example Fig. 20 and Fig. 21), (iii) a passing lane LNp (see for example Fig. 20), (iv) a right-turn lane LNr (see Fig. 21), (v) a roadside LNs (see Fig. 20 and Fig. 21) and (vi) an emergency parking zone EZ (not shown).

[0175] The road traffic markings B4 additionally include, for example, diversion zone markings, ie zebra zone markings, B461, safety zone markings B462, no-entry zone markings B463 and no-stopping zone markings B464.

[0176] The B461 diversion zone marking is painted on a road and indicates a diversion zone where it is necessary to guide the safe and smooth running of vehicles. As described above, the B461 diversion zone marking is a marking indicating a diversion zone for guiding the safe and smooth running of vehicles, and vehicles are not legally prohibited from entering the diversion zone. The B461 diversion zone marking may be provided to be adjacent to (i) an intersection Xr, (ii) a road junction, or (iii) a road fork.

[0177] The Fig. 24 illustrated safety zone markings B462 are used to indicate a safety zone. The safety zone represents a zone on a road provided to ensure the safety of, for example, pedestrians B32, so that vehicles cannot legally enter the safety zone.

[0178] The Fig. 25 illustrated no-entry zone markings B463 are used to indicate a no-entry zone. The no-entry zone represents a zone on a road that cannot be used for vehicle travel, preventing vehicles from entering the zone.

[0179] The Fig. The 26-illustrated no-stopping zone markings B464 are used to indicate a no-stopping zone. A no-stopping zone represents a zone on a road into which, depending on the situation ahead, a vehicle is likely to be stopped. This means that if any vehicle is waiting at a traffic light or is in a traffic jam, the vehicle should not be stopped within the no-stopping zone markings B464.

[0180] Fig. Figure 27 illustrates examples of traffic signs B12. Each of the traffic signs B12 is installed adjacent to one or more corresponding roads Rd or above the one or more corresponding roads Rd. Each of the traffic signs B12 consists of a display panel designed to provide necessary information to users facing the corresponding one of the traffic signs B12. The display panel of each traffic sign B12 has (i) a designed shape, (ii) one or more colors painted on at least one main surface thereof, and (iii) one or more symbols. The one or more colors include, for example, white, black, red, blue, and yellow. Fig. 27 represents the cross-hatching in red, the diagonal hatching in blue, the dot hatching in yellow, and the shaded section in black. For example, each symbol used in the B12 road signs includes one or more letters, one or more figures, and a combination of one or more letters and one or more figures.

[0181] Fig. Figure 27 illustrates examples of B12 traffic signs in Japan (JP), the European Union (EU), and the United States (US), categorized by significance. Examples of B12 traffic signs in Germany and / or France are used as typical examples of B12 traffic signs in the EU.

[0182] For example, the traffic signs in Japan, the EU, and the US that mean "NO ENTER" or "NO ENTRY" have essentially the same design. In contrast, the traffic signs in Japan, the EU, and the US that mean "STOP" have essentially the same color and symbol. The shape of the panel of the EU traffic sign is essentially the same as that of the US, but the shape of the panel of the Japanese traffic sign is different from that of the EU and the US.

[0183] The traffic signs in JP and EU which mean "MAXIMUM SPEED LIMIT" by the letter "50" have basically the same design except for the difference in letter color, but a traffic sign in US which means "SPEED LIMIT" by the letter "50" is different in the shape of the panel and color from those in JP and EU respectively.

[0184] The traffic signs in JP, the EU, and the US that mean “RAILWAY CROSSING WARNING” have a low degree of commonality in design.

[0185] The traffic signs in the EU and the US that mean "NO RIGHT TURN" have essentially the same design, and there is no corresponding traffic sign in JP.

[0186] A traffic sign from JP meaning “MOVE IN DIRECTION OF ARROW ONLY” and indicating the forward direction and the left turn direction using white arrows as in Fig. 27, also represents “NO RIGHT TURN”.

[0187] Precise recognition of the traffic signs B12 enables autonomous driving and / or advanced driving assistance to be implemented smoothly and safely. As described above, the designs of some of the traffic signs B12 vary considerably between Japan, the EU, and the US. For this reason, the storage device 22 stores in advance at least one database that stores information indicating patterns of traffic signs used, for example, in each country in the world or in each region of the world. Pattern matching between a recognized traffic sign B12 and the information stored in the at least one database enables the meaning of the recognized traffic sign B12 to be grasped.The at least one database may be configured as a common database for the entire world or consist of multiple databases provided for all countries of the world, each of the databases storing information indicating patterns of traffic signs used in the corresponding one of the countries of the world. The at least one database may be stored in a host computer instead of or in addition to the storage device 22; the driving ECU 2 is communicatively connected to the host computer via the vehicle communication module 4 based on vehicle-to-everything (V2X) technologies. This allows the driving ECU 2 to freely access the at least one database stored in the host computer.

[0188] Fig. 28 schematically illustrates a first example indicating how to use the detection results of target objects B in a situation where the host vehicle V is approaching an intersection. In this situation, the visual field display 704 may superimpose on the stop line B42 an information content M1 indicating a message "10 m to stop position" representing the distance of 10 m to the stop line B42. In addition, the HMI system 7 may output a sound message indicating "10 m to stop line" using the speaker 705. This effectively assists the driver D using the advanced driving support to reliably cause the host vehicle V to pause before the stop line B42.Alternatively, the sound message outputted by the HMI system 7 enables the one or more occupants of the host vehicle V to reliably detect the occurrence of a temporary stop event of the host vehicle V in advance, thereby avoiding a reduction in comfort for the one or more occupants of the host vehicle V.

[0189] Fig. 29 schematically illustrates a second example indicating how to use the detection results of target objects B in a situation where the host vehicle V is approaching an intersection with a traffic light B11 and the host vehicle V is about to turn right at the intersection. In this situation, the visual field display 704 of the host lane LN, i.e., the right-turn lane LNr, may superimpose on the road surface FR in front of the host vehicle V an information content M1 indicating a message “Turn right at the intersection 20 m ahead.” The HMI system 7 may also output a sound message indicating “Turn right at the intersection 20 m ahead” using the loudspeaker 705. In addition, the HMI system 7 may superimpose on the road surface FR a graphic content M2 of an arrow extending from the right-turn lane LNr to a target road Rd through the center Xrc of the intersection or thereabouts.This effectively provides the one or more occupants, including the driver D, of the host vehicle V with direction guidance of the host vehicle V and / or notification of the behavior of the host vehicle V experiencing lateral acceleration during the right turn. Therefore, this enables the one or more occupants of the host vehicle V to reliably detect the occurrence of the behavior of the host vehicle V in advance, thereby avoiding a reduction in comfort for the one or more occupants of the host vehicle V.

[0190] According to Fig. 17, the target object recognition module 2103 is operative to recognize the type and state of at least one target object B according to the recognition results of the at least one target object B and the HD map information stored in the HD map information database 5. The state of the at least one target object B includes, for example, position information about the at least one target object B. The position information about the at least one target object B includes, for example, the distance of the at least one target object B relative to the host vehicle V, the relative position of the at least one target object B relative to the host vehicle V, and, when the at least one target object B is one of the plurality of lanes LN, the position of one of the lanes LN.

[0191] In particular, the marking line detection module 2131 of the target object detection module 2103 is operative to detect the road marking lines B43 in a peripheral region around the host vehicle V, which includes the road surface FR of a road Rd located on the travel course of the host vehicle V. For example, the marking line detection module 2131 is operative to detect, for example, whether each of the vehicle road edge lines B431, the center lines B432 and / or the lane lines B433 located in Fig. 20 to Fig. 23, (i) is white or yellow, (ii) is a solid line or a dashed line, (iii) has a single-line pattern or a multi-line pattern. Since the detection technologies of the road marking lines B43 were well known at the time of filing the present application, detailed descriptions of the detection technologies for the road marking lines B43 are omitted from the present disclosure.

[0192] The road surface marking recognition module 2132 of the target object recognition module 2103 is operative to recognize the type, meaning, and position of each of the road surface markings B4 except the road marking lines B43; the recognition targets of the road surface marking recognition module 2132 include, for example, the pedestrian crossing markings B41, the stop lines B42, and the symbol markings B45.

[0193] The roadside structure recognition module 2133 is operative to recognize the type and position of at least one roadside structure B13 arranged around the host vehicle V according to, for example, at least one of (i) the recognition results based on the images captured by the cameras 34, (ii) the recognition results based on the at least one detection point data cloud acquired by the laser radar sensor 33, and (iii) the HD map information stored in the HD map information database 5.

[0194] The traffic light detection module 2134 is operative to (i) detect whether a traffic light B11 is located on the travel path of the host vehicle V, and (ii) detect the position of the traffic light B11 and which of the color signal lights are output by the traffic light B11 when it is detected that the traffic light or the signal light B11 is located on the travel path of the host vehicle V.

[0195] The traffic sign recognition module 2135 is operative to recognize the traffic signs B12 arranged around the host vehicle V.

[0196] The lane detection module 2136 is operative to perform a lane detection task for detecting the number of lanes LN on a road Rd on which the host vehicle V travels and the type of each lane LN on the road Rd. That is, the lane detection module 2136 is operative to perform the lane detection task according to, for example, (i) the detection results based on the images captured by the front camera CF of the cameras 34, (ii) the detection results based on the at least one detection point data cloud acquired by the laser radar sensor 33, and (iii) the HD map information stored in the HD map information database 5.

[0197] In particular, the lane detection module 2136 may be normally operative to perform the lane detection task according to the detection results based on the images captured by the cameras 34 or, if necessary, to perform a sensor fusion lane detection task according to a combination of (i) the detection results based on the images captured by the front camera CF of the cameras 34 and at least one of (ii) the detection results based on the at least one detection point data cloud captured by the laser radar sensor 33 and (iii) the HD map information stored in the HD map information database 5.

[0198] The pedestrian detection module 2137 is operative to detect one or more pedestrians B32 located around the host vehicle V. The surrounding vehicle detection module 2138 is operative to detect one or more other vehicles B2 located around the host vehicle V. The obstacle detection module 2139 is operative to detect one or more obstacles, such as one or more fallen objects B31 on the road surface and / or one or more pedestrians B32 located around the host vehicle V.How each of the traffic light detection module 2134, the traffic sign detection module 2135, the lane detection module 2136, the pedestrian detection module 2137, the surrounding vehicle detection module 2138, and the obstacle detection module 2139 detects one or more corresponding target objects was well known at the time of filing the present application, and therefore, detailed descriptions of how each of the modules 2134 to 2139 detects one or more corresponding target objects are omitted from the present disclosure. First embodiment

[0199] The first embodiment will be described below. In each of the following embodiments, the combination of (i) the driving ECU 2 serving as a vehicle device according to the corresponding embodiment, (ii) computer programs, i.e., computer program instructions to be executed by the driving ECU 2, i.e., the processor 21, according to the corresponding embodiment, and (iii) a storage medium storing the computer programs according to the corresponding embodiment is also collectively referred to as a present embodiment.

[0200] Fig. 30 is a schematic block diagram of a parking space detection device 2200 according to the first embodiment. The parking space detection device 2200 is configured to detect parking spaces PS located around the host vehicle V based on images captured by at least one of the cameras 34 during a parking assist operation or an automatic parking operation by the driving ECU 2. The parking assist operation may include (i) automatic parking, which parks the host vehicle V in a selected target parking space PS without any driver driving operations, and (ii) semi-automatic parking, which parks the host vehicle V in a selected target parking space PS (i) with partial driver driving operations, such as an operation of the accelerator pedal of the host vehicle V, and (ii) without other driver driving operations, such as the driver's steering operation.The parking space detection device 2200 may be implemented as at least one functional block included in the identification module 2001 shown in FIG. Fig. 17. In particular, the parking space detection device 2200 may be used as a part of the road surface character recognition unit 2132 shown in Fig. 17, or as part of the environment detection module 2106, which is shown in Fig. 17. Alternatively, the road surface sign recognition unit 2132 and the environment recognition module 2106 may serve as the parking space detection device 2200.

[0201] The following describes the details of the parking space detection device 2200 according to the present embodiment. The parking space detection device 2200 includes an annotation information generator 2201, a learning unit 2202, and a parking space determiner 2203. In addition, the parking space detection device 2200 includes an image information storage unit 2204, an image information input unit 2205, an annotation information storage unit 2206, and a learning model storage unit 2207.

[0202] The annotation information generator 2201 is configured to generate annotation information corresponding to information about each of images for training, referred to as training images or learning images, stored in the image information storage 2204, and output the annotation information to the annotation information storage 2206. The annotation information about each learning image represents the so-called ground truth data, which indicates what and where the one or more target objects B are located in the corresponding learning image.

[0203] In the present embodiment, the one or more target objects B are one or more parking spaces PS. That is, the annotation information generator 2201 is configured to generate, as the annotation information about each training image, the ground truth data, ie, ground truth annotation data, indicating what and where one or more parking spaces PS are located in the corresponding training image.

[0204] That is, each training image contains one or more parking spaces PS and each training image is paired with the corresponding ground truth annotation data that indicates what and where the one or more parking spaces PS are located in the corresponding training image.

[0205] The learning model memory 2207 contains, for example, a machine learning model LM, such as a deep neural network (DNN) model with variable parameters. The machine learning model LM stored in the learning model memory 2207 is configured to output derived annotation data indicating what and where one or more parking spaces PS are located in the input image when input data, ie, an input image, is input.

[0206] In other words, the learning unit 2202 is configured to learn a recognition rule of one or more parking spaces PS in an input image using the machine learning model LM stored in the learning model memory 2207.

[0207] In order to increase the inference accuracy, i.e., the learning accuracy, of the machine learning model LM, it is necessary that inputting many training images, each of which includes one or more actual parking spaces PS, into the machine learning model LM results in the inferred annotation data output by the machine learning model LM being substantially identical to the ground truth annotation data of the one or more actual parking spaces PS.

[0208] The learning unit 2202 is configured to perform the training of the parameters of the machine learning model LM stored in the learning model memory 2207 based on each training image stored in the image information memory 2204 and corresponding ground truth annotation information, ie, corresponding ground truth annotation data generated by the annotation information generator 2201. Then, the learning unit 2202 is configured to store the trained machine learning model LM with the trained parameters in the learning model memory 2207.

[0209] The image information input unit 2205 is configured to input one or more images captured by the at least one camera 34 into the parking space determiner 2203.

[0210] The parking space determiner 2203 is configured to execute a parking space determination task / parking space detection task using the trained machine learning model LM and the one or more images captured by the at least one camera 34 and input thereto by the image information input unit 2205.

[0211] The annotation information generator 2201 includes a bird's-eye view image generator 2211, an annotation input unit 2212, and a parking space annotation generator 2213.

[0212] The bird's-eye view image generator 2211 is configured to convert at least one training image stored in the image information storage 2204 into a bird's-eye view image using, for example, a known viewpoint conversion approach. Such a bird's-eye view image of a target is, for example, an image of the target viewed from above the target along a line of sight parallel to the direction of gravity acting on the target. That is, a bird's-eye view image of a target is an image corresponding to a top view of the target.

[0213] The annotation input unit 2212 is configured to input annotation information corresponding to the bird's-eye view image generated by the bird's-eye view image generator 2211 into the parking space annotation generator 2213. For example, the annotation input unit 2212 may perform an image recognition operation on the bird's-eye view image to automatically generate the annotation information corresponding to the bird's-eye view image. The annotation input unit 2212 may include a human-machine interface that allows an annotation information input worker to input correction information for correcting the automatically generated annotation information. Then, the annotation input unit 2212 may correct the automatically generated annotation information according to the correction information input by the annotation information input worker via the human-machine interface.

[0214] Alternatively, the annotation input unit 2212 may include such a human-machine interface that allows such an annotation information input worker to input information for generating the generated annotation information, and the annotation input unit 2212 may generate the generated annotation information according to the information input by the annotation information input worker through the human-machine interface.

[0215] The parking space annotation generator 2213 is configured to generate annotation information corresponding to at least one parking space PS included in the bird's-eye view image and output the annotation information corresponding to the at least one parking space PS included in the bird's-eye view image to the annotation information storage 2206.

[0216] In particular, the parking space annotation generator 2213 includes, for example, an angular position generator 2214 and an attribute generator 2215.

[0217] The angular position generator 214 is configured to generate, in a bird's-eye view image generated by the bird's-eye view image generator 2211, position information about parking space corner points BP and position information about a parking space center point BPC, which has the shape of a Fig. 31. The parking space corner points BP include a left front corner point BP1, a right front corner point BP2, a left rear corner point BP3, and a right rear corner point BP4.

[0218] The following describes the definition of each of the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, the right rear corner point BP4 and the parking space center point BPC with reference to Fig. 31. In Fig. 31 is for simplicity the Fig. 31 is surrounded by a rectangular parking segment B5 formed by a front segment line B501, a rear segment line B502, and a pair of side segment lines B503. However, as described later, the parking space PS is not limited to being surrounded by the rectangular parking space box B5. Specifically, the parking space PS may be formed by a substantially U-shaped parking segment B5 or by a plurality of linear lines B5 parallel to each other. Alternatively, the parking space PS may be formed as a space formed between parked vehicles B7 arranged adjacent to each other.

[0219] The front segment line B501 and the rear segment line B502 are located on respective short sides of the rectangular parking space PS. The front segment line B501 is located in front of the rear segment line B502 in a parking space direction BD. The parking space direction BD of the parking space PS, when a parked vehicle B7 is located in the parking space PS, is defined as a forward direction of the parked vehicle B7 in the parking space PS. The parking space direction BD of the parking space PS can be defined depending on the configuration and arrangement of the parking space PS. For example, when wheel stoppers B6 are provided to be closer to one of the short sides of the rectangular parking space PS, the parking space direction BD is defined from the nearer short side to the farther short side. As another example, when there are parking spaces PS arranged so that their short sides are aligned to form adjacent parking spaces,To form side-by-side parking spaces, the parking space direction BD of each parking space PS is defined from the parking space PS to the common parking corridor. As another example, if there are parking spaces PS arranged with their long sides aligned to form parallel parking spaces, the parking space direction BD of each parking space PS is defined as the direction of travel of a corresponding parked vehicle.

[0220] The pair of side segment lines B503 are located on the respective long sides of the rectangular parking space PS. When the parking space direction BD of the parking space PS is defined as the forward direction, the left side of the side segment lines B503 is referred to as a left side segment line B504, and the right side of the side segment lines B503 is referred to as a right side segment line B505.

[0221] The left front vertex BP1 is a vertex at the point where the front segment line B501 and the left-side segment line B504 intersect. The right front vertex BP2 is a vertex at the point where the front segment line B501 and the right-side segment line B505 intersect. The left back vertex BP3 is a vertex at the point where the back segment line B502 and the left-side segment line B504 intersect. The right back vertex BP4 is a vertex at the point where the back segment line B502 and the right-side segment line B505 intersect.

[0222] The parking space center point BPC is a center point of a rectangle formed by connecting the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, and the right rear corner point BP4. Specifically, the parking space center point BPC is an intersection point between a virtual straight line connecting the left front corner point BP1 and the right rear corner point BP4 and a virtual straight line connecting the right front corner point BP2 and the left rear corner point BP3. When the parking space PS is not surrounded by the rectangular parking segment B5, each corner point of the parking space PS can be defined in the same way as when the parking space PS is surrounded by the rectangular parking segment B5.

[0223] For example, if the parking space PS is surrounded by a substantially U-shaped parking segment B5 in which the front segment line B501 is missing compared to the rectangular parking segment B5, the left front corner point BP1 can be defined as the front end of the left-side segment line B504. Similarly, if the parking space PS is surrounded by a substantially U-shaped parking segment B5 in which the rear segment line B502 is missing compared to the rectangular parking segment B5, the right rear corner point BP4 can be defined as the rear end of the right-side segment line B505.When there is no parking segment B5 so that the parking space PS is formed as a space formed between parked vehicles B7 arranged adjacent to each other, each vertex of the parking space PS may be defined by a circumscribed rectangle around a parked vehicle B7 in the parking space PS or by a virtual rectangular space formed between parked vehicles B7 arranged adjacent to each other.

[0224] When parking spaces PS are included in a bird's-eye view image generated by the bird's-eye view image generator 2211, the attribute generator 2215 is configured to generate, for each parking space PS, attribute information about the corresponding parking space PS. The attribute information about each parking space PS may include, for example, parking segment information about the corresponding parking space PS. The parking segment information about each parking space PS may include, for example, (i) whether there is a corresponding parking segment B5, and (ii) whether there is a segment line component, such as a front segment line B501 or a rear segment line B502, in the corresponding parking segment B5. Other items of the attribute information except the parking segment information will be described in the second and subsequent embodiments to avoid redundant and complex descriptions.

[0225] The bird's-eye view image generator 2211 is configured to convert a training image stored in the image information memory 2204 into a bird's-eye view target image, also simply referred to as a target image (see reference symbol GS in Fig. 32). The target image GS is used to train the machine learning model LM and input the annotation information.

[0226] Fig. 32 is an example of a learning image including the host vehicle, ie, the system-installed vehicle V, which has entered a parking space PL, and a plurality of parking spaces PS arranged around the host vehicle V. Fig. 33 illustrates on a large scale part of the Fig. 32 illustrated learning images. In particular, Fig. 33 on a large scale, a selected parking space PS located diagonally to the left behind the own vehicle V and its surroundings; the selected parking space PS is a parking space in which the own vehicle V is about to be parked.

[0227] The parking space annotation generator 2213 is configured to generate, for each parking space PS included in each target image GS, the annotation information about the corresponding parking space PS in the target image GS according to the annotation information input thereto by the annotation input unit 2212. In particular, as shown in Fig. As illustrated in Figure 33, the parking space annotation generator 2213 generates, for each parking space PS, position information about at least one of the parking space vertices BP and the parking space center point BPC in each target image GS as the parking space annotation information. The parking space annotation generator 2213 is further configured to output the parking space annotation information about each parking space PS for each target image GS to the annotation information storage 2206, so that the parking space annotation information about each parking space PS for each target image GS is stored in the annotation information storage 2206.The attribute generator 2215 is configured to generate, for each parking space PS included in each target image GS, the attribute information about the corresponding parking space PS included in the corresponding target image GS and output the attribute information about each parking space PS for each target image GS to the annotation information storage 2206, so that the attribute information about each parking space PS for each target image GS is stored in the annotation information storage 2206.

[0228] The learning unit 2202 includes a bird's eye view image generator 2221 and a parking space training unit 2222.

[0229] Bird's-eye view image generator 2221 is configured to convert at least one training image stored in image information storage 2204 into a training bird's-eye view image using, for example, a known viewpoint conversion approach. The configuration and functions of bird's-eye view image generator 2221 are substantially identical to those of bird's-eye view image generator 2211. For this reason, bird's-eye view image generators 2221 and 2222 may be configured as a common module.

[0230] The parking space training unit 2222 is configured to perform the training of the parameters of the machine learning model LM stored in the learning model memory 2207 based on each target image GS generated by the bird's-eye view image generator 2221 and the annotation information, ie, ground truth data, for the corresponding target image GS generated by the parking space annotation generator 2213 and stored in the annotation information memory 2206, in order to increase the parking space derivation accuracy, ie, the parking space detection accuracy, of the machine learning model LM stored in the learning model memory 2207.

[0231] In the following, the training processing of the parameters of the machine learning model LM stored in the learning model memory 2207, which is performed by the parking space training unit 2222, will be described in detail in order to accordingly improve the accuracy of recognizing at least one parking space PS included in an input image.

[0232] First, the training processing divides a target image GS into Fig. 32 illustrated example into multiple grids GR using multiple grid lines GL. For simplicity, the training processing is divided as in Fig. As illustrated in Figure 32, the target image GS is mapped into a 10 by 10 grid GR, i.e., a 10 × 10 grid GR using nine vertical grid lines GL and nine horizontal grid lines GL. The 10 × 10 grid GR has 10 cells, i.e., 10 rows, in the vertical direction and 10 cells, i.e., 10 columns, in the horizontal direction.

[0233] Fig. 33 illustrates on a large scale a 3 × 3 grid GR, which is shown in Fig. 32 illustrated 10 × 10 grid GR. In the 3 × 3 grid GR, which is used as an example in Fig. 33, two parking space centers BPC are located in the middle cell of the 3 × 3 grid GR and the upper left cell of the 3 × 3 grid GR, respectively.

[0234] In this example, the training processing from the 3 × 3 grid GR objectifies a selected parking space PS based on the parking space center point BPC located in the middle cell of the 3 × 3 grid GR as the arrangement of the parking space center point BPC located in the middle cell of the 3 × 3 grid GR and the vertices BP1, BP2, BP3 and BP4 located in the respective of the upper middle cell, the middle right cell, the middle left cell and the lower left cell.

[0235] Then, the training processing encodes an existence of the parking space center point BPC as a first type of first encoded data Ec1, which indicates whether the parking space center point BPC exists in each cell of the 3 × 3 grid GR, which is called “CENTER: responsible” in Fig. 34. Specifically, the training processing encodes the existence of the parking space center point BPC as the first type of the first encoded data Ec1, in which a encoded data value Ecv of each of the middle cell and the upper left cell of the 3 × 3 grid GR is set to "1", and the encoded data values ​​Ecv of the other cells, that is, the upper middle cell, the upper right cell, the middle left cell, the middle right cell, the lower left cell, the lower middle cell, and the lower right cell, of the 3 × 3 grid GR are each set to "0".

[0236] Fig. 35 and Fig. 36 illustrate how the training processing encodes a horizontal position of the parking space center point BPC as a second type of the first coded data Ec1 in the 3 × 3 grid GR, which is denoted as “CENTER: Offset X” in Fig. 36 is illustrated.

[0237] The training processing according to the first embodiment defines a horizontal position of the parking space center point BPC in the middle cell of the 3×3 grid GR relative to the left side of the middle cell, and a horizontal position of the parking space center point BPC in the upper left cell of the 3×3 grid GR relative to the left side of the upper left cell. The horizontal position of any cell in the 3×3 grid GR is defined such that the left side is set to a first coded reference data value of "0" and the right side is set to a second coded reference data value of "1." Specifically, a virtual line segment of each of the middle cell and the upper left cell of the 3×3 grid GR is defined between the left side and the right side thereof. The virtual line segment passes through the corresponding parking space center point BPC and is parallel to the horizontal grid lines GL.The parking space center point BPC of each of the middle cells and the upper left cells of the 3 × 3 grid GR represents an internally dividing point of the corresponding virtual line segment. The horizontal position of the parking space center point BPC of each of the middle cells and the upper left cells of the 3 × 3 grid GR is defined as an internal division ratio expressed as a decimal number. For example, for the middle cell, because the internal division ratio of the parking space center point BPC in the middle cell is 1:1, the horizontal position of the parking space center point BPC in the middle cell is set to 0.5. Similarly, for the upper left cell, because the internal division ratio of the parking space center point BPC in the upper left cell is 2:3, the horizontal position of the parking space center point BPC in the middle cell is set to 0.4.

[0238] Then, the training processing performs an initial encoding of the five points BPC, BP1, BP2, BP3 and BP4 for the target image GS.

[0239] Specifically, the training processing encodes the horizontal position of the parking space center point BPC as the second type of the first encoded data Ec1, in which (i) the horizontal position of the middle cell is set to “0.5” as the encoded data value Ecv, (ii) the horizontal position of the upper left cell is set to “0.4” as the encoded data value Ecv, and (iii) the encoded data values ​​Ecv of the other cells are each set to “0”.

[0240] Fig. 37 and Fig. 38 illustrate how the training processing encodes a vertical position of the parking space center point BPC as a third type of the first coded data Ec1 in the 3 × 3 grid GR, which is denoted as “CENTER: Offset Y” in Fig. 38 is illustrated.

[0241] The training processing according to the first embodiment defines a vertical position of the parking space center point BPC in the middle cell of the 3×3 grid GR relative to the upper side of the middle cell, and a vertical position of the parking space center point BPC in the upper left cell of the 3×3 grid GR relative to the upper side of the upper left cell. The vertical position of any cell in the 3×3 grid GR is defined such that the upper side is set to the first coded reference data value of "0" and the lower side is set to the second coded reference data value of "1." Specifically, a virtual line segment of each of the middle cell and the upper left cell of the 3×3 grid GR is defined between the upper side and the lower side thereof. The virtual line segment passes through the corresponding parking space center point BPC and is parallel to the vertical grid lines GL.The parking space center point BPC of each of the middle cells and the upper left cells of the 3 × 3 grid GR represents an internally dividing point of the corresponding virtual line segment. The vertical position of the parking space center point BPC of each of the middle cells and the upper left cells of the 3 × 3 grid GR is defined as an internal division ratio expressed as a decimal number. For example, for the middle cell, because the internal division ratio of the parking space center point BPC in the middle cell is 1:1, the vertical position of the parking space center point BPC in the middle cell is set to "0.5." Similarly, for the upper left cell, because the internal division ratio of the parking space center point BPC in the upper left cell is 1:5, the vertical position of the parking space center point BPC in the middle cell is set to "0.2."

[0242] Then, the training processing encodes the vertical position of the parking space center point BPC as the third type of the first encoded data Ec1, in which (i) the vertical position of the middle cell is set to “0.5” as the encoded data value Ecv, (ii) the vertical position of the upper left cell is set to “0.2” as the encoded data value Ecv, and (ii) the encoded data values ​​Ecv of the other cells are each set to “0”.

[0243] Fig. 39 illustrates all the existence data elements of the parking space center point BPC as the first type of the first coded data Ec1, which are stored in Fig. 34 are illustrated, in the entire target image GS, which is shown in Fig. 32 is illustrated. Fig. 40 illustrates all horizontal position data items of the parking space center point BPC as the second type of the first coded data Ec1, which are stored in Fig. 36 are illustrated, in the entire target image GS, which is shown in Fig. 32 is illustrated. Fig. 41 illustrates all vertical position data items of the parking space center point BPC as the third type of the first coded data Ec1, which are stored in Fig. 39 are illustrated, in the entire target image GS, which is shown in Fig. 32 is illustrated.

[0244] Accordingly, (i) the existence data items, (ii) the horizontal position data items, and (iii) the vertical position data items of the parking space center point BPC have been acquired as the first, second, and third types of the first coded data Ec1 in the target image GS.

[0245] As the parking space center point BPC, (i) the existence data items, (ii) the horizontal position data items, and (iii) the vertical position data items of each of the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, and the right rear corner point BP4 were acquired as the first, second, and third types of the first coded data Ec1 in the target image GS.

[0246] That is, the first coded data Ec1 consists of (i) the existence data items, the vertical position data items, and the horizontal position data items of the parking space center point BPC, (ii) the existence data items, the vertical position data items, and the horizontal position data items of the left front corner point BP1, (iii) the existence data items, the vertical position data items, and the horizontal position data elements of the right front corner point BP2, (iv) the existence data items, the vertical position data items, and the horizontal position data elements of the left rear corner point BP3, (v) the existence data items, the vertical position data items, and the horizontal position data elements of the right rear corner point BP4.

[0247] Fig. 42 schematically illustrates a parking space annotation code Ea as the annotation information for an objectified parking space PS included in the target image GS consisting of the first coded data Ec1 regarding each of the five points BPC, BP1, BP2, BP3, and BP4.

[0248] As described above, the parking space training unit 2222 is configured to generate, for each of the five points BPC, BP1, BP2, BP3, and BP4, a total of three channel outputs of (i) the existence data, (ii) the horizontal position data, and (iii) the vertical position data as the first coded data Ec1.

[0249] As described above, the parking space training unit 2222 is configured to obtain, as a result of the first encoding of the five points BPC, BP1, BP2, BP3, and BP4 for the target image GS, (i) the first encoded data item Ec1 of the parking space center point BPC, (ii) the first encoded data item Ec1 of the left front corner point BP1, (iii) the first encoded data item Ec1 of the right front corner point BP2, (iv) the first encoded data item Ec1 of the left rear corner point BP3, and (v) the first encoded data item Ec1 of the right rear corner point BP4.

[0250] Next, the following describes a second encoding of connection vectors connecting from the parking space center point BPC to each of the vertices BP1, BP2, BP3 and BP4.

[0251] Fig. Figure 43 illustrates on a large scale a selected 3 × 3 grid GR consisting of nine cells of the 10 × 10 grid GR; the nine cells consist of a selected cell, i.e. a middle cell, defined by a square with thick lines in Fig. 42, and eight cells surrounding the selected cell. The Fig. 43 illustrated 3 × 3 grid GR is the one in Fig. 33 are identical, so that the arrangement of the five points BPC, BP1, BP2, BP3 and BP4 shown in Fig. 43, with the five points BPC, BP1, BP2, BP3 and BP4, which are shown in Fig. 33 are illustrated. The square marked with thick lines in Fig. 43 is the cell shown by the square with thick lines in Fig. 42 illustrated cell is identical.

[0252] Fig. Figure 44 illustrates an example of a connection number table Tb used to perform the second encoding. Each of Fig. 45 and Fig. 46 illustrates a result of the second encoding, which forms second encoded data Ec2.

[0253] In Fig. 43 is a connection vector directed from the parking space center point BPC as a starting point to the left front corner point BP1, illustrated by a dashed arrow. In Fig. 43, the connection vector has the left front corner point BP1 as an end point, which is located in the cell adjacent to the upper side of the middle cell in which the parking space center point BPC is located.

[0254] When the second coding of the connection vector is carried out, which is directed from the parking space center point BPC as the starting point to the left front corner point BP1, the training processing applies the Fig. 44 illustrates the connection number table Tb to the second encoding of the connection vector. The connection number table Tb comprises a 3 × 3 array corresponding to the 3 × 3 grid GR and has connection numbers Tbn of 0, 1, 2, 3, 4, 5, 6, 7, and 8 assigned respectively in the upper left cell, the upper middle cell, the upper right cell, the middle left cell, the middle cell, the middle right cell, the lower left cell, the lower middle cell, and the lower right cell. That is, the connection numbers Tbn of 0, 1, 2 are sequentially assigned to the respective cells of the upper row of the connection number table Tb, the connection numbers Tbn of 3, 4, 5 are sequentially assigned to the respective cells of the middle row of the connection number table Tb, and the connection numbers Tbn of 6, 7, 8 are sequentially assigned to the respective cells of the lower row of the connection number table Tb.

[0255] If the Fig. 44 is applied to the second coding of the connection vector directed from the parking space center point BPC as the starting point to the left front corner point BP1, because the connection vector is directed from the connection number Tbn 4 to the connection number Tbn 1, the training processing can Fig. 42 and Fig. 43 using the connection number Tbn 1.

[0256] As in Fig. 45, the training processing gives a coded data value Ecv of 0 or 1 to each of the cells of the Fig. 44 according to the connection numbers Tbn from 0 to 8, to correspondingly generate a second coded data element Ec2 of the connection vector directed from the parking space center point BPC as the starting point to the left front corner point BP1 shown in Fig. 42 and Fig. 43 are illustrated in the following procedure.

[0257] First, the training processing encodes the selected cell illustrated by the square with thick lines of the 3 x 3 grid GR, which is related to the connection vector directed from the parking space center point BPC as the starting point to the left front corner point BP1, into data consisting of "1" at the connection number Tbn 1 and "0" at each of the other connection numbers Tbn 0 and 2 to 8. The training processing performs a similar encoding operation for each of the other cells of the 10 x 10 grid GR corresponding to the target image GS to respectively generate the second coded data item Ec2 related to the connection vector directed from the parking space center point BPC as the starting point to the left front corner point BP1, each of which consists of nine-channel coded values ​​divided into Fig. 45; the second coded data elements Ec2 are arranged in conformity with the arrangement of the 10 x 10 grid GR.

[0258] Similarly, according to Fig. 43 and Fig. 44, the right front corner point BP2 is located in the cell located on the adjacent right side of the middle cell illustrated by the thick-lined square in which the parking space center point BPC is located. The cell in which the right front corner point BP2 is located corresponds to the link number Tbn 5. For this reason, the training processing encodes the selected cell illustrated by the thick-lined square of the 3 x 3 grid GR, which is related to a link vector directed from the parking space center point BPC as the starting point to the right front corner point BP2, into data consisting of "1" at the link number Tbn 5 and "0" at each of the other link numbers Tbn 0 to 4 and 6 to 8.The training processing performs a similar coding operation for each of the other cells of the 10 x 10 grid GR corresponding to the target image GS to respectively generate a second coded data item Ec2 related to the connection vector directed from the parking space center point BPC as the starting point to the right front corner point BP2, each of which consists of nine-channel coded values ​​expressed in . Fig. 45; the second coded data elements Ec2 are arranged in conformity with the arrangement of the 10 x 10 grid GR.

[0259] The training processing performs a similar coding operation for each of (i) a connection vector directed from the parking space center point BPC as the starting point to the left rear corner point BP3, and (ii) a connection vector directed from the parking space center point BPC as the starting point to the right rear corner point BP4, to respectively generate a total of four second coded data items Ec2 related to the four connection vectors from the parking space center point BPC as the starting point to the respective corner points BP1 to BP4; the four second coded data items Ec2 constitute the Fig. 46 illustrated second coded data Ec2.

[0260] Next, the following describes a third encoding of the parking segment information related to the parking segments B5 as a part of the attribute information with reference to the Fig. 47 to Fig. 54.

[0261] How Fig. 33 illustrated Fig. 47 on a large scale a selected 3 x 3 grid GR of the Fig. 32 illustrated 10 x 10 grid GR. The Fig. 47 illustrated 3 x 3 grids GR is with the in Fig. 33 are identical, so that the arrangement of the five points BPC, BP1, BP2, BP3 and BP4 shown in Fig. 47, with the five points BPC, BP1, BP2, BP3 and BP4, which are shown in Fig. 33 or Fig. 43. The parking segment information according to the first embodiment includes whether there is a segment line component, such as a front segment line B501 or a rear segment line B502, constituting the corresponding parking segment B5.

[0262] In the example of the 3 x 3 grid GR, which is shown in Fig. As illustrated in Figure 47, the parking space center points BPC are located in the middle cell and the upper left cell of the 3 x 3 grid GR, respectively. The training processing outputs a coded data value Ecv of 0 or 1 regarding the existence or non-existence of, for example, a left-side segment line B504 and a front segment line B501 to each of the middle cell and the upper left cell of the 3 x 3 grid GR.

[0263] Specifically, because the left-side segment line B504 exists to form the parking segment B5, the training processing outputs (i) the coded data value Ecv of “1” to each of the middle cell and the upper left cell of the 3 × 3 grid GR in which the parking space center point BPC exists, respectively, and (ii) the coded data value Ecv of “0” to each of the other cells of the 3 × 3 grid GR, thereby obtaining a coded result (cf. Fig. 48).

[0264] In contrast, because no front segment line B501 exists to form the parking segment B5, the training processing gives the coded data value Ecv of “0” to all cells of the 3 × 3 grid GR, thereby obtaining a coded result (cf. Fig. 49).

[0265] In particular, this consists in Fig. 47, the parking segment B5 consists only of the pair of left-side and right-side segment lines B504 and B505. Because the left-side segment line B504 exists to form the parking segment B5, the training processing outputs the coded data value Ecv "1" to each of the middle cell and the upper left cell of the 3 x 3 grid GR, in each of which the parking space center point BPC is located (cf. Fig. 48). As described above, the Fig. 47, the parking segment B5 does not consist of any front and rear segment lines B501 and B502. Because no front segment line B501 exists to form the parking segment B5, the training processing outputs the coded data value Ecv "0" to all cells of the 3 × 3 grid GR (cf. Fig. 49).

[0266] Fig. Figure 50 illustrates all coded data values ​​Ecv based on the third coding with respect to the existence or non-existence of the left-sided segment line B504, which in Fig. 48, in the entire target image GS, which is shown in Fig. 32 is illustrated. Fig. 51 illustrates all coded data values ​​Ecv based on the third coding with respect to the existence or non-existence of the right-side segment line B505 in the entire target image GS shown in Fig. 32 is illustrated. Fig. Fig. 52 illustrates all coded data values ​​Ecv based on the third coding with respect to the existence or non-existence of the posterior segment line B502 in the entire target image GS obtained in Fig. 32 is illustrated. Fig. Figure 53 illustrates all coded data values ​​Ecv based on the third coding regarding the existence or non-existence of the anterior segment line B501, which in Fig. 49, in the entire target image GS, which is shown in Fig. 32 is illustrated.

[0267] That is, all coded data values ​​Ecv based on the third coding with respect to the existence or non-existence of each of the left-side segment line B504, the right-side segment line B505, the back segment line B502, and the front segment line B501 constitute third coded data Ec3.

[0268] Fig. 54 schematically illustrates the parking space annotation code Ea for the objectified parking space PS included in the target image GS, comprising: (I) The first coded data Ec1 relating to each of the five points BPC, BP1, BP2, BP3 and BP4 (II) The second coded data Ec2 with respect to the connection vectors (III) The third coded data Ec3 relating to the existence or non-existence of the segment lines of the parking segment B5

[0269] Specifically, in the annotation information storage 2206, the ground truth annotation code Ea for each objectified parking space PS included in each target image GS was stored to be paired with the corresponding objectified parking space PS and the corresponding target image GS.

[0270] That is, the training processing calculates, for each objectified parking space PS in a target image GS input to the parking space training unit 2222, a loss function between (i) the ground truth annotation code Ea for the corresponding objectified parking space PS stored as the annotation information in the annotation information storage 2206 and (ii) the derived parking space annotation code Ea calculated for the corresponding objectified parking space PS included in the target image GS based on the machine learning model LM.

[0271] The training processing then updates the parameters of the machine learning model LM, for example, using backpropagation, to minimize the loss function accordingly. The parking space training unit 2222 is configured to repeat the training processing to train the parameters of the machine learning model LM accordingly, thereby improving the accuracy of detecting at least one parking space PS included in an input image.

[0272] Since the training processing using the loss function is already well known at the time of filing the corresponding Japanese patent application, further detailed description of the training processing is omitted.

[0273] Specifically, the annotation code Ea prepared for each target image GS includes the third coded data Ec3 regarding the existence or non-existence of the segment lines of the parking segment B5 of each parking space PS. This therefore enables the learning unit 2202 to infer, for each objectified parking space PS, whether the corresponding parking space PS has a parking segment B5 or one or more segment line components constituting the parking segment B5.

[0274] According to Fig. 30, the parking space determiner 2203 includes a bird's eye view image generator 2231, a parking space detector 2232, an edge calculator 2233, and a position corrector 2234.

[0275] The bird's-eye view image generator 2231 is configured to convert an image captured by at least one camera 34 into a bird's-eye view image using, for example, a known viewpoint conversion approach. The bird's-eye view image is used by the processor 21 to detect at least one parking space PS included in the captured image; the detected at least one parking space PS is used by the processor 21 to perform, for example, parking assistance of the host vehicle V.

[0276] The parking space detector 2232 is configured to input the bird's eye view image into the trained machine learning model LM stored in the learning model memory 2207 and to determine whether there are one or more parking spaces PS in the captured image according to the output data of the trained machine learning model LM.

[0277] In response to the determination that there are one or more parking spaces PS in the captured image according to the output data of the trained machine learning model LM, the parking space detector 2232 is configured to infer, i.e., detect, the one or more parking spaces PS and their states in the captured image according to the output data of the trained machine learning model LM. The state of each parking space PS includes information about whether the corresponding parking space PS is occupied by a parked vehicle B7, except for the host vehicle V.

[0278] In particular, the parking space detector 2232 is configured to detect, i.e., derive, for each parking space PS, location information about the corresponding parking space PS; the derived location information about each parking space PS includes (i) the position of each of the parking space center point BPC, the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, and the right rear corner point BP4, (ii) the connection vectors from the parking space center point BPC to the respective corner points BP1 to BP4, and (iii) the existence or non-existence of each segment line component of the parking segment B5 of the corresponding parking space PS.

[0279] The edge calculator 2233 is configured to calculate, in the bird's-eye view image, inner edges of each segment line component of the parking segment B5 of each actual parking space PS included in the bird's-eye view image.

[0280] The position corrector 2234 is configured to correct the derived location information about the corresponding parking space PS based on the calculated inner edges for each parking space PS in order to determine corresponding final location information about the corresponding parking space PS based on the corrected location information about each parking space PS.

[0281] The driving ECU 2 may be configured to perform, based on the final location information about each parking space PS included in the input image captured by at least one camera 34, (i) tasks for responding to an accelerator pedal misoperation, such as a task for suppressing sudden acceleration and / or a task for notifying one or more occupants of the accelerator pedal misoperation, (ii) parking assistance, and / or (iii) automatic parking. Recently, vehicles are required to have higher accuracy in detecting parking spaces PS.From these requirements, it is effective to detect different items of attribute information about each parking space PS in addition to the position information about each vertex of the corresponding parking space PS when detecting the corresponding parking space PS from an input image using a machine learning model such as a deep learning model.

[0282] In view of the above circumstances, the processor 21 of the driving ECU 2 according to the first embodiment is configured to objectify, from a target image GS, a selected parking space PS as the arrangement of a parking space center point BPC having four key points, that is, four vertices BP1, BP2, BP3, and BP4 arranged to surround the parking space center point BPC, in other words, the arrangement of four vertices BP1, BP2, BP3, and BP4 and a parking space center point BPC located within the four vertices BP1, BP2, BP3, and BP4. Then, the processor 21 learns a recognition rule of parking spaces PS from an input image using the objectified parking spaces PS.

[0283] In particular, the processor 21 is configured to train the parameters of the machine learning model LM, such as a deep learning mode, using, as the ground truth data, the parking space annotation code Ea for each training target image GS, which comprises: (I) the first type of first coded data Ec1, which includes the existence data of each of the five points BPC, BP1, BP2, BP3 and BP4 of an objectified parking space PS, (II) the second type of first coded data Ec1, which includes the horizontal position data of each of the five points BPC, BP1, BP2, BP3 and BP4 of the objectified parking space PS, (III) the third type of first coded data Ec1, which includes the vertical position data of each of the five points BPC, BP1, BP2, BP3 and BP4 of the objectified parking space PS, (IV) the second coded data Ec2 relating to the connection vectors relating to the objectified parking space PS.

[0284] Objectifying multiple sets of five points BPC, BP1, BP2, BP3, and BP4 acquired from a target image GS enables (i) the five points BPC, BP1, BP2, BP3, and BP4 of each set to be uniquely linked to the corresponding objectified parking space PS, and (ii) the attribute information to be applied to the corresponding objectified parking space PS.

[0285] In particular, the processor 21 is configured to train the parameters of the machine learning model LM using the parking space annotation code Ea for each training target image GS additionally having the attribute information, such as the third coded data Ec3 regarding the existence or non-existence of the segment lines of the parking segment B5 of the objectified parking space PS, as the ground truth data.

[0286] This allows the processor 21, when deriving a recognition rule of a parking space PS from an input image, to determine whether there is a segment line component, such as a front segment line B501 or a rear segment line B502, that forms the corresponding parking segment B5. This enables, after detecting, i.e., deriving, the parking space PS, to calculate edge lines that form the actual parking space PS from the input image, which makes it possible to correct the location of each derived vertex based on the calculated edge lines with high accuracy. This therefore improves the parking space derivation accuracy of the trained machine learning model LM to correspondingly improve the accuracy of recognizing the parking spaces PS included in the input image.This results in the parking of the host vehicle V in a selected one of the detected parking spaces PS with a higher accuracy during parking assistance or automatic parking.

[0287] Fig. 55 is a flowchart schematically illustrating, as an example of a task after detecting, ie, deriving, the parking spaces PS from an input image, a position correction routine for each derived parking space vertex BP according to the first embodiment. In Fig. 55, the reference symbol S is an abbreviation for step. Specifically, the processor 21 reads a corresponding set of program instructions from the storage device 22 and executes the program instructions in response to the completion of the derivation of the parking spaces PS from an input image to correspondingly perform the operations at steps S101 to S114 shown in the flowchart of Fig. 55. The position correction routine is described below.

[0288] When the position correction routine is started, the processor 21 initializes a parking space counter N, which is prepared in advance in the storage device 22, to 1 at step S101. The parking space counter N is a positive integer variable between 1 and Nmax. The value Nmax represents the total number of parking spaces PS detected, ie, derived, by the processor 21.

[0289] Next, the processor 21 selects one of the detected parking spaces PS as an N-th, that is, 1-th, parking space PS at step S102 and initializes a segment line component counter M, which is prepared in advance in the storage device 22, to 1 at step S103. The segment line component counter M is a positive integer which is any one of 1, 2, 3, and 4. The segment line component counter M set to 1 represents the front segment line B501, the segment line component counter M set to 2 represents the rear segment line B502, the segment line component counter M set to 3 represents the left-side segment line B504, and the segment line component counter M set to 4 represents the right-side segment line B505.

[0290] Following the operation at step S103, the position correction routine proceeds to step S104.

[0291] At step S104, the processor 21 refers to the acquired information, i.e., derived information, about the selected parking space PS using the segment line component corresponding to the segment line component counter M as a key. Specifically, when the segment line component counter M is set to 1 immediately after the operation at step S103, the processor 21 refers to the acquired information about the selected parking space PS using the front segment line B501 of the selected parking space PS as a key. Next, at step S105, the processor 21 determines whether the segment line component corresponding to the segment line component counter M is included in the acquired information, i.e., derived information, about the selected parking space PS.

[0292] When it is determined that the segment line component corresponding to the segment line component counter M is included in the acquired information on the selected parking space PS (YES at step S105), the processor 21 is programmed to perform the following operations at steps S106 to S109.

[0293] In particular, at step S106, the processor 21 calculates a circumscribed rectangle Rc (see Fig. 56) on the bird's-eye view image generated by the bird's-eye view image generator 2231. The circumscribed rectangle Rc has a pair of the detected vertices BP constituting the segment line component corresponding to the segment line component counter M. Next, at step S107, the processor 21 executes an edge calculation task for calculating the inner edges of the actual segment line component of the parking segment B5 of the selected parking space PS included in the bird's-eye view image based on luminance values ​​of the pixels included in the bird's-eye view image using a known filter such as a Sobel filter.

[0294] Next, at step S108, the processor 21 applies a known edge line generation task, such as a Hough transform, to the calculated inner edges of the actual segment line component to accordingly generate an edge line LL of the actual segment line component (see, for example, Fig. 57) to be calculated.

[0295] Following the operation at step S108, the processor 21 determines at step S109 whether the segment line component counter M is set to 4.

[0296] If it is determined that the segment line component counter M is set to any one of 1 to 3 except 4 (NO at step S109), the processor 21 recognizes that there is at least one segment line component included in the acquired information, i.e., derived information, about the selected parking space P; the edge calculation task at step S107 and the edge line calculation task at step S108 are not executed for the at least one segment line component. Then, at step S110, the processor 21 increments the segment line component counter M by 1 and thereafter returns to the operation at step S104 and iteratively performs the operations at steps S104 to S110 until the determination at step S109 is affirmative.

[0297] Otherwise, upon determining that the segment line component counter M is set to 4 (YES at step S109), the processor 21 recognizes that the edge calculation task at step S107 and the edge line calculation task at step S108 have been executed for all segment line components included in the acquired information, ie, derived information, about the selected parking space P. Then, the position correction routine proceeds to step S111.

[0298] At this time, if the edge line LL has not been calculated for at least one segment line component due to the determination at step S105 that the segment line component corresponding to the segment line component counter M is not included in the acquired information on the selected parking space PS (NO at step S105), the processor 21 calculates at least one linear line connecting between the corresponding adjacent pair of parking space vertices BP whose edge line LL was calculated at step S111.

[0299] Then, the processor 21 calculates four intersection points among the calculated four edge lines LL in which the at least one linear line may be included, at step S111.

[0300] Next, the processor 21 corrects the detected, i.e., derived, parking space vertices BP based on the machine learning model LM at step S112. Following the operation at step S112, the processor 21 determines at step S113 whether the parking space counter N is set to the value Nmax. If it is determined that the parking space counter N is not set to the value Nmax (NO at step S113), the processor 21 recognizes that at least one unselected parking space PS remains in the parking spaces PS detected, i.e., derived, by the processor 21. Then, at step S114, the processor 21 increments the parking space counter N by 1 and thereafter returns to the operation at step S102 and iteratively executes the operations at steps S102 to S114 until the determination at step S113 is affirmative.

[0301] That is, when it is determined that the parking space counter N is set to the value Nmax (YES at step S113), the processor 21 recognizes that the corner point position correction has been completed for all the parking spaces PS detected, ie, derived, by the processor 21, and accordingly the processor 21 ends the position correction routine. Second embodiment

[0302] Hereinafter, a parking space detection device 2200 according to the second embodiment of the present disclosure will be described with reference to FIG. Fig. 58 and Fig. 59 described.

[0303] Accordingly, the following mainly describes points of difference between the second embodiment and the first embodiment. Similarly, each of the following third embodiment and subsequent embodiments mainly describes points of difference from the previous embodiments. Therefore, the structure, functions, and achieved advantages of each of the subsequent embodiments are substantially identical to those of at least one of the previous embodiments, unless there are technical inconsistencies.

[0304] In an actual parking space PL, for example in Fig. 58, a situation may exist in which a part of the parking space segment B5 and / or at least one parking space vertex BP may be blocked by at least one parked vehicle B7. If the processor 21 carries out the training of the parameters of the machine learning model LM based on a training image acquired in such a situation, the processor 21 may train the parameters of the machine learning model LM such that a minor part of a parking space segment B5 appearing in the training image is a recognized target parking space PS. This may result in the processor 21 erroneously detecting a pedestrian crossing marking B41, a stop line B42, or a dashed road marking line B43 on a road Rd, for example, in Fig. 18 is illustrated, when a parking space PS is detected, ie derived.

[0305] From this point of view, the parking space annotation generator 2213 of the second embodiment may be configured to generate the annotation information, ie, the parking space annotation code Ea, for each parking space PS included in a target image GS, which includes blockage information indicating a degree to which the corresponding parking space PS is blocked by an object other than the host vehicle V, such as typically at least one parked vehicle B7. For example, the parking space annotation generator 2213 of the second embodiment is configured to Fig. 54, blockage information indicating a size of an unblocked region of the corresponding parking space PS or its parking space segment B5 is added to the parking space annotation code Ea for each parking space PS. This blockage information added to the parking space annotation code Ea for each parking space PS enables the processor 21 to determine whether to train the machine learning model LM for the corresponding parking space PS according to the size of the unblocked region of the corresponding parking space PS or its parking space segment B5. This enables a reduction in such erroneous detection of an object other than a parking space PS.

[0306] Alternatively, the parking space annotation generator 2213 of the second embodiment may be configured to generate the annotation information, i.e., the parking space annotation code Ea, for each parking space PS included in a target image GS, which includes blockage information indicating a degree to which each parking space vertex BP of the corresponding parking space PS is blocked, for example, by at least one parked vehicle B7. Then, these coded data added to the parking space annotation code Ea for each objectified parking space PS enables the processor 21 to determine whether to train the machine learning model LM for the corresponding parking space PS.In particular, the processor 21 may calculate the number of parking space vertices BP of a selected parking space PS that is blocked, for example, by at least one parked vehicle B7, and may determine not to train the machine learning model LM for the selected parking space PS if it is determined that the number of parking space vertices BP of the selected parking space PS that is blocked by the at least one parked vehicle B7 is greater than or equal to a non-training threshold, such as 3. This enables a reduction of such erroneous detection of an object other than a parking space PS while minimizing an increase in a load for generating the parking space annotation code Ea.

[0307] Fig. 59 is a flowchart schematically illustrating a portion of a training routine according to the second embodiment. In particular, the processor 21 is programmed to execute the portion of the Fig. 59 when starting to train the parameters of the machine learning model LM for each parking space PS using the corresponding parking space annotation code Ea containing the blockage information.

[0308] According to Fig. 59, the processor 21 refers to the parking space annotation code Ea for a selected parking space PS to determine whether the number of one or more blocked vertices of the selected objectified parking space PS is 0 at step S201. The blocked vertices are one or more parking space vertices BP that are blocked by at least one parked vehicle B7 so that they are difficult to see. If it is determined that the number of one or more blocked vertices of the selected parking space PS is 0 (YES at step S201), the processor 21 recognizes that the selected parking space PS is not blocked by any parked vehicle B7, so that the selected parking space PS is an empty space. Then, the processor 21 determines to train the parameters of the machine learning model LM for the selected parking space PS.

[0309] Otherwise, if it is determined that the number of one or more blocked vertices of the selected parking space PS is not 0 (NO at step S201), the processor 21 recognizes that the selected parking space PS is blocked by at least one parked vehicle B7, so that there is at least one blocked vertex. Then, the training routine proceeds to step S202.

[0310] At step S202, the processor 21 determines whether the number of one or more blocked vertices of the selected objectified parking space PS is 1. If it is determined that the number of one or more blocked vertices of the selected parking space PS is 1 (YES at step S202), the processor 21 determines to train the parameters of the machine learning model LM for the selected parking space PS.

[0311] Otherwise, if it is determined that the number of one or more blocked vertices of the selected parking space PS is not 1 (NO at step S202), the training routine proceeds to step S203.

[0312] At step S203, the processor 21 determines whether the number of one or more blocked vertices of the selected objectified parking space PS is 2. If it is determined that the number of one or more blocked vertices of the selected parking space PS is 2 (YES at step S203), the processor 21 determines to train the parameters of the machine learning model LM for the selected parking space PS.

[0313] Otherwise, if it is determined that the number of one or more blocked vertices of the selected parking space PS is not 2 (NO at step S203), the processor 21 detects that the number of one or more blocked vertices is 3 or more. Then, the processor 21 determines not to train the parameters of the machine learning model LM for the selected parking space PS, that is, the processor 21 determines not to calculate the loss function for the selected parking space PS. Third embodiment

[0314] Hereinafter, a parking space detection device 2200 according to the third embodiment of the present disclosure will be described with reference to Fig. 60 to Fig. 62 described. In Fig. 60, reference symbol Pc represents a viewpoint of at least one camera 34, referred to as a camera position Pc. In a parking assistance and / or automatic parking scene, information is required regarding whether each parking space PS is empty. That is, a target parking space PS in which the host vehicle V is about to be parked must be an empty parking space occupied by no parked vehicle B7. For this reason, it is preferable to train the parameters of the machine learning model LM for each parking space PS with the attribute information including whether the corresponding parking space PS is empty.

[0315] As in Fig. However, as illustrated in FIG. 60, there may be a situation where it is difficult to determine whether a target parking space PS that is empty is empty because a parked vehicle B7 occupies a parking space PS adjacent to the target parking space PS, blocking the target parking space PS. Assume that the processor 21 performs the training of the parameters of the machine learning model LM based on a training image acquired in such a situation. Under this assumption, the processor 21 may erroneously determine that the target parking space PS is an occupied parking space during parking assistance and / or automatic parking.

[0316] From this point of view, the parking space annotation generator 2213 of the third embodiment may be configured to generate the annotation information, ie, the parking space annotation code Ea, for each parking space PS included in a target image GS, which includes parking space empty information indicating whether each parking space PS included in the target image GS is empty.

[0317] In addition, the parking space annotation generator 2213 of the third embodiment is configured to determine how to train the parameters of the machine learning model LM for each parking space PS included in a target image GS according to the parking space empty information and a positional relationship between the camera position Pc of the host vehicle V, the corresponding parking space PS, and at least one parked vehicle B7.

[0318] Fig. 61 is a flowchart schematically illustrating preprocessing before training the parameters of the machine learning model LM. The processor 21 can serve as the annotation information generator 2201 and / or the learning unit 2202 to Fig. 61 illustrated preprocessing.

[0319] According to Fig. 61, the processor 21 initializes the parking space counter N, which has been prepared in advance in the storage device 22, to 1 at step S301. The parking space counter N is a positive integer variable between 1 and Nmax inclusive. The value Nmax represents the total number of parking spaces PS that are training candidates included in the target image GS; the parking space annotation code Ea has been prepared for each of the parking spaces PS.

[0320] Next, in step S302, the processor 21 selects one of the training candidate parking spaces PS as an Nth, i.e., 1st, parking space PS from the training candidate parking spaces PS. After the operation in step S302, the processor 21 calculates an amount of the selected parking space PS blocked by another object, such as typically at least one parked vehicle B7, in step S303. How the processor 21 calculates the blocked amount of the selected parking space PS will be described later. Next, in step S304, the processor 21 determines whether the blocked amount of the selected parking space PS is less than a predetermined threshold amount.

[0321] The following is in accordance with Fig. 62 describes how the processor 21 calculates the blocked amount of the selected parking space PS.

[0322] Fig. 62 illustrates, as an example, a situation in which a left side-by-side parking space train PSs and a right side-by-side parking space train PSs are arranged, respectively, on both sides of a part of a common parking corridor PSr in which the host vehicle V travels. Left and right are defined relative to the host vehicle V traveling in the part of the common parking corridor PSr. Each of the left and right side-by-side parking space train PSs represents a train or set of parking spaces PS arranged so that their short sides are continuously aligned in a predetermined direction parallel to the traveling direction of the common parking corridor PSr. The common parking corridor PSr represents a region through which vehicles can travel and faces at least one of the left and right side-by-side parking space train PSs.That is, the common parking corridor PSr can be defined as a region after removing several parking space trains including the left and right side-by-side parking space trains PSs from the entire region of the parking space PL.

[0323] The right side-by-side parking space train PS, which is used as an example in Fig. 62, has a first, second, third, and fourth parking space PS from the near side of the host vehicle V to the far side thereof. A parked vehicle B7 occupies a first to third parking space PS, respectively, and the fourth parking space PS is empty. The fourth parking space PS, which is empty, is referred to as a target parking space PSt. Note that Fig. 62 illustrates an exemplary situation used to simply describe how the processor 21 calculates the blocked amount of the target parking space PSt representing the selected parking space PS, and therefore, the present disclosure is not limited to this situation. Specifically, the processor 21 may calculate the blocked amount of a selected parking space PS in another situation in the same manner as in the exemplary situation.

[0324] The processor 21 can calculate the blocked amount of the target parking space PSt based on a viewing angle of the host vehicle V to the target parking space PSt.

[0325] First, as in Fig. 62, an existence range angle θe is defined as a value of the viewing angle of the at least one camera 34 located at the camera position Pc, onto the target parking space PSt in a first case in which the entire target parking space PSt is not blocked by parked vehicles B7, so that the entire target parking space PSt can be seen by the at least one camera 34. In the Fig. 62, the existence range angle θe is defined as an angle formed between a first virtual line extending from the camera position Pc and passing through the right front vertex BP2 and a second virtual line extending from the camera position Pc and passing through the left rear vertex BP3.

[0326] Next, a field of view angle θc shall be defined, which represents a field of view of the at least one camera 34 located at the camera position Pc, onto the target parking space PSt in a second case in which a part of the target parking space PSt is blocked by the parked vehicles B7. In the Fig. In the example illustrated in Figure 62, the view angle θc is defined as an angle formed between the first virtual line extending from the camera position Pc and passing through the right front corner point BP2 and a third virtual line extending from the camera position Pc and passing through the right front corner of the parked vehicle B7 located in the third parking space PS in the right side-by-side parking space train PSs. In addition, a blocked angle range θh shall be defined, which is defined by subtracting the view angle θc from the existence range angle θe.

[0327] The processor 21 may be configured to calculate, as the blocked amount of the target parking space PSt, the ratio of the blocked angular range θh to the existence range angle θe or the percentage of the blocked angular range θh to the existence range angle θe. The predetermined threshold amount used to compare with the blocked amount at step S304 can be freely set to any value, such as 0.8 or 80%, according to empirical rules and / or experiments. The threshold amount can be modified or corrected according to the training situations.

[0328] Specifically, if it is determined that the blocked amount of the selected parking space PS is greater than or equal to the predetermined threshold amount (NO at step S304), the preprocessing proceeds to step S305. At step S305, the processor 21 changes the parking space empty information about the selected parking space PS included in the parking space annotation code Ea to an "undefined state" at step S305. At this time, if the class for the parking space empty information is set to an undefined class or the training of the parameters of the machine learning model LM for the selected parking space PS cannot be performed, the preprocessing proceeds to step S306.

[0329] Otherwise, if it is determined that the blocked amount of the selected parking space PS is less than the predetermined threshold amount (YES at step S304), the preprocessing proceeds to step S306 while skipping the operation at step S305. This results in the parking space empty information about the selected parking space PS included in the parking space annotation code Ea being kept unchanged.

[0330] At step S306, the processor 21 determines whether the parking space counter N is set to the value Nmax. If it is determined that the parking space counter N is not set to the value Nmax (NO at step S306), the processor 21 recognizes that at least one unselected parking space PS remains in the training candidate parking spaces PS. Then, at step S307, the processor 21 increments the parking space counter N by 1 and then returns to the operation at step S302 and iteratively executes the operations at steps S302 to S307 until the determination at step S306 is affirmative.

[0331] That is, when it is determined that the parking space counter N is set to the value Nmax (YES at step S306), the processor 21 ends the preprocessing. Fourth embodiment

[0332] Hereinafter, a parking space detection device 2200 according to the fourth embodiment of the present disclosure will be described with reference to FIG. Fig. 63 to Fig. 68 described.

[0333] Fig. 63 illustrates a 5 x 5 grid GR of a target image GS showing a situation in which an upper side-by-side parking space train, including a parking space PS in which the host vehicle V is parked, and a lower side-by-side parking space train exist. That is, the parking spaces PS included in the upper side-by-side parking space train are adjacent to the respective parking spaces PS in the lower side-by-side parking space train, while the parking spaces PS of each adjacent pair share the common rear segment line B502. The upper side-by-side parking space train included in Fig. 63 is actually located in front of the lower side-by-side parking space train.

[0334] Assume that an address of the parking space PS in which the host vehicle V is parked is denoted as id:0, and addresses of the parking spaces PS in the lower side-by-side parking space train located at the lower side of the parking space PS with the address id:0 are denoted as id:1, id:2, id:3, and id:4, respectively. The parking space PS with the address id:0 is arranged to face the two adjacent parking spaces PS with the respective addresses id:2 and id:3. For simplicity, the left rear corner point BP3 in the parking space corner points of each parking space PS is illustrated only by a black circle, and the other parking space corner points and the parking space center points BPC are illustrated by a white circle.In a cell of the 5 x 5 grid GR, surrounded by white dashed lines and referred to as a notable cell GRt, the left rear corner point BP3 of the parking space PS with address id:0 and the left rear corner point BP3 of the parking space PS with address id:2 correspond to the same location.

[0335] The first coded data Ec1 according to the Fig. 42 illustrated first embodiment consist of (I) the data of a first type, which include the existence data of each of the five points BPC, BP1, BP2, BP3 and BP4 of an objectified parking space PS, (II) the data of a second type, which include the horizontal position data of each of the five points BPC, BP1, BP2, BP3 and BP4 of the objectified parking space PS, (III) the data of a third type, which includes the vertical position data of each of the five points BPC, BP1, BP2, BP3 and BP4 of the objectified parking space PS,

[0336] That is, the first coded data Ec1 is generated under the precondition that (i) the existence data of each point BPC, BP1, BP2, BP3 and BP4 is set to a corresponding single cell in the grid GR, (ii) the horizontal position data of each point BPC, BP1, BP2, BP3 and BP4 is set to a corresponding single cell in the grid GR, and (iii) the vertical position data of each point BPC, BP1, BP2, BP3 and BP4 is set to a corresponding single cell in the grid GR.

[0337] Since the remarkable cell GRt of the fourth embodiment is shared by both the left rear corner point BP3 of the parking space PS with the address id:0 and the left rear corner point BP3 of the parking space PS with the address id:2, it may be difficult to set (i) a first piece of information on the left rear corner point BP3 of the parking space PS with the address id:0 and (ii) a second piece of information on the left rear corner point BP3 of the parking space PS with the address id:2 to the common remarkable cell GRt under the above precondition. For this reason, encoding information on a midpoint between the left rear corner point BP3 of the parking space PS with the address id:0 and the left rear corner point BP3 of the parking space PS with the address id:2 enables the common encoded value to be set to the remarkable cell GRt.This coding may cause an encoded data value of the horizontal position or vertical position of the left rear corner point BP3 of the parking space PS with address id:0 to differ from an inherent encoded value, and similarly, an encoded data value of the horizontal position or vertical position of the left rear corner point BP3 of the parking space PS with address id:2 to differ from an inherent encoded value. This may lead to a reduction in the inference accuracy using the LM machine learning model.

[0338] Such a situation, in which parking space vertices BP of the same type of different parking spaces PS are located in a common cell of the target image GS, can occur in a specified parking space in which the parking spaces PS with the different orientations (see Fig. 31) are adjacent to each other.

[0339] From this point of view, the fourth embodiment, as in Fig. 64A illustrates typical orientations of a parking space PS using four anchor indices, ie anchor 0, anchor 1, anchor 2 and anchor 3. In Fig. 31, anchor 0, indicated by the upward arrow, represents that the orientation of the parking space direction BD of the parking space PS in the target image GS is upward, and anchor 1, indicated by the left arrow, represents that the orientation of the parking space direction BD of the parking space PS in the target image GS is left. Similarly, in Fig. 31, the anchor 2 indicated by the right arrow represents that the orientation of the parking space direction BD of the parking space PS in the target image GS is to the right, and the anchor 3 indicated by the downward arrow represents that the orientation of the parking space direction BD of the parking space PS in the target image GS is downward.

[0340] In particular, the processor 21 is configured to train the parameters of the machine learning model LM using, as the ground truth data, the parking space annotation code Ea for each training target image GS whose first coded data Ec1 additionally includes coded information indicating each typical orientation of the objectified parking space PS.

[0341] In an actual target image GS, as in Fig. 64B, the parking space PS is inclined with respect to each of the vertical and horizontal directions of the target image GS. Then, the determination of the orientation of the parking space PS uses a circumscribed rectangle Rc and a reference point PW of the parking space PS. The circumscribed rectangle Rc of the parking space PS is a virtual rectangle circumscribed around the four parking space vertices BP and has each side parallel to the corresponding one of the vertical and horizontal directions of the target image GS. The reference point PW of the parking space PS is a point on the circumscribed rectangle Rc. In the fourth embodiment, the reference point PW of the parking space PS is a point on the left corner of the circumscribed rectangle Rc. The orientation of the parking space direction BD of the parking space PS is determined based on which of the four parking space vertices BP of the parking space PS is located closest to the reference point PW.

[0342] Specifically, when the left front corner point BP1 is closest to the reference point PW, the orientation of the parking space direction BD of the parking space PS is set to anchor 0. When the right front corner point BP2 is closest to the reference point PW, the orientation of the parking space direction BD of the parking space PS is set to anchor 1. When the right rear corner point BP4 is closest to the reference point PW, the orientation of the parking space direction BD of the parking space PS is set to anchor 2. When the left rear corner point BP3 is closest to the reference point PW, the orientation of the parking space direction BD of the parking space PS is set to anchor 3.

[0343] The following describes a coding routine executed by the processor 21 with reference to Fig. 65 to Fig. 68.

[0344] If the Fig. 65 is started, the processor 21 initializes the parking space counter N, which has been prepared in advance in the storage device 22, to 1 at step S401. The parking space counter N is a positive integer variable between 1 and Nmax inclusive. The value Nmax represents the total number of parking spaces PS, which are training candidates, included in the target image GS.

[0345] Next, the processor 21 selects one of the training candidate parking spaces PS as an N-th, ie, 1-th, parking space PS at step S402.

[0346] After selecting the parking space PS in step S402, the processor 21 performs the subsequent operations in steps S403 to S408. Specifically, in step S403, the processor 21 calculates the circumscribed rectangle Rc for the selected parking space PS and, in step S404, calculates the reference point PW based on the circumscribed rectangle Rc. Next, in step S405, the processor 21 calculates distances between the reference point PW and the respective four parking space vertices BP.

[0347] Following the operation at step S405, the processor 21 performs a matching operation in which, according to the calculated distances for the respective four parking space vertices BP, it is determined at step S406 which of the anchor indices 0 to 3 corresponds to the selected parking space PS. That is, the processor 21 determines one of the anchor indices 0 to 3 that corresponds to the selected parking space PS.

[0348] The following describes in detail the matching operation at step S406 with reference to the Fig. 66 and Fig. 67. In each of the Fig. 66 and Fig. 67 corresponds to an x-axis direction of the horizontal direction of the target image GS, in which the horizontal position of any point in the grid GR in each of the Fig. 35 and Fig. 36, and a y-direction corresponds to the vertical direction of the target image GS, in which the vertical position of any point in the grid GR in each of the Fig. 35 and Fig. 36 is calculated. In Fig. 66, two parking spaces PS are referred to as parking spaces PS1 and PS2, and the circumscribed rectangle Rc for each of two parking spaces PS1 and PS2 has a first pair of sides parallel to the x-axis direction and a second pair of sides parallel to the y-axis direction. Similarly, in Fig. 67 two parking spaces PS are referred to as parking spaces PS3 and PS4 and the circumscribed rectangle Rc for each of two parking spaces PS3 and PS4 has a first pair of sides parallel to the x-axis direction and a second pair of sides parallel to the y-axis direction.

[0349] In the example of Fig. 66, the left rear corner point BP3 of the parking space PS1 and the left rear corner point BP3 of the parking space PS2 are located in the common notable cell GRt. One of the parking space corner points BP of the parking space PS1 that is closest to the reference point PW is the left front corner point BP1. For this reason, the anchor index of the parking space PS1 is set to anchor 0. In contrast, one of the parking space corner points BP of the parking space PS2 that is closest to the reference point PW is the right rear corner point BP4. For this reason, the anchor index of the parking space PS2 is set to anchor 2.

[0350] In the example of Fig. 67, the left rear corner point BP3 of the parking space PS3 and the left rear corner point BP3 of the parking space PS4 are located in the common notable cell GRt. One of the parking space corner points BP of the parking space PS3 that is closest to the reference point PW is the left front corner point BP1. For this reason, the anchor index of the parking space PS3 is set to anchor 0. In contrast, one of the parking space corner points BP of the parking space PS4 that is closest to the reference point PW is the right front corner point BP2. For this reason, the anchor index of the parking space PS4 is set to anchor 1.

[0351] According to Fig. 65, following the matching operation, at step S406, the processor 21 encodes the parking space vertices BP and the parking space center point BPC for each of the anchor indices, i.e., for each of anchor 0, anchor 1, anchor 2, and anchor 3. Specifically, at step S406, the processor 21 generates (i) first encoded data Ec1 corresponding to anchor 0, (ii) first encoded data Ec1 corresponding to anchor 1, (iii) first encoded data Ec1 corresponding to anchor 2, and (iv) first encoded data Ec1 corresponding to anchor 3.

[0352] Then, at step S407, the processor 21 generates the parking space annotation code Ea for the objectified parking space PS included in the target image GS shown in Fig. 68, shows: (I) a first coded data arrangement consisting of (i) the first coded data Ec1 having anchor 0 with respect to each of the five points BPC, BP1, BP2, BP3 and BP4, (ii) the first coded data Ec1 having anchor 1 with respect to each of the five points BPC, BP1, BP2, BP3 and BP4, (iii) the first coded data Ec1 having anchor 2 with respect to each of the five points BPC, BP1, BP2, BP3 and BP4 and (iv) the first coded data Ec1 corresponding to anchor 3 with respect to each of the five points BPC, BP1, BP2, BP3 and BP4 (II) the second coded data Ec2 with respect to the connection vectors (III) the third coded data Ec3 relating to the existence or non-existence of the segment lines of the parking segment B5

[0353] That is, the Fig. The parking space annotation code Ea according to the fourth embodiment illustrated in Fig. 68 is configured such that the Fig. 54 illustrated first coded data Ec1 are replaced by the first arrangement of coded data.

[0354] According to Fig. 65, the processor 21 determines in step S408 whether the parking space counter N is set to the value Nmax. If it is determined that the parking space counter N is not set to the value Nmax (NO in step S408), the processor 21 recognizes that at least one unselected parking space PS remains in the training candidate parking spaces PS. Then, in step S409, the processor 21 increments the parking space counter N by 1 and thereafter returns to the operation in step S402 and iteratively executes the operations in steps S402 to S409 until the determination in step S408 is affirmative.

[0355] That is, when it is determined that the parking space counter N is set to the value Nmax (YES at step S408), the processor 21 terminates the coding routine.

[0356] In particular, the processor 21 of the fourth embodiment is configured to calculate the parameters of the machine learning model LM using the parking space annotation code Ea for each training target image GS, which additionally comprises: (i) the first coded data Ec1 corresponding to anchor 0, i.e., the upward orientation, with respect to each of the five points BPC, BP1, BP2, BP3, and BP4, (ii) the first coded data Ec1 corresponding to anchor 1, i.e., the leftward orientation, with respect to each of the five points BPC, BP1, BP2, BP3, and BP4, (iii) the first coded data Ec1 corresponding to anchor 2, i.e., the rightward orientation, with respect to each of the five points BPC, BP1, BP2, BP3, and BP4, and (iv) the first coded data Ec1 corresponding to anchor 3, i.e., the downward orientation, with respect to each of the five points BPC, BP1, BP2, BP3 and BP4 as ground truth data.

[0357] Accordingly, even if parking space vertices BP of the same type, i.e., the same class, from different parking spaces PS are located in a common cell of the target image GS, this configuration enables the processor 21 to derive a recognition rule for a parking space PS from an input image for each of the individual parking space vertices BP of the same type. This therefore enables the parking space vertices BP to be detected more accurately, even if parking space vertices BP from different parking spaces PS are concentrated in one input image. Fifth embodiment

[0358] Hereinafter, a parking space detection device 2200 according to the fifth embodiment of the present disclosure will be described with reference to Fig. 69 to Fig. 75 described.

[0359] As in Fig. 69, at least one of the parking space corner points BP of a parking space PS may be located outside a target image GS. In particular, the right rear corner point BP4 of a parking space PS, which is indicated by a dashed rectangle in Fig. 69, outside the target image GS. In this case, if the right rear corner point BP4 is not encoded, so that the training of the parameters of the machine learning model LM for the right rear corner point BP4 is not performed, the accuracy of the inference result of the parking space PS may be reduced.

[0360] From this point of view, the processor 21 is preferably configured to perform an offset task for offsetting the right rear corner point BP4, which is located outside the target image GS, into the target image GS and then encode the right rear corner point BP4. As the direction in which the right rear corner point BP4 is offset to be in the target image GS, the direction in the long side of the rectangular parking space PS or in the short side of the rectangular parking space PS can typically be used.

[0361] Each of Fig. 70 to Fig. Figure 73 illustrates a corresponding example of the offset task for offsetting one of the four parking space vertices BP located outside the target image GS; the region of the target image GS is also referred to as a training input region AN. In each of Fig. 70 to Fig. 73, one of the four parking space corner points BP, which is referred to as an outer point located outside the target image GS, is represented by a black circle. In each of Fig. 70 to Fig. 73, reference character PSe represents an offset parking space whose offset segment point has been offset to be in the target image GS.

[0362] As the outer point is located outside the target image GS, which is in Fig. 70 is identical to how the outer point is located outside the target image GS, which in Fig. 71, but the offset direction of the outer point, which is in Fig. 70 is different from the offset direction of the outer point, which is shown in Fig. 71. Similarly, the outer point is located outside the target image GS, which is shown in Fig. 72 is identical to how the outer point is located outside the target image GS, which is shown in Fig. 73, but the offset direction of the outer point, which is in Fig. 72 is different from the offset direction of the outer point, which is shown in Fig. 73 is illustrated.

[0363] In the example of each of Fig. 70 and Fig. 71, the percentage referred to as an out-of-percentage of a part of a short side of the rectangular parking space PS located outside the target image GS to the entire short side is substantially 30%, and the percentage of a part of a long side of the rectangular parking space PS located outside the target image GS to the entire long side is substantially 40%. In other words, the percentage referred to as an in-image percentage of the remaining part of the short side of the rectangular parking space PS located in each of Fig. 70 and Fig. 71, the part of which is outside the target image, to the entire short side 70%, and the in-image percentage of the remaining part of the long side of the rectangular parking space PS, which in each of Fig. 70 and Fig. 71, whose part is outside the target image, to the entire long side is 60%. The offset direction for the outer point, which is in Fig. 70 is set to the direction along the long side and the offset direction for the outer point, which is in Fig. 71 is set to the direction along the short side.

[0364] Likewise, in the example of each of Fig. 72 and Fig. 73 the outside percentage of a part of a short side of the rectangular parking space PS located outside the target image GS to the entire short side is substantially 80%, and the percentage of a part of a long side of the rectangular parking space PS located outside the target image GS to the entire long side is substantially 40%. In other words, the in-image percentage of the remaining part of the short side of the rectangular parking space PS located in each of Fig. 72 and Fig. 73, the part of which is outside the target image, to the entire short side 20% and the in-image percentage of the remaining part of the long side of the rectangular parking space PS, which in each of Fig. 70 and Fig. 71, whose part is outside the target image, to the entire long side 60%. The offset direction for the outer point, which is in Fig. 72 is set to the direction along the long side and the offset direction for the outer point, which is in Fig. 73 is set to the direction along the short side.

[0365] If you compare Fig. 70 with Fig. 71 shows that the shape of the offset parking space PSe, which is to be coded and in Fig. 71 is illustrated, an original form of the actual parking space PS retains much more than the one shown in Fig. 70 is illustrated. If one compares Fig. 72 with Fig. 73 shows that the shape of the offset parking space PSe, which is to be coded and in Fig. 72 is illustrated, an original form of the actual parking space PS retains much more than the one shown in Fig. 73 is illustrated.

[0366] That is, displacing the outer point of the parking space PS to one of the partially outer short side and the partially outer long side of the parking space PS, whose out-of-image percentage is lower than the other, enables the shape of the displaced parking space PSe to retain the original shape of the parking space PS. In other words, displacing the outer point of the parking space PS to one of the partially outer short side and the partially outer long side of the parking space PS, whose in-image percentage is higher than the other, enables the shape of the displaced parking space PSe to retain the original shape of the parking space PS.

[0367] Accordingly, offsetting the outer corner point of the parking space PS into the target image in a direction determined to retain the original shape, i.e., the rectangular shape, of the parking space PS as much as possible enables both (i) improving the inference result of the parking space PS even if the parking space PS has an outer corner point, and (ii) reducing the training of the parking space PS whose outer corner point(s) have been retained as they are.

[0368] Each of Fig. 74 and Fig. 75 illustrates a corresponding example of the offset task for offsetting two of the four parking space vertices BP of the parking space PS that are located outside the target image GS. That is, when there are two or fewer outer points in the four parking space vertices BP, offsetting the two or fewer outer points of the parking space PS into the target image enables the offset parking space PSe to be encoded for training the parameters of the machine learning model LM. This enables an improvement in the inference result of the parking space PS even when the parking space PS has two or fewer outer vertices. In contrast, when there are three or more outer points in the four parking space vertices BP of the parking space PS, the processor 21 does not encode the parking space PS because the shape of the offset parking space would not retain the original rectangular shape of the parking space PS. Sixth embodiment

[0369] Hereinafter, a parking space detection device 2200 according to the sixth embodiment of the present disclosure will be described with reference to Fig. 76A to Fig. 81 described.

[0370] As in Fig. 76A, a plurality of lane lines B433, each of which has a dashed line, may be provided in a road Rd having three or more lanes in each direction in the road width direction. In this case, when the focus is placed on a selected lane LNd located at a center of the road Rd in the road width direction, which is divided by two lane lines B433 located on both sides of the selected lane LNd, a region on the selected lane LNd surrounded by a dash-dotted line X is very similar in appearance to a region shown in Fig. 76B illustrates the parking space PS formed only by a pair of left-side and right-side segment lines B503. The selected lane LNd located in the center of the road Rd in the road width direction is a lane located between the outermost lane LNd adjacent to the vehicle road edge line B431 and the passing lane LNp adjacent to the center line B432.

[0371] For this reason, an erroneous detection of the region on the selected lane LNd surrounded by the dotted line X may be considered as the Fig. 76B while the host vehicle V is traveling on the road Rd may cause the processor 21 to erroneously determine that the host vehicle V is traveling in the parking space PL. This erroneous determination may result in a reduction in the comfort of one or more occupants of the host vehicle V due to sudden, frequent warnings caused by accelerator pedal misoperations. From this point of view, training the parameters of the machine learning model LM based on the information about such a region on the road Rd that is different from the parking spaces PS may result in the occurrence of such an erroneous recognition of a region on the road Rd as a parking space PS.

[0372] From this point of view, the sixth embodiment causes the processor 21 to determine whether there is a parking space environment around the host vehicle V when executing the training of the parameters of the machine learning model LM. In addition, the processor 21 is configured to derive one or more parking spaces PS in an input image in accordance therewith, that is, to detect whether the host vehicle V is located in the parking space PL.

[0373] In particular, the parking space annotation generator 2213 of the sixth embodiment is configured to generate the annotation information, ie, the parking space annotation code Ea, for each parking space PS included in a target image GS, which additionally includes parking space annotation information indicating whether the host vehicle V is located in the parking space PL. In addition, the processor 21 determines whether the host vehicle V is located in the parking space PL when deriving one or more parking spaces PS in an input image, and discards the detected results, ie, derived results, of the parking spaces PS when it is determined that the host vehicle V is not located in the parking space PL. This configuration effectively reduces such erroneous detection and erroneous determination. The parking space annotation generator 2213 of the sixth embodiment may be configured to generate the annotation information, ie,to generate the parking space annotation code Ea for each parking space PS included in a target image GS, which additionally includes information indicating whether there is a parking space environment around the host vehicle V. This enables the processor 21, when executing the training of the parameters of the machine learning model LM, to determine whether (i) the host vehicle V is located in the parking space PL or (ii) the host vehicle V is not located therein or (iii) the host vehicle V is located in an unidentified space.

[0374] Fig. 77A is a flowchart schematically illustrating a parking space environment annotation information generation routine executed by the processor 21 serving as the annotation information generator 2201, that is, the parking space annotation generator 2213.

[0375] According to Fig. 77A, the processor 21 determines whether there is a parking space environment around the host vehicle V, that is, whether the host vehicle V is located in a parking space PL, based on the annotation information input by the annotation input unit 2212 when receiving the annotation information input by the annotation input unit 2212.

[0376] When it is determined that the host vehicle V is located in a parking space PL (YES at step S501), the parking space environment annotation information generation routine proceeds to step S502.

[0377] Otherwise, if it is determined that the host vehicle V is not located in a parking space PL (NO at step S501), the processor 21 determines that there is no parking space environment around the host vehicle V, that is, that the host vehicle V is outside a parking space PL. Then, at step S503, the processor 21 adds fourth coded data Ec4 representing that there is no parking space environment around the host vehicle V to the parking space annotation code Ea for each parking space PS included in the target image GS.

[0378] There may be a case where, even if the host vehicle V is actually located in a parking space PL, it is difficult to accurately determine from the target image GS whether the host vehicle V is located in a parking space PL, for example, due to a situation where a parking space PS is located against walls, so that the walls only appear around the parking space PS in the target image GS. From this point of view, the processor 21 determines whether there are one or more parked vehicles B7 in the target image GS based on the annotation information input by the annotation input unit 2212 at step S502.

[0379] If it is determined that there are one or more parked vehicles B7 in the target image GS (YES at step S502), the processor 21 determines that it is likely that the host vehicle V is located in a parking space PL, that is, that there is a high confidence that the host vehicle V is located in a parking space PL. Then, at step S504, the processor 21 adds the fourth coded data Ec4 representing that the host vehicle V is located in a parking space PL to the parking space annotation code Ea for each parking space PS included in the target image GS.

[0380] Alternatively, at step S502, the processor 21 determines whether there are one or more parking spaces PS in the target image GS based on the annotation information input by the annotation input unit 2212.

[0381] If it is determined that there are one or more parking spaces PS in the target image GS (YES at step S502), the processor 21 determines that it is likely that the host vehicle V is located in a parking space PL, that is, that there is a high confidence that the host vehicle V is located in a parking space PL. Then, at step S504, the processor 21 adds the fourth coded data Ec4 representing that the host vehicle V is located in a parking space PL to the parking space annotation code Ea for each parking space PS included in the target image GS.

[0382] Otherwise, if it is determined that there are no parked vehicles B7 in the target image GS (NO at step S502), the processor 21 determines that it is unlikely that the host vehicle V is located in a parking space PL, that is, there is low reliability that the host vehicle V is located in a parking space PL. Similarly, if it is determined that there are no parking spaces PS in the target image GS (NO at step S502), the processor 21 determines that it is unlikely that the host vehicle V is located in a parking space PL, that is, there is low reliability that the host vehicle V is located in a parking space PL. Then, at step S505, the processor 21 adds the fourth coded data Ec4 representing that the host vehicle V is located in an unidentified space to the parking space annotation code Ea for an objectified parking space PS included in the target image GS.

[0383] As described above, the processor 21 of the sixth embodiment is configured to add, to the parking space annotation code Ea for each parking space PS included in the target image GS, the fourth coded data Ec4 representing that (i) there is no parking space environment around the host vehicle V, (ii) the host vehicle V is located in a parking space PL, or (iii) the host vehicle V is located in an unidentified place.

[0384] Fig. 77B is a flowchart schematically illustrating a portion of a training routine according to the sixth embodiment. In particular, the processor 21 is programmed to execute the portion of the Fig. 77B when training the parameters of the machine learning model LM for each parking space PS is started using the corresponding parking space annotation code Ea including the fourth coded data Ec4.

[0385] According to Fig. 77B, at step S601, the processor 21 refers to the parking space annotation code Ea for a selected parking space PS to determine whether the fourth coded data Ec4 represents that the host vehicle V is located in a parking space PL.

[0386] If it is determined that the fourth coded data Ec4 represents that the host vehicle V is not located in a parking space PL (NO at step S601), the training routine proceeds to step S602.

[0387] Otherwise, if it is determined that the fourth coded data Ec4 represents that the host vehicle V is located in a parking space PL (YES at step S601), the processor 21 determines at step S603 to train the parameters of the machine learning model LM for the selected parking space PS in a situation where the host vehicle V is located in a parking space PL.

[0388] At step S602, the processor 21 refers to the parking space annotation code Ea for a selected parking space PS to determine whether the fourth coded data Ec4 represents that the host vehicle V is located at an unidentified place.

[0389] If it is determined that the fourth coded data Ec4 represents that the host vehicle V is located at an unidentified place (YES at step S602), the processor 21 determines at step S604 to train the parameters of the machine learning model LM for the selected parking space PS in a situation where the host vehicle V is located at an unidentified place.

[0390] Otherwise, if it is determined that the fourth coded data Ec4 represents that the host vehicle V is not located in an unidentified place (NO at step S602), the processor 21 determines at step S605 to train the parameters of the machine learning model LM for the selected parking space PS in a situation where the host vehicle V is outside a parking space PL.

[0391] Fig. Figure 78 illustrates an example of the fourth coded data Ec4 representing that the host vehicle V is located at an unidentified location or that the host vehicle V is outside a parking space PL. Specifically, the coded values ​​Ecv of all cells in the target image are set to 0. In contrast, Fig. 79 shows an example of the fourth coded data Ec4, which represents that the host vehicle V is located in a parking space PL. Specifically, the coded values ​​Ecv of all cells in the target image are set to 1.

[0392] Fig. Figure 80 schematically illustrates the parking space annotation code Ea for each parking space PS included in a target image GS, which additionally contains the Fig. 78 or Fig. 79, which indicates whether there is a parking area around the own vehicle V. For the sake of simplicity, the illustration of the second and third coded data Ec2 and Ec3 in Fig. 80 omitted. The Fig. The first coded data Ec1 illustrated in Figure 80 can be Fig. 68, which includes the first coded data elements Ec1 provided for the different orientations of each parking space PS.

[0393] Fig. 81 is a flowchart schematically illustrating a parking space detection routine, ie, a parking space derivation routine, executed by the processor 21 according to the sixth embodiment.

[0394] According to Fig. 81, ie, when starting the parking space detection routine, the processor 21 derives several parking space candidates from the input image using the machine learning model LM at step S700.

[0395] Next, at step S701, the processor 21 determines whether there is a parking space environment around the host vehicle V, that is, the host vehicle V is located in a parking space PL.

[0396] When it is determined that there is a parking space environment around the host vehicle V, that is, the host vehicle V is located in a parking space PL (YES at step S701), the processor 21 uses all the parking space candidates as the parking spaces PS at step S702.

[0397] Otherwise, if it is determined that there is no parking space environment around the host vehicle V, ie, the host vehicle V is not located in a parking space PL (NO at step S701), the processor 21 discards all parking space candidates at step S703. Seventh embodiment

[0398] Hereinafter, a parking space detection device 2200 according to the seventh embodiment of the present disclosure will be described with reference to Fig. 82 and Fig. 83 described.

[0399] The seventh embodiment describes a fifth encoding of wheel stopper information regarding the wheel stoppers B6 as a part of the attribute information.

[0400] Fig. Figure 82 illustrates an example of fifth coded data Ec5 as the result of the fifth coding. Specifically, the training processing assigns a coded data value Ecv 0 or 1 regarding the existence or non-existence of a wheel stopper B6 in each parking space PS included in a target image GS shown in Fig. 32. In particular, as in Fig. 33, Fig. 34, Fig. 38 and Fig. 82, the training processing assigns (i) the coded data value Ecv “1” to the middle cell of each selected parking space PS in which a wheel stopper B6 exists, and (ii) the coded data value Ecv “0” to each of the other cells in the target image GS.

[0401] Fig. 83 schematically illustrates the parking space annotation code Ea for each parking space PS contained in a target image GS, which additionally contains the Fig. 82 illustrated fifth coded data Ec5. For the sake of simplicity, the illustration of the second to fourth coded data Ec2 to Ec4 in Fig. 83 omitted. The Fig. The first coded data Ecv1 illustrated in Figure 83 can be Fig. 68, which includes the first coded data elements Ec1 provided for the different orientations of each parking space PS. Eighth embodiment

[0402] Hereinafter, a parking space detection device 2200 according to the eighth embodiment of the present disclosure will be described with reference to Fig. 84 and Fig. 85 described.

[0403] The eighth embodiment describes a sixth encoding of additional attribute information that differs from the parking segment information regarding the parking segments B5 and the wheel stop information regarding the wheel stoppers B6. The additional attribute information may include, for example, the existence or non-existence of painted objects in each parking space PS, such as a painted marking indicating side-by-side parking, diagonal side-by-side parking, parallel parking, an international wheelchair symbol, or an electric vehicle marking.

[0404] Fig. Figure 84 illustrates an example of sixth coded data Ec6 as the result of the sixth coding. Specifically, the training processing assigns a coded data value Ecv 0 or 1 regarding the existence or non-existence of the additional attribute information in each parking space PS included in a target image GS stored in Fig. 32. In particular, as in Fig. 33, Fig. 34, Fig. 38 and Fig. 82, the training processing assigns (i) the coded data value Ecv “1” to the middle cell of each selected parking space PS in which the attribute information exists, and (ii) the coded data value Ecv “0” to each of the other cells in the target image GS.

[0405] Fig. 85 schematically illustrates the parking space annotation code Ea for each parking space PS contained in a target image GS, which additionally contains the Fig. 84 illustrated sixth coded data Ec6. For the sake of simplicity, the illustration of the second to fifth coded data Ec2 to Ec5 in Fig. 85 omitted. The Fig. The first coded data Ecv1 illustrated in Figure 85 can be Fig. 68, which includes the first coded data elements Ec1 provided for the different orientations of each parking space PS. Ninth embodiment

[0406] Hereinafter, a parking space detection device 2200 according to the ninth embodiment of the present disclosure will be described with reference to Fig. 86 to Fig. 118 described.

[0407] The processor 21 of the ninth embodiment is configured to determine, based on a detection result or a recognition result of parking spaces PS, whether there is a parking space environment around the host vehicle V, that is, whether the host vehicle V is located in a parking space PL. The result of the parking space environment determination can be used to calculate a travel route of the host vehicle V to a target parking space PS during parking assistance or automatic parking. Alternatively, the parking space environment determination may be performed in response to receiving a training image, that is, a target image. In other words, the processor 21 may perform the parking space environment determination when generating the parking space annotation information.

[0408] The following schematically describes the parking area environment determination with reference to Fig. 86 and Fig. 87 described.

[0409] First, at step S801, the processor 21 obtains Fig. 86 a parking space train PSs. Fig. 87 illustrates an example of the parking space train PSs.

[0410] Next, at step S802, the processor 21 generates a parking environment determination group. What the parking environment determination group is and how the processor 21 generates the parking environment determination group will be described later. Next, at step S803, based on the generated parking environment determination group, the processor 21 performs the parking environment determination regarding whether the host vehicle V is located in a parking space PL.

[0411] The following describes the operations in steps S801 to S803 sequentially in detail. Parking space acquisition

[0412] The parking space train PSs represents a train or set of parking spaces PS arranged in a predetermined direction and satisfying predetermined conditions, which will be described later. Typically, the parking space train PSs represents Fig. 87 illustrates an alignment of parking spaces PS divided by parking segments B5. Each parking segment B5 has an arbitrary shape. There may be one or more parked vehicles B7 in one or more corresponding parking spaces PS, or there may be no parked vehicles B7 in the parking spaces PS. The parking space train PSs may, as shown in Fig. 88, even if there are no parking segments B5. In particular, a virtual circumscribed rectangle around each parked vehicle B7 can be defined as a virtual parking space PSv. Each virtual parking space PSv has a pair of long sides corresponding to the overall longitudinal dimension of the corresponding parked vehicle V7 and a pair of short sides corresponding to the transverse dimension, i.e., width dimension, of the corresponding parked vehicle V7. The virtual parking spaces PSv are aligned such that their spaces SP are smaller than a predetermined length, so that the parking space train PSs can be defined.

[0413] The combination of an orientation of the parking spaces PS, each of which is surrounded by the corresponding parking segment B5, and an orientation of the virtual parking spaces PSv, each of which is not surrounded by any parking segment B5, can be, as in Fig. 89, define the parking space train PSs. The parking spaces PS, each of which has the corresponding parking segment B5, and the virtual parking spaces PSv, each of which does not have a parking segment B5, are collectively referred to as parking spaces PS, unless there is a technical contradiction or otherwise specified. Accordingly, the parking spaces in Fig. 90 and subsequent figures and described below include (i) parking spaces, each of which is surrounded by the corresponding parking segment B5, and (ii) the virtual parking spaces PSv, each of which is not surrounded by any parking segment B5.

[0414] As described above, the processor 21 acquires the parking space train PSs according to the parking segments B5 and / or the parked vehicles B7.

[0415] Fig. 90 illustrates a side-by-side parking space train PSs. Fig. 91 is a diagonal side-by-side parking space train PSs. Fig. 92 is a parallel parking space train PSs. Specifically, the processor 21 of the ninth embodiment is configured to identify a parking space train PSs for each of the parking types, ie, side-by-side parking, diagonal side-by-side parking, and parallel parking. In other words, a parking space train PSs is configured not to include parking space trains of multiple types.

[0416] The following is in accordance with Fig. 93 to Fig. 97 describes an example indicating how the processor 21 obtains the parking space train PSs.

[0417] First, the processor 21 selects one of the detected parking spaces PS that are aligned in an x-direction that is Fig. 93, as a reference object PSb. In particular, the processor 21 detects the orientation of the detected parking spaces PS to determine the x-direction corresponding to the orientation direction. A selected parking space from among the parking spaces PS that form a preliminary parking space train or a final parking space train is referred to as an initial selected object PS1. Fig. 93, the reference object PSb and the initially selected object PS1 are the same parking space.

[0418] Next, the processor 21 executes a search task for searching for a nearest parking space PS to the reference object PSb in a search direction Dt corresponding to a first direction, which is a positive side of the x-axis direction, thereby finding the nearest parking space PS as a nearest object PSn. The positive side of the x-direction is the rightward direction in Fig. 93.

[0419] The search task then determines, based on the positional relationship between the reference object PSb and the nearest object PSn, whether the reference object PSb and the nearest object PSn should form a parking space train PSs (see Fig. 94). Specifically, the processor 21 determines that the reference object PSb and the nearest object PSn have the positional relationship that should form a parking space train PSs when it is determined that the following first to third continuity conditions are satisfied: (I) The first continuity condition is that the interval SP between the reference object PSb and the nearest object PSn in the x-axis direction is less than or equal to a predetermined value, for example, 2 m. (II) The second continuity condition is that a positional deviation between the reference object PSb and the nearest object PSn in the y-axis direction perpendicular to the x-axis direction is less than or equal to a predetermined value of, for example, 1.8 m. (III) The third continuity condition is that an angle Δθ formed between a center line LCb of the reference object PSb and a center line LCn of the nearest object PSn is within a predetermined angle of, for example, 30°.

[0420] In Fig. 94, the positional deviation between the reference object PSb and the nearest object PSn in the y-axis direction represents an offset amount between the reference object PSb and the nearest object PSn in the y-axis direction. In other words, when each of the reference object PSb and the nearest object PSn has opposite first and second ends in the y-axis direction, the positional deviation between the reference object PSb and the nearest object PSn in the y-axis direction represents a positional difference between the first end of the reference object PSb and the first end of the nearest object PSn in the y-axis direction. The center line LCb of the reference object PSb is a virtual line that passes through the center of the reference object PSb and is parallel to the long side thereof.The center line LCn of the nearest object PSn is a virtual line passing through the center of the nearest object PSn and parallel to its long side.

[0421] When it is determined that the first to third continuity conditions between the reference object PSb and the nearest object PSn are satisfied, the search task identifies that the reference object PSb and the nearest object PSn form a preliminary parking space train PSs, and sets the nearest object PSn as a new reference object PSb and sets the previous reference object PSb as the initial selected object PS1. This results in the reference object PSb being shifted from the initial selected object PSb to the nearest object in the search direction Dt. Fig. 95 schematically illustrates the parking spaces PS in which the location of the reference object PSb in the search direction Dt in comparison to the parking spaces PS which are in Fig. 94 are illustrated, was postponed.

[0422] Then, the search task retrieves a nearest parking space PS to the reference object PSb in the retrieval direction Dt, thereby finding the nearest parking space PS as a nearest object PSn. Thereafter, based on the positional relationship between the reference object PSb and the nearest object PSn, the search task determines whether the first to third continuity conditions between the reference object PSb and the nearest object PSn are satisfied. If it is determined that the first to third continuity conditions between the reference object PSb and the nearest object PSn are satisfied, the processor 21 adds the nearest object PSn to the provisional parking space train PSs including the initially selected object PS1, thereby identifying that the initially selected object PS1, the reference object PSb, and the nearest object PSn constitute the provisional parking space train PSs.

[0423] As described above, the processor 21 of the ninth embodiment is configured to search for a plurality of parking spaces PS that should form a parking space train PSs in the search direction Dt corresponding to the positive side of the x-axis direction, using the initial selected object PS1 as the starting point.

[0424] If it is determined that no nearest object PSn is found or at least one of the first to third continuity conditions is not satisfied, the processor 21 terminates the above search task in the search direction Dt corresponding to the positive side of the x-axis direction, using the initially selected object PS1 as the starting point. Then, the processor 21 returns the reference object PSb to the initially selected object PS1, inverts the search direction Dt so that the inverted search direction Dt corresponds to a second first direction, which is a negative side of the x-axis direction opposite to the positive side of the x-axis direction, and executes, as shown in Fig. 96, the search task for searching for a nearest parking space PS to the reference object PSb in the search direction Dt, whereby the nearest parking space PS is found as a nearest object PSn. The negative side of the x-direction is the direction to the left in Fig. 96.

[0425] Thereafter, when it is determined that no nearest object PSn is found or at least one of the first to third continuity conditions in the search task in the search direction Dt is not satisfied, the processor 21 terminates the above search task in the search direction Dt, thereby terminating the acquisition of the provisional parking space train PSs including the initial selected object PS1.

[0426] Next, the processor 21 selects a parking space PS that does not belong to the one or more identified parking space trains PSs and is not set to the reference object PSb, ie, the initial selected object PS1, and starts the search task in the search direction Dt corresponding to the positive side of the x-axis direction, using the reference object PSb, ie, the initial selected object PS1, as the starting point.

[0427] The processor 21, after obtaining a preliminary parking space train PSs, determines whether the number of parking spaces PS included in the identified preliminary parking space train PSs is greater than or equal to a predetermined threshold number.

[0428] Then, the processor 21 completes the identified preliminary parking space train PSs as a final parking space train PSs if it is determined that the number of parking spaces PSs included in the identified preliminary parking space train PSs is greater than or equal to the predetermined threshold number. Otherwise, if it is determined that the number of parking spaces PSs included in the identified preliminary parking space train PSs is less than the predetermined threshold number, the processor 21 discards the identified preliminary parking space train PSs without completing it.

[0429] Specifically, the processor 21 determines that the above predetermined conditions are met when it is determined that the first to third continuity conditions are met and the number condition indicating the number of parking spaces PS included in the parking space train PSs is greater than or equal to the predetermined threshold number. The threshold number may be set to 5, for example, but may be freely set to Q, which is a positive integer greater than or equal to 3. The threshold number may be variably set depending on one of the parking types, namely, side-by-side parking, diagonal side-by-side parking, or parallel parking. For example, Q is set to 5 for side-by-side parking, 4 for diagonal side-by-side parking, or 3 for parallel parking. The value of Q is also referred to as a parking space train acquisition threshold.

[0430] As described above, the processor 21 of the ninth embodiment is configured to select one of the detected parking spaces PS as a reference object PSb, and retrieves parking spaces PS in both the first direction and the second direction opposite to the first direction from the reference object PSb. This configuration enables the processor 21 to identify a parking space train PSs including parking spaces PS whose number is greater than or equal to the threshold number; the parking spaces PS satisfy the first to third continuity conditions. This therefore enables the processor 21 included in the host vehicle V traveling in a corridor of the parking space PL to reliably identify one or more parking space trains PSs, thereby determining with higher accuracy whether the host vehicle V is located in a parking space PL according to the identified one or more parking space trains PSs.

[0431] Next, an example of a parking space acquisition routine corresponding to the operation at step S801 shown in Fig. 86 with reference to the Fig. 98A and Fig. 98B is illustrated.

[0432] According to Fig. 98A and Fig. 98B, the processor 21 sequentially executes the operations of steps S901 to S905 of the parking space acquisition routine.

[0433] At step S901, the processor 21 obtains the detected result of the parking spaces PS from an input image or a training image. Next, at step S902, the processor 21 selects one of the detected parking spaces PS as the initial selected object PS1 from the detected parking spaces PS. After the operation at step S902, at step S903, the processor 21 sets the initial selected object PS1 as the reference object PSb. Next, at step S904, the processor 21 searches for a parking space PS in the first direction from the reference object PSb to accordingly select a nearest object PSn to the reference object PSb as the search result.

[0434] After the operation in step S904, the processor 21 determines in step S905 whether the positional relationship between the reference object PSb and the nearest object PSn satisfies the first to third continuity conditions. The determination in step S905 is negative if the positional relationship between the reference object PSb and the nearest object PSn does not satisfy at least one of the first to third continuity conditions (NO in step S905). In addition, the determination in step S905 is negative if the processor 21 does not find a parking space PS in the first direction from the reference object PSb (NO in step S905). That is, the determination in step S905 is positive (YES in step S905) if the processor 21 finds a nearest object PSn and the positional relationship between the reference object PSb and the nearest object PSn satisfies the first to third continuity conditions.

[0435] In response to the affirmative determination at step S905, the parking space acquisition routine proceeds to step S906.

[0436] At step S906, the processor 21 adds the nearest object PSn to a provisional parking space train PSs including the reference object PSb, and then determines at step S907 whether at least one unselected parking space PS exists in the search direction Dt. If it is determined that at least one unselected parking space PS exists in the search direction Dt (YES at step S907), the processor 21 shifts the selection of the reference object from the initially selected object PS1 to the nearest object PSn as the reference object PSb. Specifically, the processor 21 resets the nearest object PSn as the reference object PSb and returns the previous reference object PSb to the initially selected object PS1 at step S908. Otherwise, if it is determined that there is no unselected parking space PS in the search direction Dt (NO at step S907), the parking space acquisition routine proceeds to step S909.In addition, when the determination at step S905 is negative, the parking space acquisition routine proceeds to step S909.

[0437] At step S909, the processor 21 determines whether the search direction Dt is currently set to the second direction, that is, the search direction Dt has already been inverted from the first direction to the second direction. If it is determined that the search direction Dt is not currently set to the second direction, that is, the search direction Dt has not yet been inverted from the first direction to the second direction (NO at step S909), the parking space acquisition routine proceeds to step S910. At step S910, the processor 21 inverts the search direction Dt from the first direction to the second direction and thereafter returns to the operation at step S903. Then, the processor 21 executes the operations at steps S903 to S909 with the search direction Dt set to the second direction until the determination at step S909 is affirmative.

[0438] If the search task has been completed in each of the first and second directions, the determination at step S909 is affirmative (YES at step S909). Then, at step S911, the processor 21 determines whether the number of parking spaces PS, that is, the number of continuous parking spaces PS included in the provisional parking space train PSs, is greater than or equal to the predetermined threshold number.

[0439] If it is determined that the number of parking spaces PS, that is, the number of continuous parking spaces PS included in the provisional parking space train PSs, is greater than or equal to the predetermined threshold number (YES at step S911), the processor 21 finally acquires the provisional parking space train PSs as a final parking space train PSs at step S912. Otherwise, if it is determined that the number of parking spaces PS, that is, the number of continuous parking spaces PS included in the provisional parking space train PSs, is less than the predetermined threshold number (NO at step S911), the processor 21 discards the provisional parking space train PSs as a final parking space train PSs at step S913.

[0440] After completion of the operation at step S912 or S913, the parking space train acquisition routine proceeds to step S914. At step S914, the processor 21 determines whether there is at least one parking space PS that does not belong to the one or more final parking space trains PSs finally identified at step S912 and is not set to the reference object PSb. If it is determined that there is at least one parking space PS that does not belong to the one or more final parking space trains PSs finally identified at step S912 and is not set to the reference object PSb (YES at step S914), the processor 21 returns to the operation at step S902. This allows the processor 21 to perform the parking space train acquisition task at steps S902 to S914 in each of the first and second directions until the determination at step S914 is negative.Otherwise, if it is determined that there are no parking spaces PS that do not belong to the one or more final parking space trains PSs finally identified at step S912 and is not set to the reference object PSb (NO at step S914), the processor 21 ends the parking space train acquisition routine, which corresponds to the operation at step S801. Creating a park region

[0441] Next, the operation of creating at least one parking space environment determination group at step S802 of Fig. 86 with reference to the Fig. 99 to Fig. 103. Hereinafter, at least one parking environment determination group is referred to as at least one parking region PSg. The at least one parking region PSg is at least one virtual region which, as in Fig. 99, includes at least one parking space train PSs and a corridor PSr of the parking space PL. In particular, the processor 21 typically acquires, while the host vehicle V is traveling in the corridor PSr or will be traveling in the corridor PSr, the at least one parking region PSg formed by the corridor PSr and the parking space trains PSs located on both sides of the corridor PSr. That is, the processor 21 of the ninth embodiment is programmed to acquire, based on the parking space trains PSs identified at step S801, the at least one parking region PSg including (i) the parking space trains PSs and (ii) a region drivable by the host vehicle V and facing the parking space trains PSs.The processor 21 can acquire the at least one parking region PSg according to, as parking region acquisition conditions, the number of parking space trains PSs and / or the existence reliability of each parking space train PSs and / or the accuracy reliability. Then, the processor 21 of the ninth embodiment is programmed to determine, based on the generated at least one parking region PSg, whether the host vehicle V is located in a parking space PL. Specifically, the processor 21 of the ninth embodiment is programmed to determine whether the host vehicle V is located in a parking space PL or outside the parking space PL according to the positional relationship between the host vehicle V and the at least one parking region PSg, such as information about whether the host vehicle V has entered the at least one parking region PSg.

[0442] The processor 21 of the ninth embodiment may be programmed to expand each parking space train PSs and the at least one parking region PSg while the host vehicle V is traveling.

[0443] Fig. 99 and Fig. 100 illustrate a first example showing how the processor 21 extends a typical parking region PSg formed by the corridor PSr and the parking space trains PSs located on both sides of the corridor PSr. Fig. 99 illustrates a situation in which the own vehicle V approaches the parking region PSg, and Fig. 100 illustrates a next situation in which the own vehicle V approaches the parking region PSg closer than in the Fig. 99 illustrated situation. The processor 21 according to the Fig. 99 and Fig. The first example illustrated in Figure 100 is programmed to sequentially add to each of the parking space trains PSs forming the parking region PSg, newly detected parking spaces PS that are oriented in the same direction as the orientation of the parking spaces PS included in each of the parking space trains PSs, as the host vehicle V approaches the parking region PSg.

[0444] Fig. 101 to Fig. 103 illustrate a second example showing how the processor 21 extends a typical parking region PSg formed by the corridor PSr and the parking space trains PSs located on both sides of the corridor PSr. Fig. 101 illustrates a situation in which the owner vehicle V is driving to partially enter the parking region PSg, and Fig. 102 illustrates a next situation in which the owner vehicle V is driving to fully enter the parking region PSg. Fig. Figure 103 illustrates another situation in which the owner vehicle V is driving to enter deeper into the parking region PSg.

[0445] The processor 21 according to the Fig. 101 to Fig. The second example illustrated in Figure 103 is programmed to extend the parking region PSg, which is in the Fig. 101 while the host vehicle V is driving, in the following procedure. The parking region PSg, which is shown in Fig. 101, is formed by the corridor PSr and the parking space trains PSs, each of which has five parking spaces PS located on both sides of the corridor PSr.

[0446] The processor 21 is programmed to create a copy of the acquired parking region PSg formed by the corridor PSr and the parking space trains PSs located on both sides of the corridor PSr, to continuously connect the copied parking region PSg with the acquired parking region PSg in the traveling direction of the host vehicle V. Specifically, the processor 21 is programmed to continuously connect the copied parking region PSg, which has the same size as the size of the acquired parking region PSg, with the acquired parking region PSg in the traveling direction of the host vehicle V, thereby expanding the parking region PSg twice in the traveling direction.The processor 21 may be programmed to continuously connect a plurality of copied parking regions PSg, each of which has the same size as the acquired parking region PSg, to the acquired parking region PSg in the traveling direction of the host vehicle V, thereby expanding the parking region PSg K times in the traveling direction; K is a positive integer equal to or greater than two. In response to determining that the parking space trains PSs located on both sides of the corridor each consist of ten or more parking spaces PS, the processor 21 may be programmed to complete the virtually expanded parking region PSg.

[0447] Fig. 102 illustrates that the parking region PSg is extended twice in the direction of travel. Fig. 102 illustrates that the parking space trains PSs located on both sides of the corridor each consist of only eight parking spaces PS, so that there are two additional parking spaces PS (see dashed double-dotted lines) in each extended parking space train PSs included in the extended parking region PSg. Therefore, subtracting the two additional parking spaces PS from each extended parking space train PSs included in the extended parking region PSg, thereby correcting the size of each extended parking space train PSs and the size of the extended parking region PSg, enables the corrected parking region PSg including the corrected parking space trains PSs to be created.

[0448] As described above, the processor 21 of the ninth embodiment is configured to, when there are additional parking spaces PS or an additional parking space train PSs located adjacent to the acquired parking region PSg in a predetermined direction, expand the acquired parking region PSg in the predetermined direction so that the expanded parking region PSg includes the additional parking spaces PS or the additional parking space train PSs located adjacent to the acquired parking region PSg. This configuration makes it possible to expand the acquired parking region PSg according to additionally detected parking spaces PS or parking space train PSs while the host vehicle V travels to the acquired parking region PSg. This therefore enables proper determination of whether the host vehicle V is in or out of a parking space PL. Determine whether the vehicle is in the parking lot

[0449] Next, the following describes the operation for determining whether the host vehicle V is located in a parking space PL at step S803 of Fig. 86 with reference to the Fig. 104 to Fig. 113.

[0450] For example Fig. 104 to Fig. 106, reference symbol "ON" represents an "in-parking determination" that the host vehicle V is located in a parking space PL, and reference symbol "OFF" represents an "out-of-parking determination" that the host vehicle V is not located in a parking space PL. The processor 21 of the ninth embodiment is typically programmed to execute the in-parking determination that the host vehicle V is located in a parking space PL when it is determined that the host vehicle V has partially entered the parking region PSg, and the out-of-parking determination that the host vehicle V is not located in a parking space PL when it is determined that the host vehicle V has left or exited the parking region PSg.

[0451] Fig. 105 illustrates an exemplary situation in which at least a first and a second parking region PSg are aligned in the direction of travel of the host vehicle V, and the host vehicle V is scheduled to travel through the first and second parking regions PSg. In this exemplary situation, when the host vehicle V has reached a space between the first and second parking regions PSg, the host vehicle V has left or exited the first parking region PSg, so that the above-mentioned in-parking determination should have been temporarily switched to the out-of-parking determination. However, in this exemplary situation, there is a requirement not to switch the above-mentioned in-parking determination to the out-of-parking determination in the space between the first and second parking regions PSg.That is, in this exemplary situation, it is preferable to maintain the in-parking determination even if the host vehicle V has exited the first parking region PSg. This makes it possible to achieve a beneficial effect that is the same as in a situation shown in FIG. Fig. 106, in which the first parking region is extended in the traveling direction of the host vehicle V, in other words, the first and second parking regions are combined to form an extended parking region.

[0452] Fig. 107 illustrates a parking region PSg consisting of (i) a T-shaped corridor PSr, (ii) first and second parking space trains PSs arranged on both sides of a leg portion of the T-shaped corridor PSr so as to be parallel to the traveling direction of the host vehicle V, and (iii) a third parking space train PSs arranged to be separated from the first and second parking space trains PSs by a head portion of the T-shaped corridor PSr; the orientation of each of the first and second parking space trains PSs, which is parallel to the traveling direction of the host vehicle V, is orthogonal to the orientation of the third parking space train PSs, which is orthogonal to the traveling direction of the host vehicle V. At this time, the host vehicle V travels in the leg portion of the T-shaped corridor PSr between the first and second parking space trains PSs.

[0453] In this exemplary situation, which Fig. 107, the processor 21 may be programmed to perform the in-parking determination while the host vehicle V is traveling in the T-shaped corridor PSr. Specifically, it is assumed that the head portion of the T-shaped corridor PSr is divided into a first region R1, a second region R2, and a third region R3. The first region R1 is located to intersect with the traveling course of the host vehicle V. The second region R2 is located on the left side of the first region R1 and indicates a left-turn destination. The third region R3 is located on the right side of the first region R1 and indicates a right-turn destination.Since the first region R1 is not a high-reliability region enclosed from both sides by the parking space trains PSs, it is preferable to maintain the in-parking determination without switching the in-parking determination to the out-parking determination while the host vehicle V is traveling in the first region R1.

[0454] Fig. 108 illustrates a parking region PSg consisting of a cross-shaped corridor PSr and four parking space trains PSs arranged such that each section of the cross-shaped corridor PSr is enclosed by a corresponding pair of parking space trains PSs.

[0455] Fig. 109 illustrates a parking region PSg consisting of (i) a T-shaped corridor PSr, (ii) first and second parking space trains PSs arranged on both sides of a leg portion of the T-shaped corridor PSr so as to be orthogonal to the traveling direction of the host vehicle V, and (iii) a third parking space train PSs arranged to be separated from the first and second parking space trains PSs by a head portion of the T-shaped corridor PSr; the orientation of each of the first and second parking space trains PSs, which is orthogonal to the traveling direction of the host vehicle V, is orthogonal to the orientation of the third parking space train PSs, which is parallel to the traveling direction of the host vehicle V. At this time, the own vehicle V can travel straight through the head section of the T-shaped corridor PSr and turn left from the head section of the T-shaped corridor PSr to the leg section of the T-shaped corridor PSr.

[0456] It is preferable to maintain the in-parking destination without switching the in-parking destination to the out-parking destination while the host vehicle V is in each of the cross-shaped corridors PSr formed in Fig. 108, and the T-shaped corridors PSr, which are shown in Fig. 109 is illustrated.

[0457] Fig. 110 illustrates how a park region PSg, which is located in Fig. 7, a parking region PSg consisting of (i) a T-shaped corridor PSr, (ii) first and second parking space trains PSs arranged on both sides of a leg portion of the T-shaped corridor PSr, and (iii) a third parking space train PSs arranged to be separated from the first and second parking space trains PSs by a head portion of the T-shaped corridor PSr; the orientation of each of the first and second parking space trains PSs is orthogonal to the orientation of the third parking space train PSs.

[0458] In this situation, which Fig. 110, the processor 21 may be programmed to change one of the in-parking determination and the out-of-parking determination to the other depending on the direction of travel of the host vehicle V toward the parking region PSg. When the host vehicle V is about to enter the leg section of the T-shaped corridor PSr of the parking region PSg (see dashed line in Fig. 110), the parking space train PSs, which consists of several parking spaces PS, is located in front of the host vehicle V while intersecting with the direction of travel of the host vehicle V, i.e., perpendicular to the direction of travel of the host vehicle V. Such a situation may occur when the host vehicle V is waiting for a traffic light on a road Rd with three or more lanes in each direction. For this reason, compared to a first case where the host vehicle V enters from the left side Fig. 110 has entered the head section of the T-shaped corridor PSr, a second case in which the own vehicle V enters from the upper side or the lower side Fig. 110 has entered the head section of the T-shaped corridor PSr (see solid line in Fig. 110), a higher degree of reliability is shown that the owner's vehicle V is located in the parking space PL.

[0459] For this reason, the processor 21 may be programmed to, as in Fig. 110, (i) a first entry / exit determination line PSL located at the upper end of the head portion of the T-shaped corridor PSr, which represents the boundary of the parking region PSg at the upper side of the parking region PSg, and (ii) a second entry / exit determination line PSL located at the lower end of the head portion of the T-shaped corridor PSr, which represents the boundary of the parking region PSg at the lower side of the parking region PSg. Then, the processor 21 may be programmed to execute the in-parking determination, i.e., to make an "ON" determination that the host vehicle V is in a parking space PL, when it is determined that the host vehicle V has passed the first or second entry / exit determination line PSL to enter the parking region PSg. In contrast, the processor 21 may be programmed not to execute the in-parking determination, i.e.,not to execute an "ON" determination when it is determined that the own vehicle V has not passed the first or second entry / exit determination line PSL to enter the parking region PSg (see the dashed line in ). Fig. 110).

[0460] That is, the processor 21 of the ninth embodiment is configured to execute the in-parking determination when it is determined that the host vehicle V has entered the parking region PSg from a specified direction, that is, from the upper direction or the lower direction, and is configured not to execute the in-parking determination when it is determined that the host vehicle V has entered the parking region PSg from a direction different from the specified direction. Therefore, this makes it possible to execute the in-parking determination with higher reliability.

[0461] Fig. 111 illustrates a situation in which a first parking region PSg1, a second parking region PSg2, and a third parking region PSg3 are arranged in this order to form a substantially L-shape as a whole, and the host vehicle V is about to pass through the first, second, and third parking regions PSg1, PSg2, and PSg3 in this order along an L-shaped planned travel route TP. The first parking region PSg1 includes a first-region boundary PSL1 located at an entrance thereof, and a second-region boundary PSL2 represents the boundary between an exit of the first parking region PSg1 and an entrance of the second parking region PSg2. A third-region boundary PSL3 represents the boundary between an exit of the second parking region PSg2 and an entrance of the third parking region PSg3. Since the first, second, and third parking regions PSg1, PSg2, and PSg3 are continuously arranged, the first-region boundary PSL1 serves as an entrance / exit destination line PSL.The processor 21 may therefore be configured to maintain the in-parking determination even if the host vehicle V passes the second region boundary PSL2 or the third region boundary PSL3.

[0462] Fig. 112 illustrates a situation in which the host vehicle V is scheduled to enter a T-shaped corridor PSr in which the host vehicle V can drive straight ahead or turn left.

[0463] In Fig. 112, rectangular parking spaces PSa and PSb are located on the left side of a head portion of the T-shaped corridor PSr, such that they are separated from each other by a leg portion of the T-shaped corridor PSr, and their long sides are parallel to the straight-ahead direction of the host vehicle V. In addition, a side-by-side parking space train PSs is located on the right side of the head portion of the T-shaped corridor PSr. This therefore results in a first parking region PSg1 being formed by the head portion of the T-shaped corridor PSr and the parking spaces PSa and PSb located on the left side of the head portion of the T-shaped corridor PSr.

[0464] In addition, Fig. 112 a first and second parking space train PSs on both sides of a leg portion of the T-shaped corridor PSr, resulting in a second parking region PSg2 being formed by the leg portion of the T-shaped corridor PSr and the first and second parking space train PSs located on both sides of the leg portion of the T-shaped corridor PSr.

[0465] That is, as in Fig. 112, the first parking region PSg1 and the second parking region PSg2 partially overlap. The first parking region PSg1 has an entrance through which the host vehicle V can pass to enter the first parking region PSg1, so that the entrance of the first parking region PSg1 serves as a first-region boundary PSL1. The second parking region PSg2 has an exit through which the host vehicle V can pass to exit the second parking region PSg2, so that the exit of the second parking region PSg2 serves as a second-region boundary PSL2. That is, each of the first and second region boundaries PSL1 and PSL2 serves as an entry / exit determination line PSL. This allows the processor 21 to maintain the in-parking determination even when the host vehicle V passes boundaries between the overlapped portions of the first and second parking regions PSg1 and PSg2.

[0466] Fig. 113 illustrates a situation in which the host vehicle V, traveling straight, is about to pass through a first parking region PSg1 and a second parking region PSg2 that are continuously arranged in the traveling direction of the host vehicle V, with their parts overlapping. The first parking region PSg1 has an entrance through which the host vehicle V can pass to enter the first parking region PSg1, so that the entrance of the first parking region PSg1 serves as a first-region boundary PSL1, that is, an entry / exit determination line PSL. The first parking region PSg1 also has an exit, and the exit of the first parking region PSg1 serves as a second-region boundary PSL2. Since the second region boundary PSL2 is located in the second parking region PSg2, the exit of the first parking region PSg1, that is, the second region boundary PSL2 of the first parking region PSg1, does not serve as an entry / exit destination line PSL.This allows the processor 21 to maintain the on-parking determination even when the host vehicle V passes the second region boundary PSL2 of the first parking region PSg1. Accelerator pedal misoperation addressing routine

[0467] The processor 21 of the ninth embodiment may be configured to, when it is determined that the host vehicle V is located in a parking space PL, execute an accelerator pedal misoperation addressing routine including a drive power reduction control task of the host vehicle V and / or an accelerator pedal misoperation notification task to accordingly prevent sudden acceleration of the host vehicle V when it is determined that the host vehicle V is located in a parking space PL. Fig. 114 and Fig. 115 illustrate an example of the accelerator pedal misoperation addressing routine according to the ninth embodiment. The processor 21 may be programmed to Fig. 114 at every predetermined period of, for example, 100 ms, under a predetermined execution condition being met. The execution condition may include, for example, that the current shift position of the shift lever is set to the forward position or the reverse position.

[0468] According to Fig. 114, when starting the accelerator pedal misoperation addressing routine, the processor 21 determines at step S1001 whether the processor 21 is making the in-parking determination, that is, making an "ON" determination representing that the host vehicle V is located in a parking space PL. If it is determined that the processor 21 is making the in-parking determination such that the host vehicle V is located in a parking space PL (YES at step S1001), the accelerator pedal misoperation addressing routine proceeds to step S1002. Otherwise, if it is determined that the processor 21 is making the out-of-parking determination, that is, making an "ON" determination representing that the host vehicle V is not located in a parking space PL, the processor 21 temporarily terminates the accelerator pedal misoperation addressing routine while skipping the operations at steps S1002 and S1003.

[0469] At step S1002, the processor 21 determines whether an accelerator pedal misoperation has occurred. Specifically, the processor 21 determines whether all of the following first to fifth conditions are met, and determines that an accelerator pedal misoperation has occurred (YES at step S1002) if it is determined that all of the following first to fifth conditions are met. Conversely, the processor 21 determines that no accelerator pedal misoperation has occurred (NO at step S1002) if it is determined that at least one of the first to fifth conditions is not met.

[0470] The first condition is that the speed of the own vehicle V is less than a predetermined threshold speed.

[0471] The second condition is that an operation amount of the accelerator pedal of the own vehicle V is greater than or equal to a predetermined acceleration amount threshold.

[0472] The third condition is that an operation speed of the accelerator pedal of the own vehicle V is greater than or equal to a predetermined acceleration operation speed threshold.

[0473] The fourth condition is that no brake pedal operation is performed.

[0474] The fifth condition is that no turn signal switch 382 is operated.

[0475] If it is determined that an accelerator pedal misoperation has occurred (YES at step S1002), the accelerator pedal misoperation addressing routine proceeds to step S1003. At step S1003, the processor 21 executes a drive power reduction control subroutine included in Fig. 115, which will be described later. Otherwise, if it is determined that no accelerator pedal misoperation has occurred (NO at step S1002), the processor 21 temporarily terminates the accelerator pedal misoperation addressing routine while skipping the operation at step S1003.

[0476] The following describes the drive power reduction control subroutine at step S1003 with reference to Fig. 115.

[0477] When the drive power reduction control subroutine is started, the processor 21 executes (i) a drive power reduction control task, (ii) a strong notification task, i.e., a warning task to warn the one or more occupants in the host vehicle V of the occurrence of an accelerator pedal misoperation, and (iii) a timer task that starts counting time from an initial value of 0 in response to the execution of the drive power reduction control task and the warning task at step S1101. For example, the processor 21 instructs the drive ECU 911 to control the drive mechanism 912 to appropriately reduce the drive power generated thereby, which causes the system vehicle V to drive.

[0478] Next, at step S1102, the processor 21 determines whether the operation amount of the accelerator pedal of the host vehicle V is less than or equal to a predetermined reduction control termination threshold after executing the drive power reduction control task and the warning task at step S1101. If it is determined that the operation amount of the accelerator pedal of the host vehicle V was less than or equal to the predetermined reduction control termination threshold (YES at step...

Claims

[1] Method for learning a recognition rule of at least one parking space around a vehicle (V), the method comprising: Objectifying at least one parking space, which is at least partially included in a learning image, as at least one objectified parking space, which consists of (i) a plurality of corner points (BP) of the at least one parking space, (ii) a center point (BPC) of the at least one parking space and (iii) attribute information about the at least one parking space; Generating annotation data for the objectified at least one parking space; Starting from the learning image, learning a recognition rule of the at least one parking space based on the generated annotation data for the objectified at least one parking space; Receiving an input image around the vehicle captured by a camera (34) installed in the vehicle; and Determine whether there is at least one target parking space in the input image according to the learned detection rule. [2] Method for learning a recognition rule of at least one parking space around a vehicle (V), the method comprising: Objectifying at least one parking space, which is at least partially included in a learning image, as at least one objectified parking space, which consists of (i) a plurality of corner points (BP) of the at least one parking space, (ii) a center point (BPC) of the at least one parking space and (iii) attribute information about the at least one parking space; Generating annotation data for the objectified at least one parking space; Starting from the learning image, learning a recognition rule of the at least one parking space based on the generated annotation data for the objectified at least one parking space. [3] A method according to claim 1 or 2, wherein: the attribute information includes at least one of the following: Parking frame information relating to a parking frame (B5) of the at least one parking space; Wheel stop information about a wheel stop (B6) located in the at least one parking space; and additional attribute information relating to the at least one parking space, wherein the additional attribute information differs from the parking frame information and the wheel stop information. [4] A method according to any one of claims 1 to 3, wherein: at least one parking space includes several parking spaces, the procedure further includes: Calculating the number of one or more of the vertices of at least one of the parking spaces, wherein the one or more of the vertices of the at least one of the parking spaces is blocked by at least one object (B7) other than each of the parking spaces, wherein performing the learning does not learn the recognition rule of the at least one of the parking spaces based on the generated annotation data for the objectified at least one of the parking spaces when it is determined that the number of the one or more of the vertices of the at least one of the parking spaces is greater than or equal to a predetermined threshold number. [5] A method according to any one of claims 1 to 4, wherein: Executing learning includes: Determine whether one or two of the vertices are outside the training image; If it is determined that one or two of the vertices are outside the training image, moving one or two of the vertices into the training image; and Starting from the training image, learning the recognition rule of the at least one parking space based on the generated annotation data for the objectified at least one parking space after moving one or two of the vertices into the training image. [6] A method according to claim 5, wherein: the offsetting offsets the one or two of the vertices in the learning image in a predetermined offset direction, wherein the predetermined offset direction is selected to make a change in a shape of the at least one parking space before and after the offsetting smaller compared to another offset direction. [7] A method according to any one of claims 1 to 6, further comprising: Calculating, as a blocked amount, an amount of a part of the at least one parking space that is blocked by at least one object (B7) other than the at least one parking space, wherein performing the learning sets parking space empty information about the at least one parking space to an undefined state when it is determined that the calculated blocked amount of the portion of the at least one parking space is greater than or equal to a threshold amount. [8] A method according to claim 7, wherein: calculating the blocked amount calculates the blocked amount based on a viewing angle of the vehicle toward the at least one parking space. [9] A method according to any one of claims 1 to 8, wherein: performing the learning of typical orientations (anchor 0, anchor 1, anchor 2 and anchor 3) that defines at least one parking space and performs the learning for each of the typical orientations. [10] A method according to claim 9, wherein: performing the learning encodes each of the vertices and the center point of the at least one parking space for each of the typical orientations to obtain encoded data for each typical orientation, and performing the learning using the encoded data for each typical orientation. [11] Method according to one of claims 1 to 10, further comprising: Recognizing, based on the learning image, more than one parking space; Selecting a parking space from the detected parking spaces as a reference object (PSb); Searching, starting from the reference object, for at least one parking space train (PSs) in at least one of a predetermined first direction and a second direction opposite to the first direction, wherein the at least one parking space train consists of selected parking spaces included in the plurality of parking spaces and continuously aligned in at least one of the first direction or the second direction to find the at least one parking space train; and Performing, based on information regarding the at least one parking space train, a parking space environment determination regarding whether the vehicle is located in a parking space. [12] A method according to claim 11, wherein: Carrying out the parking environment determination includes: Obtaining, based on the at least one parking space train, a parking region (PSg) including the at least one parking space train and a corridor (PSr) arranged to face the parking space train, wherein the vehicle is drivable in the corridor; Determining that the vehicle is in a parking space when it is determined that the vehicle has entered the parking region; and Determine that the vehicle is not in a parking space when it is determined that the vehicle has exited the parking region. [13] A method according to any one of claims 1 to 10, further comprising: Recognizing, based on the learning image, more than one parking space; Obtaining, based on the plurality of parking spaces, at least one parking space train (PSs) consisting of selected parking spaces included in the plurality of parking spaces and continuously aligned in a predetermined direction; Obtaining, based on the at least one parking space train, a parking region (PSg) including the at least one parking space train and a corridor (PSr) arranged to face the parking space train, wherein the vehicle is drivable in the corridor; Expanding, when it is determined that there is at least one additional parking space or at least one additional parking space train located adjacent to the acquired parking region in the predetermined direction, the parking region in the predetermined direction; and Performing, based on information regarding the extended parking region, a parking environment determination regarding whether the vehicle is in a parking space. [14] A method according to claim 13, wherein: obtaining the at least one parking space train obtains a first and second parking space train as the at least one parking space train; and obtaining the parking region obtains the parking region configured such that the first and second parking space trains are arranged on both sides of the corridor. [15] A method according to claim 13 or 14, wherein: Carrying out the parking environment determination includes: Determining that the vehicle is in a parking space when it is determined that the vehicle has entered the extended parking region; and Determine that the vehicle is not in a parking space when it is determined that the vehicle has exited the extended parking region. [16] A method according to any one of claims 12 to 15, wherein: executing the parking space environment determination executes the parking space environment determination that the vehicle is in a parking space when it is determined that the vehicle entered the parking region from a specified direction, and does not execute the parking space environment determination when it is determined that the vehicle entered the parking region from a direction different from the specified direction. [17] A method according to any one of claims 11 to 16, further comprising: when, when it is determined that the vehicle is in a parking space, sudden acceleration due to an accelerator pedal misoperation is detected, executing an accelerator pedal misoperation addressing task including at least one of (i) reducing the driving power of the vehicle and (ii) notifying one or more occupants included in the vehicle of an occurrence of the accelerator pedal misoperation. [18] A method according to claim 17, wherein: executing the accelerator pedal misoperation addressing task determines whether the accelerator pedal misoperation has continued for a predetermined time after executing the accelerator pedal misoperation addressing task, and executing the accelerator pedal misoperation addressing task again if it is determined that the accelerator pedal misoperation has continued for the predetermined time. [19] A method according to claim 17 or 18, wherein: executing the accelerator pedal misoperation addressing task after the accelerator pedal misoperation is canceled, determining whether a new accelerator pedal misoperation exists, and executing the accelerator pedal misoperation addressing task again if it is determined that a new accelerator pedal misoperation exists after the accelerator pedal misoperation is canceled. [20] Device (2) for learning a recognition rule of at least one parking space around a vehicle (V), the device comprising: a storage device (22) that stores learning program instructions; and a processor (21) configured to execute the tutorial instructions to accordingly: to objectify at least one parking space, which is at least partially included in a learning image, as at least one objectified parking space, which consists of (i) a plurality of corner points (BP) of the at least one parking space, (ii) a center point (BPC) of the at least one parking space and (iii) attribute information about the at least one parking space; to generate annotation data for the objectified at least one parking space; Starting from the learning image, learning a recognition rule for the at least one parking space based on the generated annotation data for the objectified at least one parking space.

Images