Parking space learning method and equipment

Abstract

The invention relates to a parking space learning method and equipment. A processor of an apparatus for learning a recognition rule of at least one parking space around a vehicle is configured to execute learning program instructions to object the at least one parking space, which is at least partially included in a learning image, into at least one objectified parking space, respectively, the at least one objectified parking space comprises: (i) a plurality of angular points (BP) of the at least one parking space; (ii) a center point (BPC) of at least one parking space; and (iii) attribute information about the at least one parking space. The processor is configured to execute the learning program instructions to correspondingly generate annotation data for the objectified at least one parking space, and to perform learning of an identification rule for the at least one parking space according to the learning image based on the generated annotation data for the objectified at least one parking space.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based upon and claims the benefit of priority from Japanese patent application No. 2023-168680, filed on September 28, 2023, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to a parking space learning method and apparatus. Background Art

[0004] Research and development of driving assistance technology and / or autonomous driving technology has been actively carried out. Various types of proposals related to driving assistance technology and / or autonomous driving technology have been implemented, one of which is disclosed in internal patent publication WO 2019 / 202397.

[0005] Driving assistance technologies are intended to reduce the burden on the driver so that the driver can drive the vehicle comfortably and safely. For example, these driving assistance technologies include technologies related to following distance control, lane keeping assist control, lane change assist control, parking assist control, obstacle warning, collision avoidance assist control or other vehicle control technologies. These autonomous driving technologies are intended to enable the vehicle to travel automatically, that is, autonomously, without the need for the driver's driving operation. Various sensing devices such as camera devices and / or radar devices are installed in vehicles with such driving assistance technologies and / or driving assistance technologies. Various sensing devices are configured to detect the surrounding conditions around the vehicle. Autonomous control operations of autonomous vehicles in motion, such as autonomous steering, autonomous driving and / or autonomous braking, are performed based on map information of visual representations of surrounding conditions detected by sensing devices and / or areas around the current position of the autonomous vehicle, for example. In particular, high-accuracy map information including road data for each lane around the autonomous vehicle can be used to perform autonomous control operations of autonomous vehicles in motion, so that the safety and reliability of autonomous control operations of autonomous vehicles in motion can be improved. Summary of the invention

[0006] In view of the improvement in accuracy of object detection and / or object recognition by sensing devices and the improvement in user convenience, research and development of these autonomous driving technologies and driving assistance technologies have been accelerated recently. That is, further improvement of these autonomous driving technologies and driving assistance technologies enables advanced driving assistance vehicles and / or autonomous driving vehicles to be popularized earlier and more widely.

[0007] In view of the above situation, the present disclosure provides a device for learning a recognition rule of at least one parking space around a vehicle. The device includes: a memory device storing a learning program instruction; and a processor configured to execute the learning program instruction to objectify at least one parking space at least partially included in a learning image into at least one objectified parking space, the at least one objectified parking space including: (i) multiple corner points of the at least one parking space; (ii) a center point of the at least one parking space; and (iii) attribute information about the at least one parking space. The processor is configured to: execute the learning program instruction to generate annotation data for the objectified at least one parking space, and perform learning of the recognition rule of the at least one parking space according to the learning image based on the generated annotation data for the objectified at least one parking space.

[0008] The present disclosure provides a method for learning a recognition rule of at least one parking space around a vehicle, the method comprising:

[0009] (1) objectifying at least one parking space at least partially included in a learning image into at least one objectified parking space, wherein the at least one objectified parking space includes: (i) a plurality of corner points of the at least one parking space; (ii) a center point of the at least one parking space; and (iii) attribute information about the at least one parking space;

[0010] (2) generating annotation data for at least one objectified parking space;

[0011] (3) Based on the generated annotation data for the objectified at least one parking space, learning of a recognition rule for the at least one parking space is performed according to the learning image.

[0012] The present disclosure provides a processor program product that can be read by a processor installed in a vehicle. The processor program product includes learning program instructions that cause the processor to:

[0013] (1) objectifying at least one parking space at least partially included in a learning image into at least one objectified parking space, wherein the at least one objectified parking space includes: (i) a plurality of corner points of the at least one parking space; (ii) a center point of the at least one parking space; and (iii) attribute information about the at least one parking space;

[0014] (2) generating annotation data for at least one objectified parking space;

[0015] (3) Based on the generated annotation data for the objectified at least one parking space, learning of a recognition rule for the at least one parking space is performed according to the learning image.

[0016] Note that each parenthesized reference numeral assigned to a corresponding element in the present disclosure merely represents an example of a relationship between the corresponding element and a corresponding specific element described in the following embodiments described later, and thus the present disclosure is not limited to the parenthesized reference numerals. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Other aspects of the present disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 is a plan view schematically showing a vehicle in which a vehicle system includes a driving ECU serving as a control device according to a basic embodiment;

[0019] Figure 2 It is shown Figure 1 A schematic block diagram of the overall structure of the vehicle system shown in ;

[0020] Figure 3 is a schematic diagram showing the use of Figure 2 A plan view of one of the sonar sensors shown in FIG. 1 performing target object detection;

[0021] Figure 4 is a schematic diagram showing the use of Figure 2 A plan view of a pair of sonar sensors performing target object detection as shown in FIG.

[0022] Figure 5 is a schematic diagram showing the use of Figure 2 A plan view of each sonar sensor performing target object detection as shown in FIG.

[0023] Figure 6 is a schematic diagram showing the use of Figure 2 A plan view of a first sonar sensor performing parking space detection as shown in FIG.

[0024] Figure 7 is a schematic diagram showing the use of Figure 2 A plan view of a radar sensor performing target object detection as shown in FIG.

[0025] Figure 8 is a schematic diagram showing the use of Figure 2 A plan view of a lidar sensor performing target object detection as shown in FIG.

[0026] Fig. 9 It is schematically shown Figure 8 A perspective view of the configuration and operation of a lidar sensor as shown in;

[0027] Fig.10 is a schematic diagram showing the use of Figure 2FIG. 1 shows a camera device performing target object detection;

[0028] Fig.11 is a schematic diagram showing the use of Figure 2 FIG. 1 shows a camera device performing target object detection;

[0029] Fig.12 is a schematic diagram showing the use of Figure 2 A diagram showing a driver state monitor performing driver state detection;

[0030] Fig.13 is a schematic diagram showing the use of Figure 2 A diagram showing a driver state monitor performing driver state detection;

[0031] Fig.14 It is shown Figure 2 A diagram of a schematic configuration of a head-up display shown in FIG.

[0032] Fig.15 is a schematic diagram showing the following: Fig.14 The driver's perspective shown in FIG. 1 is viewing (i) the road surface in front of the own vehicle through the front windshield and (ii) a portion of the instrument panel;

[0033] Fig.16 It is indicated by Figure 2 A block diagram of a schematic functional configuration implemented by the driving ECU shown in FIG.

[0034] Fig.17 It is indicated by Fig.16 A block diagram of a schematic functional configuration implemented by the identification unit shown in ;

[0035] Fig.18 is a schematic diagram showing a first example of a target object to be identified by a basic implementation;

[0036] Fig.19 is a schematic diagram showing a second example of a target object to be identified by a basic implementation;

[0037] Fig. 20 is a schematic diagram showing a third example of a target object to be identified by a basic implementation;

[0038] Fig.21 is a schematic diagram showing a fourth example of a target object to be identified by a basic implementation;

[0039] Fig. 22 is a schematic diagram showing a fifth example of a target object to be identified by a basic implementation;

[0040] Fig.23is a schematic diagram showing a sixth example of a target object to be identified by a basic implementation;

[0041] Fig.24 is a schematic diagram showing a fifth example of a target object to be identified by a basic implementation;

[0042] Fig.25 is a schematic diagram showing a sixth example of a target object to be identified by a basic implementation;

[0043] Fig.26 is a schematic diagram showing a seventh example of a target object to be identified by a basic implementation;

[0044] Fig. 27 is a diagram schematically showing eight examples of traffic signs as target objects;

[0045] Fig.28 is a schematic diagram showing a first example of indicating how to use a recognition result of a target object;

[0046] Fig.29 is a schematic diagram showing a second example of indicating how to use the recognition result of the target object;

[0047] Fig.30 is a schematic block diagram of a parking space detection device according to a first embodiment;

[0048] Fig.31 It is shown by Fig.30 A schematic diagram of annotation information generated by a parking space annotation generator shown in ;

[0049] Fig.32 is an example of a learning image including the own vehicle that has entered a parking lot and a plurality of parking spaces arranged around the own vehicle;

[0050] Fig.33 Shown in large scale Fig.32 a view of a portion of a target image shown in ;

[0051] Fig.34 It is shown Fig.33 A diagram encoding the presence of parking space centers in the grid shown in ;

[0052] Fig.35 It is shown Fig.33 A view of the horizontal positions of the centers of parking spaces in the grid shown in FIG.

[0053] Fig.36 It is shown Fig.35 A diagram of the coding of the horizontal position of the center of the parking space shown in ;

[0054] Fig.37 It is shown Fig.33 A view of the vertical positions of the centers of parking spaces in the grid shown in FIG.

[0055] Fig.38 It is shown Fig.37 a first coded diagram of the vertical position of the center of the parking space shown in;

[0056] Fig.39 It is shown in Fig.32 A first coded map of the presence of each parking space center in the entire target image shown in ;

[0057] Fig.40 It is shown in Fig.32 A first encoded map of the horizontal position of the center of each parking space in the entire target image shown in FIG.

[0058] Fig.41 It is shown in Fig.32 a first coded map of the vertical position of the center of each parking space in the entire target image shown in FIG.

[0059] Fig.42 It is shown that the Figure 39 to Figure 41 A diagram of a parking space annotation code of the first encoded data of the encoding result shown in FIG.

[0060] Fig.43 is shown in large scale and consists of Fig.42 A view of a selected 3×3 grid consisting of 9 cells of the selected cell and 8 cells surrounding the selected cell;

[0061] Fig.44 is schematically shown for executing Fig.43 A diagram of a connection number table for a second encoding of a connection vector shown in ;

[0062] Fig.45 is a schematic diagram showing the use of Fig.44 The connection number table shown in Fig.43 A diagram showing the encoding result of the concatenated vector shown in ;

[0063] Fig.46 It is shown that the first coded data and the Fig.45 A diagram of a parking space annotation code of the second encoded data of the encoding result shown in FIG.

[0064] Fig.47 is a view showing the selected 3×3 grid in a large scale and showing information on segment lines as part of attribute information;

[0065] Fig.48 Is to show about Fig.47A view of a third encoding of the information of the left segmented line shown in ;

[0066] Fig.49 Is to show about Fig.47 A third encoded view of the information of the front segment line shown in ;

[0067] Fig.50 Is to show about Fig.32 A third coded diagram of information of the left segment line in the entire target image shown in FIG.

[0068] Fig.51 Is to show about Fig.32 A third coded diagram of information of the right segment line in the entire target image shown in FIG.

[0069] Fig.52 Is to show about Fig.32 A third coded diagram of information of the left segment line in the entire target image shown in FIG.

[0070] Fig.53 Is to show about Fig.32 A third encoded diagram of information of the front segmentation line in the entire target image shown in ;

[0071] Fig.54 The first coded data, the second coded data and the Figure 50 to Figure 53 A diagram of a parking space annotation code of the third encoded data of the encoding result shown in FIG.

[0072] Fig.55 is a flowchart schematically showing a position correction routine according to the first embodiment as an example of a task after detecting a parking space from an input image;

[0073] Fig.56 It is schematically shown Fig.55 A view of the bounding rectangle calculation task shown in ;

[0074] Fig.57 It is schematically shown Fig.55 A view of the edge line generation task and the edge line calculation task shown in ;

[0075] Fig.58 is a perspective view showing a situation where a portion of a parking space segment and / or at least one parking space corner point may be obscured by at least one parked vehicle;

[0076] Fig.59 is a flow chart schematically illustrating a portion of a training routine according to a second embodiment;

[0077] Fig.60is a view schematically showing an image indicating a situation where it is difficult to determine whether an empty target parking space obstructs the target parking space due to a parked vehicle occupying a parking space adjacent to the target parking space according to a third embodiment;

[0078] Fig.61 is a flowchart schematically illustrating preprocessing before training of parameters of a machine learning model according to a third embodiment;

[0079] Fig.62 is a plan view schematically showing a training routine according to the third embodiment in a case where a left-side parallel parking row and a right-side parallel parking row are arranged on both sides of a portion of a public parking passage in which the own vehicle according to the third embodiment travels, respectively;

[0080] Fig.63 is a view schematically showing an image of a case where parking space corner points of the same type indicating different parking spaces are located in a common cell of a target image according to a fourth embodiment;

[0081] Fig.64A and Fig.64B Each is a diagram schematically showing the definition of a typical orientation of a parking space according to a fourth embodiment;

[0082] Fig.65 is a flowchart schematically showing an encoding routine according to a fourth embodiment;

[0083] Fig.66 It is schematically shown Fig.65 A diagram of an example of an encoding routine shown in;

[0084] Fig.67 It is schematically shown Fig.65 A diagram of another example of an encoding routine shown in;

[0085] Fig.68 is a diagram showing a parking space annotation code including a first encoded data set additionally including fourth encoded data having an encoding result according to a fourth embodiment;

[0086] Fig.69 is an image schematically showing a case in which at least one of parking space corner points indicating a parking space may be located outside a target image and an offset task in this case according to a fifth embodiment;

[0087] Figure 70 to Figure 73 Each shows a corresponding example of an offset task for offsetting one parking space corner point located outside a target image among four parking space corner points according to the fifth embodiment;

[0088] Fig.74 and Fig.75shows a corresponding example of an offset task for offsetting two parking space corner points located outside a target image among four parking space corner points of a parking space according to a fifth embodiment;

[0089] Fig.76A is a plan view schematically showing determination of whether a parking lot environment exists around an own vehicle according to a sixth embodiment;

[0090] Fig.76B is an enlarged diagram schematically showing determination of whether a parking lot environment exists around an own vehicle according to a sixth embodiment;

[0091] Fig.77A is a flowchart schematically showing a parking environment annotation information generation routine according to a sixth embodiment;

[0092] Fig.77B is a flow chart schematically illustrating a portion of a training routine according to a sixth embodiment;

[0093] Fig.78 is a diagram showing an example of fourth coded data indicating that the own vehicle is located in an unmarked place or the own vehicle is located outside a parking lot according to the sixth embodiment;

[0094] Fig.79 is a diagram showing an example of fourth encoded data indicating that the own vehicle is located in a parking lot according to the sixth embodiment;

[0095] Fig.80 is a schematic diagram showing a method for each parking space included in the target image to additionally include Fig.78 or Fig.79 A diagram of a parking space annotation code of the fourth encoded data shown in ;

[0096] Fig.81 is a flowchart schematically showing a parking space detection routine (ie, parking space inference routine) executed by a processor according to a sixth embodiment;

[0097] Fig.82 is a diagram schematically showing an example of fifth encoded data as a result of fifth encoding according to the seventh embodiment;

[0098] Fig.83 is a schematic diagram showing a method for each parking space included in the target image to additionally include Fig.82 A diagram of a parking space annotation code of the fifth encoded data shown in ;

[0099] Fig.84 is a diagram schematically showing an example of fifth encoded data as a result of sixth encoding according to the eighth embodiment;

[0100] Fig.85 is a schematic diagram showing a method for each parking space included in the target image to additionally include Fig.84 A diagram of a parking space annotation code of the sixth coded data shown in ;

[0101] Fig.86 is a flowchart schematically showing a parking lot environment determination routine according to a ninth embodiment;

[0102] Fig.87 is a plan view schematically showing an example of a parking space array;

[0103] Fig.88 is a plan view schematically showing another example of a parking space array;

[0104] Fig.89 is a plan view schematically showing another example of a parking space array;

[0105] Fig.90 is a plan view schematically showing an example of a parallel parking space array;

[0106] Fig.91 is a plan view schematically showing an example of a diagonal parallel parking space array;

[0107] Fig.92 is a plan view schematically showing an example of a parallel parking space row;

[0108] Fig.93 It is used to describe the Fig.86 A plan view of a parking space array acquisition routine in a parking lot environment determination routine shown in FIG.

[0109] Fig.94 It is used to describe the Fig.86 A plan view of a parking space array acquisition routine in a parking lot environment determination routine shown in FIG.

[0110] Fig.95 It is used to describe the Fig.86 A plan view of a parking space array acquisition routine in a parking lot environment determination routine shown in FIG.

[0111] Fig.96 It is used to describe the Fig.86 A plan view of a parking space array acquisition routine in a parking lot environment determination routine shown in FIG.

[0112] Fig.97 It is used to describe the Fig.86 A plan view of a parking space array acquisition routine in a parking lot environment determination routine shown in FIG.

[0113] Fig.98Aand Fig.98B It is schematically shown Fig.86 A joint flow chart of the parking space list acquisition routine shown in ;

[0114] Fig.99 and Fig.100 are plan views each showing a first example of how the processor expands a typical parking area consisting of an aisle and rows of parking spaces located on both sides of the aisle;

[0115] Figure 101 to Figure 103 are plan views each showing a second example of how the processor expands a typical parking area consisting of an aisle and rows of parking spaces located on both sides of the aisle;

[0116] Figures 104 to 113 are shown in Fig.86 A plan view of the operation of determining whether the own vehicle is located in the parking lot in step S803;

[0117] Fig.114 is a flowchart schematically showing an accelerator malfunction solution routine according to a ninth embodiment;

[0118] Fig.115 is schematically shown in Fig.114 A flowchart of the drive capability reduction control subroutine shown in step S1003 of FIG.

[0119] Fig.116 is a flowchart schematically showing a routine for assisting the travel of the own vehicle to a selected target parking space during parking assistance according to a ninth embodiment;

[0120] Fig.117 is schematically shown in Fig.116 A plan view of the parking space search route shown in step S1203; and

[0121] Fig.118 is schematically shown in Fig.116 The plan view of the parking entrance route generation operation shown in step S1205. DETAILED DESCRIPTION

[0122] The following describes the basic implementation of the present disclosure and the first to ninth implementations with reference to the accompanying drawings. The configurations, functions and / or examples described in the following implementations can be freely modified. In the implementations and their modifications, the same reference numerals are assigned to equivalent or identical components between the various implementations, between each implementation and its modifications, and between the various modifications. Among equivalent or identical components, the description of the previous component can be directly used to describe the latter component or subsequent multiple components, as long as (i) no technical contradiction occurs and / or there is no additional description.

[0123] Overall configuration of the vehicle

[0124] Figure 1 A vehicle V on which a system is installed to which the present disclosure is applied according to a basic embodiment is shown. The vehicle V on which the system is installed is a four-wheeled motor vehicle. The vehicle V on which the system is installed has a vehicle body V1 having a substantially rectangular shape in a plan view. The vehicle body, i.e., the vehicle body V1, has a first center line LC1, which is defined as a virtual line passing through a center point VC of the vehicle V and extending in the front-to-rear direction, i.e., the longitudinal direction, of the vehicle V on which the system is installed. The vehicle V on which the system is installed has a second center line LC2, which is defined as a virtual line passing through a center point VC of the vehicle and extending in the left-to-right direction, i.e., the width direction, of the vehicle V on which the system is installed. The center point VC of the vehicle V on which the system is installed is defined as the three-dimensional center point of the vehicle body V1.

[0125] exist Figure 1 , the width direction of the vehicle V on which the system is mounted (which will be referred to as the vehicle width direction) corresponds to the horizontal direction therein, and the vehicle height direction is defined as the height direction of the vehicle V on which the system is mounted, and is parallel to the gravity direction of the vehicle V on which the system is mounted in the case where the vehicle V on which the system is mounted is movably and stably mounted on a horizontal surface. The longitudinal direction of the vehicle V on which the system is mounted (which will be referred to as the vehicle longitudinal direction) is defined as being perpendicular to both the vehicle width direction and the vehicle height direction.

[0126] In a vehicle V in which the system is installed, if Figure 1 The front direction, rear direction, left direction, and right direction are defined as shown in . That is, the longitudinal direction of the vehicle is synonymous with the front-rear direction, and the width direction of the vehicle is synonymous with the left-right direction. Depending on the situation where the vehicle V on which the system is installed is installed on any road and / or the condition under which the vehicle V on which the system is installed is traveling, the vehicle height direction may not be parallel to the gravity direction of the vehicle V on which the system is installed. The vehicle height direction is generally parallel to the gravity direction of the vehicle V on which the system is installed.

[0127] The vehicle body V1 defines therein an interior V2 serving as a compartment for one or more occupants including a driver of the vehicle V in which the system is installed.

[0128] The vehicle body V1 has a front end portion, a rear end portion, a left side portion, a right side portion, a top portion, and four corners including a left front corner, a right front corner, a left rear corner, and a right rear corner.

[0129] The vehicle V includes four wheels V3, i.e., four wheels V3a, V3b, V3c, and V3d mounted to the respective four corners of the vehicle body V1. Specifically, the wheel V3a, i.e., the left front wheel V3a, is mounted to the left front corner of the vehicle body V1, the wheel V3b, i.e., the right front wheel V3b, is mounted to the right front corner of the vehicle body V1, the wheel V3c, i.e., the left rear wheel V3c, is mounted to the left rear corner of the vehicle body V1, and the wheel V3d, i.e., the right rear wheel V3d, is mounted to the right rear corner of the vehicle body V1. The vehicle V on which the system is installed is not limited to such a four-wheeled motor vehicle, and a three-wheeled motor vehicle or a six-wheeled or eight-wheeled vehicle such as a truck can be used as the vehicle V on which the system is installed. The number of drive wheels among the wheels of the vehicle V on which the system is installed can be freely determined, and each drive wheel can be freely positioned to the vehicle body V1 of the vehicle V on which the system is installed.

[0130] The vehicle V in which the system is installed includes a front bumper V12 installed to the front end portion, i.e., the front side V11, of the vehicle body V1. The vehicle V in which the system is installed includes a rear bumper V14 installed to the rear end portion, i.e., the rear side V13, of the vehicle body V1. The vehicle body V1 in which the system is installed includes a vehicle body panel V15 arranged to constitute the left and right sides and the top of the vehicle body V1. The vehicle body panel V15 includes a door panel V16. Figure 1 In the basic embodiment shown in , the body panel V15 includes a front pair of left and right door panels V16 and a rear pair of left and right door panels V16; the front pair and the rear pair of left door panels V16 are located at the left side of the body V1, and the front pair and the rear pair of right door panels V16 are located at the right side of the body V1. The vehicle V in which the system is installed includes a pair of door mirrors V17 mounted to the front pair of corresponding left and right doors V16. The body V1 includes a front windshield V18 covering the front side of the interior V2.

[0131] The front windshield V18 has a generally plate-like shape and is made of translucent glass or translucent synthetic resin. The front windshield V18 is attached to the vehicle body panel V15 and is inclined so that when viewed in a side view parallel to the vehicle width direction, the bottom of the front windshield V18 is positioned closer to the front of the vehicle body V1 than the top of the front windshield V18.

[0132] The vehicle V on which the system is installed includes an instrument panel V21 and a plurality of seats V22 including a driver's seat V23 on which a driver D sits. The instrument panel V21 is disposed in the front portion of the interior V2, and the seat V22 is located behind the instrument panel V21. The vehicle V on which the system is installed includes a steering wheel V24 located in front of the driver's seat V23. The driver D holds the steering wheel V24 and performs steering manipulation on the steering wheel V24 to thereby control the steering manipulation of the vehicle V on which the system is installed. The steering wheel V24 generally has a substantially annular shape, a substantially elliptical shape, or a substantially polygonal annular shape, but may have a rod-like shape or a joystick shape.

[0133] Overall configuration of vehicle systems

[0134] The vehicle V in which the system is installed includes a vehicle system 1 installed therein. The vehicle system 1 installed in the vehicle V in which the system is installed is configured to be used as a driving automation system or an autonomous driving system in the vehicle V in which the system is installed. The vehicle V in which the system is installed will also be referred to as its own vehicle V. The autonomous driving system is configured to implement one of the levels 1 to 5 included in all six autonomous driving levels (levels 0 to 5) defined in the SAEJ2016 standard issued by SAE International; SAE is the abbreviation of the "Society of Automotive Engineers". Any level X in the autonomous driving levels 0 to 5 will also be referred to as SOE level X. That is, the variable X can take any one of 0, 1, 2, 3, 4 and 5. The higher the SOE level X, the higher the driving automation level. In other words, the more dynamic driving tasks performed by the driving system, the higher the autonomous driving level. When the autonomous driving level is changed to a higher level, the autonomous driving level is improved. In contrast, the lower the SOE level X, the lower the autonomous driving level. In other words, the fewer the number of dynamic driving tasks performed by the autonomous driving system, the lower the autonomous driving level. When the autonomous driving level is changed to a lower level, the autonomous driving level is reduced.

[0135] Definition of driver assistance and autonomous driving

[0136] SAE levels 0 to 5 will be described in detail below.

[0137] In the following, the driver D is an occupant who manages and performs dynamic driving tasks. The dynamic driving tasks show all operating functions and all maneuvering functions that need to be performed by the driver D in real time when the driver D drives the own vehicle V on a traffic road, except for the strategic function. The overall driving action can be classified into operating functions, maneuvering functions, and strategic functions.

[0138] The strategic function may include a function of planning a travel itinerary and selecting one or more locations through the planned travel itinerary. Specifically, the strategic function may include a function of determining or selecting a travel itinerary that shows (i) whether to go to a destination, (ii) when to go to a destination, and (iii) how to go to a destination.

[0139] The maneuvering function may include the function of determining various maneuvers in various traffic situations, and the various maneuvers may include, during planned driving, (i) determining whether and when to overtake during planned driving, (ii) determining whether and when to change lanes, (iii) selectively setting an appropriate speed for the own vehicle V, and (iv) checking the rearview mirror.

[0140] The operational function may include the driver's immediate reaction, such as steering operation, braking operation, acceleration operation and / or minor adjustments to these operations to maintain the position of the own vehicle V in the corresponding lane of the road and / or avoid at least one obstacle and / or at least one dangerous event on the path of the moving own vehicle V. OEDR is an abbreviation for "object and event detection and response" and can be referred to as "object and event detection and response". OEDR includes monitoring of the driving environment around the own vehicle V. Monitoring of the driving environment around the own vehicle V may include detection, identification and classification of one or more objects and / or events. Monitoring of the driving environment around the own vehicle V may additionally include response preparation for one or more objects and / or events. Operational design domain (ODD) conditions (which will also be referred to as "specific domain conditions") represent specific conditions in which a given autonomous driving system or its features are designed to function. ODD conditions may include, for example, at least one of a plurality of restrictive conditions, the plurality of restrictive conditions including, for example, geographical conditions, environmental conditions, speed conditions and time conditions.

[0141] Level 0 means no autonomous driving, which means that the driver D performs all dynamic driving tasks.

[0142] Level 1 indicates driving assistance, which means that the autonomous driving system can continuously perform the lateral vehicle motion control subtask or the longitudinal vehicle motion control subtask in the dynamic driving task under specific ODD conditions. The longitudinal vehicle motion control subtask includes, for example, forward / reverse operation, acceleration / deceleration operation, and stop operation. The lateral vehicle motion control task includes, for example, steering operation. In particular, the autonomous driving system is configured not to perform both the lateral vehicle motion control subtask and the longitudinal vehicle motion control subtask.

[0143] Level 2 represents partial autonomous driving or advanced driving assistance, which means that the autonomous driving system can sustainably perform both the lateral vehicle motion control subtask and the longitudinal vehicle motion control subtask in the dynamic driving task under specific ODD conditions while expecting the driver D to complete the OEDR subtask and supervise the autonomous driving system.

[0144] Level 3 indicates conditional autonomous driving, which means that the driving automation system can continuously perform all dynamic driving tasks under specific ODD conditions. Under specific ODD conditions, the driver D is not required to perform one of the OEDR subtasks of monitoring the traffic environment around the own vehicle V, but in the case where it is difficult for the driving automation system to continue at level 3, the driving automation system requests the driver D to control the own vehicle V with sufficient time, so that the driver D needs to respond to the request smoothly.

[0145] Level 4 indicates high automation, which means that the driving automation system can continuously perform all dynamic driving tasks under specific ODD conditions. In the case where the driving automation system has difficulty continuing at level 4, the driving automation system will resolve the difficulty.

[0146] Level 5 indicates full automation, which means that the driving automation system on the own vehicle V can continuously perform all dynamic driving tasks without restriction under all ODD conditions. In the event that the driving automation system has difficulty continuing at level 5, the driving automation system resolves the difficulty without restriction under all ODD conditions.

[0147] The vehicle system 1 is configured to perform various driving control tasks and various notification tasks based on various driving tasks during driving of the own vehicle V. Specifically, the vehicle system 1 is configured as an autonomous driving system that implements driving assistance for the own vehicle V and / or autonomous driving of the own vehicle V. Autonomous driving corresponds to each of SAE levels 3 to 5. That is, autonomous driving in each of SAE levels 3 to 5 means that the vehicle system 1 is used as an autonomous driving system to perform all dynamic driving tasks in the corresponding level of SAE levels 3 to 5. In contrast, driving assistance corresponds to each of SAE levels 1 and 2. That is, autonomous driving in each of SAE levels 1 and 2 means that the vehicle system 1 is used as an autonomous driving system to perform a part of the dynamic driving tasks in the corresponding level of SAE levels 1 and 2. That is, driving assistance may include both SAE level 1 "driver assistance" and SAE level 2 "partial autonomous driving" or "advanced driving assistance", unless the expression "driving assistance of SAE level 1" is used or driving assistance is used to distinguish from partial autonomous driving of SAE level 2.

[0148] The vehicle system 1 of the basic embodiment may be configured to perform: (i) autonomous driving at each of SAE Levels 3 to 5, (ii) partially autonomous driving at SAE Level 2, i.e., advanced driving assistance, and driving assistance at SAE Level 1. Driving assistance that may be performed by the vehicle system 1 of the basic embodiment may include hands-off driving. Hands-off driving enables the vehicle system 1 to automatically move the own vehicle V forward or backward, steer the own vehicle V, accelerate or decelerate the own vehicle V, change lanes of the own vehicle V, and / or stop the own vehicle V, as long as the driver D appropriately handles the intervention request issued from the vehicle system 1.

[0149] Hands-off driving requests the driver D to monitor the road conditions around the own vehicle V, the traffic conditions around the own vehicle V, and information about whether there are one or more obstacles around the own vehicle V, without requesting the driver D to be in a hands-on state. The hands-on state indicates a state of the driver D in which the driver D is ready to perform steering manipulation on the own vehicle V, that is, ready to intervene in the lateral vehicle motion control subtask. The hands-on state generally indicates a state of the driver D in which the driver D sits on the driver's seat V23 in a posture capable of driving the own vehicle V and is ready to operate the steering wheel V24 with his / her hands. The driver D in the hands-on state holds the steering wheel V24 with his / her hands. The state in which the driver D touches the steering wheel V24 with his / her hands while being ready to hold the steering wheel V24 is applicable to the hands-on state. For example, the state in which the driver D is operating the steering wheel V24, that is, the state in which the driver D is actively operating the steering wheel V24 is applicable to the hands-on state. The state in which the driver D holds the steering wheel V24 relative to the control steering manipulation of the steering wheel V24 by the vehicle system 1 is applicable to the hands-on state.

[0150] Vehicle system configuration

[0151] The following reference Figure 1 as well as Figure 2 The overall configuration of the vehicle system 1 is described.

[0152] The vehicle system 1 includes: a driving electronic control unit (ECU) 2, a driving information input unit 3, a vehicle communication module (in other words, a data communication module (DCM)) 4, a high-definition (HD) map database 5, a navigation system 6, a human-machine interface (HMI) system 7, a lighting system 8 and a motion control system 9.

[0153] The vehicle system 1 is configured such that a driving ECU 2, a driving information input unit 3, a vehicle communication module 4, an HD map database 5, a navigation system 6, an HMI system 7, a lighting system 8, and a motion control system 9 are communicatively connected to each other via a vehicle communication network 10. The vehicle communication network 10 includes a main network conforming to one of various communication standards, such as (International Registration No. 1048262A). In addition to the main network of the vehicle communication network 10, the vehicle communication network 10 may also include a sub-network compliant with a local interconnect network (LIN) or FlexRay.

[0154] Driving ECU

[0155] The driving ECU 2 (which is installed in the vehicle V in which the system is installed) serves as a control device according to the present disclosure and is configured to control the overall operation in the vehicle system 1. The driving ECU 2 is configured as an autonomous driving / advanced driver assistance system (AD / ADAS) ECU that serves as both a driving assistance controller and an autonomous driving controller. Specifically, the driving ECU 2 includes at least one processor 21 and at least one memory device 22 that is communicatively connected to the at least one processor 21.

[0156] At least one processor 21, which will be simply referred to as the processor 21, includes a central processing unit (CPU) or a microprocessing unit (MPU). At least one memory device 22 may include, for example, at least a read-only memory (ROM) and a random access memory (RAM) selected from various nonvolatile memory devices (such as ROM, RAM, and nonvolatile rewritable recording media). Such a recording medium may be referred to as a storage medium. These nonvolatile rewritable recording media, such as flash ROM or EEPROM, make the information stored therein rewritable when power is on, and keep the information unwritable when power is off. EEPROM is an abbreviation of electronically erasable and programmable ROM. At least one nonvolatile rewritable memory device in the ROM or nonvolatile rewritable memory device included in the memory device 22 pre-stores data and program instructions for the processor 21. The driving ECU 2 is configured to read the program instructions stored in the memory device 22, and execute the read program instructions to perform various tasks and operations accordingly, including the own vehicle control operation and the notification operation to the occupant.

[0157] Driving information input unit

[0158] The driving information input unit 3 is configured to input information required for the driving ECU 2 to perform various operations and tasks. Specifically, the driving information input unit 3 may include at least one sonar sensor 31, a radar sensor 32, a laser radar sensor (LIDAR) 33, at least one camera 34, an operation sensor 35, a behavior sensor 36, a driver state monitor 37, an operation switch 38, and a locator 39. The sonar sensor 31, the radar sensor 32, the laser radar 33, and the camera 34 are collectively referred to as a surrounding monitor sensor or an ADAS sensor. The components of the driving information input unit 3 are described in sequence below.

[0159] Sonar Sensor

[0160] At least one sonar sensor (which will be simply referred to as a sonar sensor) 31 is an ultrasonic distance sensor mounted to the vehicle body V1. The sonar sensor 31 is an ultrasonic sensor, and as Figure 3The sonar sensor 31 is configured to transmit a sonar detection wave Wsp in an ultrasonic frequency band toward an external space outside the own vehicle V. The sonar sensor 31 is configured to receive a sonar echo Wsr generated by reflection of the sonar detection wave Wsp by the target object B to detect various information about the target object B accordingly. In this specification, for the sake of convenience, a reference numeral B is assigned to any target object, but target objects assigned the same reference numeral B may not necessarily represent the same target object.

[0161] Specifically, the sonar sensor 31 is configured to calculate the distance of the object B from the sonar sensor 31 based on the time of flight (TOF) and the speed of sound. TOF represents the time defined from the time when the sonar detection wave (i.e., pulse) Wsp is emitted to the time when the sonar echo Wsr is received through the propagation path Ls of the pulses Wsp and Wsr; therefore, TOF will also be referred to as the propagation time.

[0162] If the driving information input unit 3 includes a pair of sonar sensors 31, that is, a pair of first sonar sensor 311 and second sonar sensor 312 (see Figure 4 ), then the known triangulation method is used to measure the relative distance of the target object B relative to the own vehicle V. Figure 4 In FIG. 1 , an X-axis is defined along a virtual line connecting between the first sonar 311 and the second sonar 312, and a Y-axis is defined to be perpendicular to the X-axis. The X-axis and the Y-axis extend along a reference horizontal plane perpendicular to the vehicle height direction.

[0163] The first sonar sensor 311 is configured to emit a sonar detection wave Wsp, and each of the first sonar sensor 311 and the second sonar sensor 312 is configured to receive a sonar echo Wsr generated by reflecting the sonar detection wave Wsp from the target object B. The position of the target object B in a two-dimensional XY coordinate system composed of an X-axis and a Y-axis can be calculated based on a first TOF through the first propagation path Ls1 and a second TOF through the second propagation path Ls2. The first propagation path Ls1 is defined as a propagation path through which an ultrasonic wave (pulse) emitted from the first sonar 311 as a sonar detection wave Wsp propagates through the target object B to return to the first sonar 311 as a sonar echo Wsr. The ultrasonic wave (pulse) propagated through the propagation path Ls1 will also be referred to as a direct wave (pulse). The second propagation path Ls2 is defined as a propagation path through which an ultrasonic wave (pulse) emitted from the first sonar 311 as a sonar detection wave Wsp propagates through the target object B to reach the second sonar 312 as a sonar echo Wsr. The ultrasonic wave (pulse) propagating through the propagation path Ls2 will also be referred to as an indirect wave (pulse).

[0164] For example, the driving information input unit 3 according to the basic embodiment includes a plurality of sonar sensors 31 (see Figure 1 ). Specifically, the sonar sensor 31 includes a first front sonar SF1, a second front sonar SF2, a third front sonar SF3, and a fourth front sonar SF4 mounted to the front bumper V12. Similarly, the sonar sensor 31 includes a first rear sonar SR1, a second rear sonar SR2, a third rear sonar SR3, and a fourth rear sonar SR4 mounted to the rear bumper V14. In addition, the sonar sensor 31 includes a first side sonar SS1, a second side sonar SS2, a third side sonar SS3, and a fourth side sonar SS4. The first side sonar SS1 and the third side sonar SS3 are mounted to the left side portion of the vehicle body V1, and the second side sonar SS2 and the fourth side sonar SS4 are mounted to the right side portion of the vehicle body V1.

[0165] If there is no need to identify any of the sonars SF1 to SS4 , the first to fourth front sonars SF1 to SF4 , the first to fourth rear sonars SR1 to SR4 , and the first to fourth side sonars SS1 to SS4 will also be simply collectively referred to as the sonar sensor 31 or multiple sonar sensors 31 .

[0166] Figure 5 A predetermined detection area, that is, a predetermined sensing area, of each of the sonar sensors 31 is shown.

[0167] The following reference Figure 1 and Figure 5 The sonar sensor 31 will be described sequentially.

[0168] The first front sonar SF1 is mounted to a portion of the front bumper V12 that is closer to its left edge than the right edge of the front bumper V12 in the vehicle width direction, and is configured to emit a sonar detection wave Wsp obliquely to the left front. The second front sonar SF2 is mounted to a portion of the front bumper V12 that is closer to its right edge than the left edge of the front bumper V12 in the vehicle width direction, and is configured to emit a sonar detection wave Wsp obliquely to the right front. The first front sonar SF1 and the second front sonar SF2 are arranged symmetrically with respect to the first center line LC1. Each of the first front sonar SF1 and the second front sonar SF2 has a predetermined detection area Rsc, and the detection area Rsc of each of the first front sonar SF1 and the second front sonar SF2 (also referred to as the front corner detection area Rsc) is designed so that the detection range of the front corner detection area Rsc is set to, for example, 60 cm or about 60 cm. The detection range of the sonar sensor 31 represents the maximum measurable range (distance) of the sonar sensor 31.

[0169] The third front sonar SF3 and the fourth front sonar SF4 are mounted to the middle portion of the front bumper V12 to be aligned in the vehicle width direction. Specifically, the third front sonar SF3 is arranged between the first front sonar SF1 and the first center line LC1, and is configured to emit a sonar detection wave Wsp substantially forward, and the fourth front sonar SF4 is arranged between the second front sonar SF2 and the first center line LC1, and is configured to emit a sonar detection wave Wsp substantially forward. The third front sonar SF3 and the fourth front sonar SF4 are arranged symmetrically with respect to the first center line LC1. Each of the third front sonar SF3 and the fourth front sonar SF4 has a predetermined detection area Rsf, and the detection area Rsf of each of the third front sonar SF3 and the fourth front sonar SF4 (also referred to as the front detection area Rsf) is designed so that the detection range of the front detection area Rsf is set to, for example, 1m or about 1m.

[0170] The first front sonar SF1 and the third front sonar SF3 mounted to the front bumper V12 on the left side relative to the first center line LC1 are arranged at different positions in the vehicle width direction, that is, in the horizontal direction. The first front sonar SF1 and the third front sonar SF3 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the first front sonar SF1 and the third front sonar SF3 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the first front sonar SF1 and the third front sonar SF3 by a target object as a reception echo.

[0171] Specifically, the first front sonar SF1 is arranged to receive (i) a direct echo generated by the target object reflecting the sonar detection wave Wsp emitted from the first front sonar SF1 and (ii) an indirect echo generated by the target object reflecting the sonar detection wave Wsp emitted from the third front sonar SF3. Similarly, the third front sonar SF3 is arranged to receive (i) a direct echo generated by the target object reflecting the sonar detection wave Wsp emitted from the third front sonar SF3 and (ii) an indirect echo generated by the target object reflecting the sonar detection wave Wsp emitted from the first front sonar SF1.

[0172] Similarly, the third front sonar SF3 and the fourth front sonar SF4 mounted to the middle portion of the front bumper V12 in the vehicle width direction are arranged at different positions in the vehicle width direction, i.e., in the horizontal direction. The third front sonar SF3 and the fourth front sonar SF4 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the third front sonar SF3 and the fourth front sonar SF4 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the third front sonar SF3 and the fourth front sonar SF4 by a target object as a reception echo.

[0173] The second front sonar SF2 and the fourth front sonar SF4 mounted to the front bumper V12 on the right side with respect to the first center line LC1 are arranged at different positions in the vehicle width direction, i.e., in the horizontal direction. The second front sonar SF2 and the fourth front sonar SF4 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the second front sonar SF2 and the fourth front sonar SF4 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the second front sonar SF2 and the fourth front sonar SF4 by a target object as a reception echo.

[0174] The first rear sonar SR1 is mounted to a portion of the rear bumper V14 that is closer to its left edge than the right edge of the rear bumper V14 in the vehicle width direction, and is configured to emit a sonar detection wave Wsp obliquely to the left rear. The second rear sonar SR2 is mounted to a portion of the rear bumper V14 that is closer to its right edge than the left edge of the rear bumper V14 in the vehicle width direction, and is configured to emit a sonar detection wave Wsp obliquely to the right rear. The first rear sonar SR1 and the second rear sonar SR2 are arranged symmetrically with respect to the first center line LC1. Each of the first rear sonar SR1 and the second rear sonar SR2 has a predetermined detection area Rsd, and the detection area Rsd of each of the first rear sonar SR1 and the second rear sonar SR2 (also referred to as the rear corner detection area Rsd) is designed so that the detection range of the rear corner detection area Rsd is set to, for example, 60 cm or about 60 cm.

[0175] The third rear sonar SR3 and the fourth rear sonar SR4 are mounted to the middle portion of the rear bumper V14 to be aligned in the vehicle width direction. Specifically, the third rear sonar SR3 is arranged between the first rear sonar SR1 and the first center line LC1, and is configured to emit a sonar detection wave Wsp substantially to the rear, and the fourth rear sonar SR4 is arranged between the second rear sonar SR2 and the first center line LC1, and is configured to emit a sonar detection wave Wsp substantially to the rear. The third rear sonar SR3 and the fourth rear sonar SR4 are arranged symmetrically with respect to the first center line LC1. Each of the third rear sonar SR3 and the fourth rear sonar SR4 has a predetermined detection area Rsr, and the detection area Rsr of each of the third rear sonar SR3 and the fourth rear sonar SR4 (also referred to as the rear detection area Rsr) is designed so that the detection distance of the rear detection area Rsr is set to, for example, 1.5 m or about 1.5 m.

[0176] The first rear sonar SR1 and the third rear sonar SR3 mounted to the rear bumper V14 on the left side with respect to the first center line LC1 are arranged at different positions in the vehicle width direction, that is, in the horizontal direction. The first rear sonar SR1 and the third rear sonar SR3 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the first rear sonar SR1 and the third rear sonar SR3 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the first rear sonar SR1 and the third rear sonar SR3 by a target object as a reception echo.

[0177] Specifically, the first rear sonar SR1 is arranged to receive (i) a direct echo generated by the target object reflecting the sonar detection wave Wsp emitted from the first rear sonar SR1 and (ii) an indirect echo generated by the target object reflecting the sonar detection wave Wsp emitted from the third rear sonar SR3. Similarly, the third rear sonar SR3 is arranged to receive (i) a direct echo generated by the target object reflecting the sonar detection wave Wsp emitted from the third rear sonar SR3 and (ii) an indirect echo generated by the target object reflecting the sonar detection wave Wsp emitted from the first rear sonar SR1.

[0178] Similarly, the third rear sonar SR3 and the fourth rear sonar SR4 mounted to the middle portion of the rear bumper V14 in the vehicle width direction are arranged at different positions in the vehicle width direction, i.e., in the horizontal direction. The third rear sonar SR3 and the fourth rear sonar SR4 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the third rear sonar SR3 and the fourth rear sonar SR4 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the third rear sonar SR3 and the fourth rear sonar SR4 by a target object as a reception echo.

[0179] The second rear sonar SR2 and the fourth rear sonar SR4 mounted to the rear bumper V14 on the right side with respect to the first center line LC1 are arranged at different positions in the vehicle width direction, i.e., in the horizontal direction. The second rear sonar SR2 and the fourth rear sonar SR4 adjacent to each other in the vehicle width direction are arranged to have a predetermined positional relationship that enables one of the second rear sonar SR2 and the fourth rear sonar SR4 to receive a sonar echo generated by reflection of the sonar detection wave Wsp emitted from the other of the second rear sonar SR2 and the fourth rear sonar SR4 by a target object as a reception echo.

[0180] Each of the first side sonar SS1 and the third side sonar SS3 is mounted to a portion of the left side portion of the vehicle body V1, and is configured to transmit the sonar detection wave Wsp to the left side relative to the own vehicle V. Similarly, each of the second side sonar SS2 and the fourth side sonar SS4 is mounted to a portion of the right side portion of the vehicle body V1, and is configured to transmit the sonar detection wave Wsp to the right side relative to the own vehicle V. Each of the first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4 is arranged to receive only a direct echo generated by reflection of the sonar detection wave Wsp emitted from a corresponding one of the first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4. Each of the first side sonar SS1, the second side sonar SS2, the third side sonar SS3 and the fourth side sonar SS4 has a predetermined detection area Rss, and the detection area Rss of each of the first side sonar SS1 to the fourth side sonar SS4 (also referred to as the side detection area Rss) is designed so that the detection distance of the side detection area Rss is set to, for example, within the range from 2m to 3m, inclusive.

[0181] The first side sonar SS1 is arranged between the first front sonar SF1 and the door mirror V17 mounted to the front pair of left door panels V16, which will also be referred to as the left door mirror V17. The first side sonar SS1 is configured to emit a sonar detection wave Wsp to the left relative to the own vehicle V. The second side sonar SS2 is arranged between the second front sonar SF2 and the door mirror V17 mounted to the front pair of right door panels V16, which will also be referred to as the right door mirror V17. The second side sonar SS2 is configured to emit a sonar detection wave Wsp to the right relative to the own vehicle V. The first side sonar SS1 and the second side sonar SS2 are arranged symmetrically relative to the first center line LC1. The first side sonar SS1 and the second side sonar SS2 may be mounted to the vehicle body panel V15, or to portions of the corresponding left and right edges of the front bumper V12 in the vehicle width direction; a portion of each of the left and right edges of the front bumper V12 to which a corresponding one of the first side sonar SS1 and the second side sonar SS2 extends rearward in the vehicle longitudinal direction.

[0182] The third side sonar SS3 is arranged between the first rear sonar SR1 and a pair of rear left door panels V16. The third side sonar SS3 is configured to emit sonar detection waves Wsp to the left relative to the own vehicle V. The fourth side sonar SS2 is arranged between the second rear sonar SR2 and a pair of rear right door panels V16. The fourth side sonar SS4 is configured to emit sonar detection waves Wsp to the right relative to the own vehicle V. The third side sonar SS3 and the fourth side sonar SS4 are arranged symmetrically with respect to the first center line LC1. The third side sonar SS3 and the fourth side sonar SS4 can be mounted to the vehicle body panel V15, or to the corresponding left and right edge portions of the rear bumper V14 in the vehicle width direction; the corresponding one of the third side sonar SS3 and the fourth side sonar SS4 extends forward in the vehicle longitudinal direction to each of the left and right edges of the rear bumper V14.

[0183] Figure 6 An example is schematically shown in which the driving ECU 2 detects a parking space PS located on the left side relative to the own vehicle V for performing parallel parking using the first side sonar SS1. The parking space PS is a rectangular frame-shaped area in a plan view and represents a space in which a vehicle can be parked. The parking space PS may also be referred to as a parking space or a parking lot. Figure 6 1 shows a sonar detection point Psr, which is a point where it is estimated that the sonar detection wave Wsp emitted from any sonar sensor 31 will be reflected; the sonar detection point Psr may also be referred to as a ranging point Psr. Figure 6 As shown, the driving ECU 2 can detect the parking space PS based on the distribution of sonar detection points Psr acquired based on sonar echoes Wsr from target objects B that are parked vehicles arranged side by side in the straight forward moving direction of the own vehicle V.

[0184] Specifically, when Figure 6 When acquiring adjacent rows of sonar detection points Psr arranged in the traveling direction of the own vehicle V in the case of performing parallel parking of the own vehicle V as shown, the driving ECU 2 can determine whether the length of the space between each adjacent row pair of sonar detection points Psr exceeds the entire width of the own vehicle V, and in response to determining that the length of the space between at least one adjacent row pair of sonar detection points Psr exceeds the entire width of the own vehicle V, detect the space between at least one adjacent row pair of sonar detection points Psr as the position of the parking space PS.

[0185] Radar Sensor

[0186] Reference Figure 1 and Figure 2The radar sensor 32 is configured to transmit millimeter radar waves or submillimeter radar waves and receive reflected radar waves to detect the target object B accordingly. The radar sensor 32 of the basic embodiment is mounted to the middle of the front portion V11 of the vehicle body V1 in the vehicle width direction. Figure 7 The radar sensor 32 shown is a long-range radar having a substantially fan-shaped radar detection area Rg1 in a plan view. The fan-shaped radar detection area Rg1 has a predetermined detection range from 10m to 250m (inclusive), and has a radar scanning angle θr1 around the forward moving direction Df of the own vehicle V. The radar sensor 32 is configured as a phased array radar sensor or a beamforming radar sensor. The forward moving direction Df represents the direction of a virtual line extending forward from the first center line LC1, and represents the driving direction of the own vehicle V when the current shift position of the shift lever is set to any position other than the reverse position. The forward moving direction Df of the own vehicle V matches the driving direction of the own vehicle V when the own vehicle V is moving straight forward, and becomes a tangent direction to the driving path of the own vehicle V when the own vehicle V is traveling around a curve.

[0187] The radar sensor 32 is configured to transmit a radar detection wave Wrp, scan the transmitted radar detection wave Wrp within a radar scanning angle θr1 , and receive a radar wave Wrr generated by reflection of the radar detection wave Wrp by a target object B located in the radar detection area Rg1 .

[0188] Specifically, the radar sensor 32 includes an FMCW radar device equipped with an array antenna, and is configured to detect (i) the distance from it to the target object B, (ii) the azimuth angle θa of the target object B, and (iii) the relative speed of the target object B with respect to the own vehicle V, based on the frequency difference between the transmitted millimeter wave and the received millimeter wave and / or the phase difference between the transmitted millimeter wave and the received millimeter wave. The azimuth angle θa of the target object B is defined as the angle formed by a first virtual line generated by extending the first center line LC1 toward the front of the own object V and a second virtual line connecting the radar sensor 32 and the target object B; the first center line CL1 represents the center line of the radar detection range Rg1.

[0189] The relative speed of the target object B with respect to the own vehicle V is defined as the difference between the moving speed vb of the target object B and the traveling speed vm of the own vehicle V.

[0190] Specifically, the radar sensor 32 is configured to transmit a radar detection wave Wrp generated based on a transmission signal having a predetermined modulation frequency, and receive a radar wave Wrr generated by reflecting the radar detection wave Wrp from the target object B, so as to detect a received signal representing the frequency characteristics of the received radar wave Wrr based on the received radar wave Wrr. Then, the radar sensor 32 is configured to calculate the deviation between the modulation frequency of the transmission signal and the frequency of the received signal, so as to generate a beat signal based on the corresponding frequency deviation. The radar sensor 32 is configured to perform a first Fourier transform on the beat signal, so as to calculate the frequency-power spectrum of each beat signal in the beat signal. Then, the radar sensor 32 is configured to analyze the frequency-power spectrum of each beat signal, so as to obtain the beat frequency, and calculate the distance of the target object B from the radar sensor 32 and the relative speed of the target object B relative to the own vehicle V based on the beat frequency.

[0191] In addition to the long-range radar sensor 32, a medium-range radar sensor or a short-range radar sensor may be installed in the vehicle system 1. Such a medium-range radar sensor has, for example, a predetermined detection range from 1 m to 100 m (inclusive), and such a short-range radar sensor has, for example, a predetermined detection range from 15 cm to 30 m.

[0192] LiDAR Sensor

[0193] like Figure 8 As shown, the laser radar sensor 33 is configured to illuminate the exterior of the own vehicle V with detection light Lp, i.e., laser light in the infrared band, and receive reflected light Lr generated by the target object B reflecting the detection light Lp, so as to detect the target object B accordingly. The laser radar sensor 33 may also be referred to as LIDAR, which is an abbreviation of light detection and ranging or laser imaging detection and ranging. The vehicle system 1 of the basic embodiment includes a long-range laser radar sensor 33 including a scanning LIDAR as the laser radar sensor 33. As shown in FIG. Figure 1 As shown, the laser radar sensor 33 is mounted to the center of the front portion V11 of the vehicle body V1 in the vehicle width direction.

[0194] The laser radar sensor 33 has a substantially fan-shaped LIDAR detection area Rg2 in a plan view. The fan-shaped radar detection area Rg2 has a predetermined radius of, for example, 200 m or more, and has a LIDAR scanning angle θr2 around the forward moving direction Df of the own vehicle V. The laser radar sensor 33 is configured to horizontally scan the laser detection light Lp within the LIDAR scanning angle θr2 to detect the target object B located in the LIDAR detection area Rg2 accordingly.

[0195] Fig. 92 shows a schematic configuration of the laser radar sensor 33. The laser radar 33 includes a light emitting unit 331, a scanning unit 332, and a light receiving unit 333.

[0196] The light emitting unit 331 is configured to emit the detection light Lp. The scanning unit 332 includes a MEMS mirror unit and a MEMS mechanism, wherein the MEMS mirror unit includes at least one mirror located on the light path of the detection light Lp emitted by the light emitting unit 331, and the MEMS mechanism, for example, rotates at least one mirror to correspondingly change the direction of the light reflected by the at least one mirror; MEMS is an abbreviation of micro-electromechanical system. Specifically, the scanning unit 332 is configured to electrically control the MEMS mechanism so that the MEMS mechanism rotates at least one mirror, thereby scanning the detection light Lp emitted by the light emitting unit 331 in both the horizontal scanning direction Ds and the vertical scanning direction Dh.

[0197] The light receiving unit 333 includes a light receiving sensor 334 as a two-dimensional image sensor. Specifically, the light receiving sensor 334 includes a plurality of light receiving elements 335 arranged two-dimensionally in a horizontal direction corresponding to the horizontal scanning direction Ds and in a vertical direction corresponding to the vertical scanning direction Dh. Each of the light receiving elements 335 including an avalanche photodiode (APD) or a single photon avalanche diode (SPAD) is configured to detect a corresponding portion of the reflected light Lr generated by at least one target object B reflecting the detection light Lp.

[0198] The laser radar sensor 33 is configured to generate at least one detection point data cloud based on the reflected light Lr received by the light receiving unit 333, and detect at least one target object B based on the at least one detection point data cloud.

[0199] At least one detection point data cloud represents a two-dimensional array of a plurality of LIDAR detection points Prr, such as an image (i.e., a frame image), which are close to each other and arranged two-dimensionally in the horizontal scanning direction Ds and the vertical scanning direction Dh (see Fig. 9 ). Each LIDAR detection point Prr arranged to the corresponding light receiving element 335 represents a point of at least one target object B estimated to be a part of the reflected detection light Lp; a part of the reflected light Lr received by the corresponding light receiving element 335 corresponds to a part of the detection light Lp. Therefore, this results in the LIDAR detection points Prr arranged two-dimensionally in the horizontal scanning direction Ds and the vertical scanning direction Dh including position information and range information on a two-dimensional array. Therefore, this enables at least one detection point data cloud to also be referred to as at least one range point data cloud.

[0200] That is, the laser radar sensor 33 is configured to detect at least one detection point data cloud including LIDAR detection points Prr arranged two-dimensionally like a frame image in the horizontal scanning direction Ds and the vertical scanning direction Dh. Therefore, this enables the laser radar sensor 33 to be also called an image sensor.

[0201] That is, if there are target objects B, the laser radar sensor 33 is configured to detect, for each of the target objects B, a detection point data cloud including LIDAR detection points Prr arranged two-dimensionally like a frame image in the horizontal scanning direction Ds and the vertical scanning direction Dh.

[0202] Camera device

[0203] Reference Figure 1 and Figure 2 , at least one camera device 34 serving as an image sensor is mounted to the own vehicle V. The at least one camera device 34 is configured to capture images of a peripheral field of view around the own vehicle V while moving together with the own vehicle V. That is, the at least one camera device 34 is configured to generate image information about each captured image around the own vehicle V, which will also be referred to as image data.

[0204] The at least one camera device 34 is configured as a digital camera device including an image sensor such as a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. The image sensor of the at least one camera device 34 includes a plurality of photosensitive elements such as photodiodes, which respectively correspond to a plurality of pixels arranged two-dimensionally in both the vertical direction (i.e., vehicle height direction) and the horizontal direction (i.e., vehicle width direction) of the own vehicle V.

[0205] The driving information input unit 3 of the basic embodiment includes a plurality of cameras, namely, a front camera CF, a rear camera CB, a left camera CL, and a right camera CR mounted to the own vehicle V. If it is not necessary to identify any of the cameras CF, CR, CL, and CR, the front camera CF, the rear camera CR, the left camera CL, and the right camera CR will also be simply collectively referred to as one camera 34 or a plurality of cameras 34.

[0206] The front camera device CF is mounted to the upper end of the front windshield V18 approximately in the middle of the vehicle width direction at the top of the interior V2, and has a forward field of view in front of the own vehicle V. That is, the front camera device CF is located on the first center line LC1 in a plan view. The front camera device CF can be mounted to the front portion V11 of the vehicle body V1. The front camera device CF is configured to capture an image of the forward field of view to obtain information about the captured image of the forward field of view accordingly.

[0207] The rear camera CB is mounted to the substantially middle of the rear end V13 of the vehicle body V1 in the vehicle width direction and has a rear view field behind the own vehicle V. The rear camera CR is configured to capture an image of the rear view field to accordingly acquire information about the captured image of the rear view field.

[0208] The left camera CL is mounted to the left door mirror V17 and has a left field of view at the left side of the own vehicle V. The left camera CL is configured to capture an image of the left field of view to correspondingly acquire information about the captured image of the left field of view.

[0209] The right camera CR is mounted to the right door mirror V17 and has a right field of view at the right side of the own vehicle V. The right camera CR is configured to capture an image of the right field of view to correspondingly acquire information about the captured image of the right field of view.

[0210] The image captured by each camera 34 includes two-dimensionally arranged pixels, which respectively correspond to the two-dimensionally arranged photosensitive elements of the corresponding camera 34 .

[0211] The driving ECU 2 can recognize the target object B based on the image captured by any of the cameras 34 , that is, determine at least the position of the target object B and the type of the target object B based on the image captured by any of the cameras 34 .

[0212] Fig.10 An example is shown of how the driving ECU 2 recognizes a target object B included in a front field of view based on an image of the front field of view captured by the front camera CF; the front field of view includes a road Rd in front of the own vehicle V. In the specification, for convenience, reference numeral Rd is assigned to any road, but roads assigned the same reference numeral Rd may not necessarily represent the same road.

[0213] Specifically, the driving ECU 2 sets a detection area Aw in the captured image. The detection area Aw represents a part or all of the entire area of ​​the captured image, that is, the entire viewing angle area of ​​the front camera device CF. Then, the driving ECU 2 obtains the feature point image Gp based on the pixel feature parameters of the image data included in the detection area Aw. The pixel feature parameters of the image data represent the feature parameters of each pixel constituting the image data based on the corresponding received light, and may include, for example, the brightness, contrast, and hue of each pixel of the image data. The brightness of each pixel may be referred to as the brightness of the corresponding pixel, and the hue of each pixel may be referred to as the chromaticity of the corresponding pixel.

[0214] The feature point image Gp includes feature points Pt arranged two-dimensionally in the horizontal and vertical directions like the frame image; each of the feature points Pt is extracted from the image data based on, for example, the difference and / or change gradient between corresponding adjacent pixel pairs of the image data included in the detection area Aw. The feature point Pt characterizes the shape of the target object B included in the detection area Aw of the captured image. In other words, the feature point Pt is a characteristic point of the image data included in the detection area Aw, that is, a characteristic pixel.

[0215] The driving ECU 2 performs a pattern matching task of matching one or more feature point clouds Pg (each feature point cloud Pg is a set of corresponding feature points Pt) with a predetermined pattern stored therein, so as to identify, for each of the feature point clouds Pg, the type of a corresponding one of the target objects B based on the corresponding one of the feature point clouds Pg.

[0216] Various methods for extracting feature points Pt from the image data included in the detection area Aw are known. For example, an extraction method using a Sobel filter, an extraction method using a Laplacian filter, an extraction method using a Canny algorithm can be used to extract feature points Pt from the image data included in the detection area Aw. Therefore, a detailed description of one of the known extraction methods used in the specification is omitted. Extracting feature points Pt from the image data included in the detection area Aw can be expressed as detecting feature points Pt from the image data included in the detection area Aw.

[0217] The driving ECU 2 may estimate the relative position of the recognized target object B with respect to the own vehicle V and / or the distance of the recognized target object B with respect to the own vehicle V, ie, the range.

[0218] Fig.11 An example is shown in which the driving ECU 2 calculates the estimated point Pb as the estimated result of the three-dimensional position of each feature point Pt corresponding to the target object B using, for example, a monocular motion stereo method, in other words, a structure from motion (SFM) method. The present disclosure may calculate the estimated point Pb as the estimated result of the three-dimensional position of each feature point Pt corresponding to the target object B using one of the other calculation methods selected. That is, each camera 34 may be configured as a compound eye stereo camera.

[0219] Assume that Fig.11As shown, the camera 34 moves together with the own vehicle V toward the own vehicle moving direction Dv of the own vehicle V. The position of the camera 34 at time t1 is referred to as a first camera position Pc1, and the position of the camera 34 at time t2 after time t1 is referred to as a second camera position Pc2. The first captured image A1 is an image captured by the camera 34 at the first camera position Pc1, and the second captured image A2 is an image captured by the camera 34 at the second camera position Pc2.

[0220] The first feature point Pt1 is a feature point Pt extracted from the first captured image A1, and the second feature point Pt2 is extracted from the second captured image A2, estimated to correspond to the first feature point Pt1 at time t1. That is, the second feature point Pt2 is a point on the target object B to which the point corresponding to the first feature point Pt1 is estimated to have moved within the elapsed time (t2-t1) from time t1 to time t2. The driving ECU 2 can use one of the well-known methods such as the optical flow method to determine whether the first feature point Pt1 and the second feature point Pt1 are based on the same point on the target object B, that is, whether the first feature point Pt1 on the target object B corresponds to the second feature point Pt2 thereon. Then, the driving ECU 2 defines a first line L1 passing through the first camera position Pc1 and the first feature point Pt1 and a second line L2 passing through the second camera position Pc2 and the second feature point Pt2, and calculates the intersection of the first line L1 and the second line L2 as the estimated point Pb. The estimated point Pb represents a point on the target object B corresponding to both the first feature point Pt1 and the second feature point Pt2 in a three-dimensional coordinate system defined relative to the own vehicle V. If the estimated point Pb is a stationary point, the estimated point Pb satisfies the epipolar constraint. The epipolar constraint is an epipolar geometric constraint, that is, the first camera position Pc1, the second camera position Pc2, and the estimated point Pb are located on the same plane Π at any time t1 or t2.

[0221] Various sensors

[0222] Reference Figure 1 and Figure 2, the operation sensors 35 are each arranged in the own vehicle V, and are used to output parameters indicating the corresponding driver operation state of the own vehicle V. The parameters to be output from each operation sensor 35 (each indicating the corresponding driver operation state of the own vehicle V) may include, for example, (i) the driver operation amount of the accelerator pedal of the own vehicle V, (ii) the driver operation amount of the brake pedal of the own vehicle V, (iii) the current gear position set by the driver of the shift lever, (iv) the driver operation steering angle of the steering wheel V24, and (v) the steering torque applied by the driver of the steering wheel V24. That is, the operation sensor 35 includes known sensors, including, for example, an accelerator pedal sensor, a brake pedal sensor, a gear position sensor, a steering angle sensor, and a steering torque sensor. That is, for the sake of simple explanation and simple description, these known sensors are collectively referred to as the operation sensor 35. The operation sensor 35 may include a steering wheel sensor, which is used to detect information indicating whether the driver D is holding the steering wheel V24.

[0223] The behavior sensors 36 are each provided in the own vehicle V, and are used to output parameters indicating the corresponding driving behavior of the own vehicle V. The parameters to be output from the respective behavior sensors 36 (each indicating the corresponding behavior of the own vehicle V) may include, for example, (i) the speed of the own vehicle V, (ii) the yaw rate of the own vehicle V, (iii) the acceleration of the own vehicle V in the longitudinal direction, and (iv) the acceleration of the own vehicle V in the vehicle width direction. That is, the behavior sensors 36 include known sensors, including, for example, a vehicle speed sensor, a yaw rate sensor, and an acceleration sensor. That is, for the sake of simple explanation and simple description, these known sensors are collectively referred to as behavior sensors 36.

[0224] Driver Status Monitor

[0225] Reference Figure 1 , Figure 2 , Fig.12 and Fig.13 The driver state monitor 37 is configured to sequentially capture images of the driver D and sequentially detect a driver's state parameter representing the driver's state based on the captured images of the driver D. The driver's condition includes, for example, information indicating whether the driver D is in a sober state.

[0226] Specifically, the driver state monitor 37 includes a driver monitor camera having a predetermined field of view; the driver monitor camera is located in the interior V2 so that at least the head D1 of the driver D2 sitting on the driver seat V23 is located within the field of view of the driver monitor camera. This enables the driver monitor camera to capture an image of the face D2 of the driver D from the front. The driver monitor camera may be configured as a near infrared camera.

[0227] The driver state monitor 37 comprises an image processing unit configured to perform image processing tasks on images captured by the driver monitor camera to detect the driver's state parameters accordingly.

[0228] like Fig.12 and Fig.13 As shown, the state parameters of the driver to be detected by the driver state monitor 37 include, for example, (i) the direction of the face D2 of the driver D, (ii) the degree of opening of each driver's eye D3, and (iii) the position of each pupil D4 of the driver D. The direction of the face D2 of the driver D is defined by the deflection angle θy and the pitch angle θp of the face D2 of the driver D. The deflection angle θy represents the rotation angle of the face D2 of the driver D around the vertical axis Dx1, which extends vertically through the head D1 of the driver D. When the face D2 of the driver D faces forward, the deflection angle θy is 0°. That is, when the face D2 of the driver D faces the left side relative to the front, the deflection angle θy becomes a corresponding positive angle, and when the face D2 of the driver D faces the right side relative to the front, the deflection angle θy becomes a corresponding negative angle.

[0229] The pitch angle θp represents the rotation angle of the face D2 of the driver D around the horizontal axis Dx2, which horizontally extends through the face D2 of the driver D. When the face D2 of the driver D faces forward, the pitch angle θp is 0°. That is, when the face D2 of the driver D is upward relative to the front, the pitch angle θp becomes a corresponding positive angle, and when the face D2 of the driver D is downward relative to the front, the pitch angle θp becomes a corresponding negative angle.

[0230] Fig.12 is an example of an image captured by the driver state monitor 37 in which the driver D is awake, and Fig.13 is an example of an image captured by the driver state monitor 37 in which the driver D is not in a wakeful state. Fig.12 and Fig.13The driver's state parameters, (i) the direction of the driver D's face D2, (ii) the degree of openness of each driver's eye D3, and (iii) the position of each pupil D4 of the driver D are shown, so that the driving ECU 2 can detect the driver's state, including, for example, (i) information about whether the driver D is in a waking state and (ii) information about the driver's line of sight D5.

[0231] Operation switch

[0232] Reference Figure 1 and Figure 2 The operation switches 38 are various switches that can be operated by the driver E when the driver E drives the own vehicle E, and are provided at predetermined positions in the interior V2. Each of the operation switches 38 is a switch whose operation amount and operation state are exempted from detection by the operation sensor 35. Specifically, the operation switch 38 includes, for example, an ignition switch 381, a flasher switch 382, ​​and one or more AD / ADAS switches 383. The ignition switch 381, which may be referred to as a start switch or a power switch, is a switch for activating or deactivating the vehicle V (i.e., the vehicle system 1) in which the system is installed. The flasher switch 382 is configured to detect the operation state of the flasher lever of the own vehicle V by the driver. The AD / ADAS switch 383 is a switch that enables the driver E to input various instructions related to the driving assistance or autonomous driving of the own vehicle V to the driving ECU 2 when operating them; the various instructions related to the driving assistance or autonomous driving of the own vehicle V include, for example, a start instruction, a stop instruction, a level setting instruction, and a function selection instruction.

[0233] The positioner 39 is configured to acquire high-accuracy position information about the own vehicle V. Specifically, the positioner 39 is configured as a complex positioning system, and includes a GNSS receiver 391 , an inertial detector 392 , and a positioner ECU 393 .

[0234] GNSS is an abbreviation for Global Navigation Satellite System, and the high-accuracy position information about the own vehicle V is position information about the own vehicle V that has at least a position accuracy that can be used by advanced driving assistance of SAL level 2 or higher, in other words, a position accuracy with an error of less than or equal to 10 cm. As the positioner 39, a commercially available positioning system such as the POSLV system for land vehicles, in other words, a positioning orientation system for land vehicles manufactured by Applanix Corporation can be used.

[0235] The GNSS receiver 391 may be configured to receive a navigation signal transmitted from at least one positioning satellite (i.e., at least one artificial satellite). Specifically, the GNSS receiver 391 is configured to be able to receive a navigation signal from a positioning satellite included in at least one GNSS selected from GPS, QZSS, GLONASS, GLONASS, Galileo, IRNSS, and BeiDou navigation satellite systems. GPS is an abbreviation for Global Positioning System, QZSS is an abbreviation for Quasi-Zenith Satellite System, GLONASS is an abbreviation for Global Navigation Satellite System, and IRNSS is an abbreviation for Indian Regional Navigation Satellite System.

[0236] The inertial detector 392 is configured to detect (i) linear accelerations acting on the own vehicle V in respective three axes corresponding to the vehicle longitudinal direction, the vehicle width direction, and the vehicle height direction, and (ii) angular velocities acting on the own vehicle V around the respective three axes. For example, the positioner 39 has a substantially box-shaped housing, and the inertial detector 392 includes a three-axis accelerometer and a three-axis gyro sensor installed in the housing.

[0237] The locator ECU 393 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The locator ECU 393 is configured to sequentially determine the current position and / or current orientation of the own vehicle V based on the navigation signal received by the GNSS receiver 391 and the linear acceleration and angular velocity detected by the inertial detector 392.

[0238] Vehicle Communication Module

[0239] The vehicle communication module 4 , which will also be referred to as DCM 4 , may be configured to communicate information with base stations located around the own vehicle V using wireless communications conforming to a predetermined communication standard such as Long Term Evolution (LTE) or fifth generation (5G).

[0240] Specifically, the vehicle communication module 4 is configured to obtain traffic information, such as traffic jam information, from a detection server and / or a predetermined database in a cloud computing environment. The traffic jam information includes, for example, the location and length of at least one traffic jam section. Specifically, the traffic jam information includes, for example, various information items about at least one traffic jam section, such as the head of at least one traffic jam section, the tail of at least one traffic jam section, the estimated length of at least one traffic jam section, and the estimated time of at least one traffic jam section. The traffic information will also be referred to as road traffic information.

[0241] Additionally, the vehicle communication module 4 is configured to retrieve the latest HD map information from at least one of the probe servers and store the HD map information in the HD map database 5 .

[0242] HD Map Database

[0243] The HD map database 5 mainly includes one or more nonvolatile rewritable memories, and is configured to store rewritable HD map information while maintaining the stored HD map information even if power is cut off supplied to the HD map database 5. The HD map information will also be referred to as HD map data.

[0244] The HD map information includes higher definition map information than the map information stored in the standard (SD) map database 601 of the navigation system 6. That is, the higher definition map information has a position error lower than or equal to an error of about several meters of the map information stored in the SD map database 601.

[0245] Specifically, the HD map information database 5 stores, as HD map information, map information usable for example for advanced driving assistance or autonomous driving, including, for example, (i) information on three-dimensional road shapes, (ii) information on the number of lanes in each road, and (iii) information on road traffic regulations. The HD map information is stored in the HD map information database 5 to comply with a predetermined standard, such as ADASIS.

[0246] Navigation system

[0247] The navigation system 6 is configured to calculate a planned travel route from the current position of the own vehicle V to the destination. The navigation system 6 of the basic embodiment is configured to calculate the planned travel route based on (i) a destination input by the driver D through, for example, the HMI system 7, (ii) HD map information stored in the HD map database 5 or SD map information stored in the SD map database 601, and (iii) position information about the own vehicle V, such as the current position and current orientation of the own vehicle V. The navigation system 6 is further configured to provide various information including the planned travel route to one or more selected components of the vehicle system 1, such as the driving ECU 2 and / or the HMI system 7, via the vehicle communication network 10. That is, the navigation system 6 is capable of instructing the HMI system 7 to sequentially display navigation images showing, for example, a map on which the current position of the own vehicle V and the planned travel route appear, respectively.

[0248] HMI System

[0249] The HMI system 7 is designed as a vehicle HMI system, and is configured to enable information communication between the own vehicle V and one or more occupants including a driver D of the own vehicle V.

[0250] Specifically, the HMI system 7 is configured to at least visually provide (i.e., display) various information items to one or more passengers, and enable the passengers to input information input related to the provided information items. The various information items to be provided to one or more passengers include, for example, various guide information items, information items about input operation guidance, notifications of input information, and / or warnings.

[0251] The HMI system 7 typically includes I / O components mounted to the steering wheel V24 or mounted in the dashboard V21 (the so-called "dashboard HMI"). At least one of the I / O components of the HMI system 7 may be mounted to at least one portion of the interior V2 other than the dashboard V21, such as a unit in the interior V2 or a center console between the driver's seat V23 and a passenger seat V22 adjacent to the driver's seat V23.

[0252] The HMI system 7 includes an HMI control unit (HCU) 701 , an instrument panel 702 , a main display device 703 , a head-up display 704 , a speaker 705 , and an operating device 706 .

[0253] The HCU 701 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The HCU 701 is configured to perform overall control of display output and / or auditory output through the HMI system 7. That is, the HCU 701 is configured to control the operation of each of the instrument panel 702, the main display device 703, the head-up display 704, and the speaker 705.

[0254] The instrument panel 702 is installed in the instrument panel V21 so as to be arranged to face the driver's seat V23. The instrument panel 702 is configured to display metering values ​​including, for example, the speed of the own vehicle V, the temperature of the coolant, and the fuel level. The instrument panel 702 is further configured to display various information items including, for example, the current date and time, the outside temperature, and the receivable radio broadcasts.

[0255] The main display device 703, also referred to as a central information display (CID) device, is installed in the middle of the instrument panel V21 in the vehicle width direction, which enables one or more passengers to visually recognize information displayed thereon.

[0256] The main display device 703 has a housing and a screen mounted on the housing, and can be configured to continuously display a navigation image generated by the navigation system 6 on the screen, the navigation image showing, for example, a map on which the current position and the planned driving route of the own vehicle V appear, respectively. The main display device 703 can also be configured to display various information and content different from the navigation image on the screen. For example, the main display device 703 can be configured to display a driving mode setting image on which icons of multiple driving modes are selectively displayed; the multiple driving modes include a comfort driving mode, a normal driving mode, a sports driving mode, and a circuit driving mode. The main display device 703 is further configured to display a second task image on the screen on which icons of multiple second tasks are selectively displayed; the second task other than the driving task of the own vehicle V can be used by the driver D during the autonomous driving of the own vehicle V. For example, the second task includes (i) a task of reading a digital book, (ii) a task of operating a mobile communication terminal, and (iii) a task of watching video content (e.g., a movie, a concert video, a music video, or a television broadcast). The second task can be referred to as a secondary activity or other task.

[0257] Reference Figure 1 , Figure 2 , Fig.14 and Fig.15 , the head-up display 704 is configured to display a virtual image M including letters, numbers and / or characters in the driver's front field of view. The head-up display 704 of the basic embodiment is configured to project the virtual image M in front of the driver D using AR technology to superimpose the virtual image M on the front scene including the road surface FR located at the driving route of the own vehicle V; AR is an abbreviation for augmented reality. The task of superimposing the virtual image M on the front scene can display information included in the virtual image M, which is related to at least one focus target included in the front scene to be linked to the at least one focus target in position.

[0258] For example, the task of superimposing the virtual image M on the foreground scene may display the information included in the virtual image M while superimposing the information on at least one focus target, or display the information included in the virtual image M while being adjacent to at least one focus target. The at least one focus target is, for example, at least one target object that the driver D driving the own vehicle V should focus on, that is, at least one target object that the driver D driving the own vehicle V should pay attention to. The at least one focus target includes, for example, road surface markings (road markings), road signs, preceding vehicles and / or pedestrians. For example, the head-up display 704 may be configured to superimpose the planned driving route, the driving direction of the own vehicle V, traffic information, and other similar information on the front road surface FR as the focus target.

[0259] The area on the front windshield V18 on which the virtual image M is projected will be referred to as a projection area AP.

[0260] like Fig.14 As shown, the head-up display 704 has a depression angle AD, for example, a positive angle greater than 0°. Fig.14 In the side view shown, the depression angle AD of the head-up display 704 is defined as an angle formed between a virtual horizontal plane passing through the viewpoint EP of the driver D and a virtual plane passing through the viewpoint EP of the driver D and the top edge of the projection area AP. The depression angle AD becomes a positive value when the viewpoint EP looks down at the top edge of the projection area AP, or becomes a negative value when the viewpoint EP looks up at the top edge of the projection area AP.

[0261] The head-up display 704 has a vertical viewing angle AV and a horizontal viewing angle, and the vertical viewing angle AV defines the vertical width of the projection area AP, while the horizontal viewing angle defines the horizontal width of the projection area AP. The horizontal viewing angle is set to be larger than the vertical viewing angle AV so that the projection area AP has a substantially rectangular shape. In the example Fig.14 In the left side view shown, the vertical viewing angle AV may be defined as the angle formed between a virtual line passing through the left viewpoint EP of the driver D and the top edge of the projection area AP and a virtual line passing through the left viewpoint EP of the driver D and the bottom edge of the projection area AP. That is, the virtual viewing angle AV represents an angle range in the vertical direction within which the left viewpoint EP can view the virtual image M, which may be referred to as a vertical viewing angle in the vertical direction.

[0262] like Fig.14 As shown, the head-up display 704 includes a projector 741 and a magnifying optical system 742. The projector 741 is configured to generate a light-based image LV based on a display image signal generated by the HCU 701, and transmit the light-based image to the magnifying optical system 742. The magnifying optical system 742 includes a plurality of optical components, the plurality of optical components including a concave mirror, and an actuator configured to control the alignment of the optical components. The magnifying optical system 742 is configured to project an enlarged image of the light-based image LV onto the front windshield V18 using the optical components, while controlling the alignment of the optical components by the actuator based on the position of the viewpoint EP detected by the driver state monitor 37 to adjust the projection state of the enlarged image on the front windshield V18 accordingly. This causes the enlarged image based on the light-based image LV to be projected in the projection area AP of the front windshield V18. This enables the driver D to visually identify the virtual image M based on the reflected light of the enlarged image projected in the projection area AP of the front windshield V18.

[0263] Fig.15An example of a virtual image M is shown. The virtual image M includes, for example, a transparent base image, one or more information contents M1 contained in the transparent base image, and one or more graphic contents M2 contained in the transparent base image. The one or more information contents M1 are information contents showing, for example, the maximum speed limit of the road Rd on which the own vehicle V is traveling, the current speed of the own vehicle V, the distance to the destination, the estimated arrival time to the destination, and the name of at least one building appearing in the foreground scene. The one or more graphic contents M2 are graphic contents of indicating lines and / or arrows, which are used to display information indicating (i) the driving direction of the own vehicle V and (ii) lane selection.

[0264] The head-up display 704 may be configured to display a virtual image M including overlay content and non-overlay content. Overlay content is image content that is linked to and overlays one or more specific focus targets included in the foreground scene. In contrast, non-overlay content is image content that is not linked to and overlays one or more specific focus targets included in the foreground scene.

[0265] exist Fig.15 In the virtual image M shown as an example, linear graphic content as an example of graphic content M2 is superimposed on the lane in which the own vehicle V is traveling, and information content indicating the maximum speed limit of the current road of the own vehicle V as an example of information content M1 is located at a predetermined position of the virtual image M and is not superimposed on the graphic content M2.

[0266] The above-described HMI system 7 functions as a notification unit for notifying one or more passengers including the driver D in the own vehicle V of information.

[0267] Reference Figure 1 and Figure 2 , the speaker 705 is configured to output a voice related to the information indicated by each of the instrument panel 702, the main display device 703, and the head-up display 704. The speaker 705 may be configured to output a voice unrelated to the information indicated by each of the instrument panel 702, the main display device 703, and the head-up display 704, such as music and / or radio sound.

[0268] The operation device 706 is an input device that is not included in the operation switch 38, and the operation amount and operation state of the operation device 706 are exempted from detection by the operation sensor 35. Specifically, the operation device 706 includes, for example, switches mounted to the periphery of the screen of the housing of the main display device 703, and a transparent touch panel mounted to cover the screen of the main display device 703. The operation device 703 may include switches mounted to the spokes of the steering wheel V24 and a pointing device such as a touch panel mounted to the center console.

[0269] The switches, pointing device, and touch panel of the operating device 706 enable one or more passengers who operate them to input various information items corresponding to the switches, pointing device, and touch panel, respectively.

[0270] The lighting system 8 includes a body ECU 801 , a headlight 802 , and a flasher 803 .

[0271] The body ECU 801 includes a vehicle microcomputer composed of a CPU, ROM, RAM, an input / output (I / O) interface, and other peripheral devices. The body ECU 801 is configured to control how the headlight 802 is lit according to information input from the driving ECU 2 and / or the driving information input unit 3, and to control how the flasher 803 is lit according to information input from the driving ECU 2 and / or the driving information input unit 3, particularly information input from the flasher switch 382.

[0272] The motion control system 9 is configured to control the motion of the own vehicle V, that is, the running behavior of the own vehicle V, according to information input by the driving ECU 2 and / or the driving information input unit 3 .

[0273] Specifically, the motion control system 9 includes, for example, a driving system 91 , a gear shifting system 92 , a braking system 93 , and a steering system 94 .

[0274] The driving system 91 includes a driving ECU 911 and a driving mechanism 912. The driving ECU 911 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface and other peripheral devices. The driving ECU 911 is configured to receive an accelerator signal indicating an acceleration request from an accelerator pedal sensor or a driving ECU 2, and controls the operation of the driving mechanism 912 according to the acceleration request. The driving mechanism 912 is configured to generate a driving power for the vehicle V on which the system is installed to travel. Specifically, the driving mechanism 912 includes an engine, i.e., an internal combustion engine, and one or more motors. That is, the vehicle V on which the system is installed is any one of a gasoline fuel vehicle, a diesel engine vehicle, a biofuel vehicle, a hydrogen engine vehicle, a hybrid vehicle, a battery electric vehicle (BEV), a fuel cell vehicle or other vehicles.

[0275] The shift system 92 includes a shift ECU 921 and a shift mechanism 922. The shift ECU 921 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The shift ECU 921 is configured to receive a shift position signal indicating a current shift position set by a shift lever or a driving ECU 2 from a shift position sensor, and control the operation of the shift mechanism 922 according to the current shift position of the shift lever. The shift mechanism 922 is provided between the drive wheel of the wheel V3 and the driving mechanism 912, and includes an automatic transmission. Specifically, the shift ECU 921 is configured to control the shift mechanism 922 to perform, based on a shift position signal indicating a current shift position set by the shift lever or the driving ECU 2, (i) a first task: to transfer the forward driving power generated by the driving mechanism 912 to the drive wheels so that the own vehicle V moves forward, (ii) a second task: to transfer the reverse driving power generated by the driving mechanism 912 to the drive wheels so that the own vehicle V moves backward, (iii) a third task: to cut off the driving power of the drive wheels to stop the own vehicle V accordingly, and / or (iv) a fourth task: to change the speed ratio between the input speed from the driving mechanism 912 to the shift mechanism 922 and the output speed from the shift mechanism 922 to the drive wheels when the own vehicle V moves forward. The shift system 92 may be configured as a so-called shift-by-wire configuration.

[0276] The braking system 93 includes a braking ECU 931 and a braking mechanism 932. The braking ECU 931 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface, and other peripheral devices. The braking ECU 931 is configured to receive a braking signal indicating a braking request from a brake pedal sensor or a driving ECU 2, and control the operation of the braking mechanism 932 according to the braking request. The braking mechanism 932 includes a friction mechanism of each wheel in the wheels V3. That is, the braking ECU 931 is configured to control the friction mechanism of each wheel V3 according to the braking request to apply friction force to each wheel V3 accordingly, thereby decelerating the own vehicle V. The braking mechanism 932 may include: a regenerative braking mechanism, which is configured to rotate the drive wheel by the kinetic energy of the own vehicle V, to decelerate the own vehicle V accordingly due to the load of the drive wheel rotation, and convert the kinetic energy of the own vehicle V into electricity. The braking system 93 may be configured as a so-called wire control brake configuration.

[0277] The steering system 94 includes a steering ECU 941 and a steering mechanism 942. The steering ECU 941 includes a vehicle microcomputer composed of a CPU, a ROM, a RAM, an input / output (I / O) interface and other peripheral devices. The steering ECU 941 is configured to receive a steering signal indicating a steering request from a steering angle sensor or a driving ECU 2, and control the operation of the steering mechanism 942 according to the steering request. That is, the steering ECU 941 is configured to control the steering mechanism 942 to change each steering wheel, such as the direction of each front wheel V3a, V3b, according to the steering request, so as to change the driving direction of the own vehicle V accordingly. The steering mechanism 942 can be configured to change the direction of each of the front and rear wheels V3a, V3b, V3c and V3d. That is, the own vehicle V can be configured as a four-wheel steering vehicle. The steering system 94 can be configured as a so-called wire control steering configuration.

[0278] Driving control based on the identification result of at least one of the ADAS sensors 31 to 34

[0279] Fig.16 and Fig.17 The example of the functions implemented by the driving ECU 2 is schematically shown. That is, the processor 21 executes the program instructions stored in the memory device 22 to implement the corresponding Fig.16 and Fig.17 Functions shown.

[0280] An outline of autonomous driving of the own vehicle V performed by the driving ECU 2 based on the identification result of at least one of the ADAS sensors 31 to 34 around the own vehicle V is described below.

[0281] Note that in a basic embodiment, "identification" conceptually includes "detection", "classification" and "recognition". Detection is to find the target object B based on at least one detection point data cloud and / or an image captured by the camera device 34. Detection of the target object B is to determine the existence of the target object B without identifying the shape and / or attributes of the target object B. Classification of the target object B is to classify the shape and / or attributes of the detected target object B into various object types, such as one of "human", "vehicle", "building", etc. In other words, a classified target object is a target object that has been detected and classified into one of the various object types. Recognition of the target object B is to determine whether the detected and classified target object B should be considered when driving control of the own vehicle V.

[0282] Detection of an object may include sensing the object. Detection may conceptually include classification and / or identification, classification may conceptually include detection and / or identification, and identification may conceptually include detection and classification.

[0283] Reference Fig.16 , the driving ECU 2 includes an identification module 2001 , an operation determiner 2002 , and a control signal output module 2003 as functional components implemented by the processor 21 .

[0284] The identification module 2001 is operable to perform an identification task of one or more target objects B around the own vehicle V according to information items input from the surrounding monitoring sensors 31 to 34, the operation sensor 35, and the behavior sensor 36. The operation determiner 2002 is operable to determine one or more control tasks currently required to control the own vehicle V based on the identification result of the identification module 2001 and the information items input from the operation sensor 35 and the behavior sensor 36. The one or more control tasks include, for example, a collision avoidance task, an emergency stop task, and a warning task for the driver D.

[0285] The control signal output module 2003 is operable to output control signals based on the determined control tasks to selected components of the vehicle system 1. The control signals include, for example, signals indicating a steering manipulation amount, a braking signal indicating a braking amount, and a message code signal indicating a warning message.

[0286] Reference Fig.17 The identification unit 120 includes an input information acquisition module 2101, an input information processing module 2102, a target object recognition module 2103, a own lane recognition module 2104, an intersection recognition module 2105 and a surrounding environment recognition module 2106.

[0287] The information acquisition module 2101 acquires information items input from the surrounding monitoring sensors 31 to 34, the operation sensor 35, and the behavior sensor 36, and stores the acquired information items in sequence. The input information processing module 2102 applies one or more predetermined tasks, such as a noise removal task and / or a coordinate conversion task, to the information items stored in the information acquisition module 2101.

[0288] The target object recognition module 2103 is operable to perform a target object recognition task according to the information item subject to a predetermined task.

[0289] Specifically, the target object recognition module 2103 includes, for example, a marking line recognition module 2131, a road surface marking recognition module 2132, a roadside structure recognition module 2133, a traffic light recognition module 2134, a traffic sign recognition module 2135, a lane recognition module 2136, a pedestrian recognition module 2137, a surrounding vehicle recognition module 2138 and an obstacle recognition module 2139.

[0290] exist Fig.18 or Fig.19 , a target object B to be identified by the identification module 2001 is shown as an example.

[0291] Reference Fig.18 , the target object B broadly includes, for example, traffic-related three-dimensional objects B1, other vehicles B2, general three-dimensional objects B3, and road surface markings B4. The target object B to be identified by the identification module 2001 further includes, for example, Fig.19 The parking space segmentation lines B5, wheel stoppers B6, and parked vehicles B7 shown in relation to the detection of the parking space PS and / or the execution of the parking assistance. The parking space segmentation lines B5 are, for example, painted lines for defining the shape and area of ​​each parking space PS. Each of the wheel stoppers B6 is a block-shaped solid installed in each parking space PS for stopping the wheels of the corresponding parked vehicle B7. Each of the parked vehicles B7 is a vehicle parked in the corresponding parking space PS.

[0292] The traffic-related three-dimensional objects B1 are three-dimensional objects for road traffic safety, such as traffic lights B11 and traffic signs B12. Each of the other vehicles B2 can become a target vehicle tracked by the traveling own vehicle V, or an obstacle to the traveling own vehicle V, so the parked vehicle B7 is excluded from the other vehicles B2. The general three-dimensional objects B3 are entity objects other than the traffic-related three-dimensional objects B1, other vehicles B2, wheel stoppers B6, and parked vehicles V7, and can mainly constitute obstacles.

[0293] The lane LN may be identified based on the recognition result of the target object B acquired based on the captured image. Specifically, although the parking space PS or the lane LN is different from the target object B that is the direct recognition target of the ADAS sensors 31 to 34, these parking spaces PS and the lane LN may be used as indirect recognition targets based on the recognition result of the target object B. The lane LN that is the indirect target includes, for example, the own lane LNm in which the own vehicle V is traveling and the oncoming lane LNc. The recognition target to be recognized by the identification module 2001 will be described in detail later.

[0294] The own lane recognition module 2104 is operable to recognize the position of the own vehicle V in the road in the width direction of the road based on the recognition result of the target object B by the target object recognition module 2103; the road is the road on which the own vehicle V is traveling. The width direction of the road is referred to as the road width direction, and the direction perpendicular to the road width direction is referred to as the road extension direction. The road extension direction is a direction extending along the road, and may be referred to as a road extension direction or a road elongation direction. If the road Rd includes a plurality of lanes LN, the own lane recognition module 2104 is operable to recognize any one of the plurality of lanes LN arranged in the road width direction as the own lane LNm.

[0295] The intersection recognition module 2105 is operable to recognize an intersection Xr around the own vehicle V based on the recognition result of the target object B by the target object recognition module 2103 .

[0296] Specifically, the intersection identification module 2105 is operable to identify the intersection Xr that the own vehicle V is approaching based on (i) whether there is a traffic light B11, (ii) if it is determined that there is a traffic light B11, which color signal light is output from the traffic light B11, (iii) whether there is a stop line B42 which is one of the road surface markings B4, (iv) the position of the intersection entrance Xr1, (v) the position of the intersection exit Xr2, and (vi) each traffic sign or mark indicating the corresponding direction of travel.

[0297] The surrounding environment recognition module 2106 is operable to recognize the surrounding environment around the own vehicle V, such as how one or more obstacles are located around the own vehicle V, based on the recognition result of the target object B by the target object recognition module 2103.

[0298] These recognition results of the recognition modules 2103 to 2106 are used for the operation determiner 2002, which determines one or more control tasks required to currently control the own vehicle V.

[0299] Fig.18It is an example of a foreground scene from the own vehicle V in a situation where the own vehicle V is approaching an intersection Xr constituting an intersection and the intersection Xr includes a traffic light B11. The intersection Xr constituting an intersection is simply referred to as a cross intersection Xr. For the HMI system 7, the recognition results of the traffic lights B11 and traffic signs B12 included in the traffic-related three-dimensional object B1 can be used to notify the driver D of information and / or output a warning to the driver D. For the driving ECU 2, the recognition results of the traffic lights B11 and traffic signs B12 included in the traffic-related three-dimensional object B1 can also be used in the autonomous driving of the own vehicle V and / or the driving assistance of the own vehicle V to perform motion control of the own vehicle V. In addition to the traffic lights B11 and traffic signs B12, the traffic-related three-dimensional object B1 also includes roadside structures B13 such as guardrails and curbs. The traffic-related entity object B11 also includes such as Fig. 20 The road spikes B14 and the poles B15 are shown. The road spikes may be recognized as obstacles depending on (i) the height of the road spikes and / or (ii) how the road spikes are arranged around the own vehicle V. When the driving ECU 2 performs travel control of the own vehicle V, the poles B15 are recognized as obstacles.

[0300] Reference Fig.18 , the general three-dimensional object B3 includes, for example, a fallen object on the road surface B31, a pedestrian B32, a cyclist B33, and a building B34. When the driving ECU 2 performs the driving control of the own vehicle V, the fallen object B31, the pedestrian B32, and the cyclist B33 on the road surface are recognized as obstacles. The building B34 is not recognized as an obstacle because it is located outside the road Rd, but when the head-up display 704 displays the virtual image M, it may become the content superimposed on the foreground scene.

[0301] The road surface markings B4 include, for example, a pedestrian crossing marking B41, a stop line B42, and a road dividing line B43. The road surface markings B4 also include Fig.21 The letter mark B44 and the symbol mark B45 are shown. The letter mark B44 includes, for example, a number indicating the maximum speed limit, a vehicle traffic area such as "bus only" or a restriction letter such as "STOP". The symbol mark B45 includes, for example, an arrow indicating the corresponding mandatory direction and a warning symbol for a pedestrian crossing ahead.

[0302] The recognition result of the road traffic mark B4 is used to estimate the position of each of the intersection entrance Xr1 and the intersection exit Xr2 and the position of the intersection center Xrc. The intersection entrance Xr1 represents the edge of the focused intersection that the own vehicle V is about to enter. The focused intersection is the nearest intersection Xr that (i) the own vehicle V is approaching and (ii) the own vehicle V plans to pass through or may pass through. The intersection that the own vehicle V may pass through is an intersection that the own vehicle V is estimated to pass with a high probability based on (i) the distance from the own vehicle V to the intersection and (ii) the speed of the own vehicle V when the navigation system 6 has not determined the planned driving route and destination of the own vehicle V. The intersection exit Xr2 represents the edge of the focused intersection that the own vehicle V is about to leave. The intersection center Xrc is the center of the focused intersection.

[0303] like Fig. 22 and Fig.23 As shown, the road demarcation lines B43 include, for example, vehicle-road edge lines B431, center lines B432, and lane lines B433. Vehicle-road edge lines B431 are provided for each vehicle-road, and they respectively show the left edge and the right edge of the corresponding vehicle-road in the road width direction. The vehicle-road of the road Rd represents the portion of the road Rd where vehicles can travel. Each center line B432 represents a line running downward along the center of the two-way road, and divides the two-way road into two parts in corresponding opposite directions of travel. Each center line B432 is composed of a single straight line pattern or a single dotted line pattern, a double line pattern, or a triple line pattern. Lane lines B433 are provided for a road Rd having two or more lanes in each direction, and they divide the road Rd into two or more lanes in each direction. Each of the lane lines B433 is a white painted line or a yellow painted line. As an example, Fig. 20 The center line B432 is shown to be composed of a three-line pattern consisting of a solid white line and a pair of yellow lines on both sides of the white line. The road studs B14 and the rods B15 are installed on Fig. 20 The white line of the three-line pattern of the center line B432 shown. As another example, Fig.21 It is shown that the center line B432 is composed of a double line pattern composed of a white solid line and the lane line B433 is composed of a white dashed line and a yellow solid line extending from the white dashed line. Based on the recognition results of the roadside structure B13 and the road division line B43, it is possible to recognize (i) the driving lane LNd or the own lane LNm in which the own vehicle V is traveling (see, for example Fig. 20 and Fig.21 ), (ii) the oncoming lane LNc (see e.g. Fig. 20 and Fig.21 ), (iii) through lane LNp (see e.g. Fig. 20), (iv) right turn lane LNr (see Fig.21 ), (v) shoulder LNs (see Fig. 20 and Fig.21 ), and (vi) an emergency stop zone EZ (not shown).

[0304] The road traffic signs B4 further include, for example, diversion area signs, namely, a zebra crossing area sign B461 , a safety area sign B462 , a no-entry area sign B463 , and a no-stop area sign B464 .

[0305] The diversion area mark B461 is painted on the road and indicates a diversion area where the safe and smooth operation of the vehicle needs to be guided. As described above, the diversion area mark B461 is a mark indicating a diversion area for guiding the safe and smooth operation of the vehicle, and the vehicle is not legally prohibited from entering the diversion area. The diversion area mark B461 can be set adjacent to (i) an intersection Xr, (ii) a junction of roads, or (iii) a fork in the road.

[0306] Fig.24 The safety zone mark B462 shown is used to indicate a safety zone. The safety zone represents an area set up on the road to ensure the safety of, for example, pedestrians B32, so vehicles cannot legally enter the safety zone.

[0307] Fig.25 The prohibited entry area mark B463 shown is used to indicate a prohibited entry area. The prohibited entry area indicates an area on the road that cannot be used for vehicle driving, so vehicles cannot enter the prohibited entry area.

[0308] Fig.26 The shown no-stopping area mark B464 is used to indicate a no-stopping area. The no-stopping area represents an area on the road that a vehicle should not enter when the vehicle may have to stop according to the conditions in front of the vehicle. That is, when any vehicle is waiting for a traffic light or is in a traffic jam, the vehicle should not stop within the no-stopping area mark B464.

[0309] Fig. 27 An example of a traffic sign B12 is shown. Each of the traffic signs B12 is installed adjacent to a corresponding one or more roads Rd, or installed above a corresponding one or more roads Rd. Each of the traffic signs B12 is composed of a display panel designed to provide corresponding necessary information to a user facing a corresponding one of the traffic signs B12. The display panel of each traffic sign B12 has (i) a designed shape, (ii) one or more colors painted on at least one main surface thereof, and (iii) one or more symbols. The one or more colors include, for example, white, black, red, blue, and yellow. Fig. 27In the traffic sign B12, cross hatching indicates red, oblique hatching indicates blue, dot hatching indicates yellow, and shaded portions indicate black. Each symbol used in the traffic sign B12 includes, for example, one or more letters, one or more numbers, and a combination of one or more letters and one or more numbers.

[0310] Fig. 27 Examples of traffic signs B12 in Japan (JP), the European Union (EU), and the United States (US) are shown, classified by meaning. As typical examples of traffic signs B12 in the EU, examples of traffic signs B12 in Germany and / or France are used.

[0311] For example, the corresponding traffic signs of JP, EU, and US meaning "Do Not Enter" or "No Entry" have substantially the same design. In contrast, the corresponding traffic signs of JP, EU, and US meaning "Stop" have substantially the same color and symbol, and the shape of the plate of the traffic sign of EU is substantially the same as that of US, but the shape of the plate of the traffic sign of JP is different from that of EU and US.

[0312] The corresponding JP and EU traffic signs using the letters "50" to mean "maximum speed limit" have basically the same design except for the color of the letters, but the US traffic sign using the letters "50" to mean "speed limit" is different from the corresponding JP and EU in the shape and color of the plate.

[0313] The corresponding traffic signs of JP, EU, and US meaning "Railway Crossing Caution" have a low level of commonality in design.

[0314] The corresponding traffic signs of EU and US meaning "No Right Turn" have basically the same design, but JP has no corresponding traffic sign.

[0315] JP's traffic sign means "drive only in the direction of the arrow" and uses white arrows to indicate the forward direction and left turn direction, such as Fig. 27 As shown, it similarly means "no right turn".

[0316] The accurate recognition of the traffic sign B12 enables smooth and safe implementation of autonomous driving and / or advanced driving assistance. As described above, the design of some of the traffic signs B12 varies greatly between JP, EU and US. For this reason, the memory device 22 pre-stores at least one database, which stores information indicating patterns of traffic signs used in, for example, each of all countries in the world or each of all regions in the world. Pattern matching between the identified traffic sign B12 and the information stored in at least one database enables detection of the meaning of the identified traffic sign B12. The at least one database may be configured as a public database for the whole world, or may be composed of a plurality of databases provided for all countries in the world, each of which stores information indicating patterns of traffic signs used in a corresponding one of the countries in the world. Instead of the memory device 22 or in addition to the memory device 22, at least one database may be stored in a host computer; the driving ECU 2 is communicatively connected to the host computer through the vehicle communication module 4 based on vehicle-to-everything technology (V2X). This enables the driving ECU 2 to freely access at least one database stored in the host computer.

[0317] Fig.28 A first example is schematically shown indicating how to use the recognition result of the target object B in a situation where the own vehicle V is approaching an intersection. In this situation, the head-up display 704 can superimpose the information content M1 indicating the message "10m from the stop position" indicating the distance 10m from the stop line B42 on the stop line B42. In addition, the HMI system 7 can output a voice message indicating "10m from the stop line" using the speaker 705. This effectively assists the driver D who uses the advanced driving assistance to reliably stop the own vehicle V before the stop line B42. Alternatively, the voice message output from the HMI system 7 enables one or more passengers of the own vehicle V to reliably grasp the occurrence of the temporary stop event of the own vehicle V in advance, thereby avoiding a reduction in the comfort of one or more passengers of the own vehicle V.

[0318] Fig.29A second example is schematically shown indicating how to use the recognition result of the target object B in a situation where the own vehicle V is approaching an intersection with a traffic light B11 and the own vehicle V is going to turn right at the intersection. In this situation, the head-up display 704 can superimpose information content M1 indicating a message "turn right at the intersection 20m ahead" on the own lane LN, that is, the right-turn lane LNr of the road surface FR in front of the own vehicle V. The HMI system 7 can also output a voice message indicating "turn right at the intersection 20m ahead" using the speaker 705. In addition, the HMI system 7 can superimpose graphic content M2 of an arrow extending from the right-turn lane LNr through the center Xrc of the intersection or its vicinity to the target road Rd on the road surface FR. This effectively provides one or more passengers including the driver D of the own vehicle V with advance notice of the travel direction guidance of the own vehicle V and / or the behavior of the own vehicle V undergoing lateral acceleration during the right turn. This therefore enables one or more passengers of the own vehicle V to reliably grasp the occurrence of the behavior of the own vehicle V in advance, thereby avoiding a reduction in the comfort of one or more passengers of the own vehicle V.

[0319] Reference Fig.17 The target object recognition module 2103 is operable to recognize the type and state of at least one target object B based on the recognition result of the at least one target object B and the HD map information stored in the HD map information database 5. The state of the at least one target object B includes, for example, position information about the at least one target object B. The position information about the at least one target object B includes, for example, the distance of the at least one target object B relative to the own vehicle V, the relative position of the at least one target object B relative to the own vehicle V, and the position of one of the lanes LN in the case where the at least one target object B is one of the multiple lanes LN.

[0320] Specifically, the marking line recognition module 2131 of the target object recognition module 2103 is operable to recognize a road marking line B43 in a peripheral area around the own vehicle V, the peripheral area including a road surface FR of a road Rd located at a travel route of the own vehicle V. For example, the marking line recognition module 2131 is operable to recognize, for example Figure 20 to Figure 23 Each of the vehicle-road edge line B431, the center line B432 and / or the lane line B433 shown is (i) white or yellow, (ii) solid or dashed, (iii) single-line or multi-line. Because the recognition technology of the road marking line B43 is well known at the time of filing this application, a detailed description of the recognition technology of the road marking line B43 is omitted in the present disclosure.

[0321] The road surface marking recognition module 2132 of the target object recognition module 2103 is operable to recognize the type, meaning and position of each of the road surface markings B4 except the road marking line B43; the recognition targets of the road surface marking recognition module 2132 include, for example, a crosswalk marking B41, a stop line B42 and a symbol marking B45.

[0322] The roadside structure identification module 2133 is operable to identify the type and location of at least one roadside structure B13 located around the own vehicle V based on, for example, (i) a recognition result based on an image captured by the camera device 34, (ii) a recognition result based on at least one detection point data cloud detected by the lidar sensor 33, and (iii) HD map information stored in the HD map information database 5.

[0323] The traffic light identification module 2134 is operable to identify (i) whether the traffic light B11 is located at the driving route of the own vehicle V, and (ii) if it is identified that the traffic light B11 is located at the driving route of the own vehicle V, identify the location of the traffic light B11 and which of the color signal lights is output from the traffic light B11.

[0324] The traffic sign recognition module 2135 is operable to recognize a traffic sign B12 located around the own vehicle V.

[0325] The lane recognition module 2136 is operable to perform a lane recognition task: identifying the number of lanes LN in the road Rd on which the own vehicle V is traveling and the type of each lane LN in the road Rd. That is, the lane recognition module 2136 is operable to perform the lane recognition task based on, for example, (i) a recognition result based on an image captured by the front camera CF of the camera 34, (ii) a recognition result based on at least one detection point data cloud detected by the laser radar sensor 33, and (iii) the HD map information stored in the HD map information database 5.

[0326] Specifically, the lane recognition module 2136 can operate normally to perform a lane recognition task based on a recognition result based on an image captured by the camera device 34, or, if necessary, perform a sensor fusion lane recognition task based on a combination of (i) a recognition result based on an image captured by the front camera CF of the camera device 34 and (ii) a recognition result based on at least one detection point data cloud detected by the lidar sensor 33 and (iii) at least one of the HD map information stored in the HD map information database 5.

[0327] The pedestrian recognition module 2137 is operable to recognize one or more pedestrians B32 located around the own vehicle V. The surrounding vehicle recognition module 2138 is operable to recognize one or more other vehicles B2 located around the own vehicle V. The obstacle recognition module 2139 is operable to recognize one or more obstacles, such as one or more fallen objects B31 on the road surface and / or one or more pedestrians B32 located around the own vehicle V. How each of the traffic light recognition module 2134, the traffic sign recognition module 2135, the lane recognition module 2136, the pedestrian recognition module 2137, the surrounding vehicle recognition module 2138 and the obstacle recognition module 2139 recognizes the corresponding one or more target objects is well known at the time of filing this application, and therefore a detailed description of how each of the modules 2134 to 2139 recognizes the corresponding one or more target objects is omitted in this disclosure.

[0328] First embodiment

[0329] The first embodiment is described below. In each of the following embodiments, the combination of (i) the driving ECU 2 used as a vehicle device according to the corresponding embodiment, (ii) the computer program, i.e., computer program instructions to be executed by the driving ECU 2, i.e., the processor 21 according to the corresponding embodiment, and (iii) the storage medium storing the computer program according to the corresponding embodiment will also be collectively referred to as the present embodiment.

[0330] Fig.30 is a schematic block diagram of a parking space detection device 2200 according to the first embodiment. The parking space detection device 2200 is configured to detect a parking space PS located around the own vehicle V based on an image captured by at least one of the cameras 34 during a parking assist operation or an automatic parking operation by the driving ECU 2. The parking assist operation may include: (i) automatic parking of the own vehicle V in a selected target parking space PS without a driving operation by the driver; and (ii) semi-automatic parking of the own vehicle V in the selected target parking space PS under the following circumstances: (i) partial driving operation by the driver (e.g., operation of an accelerator pedal of the own vehicle V); and (ii) no driving operation by other drivers (e.g., steering operation by the driver). The parking space detection device 2200 may be implemented as included in Fig.17 Specifically, the parking space detection device 2200 can be implemented as Fig.17 A portion of the road surface marking recognition unit 2132 shown or Fig.17A portion of the surrounding environment recognition module 2106 is shown. Alternatively, the road surface marking recognition unit 2132 and the surrounding environment recognition module 2106 can be used as the parking space detection device 2200.

[0331] The following describes the details of the parking space detection device 2200 according to the present embodiment. The parking space detection device 2200 includes an annotation information generator 2201, a learning unit 2202, and a parking space determiner 2203. In addition, the parking space detection device 2200 includes an image information storage device 2204, an image information input unit 2205, an annotation information storage device 2206, and a learning model storage device 2207.

[0332] The annotation information generator 2201 is configured to generate annotation information corresponding to information about each of the images for training (which will be referred to as training images or learning images) stored in the image information storage device 2204, and output the annotation information to the annotation information storage device 2206. The annotation information about each learning image represents so-called ground truth data indicating the content and position of one or more target objects B in the corresponding learning image.

[0333] In this embodiment, the one or more target objects B are one or more parking spaces PS. That is, the annotation information generator 2201 is configured to generate ground truth data indicating the content and position of the one or more parking spaces PS in the corresponding learning image, i.e., ground truth annotation data as annotation information about each learning image.

[0334] That is, each learning image includes one or more parking spaces PS, and each learning image is paired with corresponding ground truth annotation data indicating the content and location of the one or more parking spaces PS in the corresponding learning image.

[0335] The learning model storage device 2207 includes, for example, a machine learning model LM, such as a deep neural network model (DNN model) with variable parameters. The machine learning model LM stored in the learning model storage device 2207 is configured to output inferred annotation data indicating the content and position of one or more parking spaces PS in the input image when input data, i.e., an input image, is input thereto.

[0336] In other words, the learning unit 2202 is configured to learn recognition rules of one or more parking spaces PS in the input image using the machine learning model LM stored in the learning model storage device 2207.

[0337] In order to improve the inference accuracy of the machine learning model LM, that is, the learning accuracy, it is necessary to input many learning images (each learning image includes one or more actual parking spaces PS) into the machine learning model LM, so that the inferred annotation data output from the machine learning model LM is basically the same as the ground truth annotation data of one or more actual parking spaces PS.

[0338] The learning unit 2202 is configured to perform training of parameters of the machine learning model LM stored in the learning model storage device 2207 based on each learning image stored in the image information storage device 2204 and the corresponding ground truth annotation information, i.e., the corresponding ground truth annotation data, generated by the annotation information generator 2201. Then, the learning unit 2202 is configured to store the trained machine learning model LM having the trained parameters in the learning model storage device 2207.

[0339] The image information input unit 2205 is configured to input one or more images captured by at least one camera 34 to the parking space determiner 2203 .

[0340] The parking space determiner 2203 is configured to perform a parking space determination / detection task using the trained machine learning model LM and one or more images captured by at least one camera 34 and input thereto by the image information input unit 2205 .

[0341] The annotation information generator 2201 includes a bird's-eye view image generator 2211 , an annotation input unit 2212 , and a parking space annotation generator 2213 .

[0342] The bird's-eye image generator 2211 is configured to convert at least one learning image stored in the image information storage device 2204 into a bird's-eye image using, for example, a known viewpoint conversion method. Such a bird's-eye image of the target is, for example, an image of the target observed from above the target along a line of sight parallel to the direction of gravity applied to the target. That is, the bird's-eye image of the target is an image corresponding to a plan view of the target.

[0343] The annotation input unit 2212 is configured to input annotation information corresponding to the bird's-eye image generated by the bird's-eye image generator 2211 to the parking space annotation generator 2213. For example, the annotation input unit 2212 may perform an image recognition operation on the bird's-eye image to automatically generate annotation information corresponding to the bird's-eye image. The annotation input unit 2212 may include a human-machine interface that enables an annotation information input worker to input correction information for correcting the automatically generated annotation information. Then, the annotation input unit 2212 may correct the automatically generated annotation information according to the correction information input by the annotation information input worker through the human-machine interface.

[0344] Alternatively, the annotation input unit 2212 may include a human-machine interface that enables such an annotation information input staff to input information used to generate the generated annotation information, and the annotation input unit 2212 may generate the generated annotation information based on the information input by the annotation information input staff through the human-machine interface.

[0345] The parking space annotation generator 2213 is configured to generate annotation information corresponding to at least one parking space PS included in the bird's-eye image, and output the annotation information corresponding to the at least one parking space PS included in the bird's-eye image to the annotation information storage device 2206 .

[0346] Specifically, the parking space annotation generator 2213 includes, for example, an angle position generator 2214 and an attribute generator 2215 .

[0347] The angular position generator 214 is configured to generate a component in the bird's-eye view image generated by the bird's-eye view image generator 2211. Fig.31 The shape of the parking space PS is shown with position information about parking space corner points BP and position information about parking space center point BPC. The parking space corner points BP include left front corner point BP1, right front corner point BP2, left rear corner point BP3 and right rear corner point BP4.

[0348] Refer to the following Fig.31 The following describes the definition of each of the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, the right rear corner point BP4, and the parking space center point BPC. Fig.31 For simplicity, Fig.31 The parking space PS shown in FIG. 1 is surrounded by a rectangular parking segment B5 formed by a front segment line B501, a rear segment line B502, and a pair of side segment lines B503. However, as described later, the parking space PS is not limited to being surrounded by a rectangular parking space frame B5. Specifically, the parking space PS may be composed of a substantially U-shaped parking segment B5 or of a plurality of linear lines B5 parallel to each other. Alternatively, the parking space PS may be composed of a space formed between parked vehicles B7 arranged adjacent to each other.

[0349] The front segmentation line B501 and the rear segmentation line B502 are located at the corresponding short sides of the rectangular parking space PS. The front segmentation line B501 is located in front of the rear segmentation line B502 in the parking space direction BD. If there is a parked vehicle B7 in the parking space PS, the parking space direction BD of the parking space PS is defined as the forward direction of the parked vehicle B7 in the parking space PS. The parking space direction BD of the parking space PS may be defined according to the configuration and arrangement of the parking space PS. For example, if the wheel stopper B6 is disposed closer to one of the short sides of the rectangular parking space PS, the parking space direction BD is defined as from the closer short side to the farther short side. As another example, if there are parking spaces PS arranged so that their short sides are aligned to constitute a parallel parking area, the parking space direction BD of each parking space PS is defined as from the parking space PS to the public parking passage. As another example, if there are parking stalls PS arranged so that their long sides are aligned to constitute a parallel parking area, the parking stall direction BD of each parking stall PS is defined as the traveling direction of the corresponding parked vehicle.

[0350] The pair of side segment lines B503 are located at the corresponding long sides of the rectangular parking space PS. If the parking space direction BD of the parking space PS is defined as the forward direction, the left segment line among the side segment lines B503 will be referred to as the left segment line B504, and the right segment line among the side segment lines B503 will be referred to as the right segment line B505.

[0351] The left front corner point BP1 is a corner point at the position where the front segment line B501 and the left segment line B504 intersect each other. The right front corner point BP2 is a corner point at the position where the front segment line B501 and the right segment line B505 intersect each other. The left rear corner point BP3 is a corner point at the position where the rear segment line B502 and the left segment line B504 intersect each other. The right rear corner point BP4 is a corner point at the position where the rear segment line B502 and the right segment line B505 intersect each other.

[0352] The parking space center point BPC is the center point of a rectangle formed by connecting the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, and the right rear corner point BP4. Specifically, the parking space center point BPC is the intersection point between a virtual straight line connecting the left front corner point BP1 and the right rear corner point BP4 and a virtual straight line connecting the right front corner point BP2 and the left rear corner point BP3. If the parking space PS is not surrounded by the rectangular parking segment B5, each corner point of the parking space PS may be defined in the same manner as in the case where the parking space PS is surrounded by the rectangular parking segment B5.

[0353] For example, if the parking space PS is surrounded by a substantially U-shaped parking segment B5 and lacks the front segment line B501 compared to the rectangular parking segment B5, the left front corner point BP1 may be defined as the front end of the left segment line B504. Similarly, if the parking space PS is surrounded by a substantially U-shaped parking segment B5 and lacks the rear segment line B502 compared to the rectangular parking segment B5, the right rear corner point BP4 may be defined as the rear end of the right segment line B505. If there is no parking segment B5, so that the parking space PS is constructed as a space formed between the parked vehicles B7 arranged adjacent to each other, each corner point of the parking space PS may be defined by a circumscribed rectangle around the parked vehicles B7 in the parking space PS or by a virtual rectangular space formed between the parked vehicles B7 arranged adjacent to each other.

[0354] If the parking space PS is included in the bird's-eye image generated by the bird's-eye image generator 2211, the attribute generator 2215 is configured to generate attribute information about the corresponding parking space PS for each parking space PS. The attribute information about each parking space PS may include, for example, parking segment information about the corresponding parking space PS. The parking segment information about each parking space PS may include, for example, (i) whether the corresponding parking segment B5 exists, and (ii) whether a segment line component, such as the front segment line B501 or the rear segment line B502, exists in the corresponding parking segment B5. In order to avoid redundant and complicated descriptions, other items of attribute information other than the parking segment information will be described in the second embodiment and subsequent embodiments.

[0355] The bird's-eye view image generator 2211 is configured to convert the learning image stored in the image information storage device 2204 into a bird's-eye view target image, which is also referred to as a target image (see Fig.32 The target image GS is used to train the machine learning model LM and input annotation information.

[0356] Fig.32 1 is an example of a learning image including the own vehicle, that is, the system-installed vehicle V, which has entered the parking lot PL, and a plurality of parking spaces PS arranged around the own vehicle V. Fig.33 Shown on a large scale Fig.32 A portion of the learning image is shown. Specifically, Fig.33 The selected parking space PS and its surroundings located diagonally to the left and rear of the host vehicle V are shown on a large scale; the selected parking space PS is the parking space where the host vehicle V is to be parked.

[0357] The parking space annotation generator 2213 is configured to generate annotation information about the corresponding parking space PS in the target image GS according to the annotation information input thereto from the annotation input unit 2212 for each parking space PS included in each target image GS. Specifically, Fig.33 As shown, the parking space annotation generator 2213 generates, for each parking space PS, position information about at least one of the parking space corner point BP and the parking space center BPC in each target image GS as parking space annotation information. The parking space annotation generator 2213 is also configured to output the parking space annotation information about each parking space PS of each target image GS to the annotation information storage device 2206, so that the parking space annotation information about each parking space PS of each target image GS is stored in the annotation information storage device 2206. The attribute generator 2215 is configured to generate, for each parking space PS included in each target image GS, attribute information about the corresponding parking space PS included in the corresponding target image GS, and output the attribute information about each parking space PS of each target image GS to the annotation information storage device 2206, so that the attribute information about each parking space PS of each target image GS is stored in the annotation information storage device 2206.

[0358] The learning unit 2202 includes a bird's-eye view image generator 2221 and a parking space training unit 2222 .

[0359] The bird's-eye image generator 2221 is configured to convert at least one learning image stored in the image information storage device 2204 into a training bird's-eye image using, for example, a known viewpoint conversion method. The configuration and function of the bird's-eye image generator 2221 are substantially the same as those of the bird's-eye image generator 2211. Therefore, the bird's-eye image generators 2221 and 2222 may be configured as a common module.

[0360] The parking space training unit 2222 is configured to perform training on parameters of the machine learning model LM stored in the learning model storage device 2207 based on each target image GS generated by the bird's-eye image generator 2221 and the annotation information (i.e., ground truth data) for the corresponding target image GS generated by the parking space annotation generator 2213 and stored in the annotation information storage device 2206, so as to improve the parking space reasoning accuracy of the machine learning model LM stored in the learning model storage device 2207, i.e., the parking space recognition accuracy.

[0361] The following describes in detail the training process of the parameters of the machine learning model LM stored in the learning model storage device 2207, which is performed by the parking space training unit 2222 to accordingly improve the accuracy of identifying at least one parking space PS included in the input image.

[0362] First, the training process uses multiple grid lines GL will be as follows Fig.32 The target image GS shown as an example is divided into a plurality of grids GR. For the sake of simplicity and description, Fig.32 As shown, the training process divides the target image GS into a 10 by 10 grid GR, i.e., a 10×10 grid GR using nine vertical grid lines GL and nine horizontal grid lines GL. The 10×10 grid GR has 10 cells in the vertical direction, i.e., 10 rows, and 10 cells in the horizontal direction, i.e., 10 columns.

[0363] Fig.33 Shown on a large scale are the Fig.32 3×3 grid GR in the 10×10 grid GR shown in FIG. Fig.33 In the 3×3 grid GR shown as an example in , the two parking space centers BPC are respectively located in the central cell of the 3×3 grid GR and the top left cell of the 3×3 grid GR.

[0364] In this example, the training process objectifies the parking space PS selected based on the parking space center BPC located in the central cell of the 3×3 grid GR into a set of corner points BP1, BP2, BP3 and BP4 located in the corresponding top middle cell, middle right cell, middle left cell and bottom left cell.

[0365] Then, the training process encodes the existence of the parking space center BPC as the first type of first encoded data Ec1 indicating whether the parking space center BPC exists in each cell of the 3×3 grid GR, which is Fig.34 is shown as "Center: Responsible". Specifically, the training process encodes the existence of the parking space center BPC as the first type of first encoded data Ec1, in which the encoded data value Ecv of each of the center cell and the top left cell of the 3×3 grid GR is set to "1" and the encoded data values ​​Ecv of the other cells of the 3×3 grid GR (i.e., the top middle cell, the top right cell, the middle left cell, the middle right cell, the top left cell, the bottom middle cell, and the bottom right cell) are respectively set to "0".

[0366] Fig.35 and Fig.36 shows how the training process encodes the horizontal position of the parking space center BPC into the first encoded data Ec1 of the second type in the 3×3 grid GR. Fig.36 is shown as "Center:OffsetX".

[0367] The training process according to the first embodiment defines the horizontal position of the parking space center BPC in the center cell of the 3×3 grid GR relative to the left side of the center cell, and the horizontal position of the parking space center BPC in the top left cell of the 3×3 grid GR relative to the left side of the top left cell. The horizontal position of any cell in the 3×3 grid GR is defined so that the left side is set to the first reference encoded data value "0" and the right side is set to the second reference encoded data value "1". Specifically, a virtual line segment between the left and right sides of each of the center cell and the top left cell of the 3×3 grid GR is defined; the virtual line segment passes through the corresponding parking space center BPC and is parallel to the horizontal grid line GL. The parking space center BPC of each of the center cell and the top left cell of the 3×3 grid GR represents an internal division point of the corresponding virtual line segment. The horizontal position of the parking space center BPC of each of the center cell and the top left cell of the 3×3 grid GR is defined as an internal division ratio expressed as a decimal. For example, for the center cell, since the internal division ratio of the parking space center BPC in the center cell is 1:1, the horizontal position of the parking space center BPC in the center cell is set to 0.5. Similarly, for the top left cell, since the internal division ratio of the parking space center BPC in the top left cell is 2:3, the horizontal position of the parking space center BPC in the center cell is set to 0.4.

[0368] Then, the training process performs the first encoding of the five points BPC, BP1, BP2, BP3, and BP4 with respect to the target image GS.

[0369] Specifically, the training process encodes the horizontal position of the parking space center BPC into the second type of first encoded data Ec1, where (i) the horizontal position of the center unit is set to "0.5" as the encoded data value Ecv, (ii) the horizontal position of the top left unit is set to "0.4" as the encoded data value Ecv, and (iii) the encoded data values ​​Ecv of other units are respectively set to "0".

[0370] Fig.37 and Fig.38 shows how the training process encodes the vertical position of the parking space center BPC into the third type of first encoded data Ec1 in the 3×3 grid GR. Fig.38 Shown as "Center:Offset Y".

[0371] The training process according to the first embodiment defines the vertical position of the parking space center BPC in the central cell of the 3×3 grid GR relative to the top side of the central cell, and the vertical position of the parking space center BPC in the top left cell of the 3×3 grid GR relative to the top side of the top left cell. The vertical position of any cell in the 3×3 grid GR is defined so that the top side is set to the first reference encoded data value "0" and the bottom side is set to the second reference encoded data value "1". Specifically, a virtual line segment between the top side and the bottom side of each of the central cell and the top left cell of the 3×3 grid GR is defined; the virtual line segment passes through the corresponding parking space center BPC and is parallel to the vertical grid line GL. The parking space center BPC of each of the central cell and the top left cell of the 3×3 grid GR represents an internal division point of the corresponding virtual line segment. The vertical position of the parking space center BPC of each of the central cell and the top left cell of the 3×3 grid GR is defined as an internal division ratio expressed as a decimal. For example, for the center cell, since the internal division ratio of the parking space center BPC in the center cell is 1:1, the vertical position of the parking space center BPC in the center cell is set to "0.5". Similarly, for the top left cell, since the internal division ratio of the parking space center BPC in the top left cell is 1:5, the vertical position of the parking space center BPC in the center cell is set to "0.2".

[0372] The training process then encodes the vertical position of the parking space center BPC into the third type of first encoded data Ec1, in which (i) the vertical position of the center unit is set to "0.5" as the encoded data value Ecv, (ii) the vertical position of the top left unit is set to "0.2" as the encoded data value Ecv, and (ii) the encoded data values ​​Ecv of other units are respectively set to "0".

[0373] Fig.39 Shown in Fig.32 The entire target image GS shown in Fig.34 All existing data items of the parking space center BPC of the first type of first encoded data Ec1 shown in FIG. Fig.40 Shown in Fig.32 The entire target image GS shown in Fig.36 All horizontal position data items of the parking space center BPC of the second type first encoded data Ec1 shown in FIG. Fig.41 Shown in Fig.32 The entire target image GS shown in Fig.39 All vertical position data items of the parking space center BPC of the third type of first encoded data Ec1 shown in FIG.

[0374] Therefore, (i) existence data items, (ii) horizontal position data items and (iii) vertical position data items of the parking space center BPC of the first type of first encoded data Ec1, the second type of first encoded data Ec1 and the third type of first encoded data Ec1 in the target image GS have been obtained.

[0375] Similar to the parking space center BPC, (i) existence data items, (ii) horizontal position data items and (iii) vertical position data items of each of the left front corner point BP1, right front corner point BP2, left rear corner point BP3 and right rear corner point BP4 of the first type of first encoded data Ec1, the second type of first encoded data Ec1 and the third type of first encoded data Ec1 in the target image GS have been obtained.

[0376] That is, the first encoded data Ec1 includes: (i) the existence data item, vertical position data item and horizontal position data item of the parking space center BPC; (ii) the existence data item, vertical position data item and horizontal position data item of the left front corner point BP1; (iii) the existence data item, vertical position data item and horizontal position data item of the right front corner point BP2; (iv) the existence data item, vertical position data item and horizontal position data item of the left rear corner point BP3; (v) the existence data item, vertical position data item and horizontal position data item of the right rear corner point BP4.

[0377] Fig.42 The parking space annotation code Ea as annotation information of the objectified parking space PS included in the target image GS is schematically shown, and includes first encoded data Ec1 associated with each of five points BPC, BP1, BP2, BP3, and BP4.

[0378] As described above, the parking space training unit 2222 is configured to generate a total of three-channel outputs of (i) existence data, (ii) horizontal position data, and (iii) vertical position data as the first encoded data Ec1 for each of the five points BPC, BP1, BP2, BP3, and BP4.

[0379] As described above, the parking space training unit 2222 is configured to obtain as a result of the first encoding of the five points BPC, BP1, BP2, BP3 and BP4 of the target image GS: (i) the first encoded data item Ec1 of the parking space center BPC; (ii) the first encoded data item Ec1 of the left front corner point BP1; (iii) the first encoded data item Ec1 of the right front corner point BP2; (iv) the first encoded data item Ec1 of the left rear corner point BP3; and (v) the first encoded data item Ec1 of the right rear corner point BP4.

[0380] Next, the second encoding of the connection vector connecting from the parking space center BPC to each of the corner points BP1, BP2, BP3, and BP4 is described below.

[0381] Fig.43 A selected 3×3 grid GR comprising nine cells in a 10×10 grid GR is shown on a large scale; the nine cells comprised of Fig.42 The selected cell (ie, the central cell) and the eight cells surrounding the selected cell are shown by the thick-line square in . Fig.43 The 3×3 grid GR shown in Fig.33 The 3×3 grid GR shown in FIG. Fig.43 The arrangement of the five points BPC, BP1, BP2, BP3 and BP4 shown in Fig.33 The arrangement of the five points BPC, BP1, BP2, BP3 and BP4 shown in FIG. 1 is the same. Fig.43 The thick square in the figure shows the unit Fig.42 The units shown by the bold squares in FIG. 1 are identical.

[0382] Fig.44 An example of the connection number table Tb for performing the second encoding is shown. Fig.45 and Fig.46 Each of shows the result of the second encoding constituting the second encoded data Ec2.

[0383] exist Fig.43 In FIG. 1 , the connection vector from the parking space center BPC as the starting point to the left front corner point BP1 is shown by a dotted arrow. Fig.43 , the connection vector has the left front corner point BP1 as an end point located in a cell positioned adjacent to the upper side of the center cell where the parking space center BPC is located.

[0384] When performing the second encoding of the connection vector directed from the parking space center BPC as the starting point to the left front corner point BP1, the training process will Fig.44 The connection number table Tb shown in is applied to the second encoding of the connection vector. The connection number table Tb has a 3×3 array corresponding to the 3×3 grid GR, and has connection numbers Tbn of 0, 1, 2, 3, 4, 5, 6, 7, and 8 respectively assigned to the top left unit, the top middle unit, the top right unit, the middle left unit, the center unit, the middle right unit, the bottom left unit, the bottom middle unit, and the bottom right unit. That is, the connection numbers Tbn of 0, 1, 2 are sequentially assigned to the respective cells of the top row of the connection number table Tb, the connection numbers Tbn of 3, 4, 5 are sequentially assigned to the respective cells of the middle row of the connection number table Tb, and the connection numbers Tbn of 6, 7, 8 are sequentially assigned to the respective cells of the bottom row of the connection number table Tb.

[0385] When the general Fig.44 When the connection number table Tb shown in FIG. 1 is applied to the second encoding of the connection vector from the parking space center BPC as the starting point to the left front corner point BP1, since the connection vector points from the connection number Tbn of 4 to the connection number Tbn of 1, the training process can be represented by the connection number Tbn of 1. Fig.42 and Fig.43 The connection vector shown in .

[0386] like Fig.45 As shown, the training process is directed to the connection number Tbn from 0 to 8. Fig.44 Each of the cells of the 10×10 grid GR shown in FIG. 1 is assigned a coded data value Ecv of 0 or 1 to generate a corresponding direction from the parking space center BPC as the starting point in the following process: Fig.42 and Fig.43 The second encoded data item Ec2 of the connection vector of the left front corner point BP1 shown in .

[0387] First, the training process encodes the selected cell indicated by the thick-line square of the 3×3 grid GR into data including “1” when the connection number Tbn is 1 and “0” when the other connection numbers Tbn are 0 and each of 2 to 8, which is associated with the connection vector pointing from the parking space center BPC as the starting point to the left front corner point BP1. The training process performs a similar encoding operation on each of the other cells of the 10×10 grid GR corresponding to the target image GS to correspondingly generate second encoded data items Ec2 associated with the connection vector pointing from the parking space center BPC as the starting point to the left front corner point BP1, each of which includes Fig.45 The nine-channel encoded values ​​shown in FIG. 1 ; the second encoded data items Ec2 are arranged in accordance with the arrangement of a 10×10 grid GR.

[0388] Similarly, refer to Fig.43 and Fig.44, the right front corner point BP2 is located in the cell on the adjacent right-hand side of the center cell indicated by the thick-line square where the parking space center BPC is located. The cell where the right front corner point BP2 is located corresponds to the connection number Tbn of 5. To this end, the training process encodes the selected cell indicated by the thick-line square of the 3×3 grid GR into data including "1" when the connection number Tbn is 5 and "0" when each of the other connection numbers Tbn is 0 to 4 and 6 to 8, which is associated with the connection vector pointing from the parking space center BPC as the starting point to the right front corner point BP2. The training process performs a similar encoding operation for each of the other cells of the 10×10 grid GR corresponding to the target image GS to correspondingly generate a second encoded data item Ec2 associated with the connection vector pointing from the parking space center BPC as the starting point to the right front corner point BP2, each of the second encoded data items Ec2 includes Fig.45 The nine-channel encoded values ​​shown in FIG. 1 ; the second encoded data items Ec2 are arranged in accordance with the arrangement of a 10×10 grid GR.

[0389] The training process performs similar encoding operations for each of (i) the connection vector pointing from the parking space center BPC as the starting point to the left rear corner point BP3 and (ii) the connection vector pointing from the parking space center BPC as the starting point to the right rear corner point BP4, so as to generate a total of four second encoded data items Ec2 associated with the four connection vectors from the parking space center BPC as the starting point to the respective corner points BP1 to BP4; the four second encoded data items Ec2 constitute Fig.46 The second encoded data Ec2 shown in .

[0390] Next, refer to Figures 47 to 54 A third code describing the parking section information related to the parking section B5 as a part of the attribute information.

[0391] Similar to Fig.33 , Fig.47 Shown on a large scale Fig.32 The selected 3×3 grids GR among the 10×10 grids GR shown in . Fig.47 The 3×3 grid GR shown in Fig.33 The 3×3 grid GR shown in FIG. Fig.47 The arrangement of the five points BPC, BP1, BP2, BP3 and BP4 shown in Fig.33 or Fig.43 The arrangement of the five points BPC, BP1, BP2, BP3 and BP4 shown in FIG. 1 is the same. The parking segment information according to the first embodiment includes whether there is a segment line component constituting the corresponding parking segment B5, such as the front segment line B501 or the rear segment line B502.

[0392] exist Fig.47 In the example of the 3×3 grid GR shown in , the parking space center BPC is located at the center cell and the top left cell of the 3×3 grid GR, respectively. The training process assigns a coded data value Ecv of 0 or 1 associated with the presence or absence of, for example, the left segment line B504 and the front segment line B501 to each of the center cell and the top left cell of the 3×3 grid GR.

[0393] Specifically, since there is a left segment line B504 constituting the parking segment B5, the training process (i) assigns a coded data value Ecv of "1" to each of the center cell and the top left cell of the 3×3 grid GR in which the parking space center BPC exists, and (ii) assigns a coded data value Ecv of "0" to each of the other cells of the 3×3 grid GR, thereby obtaining a coded result (see Fig.48 ).

[0394] On the contrary, since there is no front segment line B501 constituting the parking segment B5, the training process assigns the encoding data value Ecv of "0" to all cells of the 3×3 grid GR, thereby obtaining the encoding result (see Fig.49 ).

[0395] Specifically, Fig.47 The parking segment B5 shown in FIG. 1 includes only a pair of left segment lines B504 and right segment lines B505. Since there is the left segment line B504 constituting the parking segment B5, the training process assigns a coded data value Ecv of "1" to each of the center cell and the top left cell of the 3×3 grid GR where the parking space center BPC is located (see FIG. Fig.48 ). As mentioned above, Fig.47 The parking segment B5 shown in FIG does not include the front segment line B501 and the rear segment line B502. Since there is no front segment line B501 constituting the parking segment B5, the training process assigns the encoded data value Ecv of "0" to all cells of the 3×3 grid GR (see Fig.49 ).

[0396] Fig.50 Shown in Fig.32 The entire target image GS shown in Fig.48 The presence or absence of the left segment line B504 shown in FIG. 5 is related to all the encoded data values ​​Ecv of the third encoding. Fig.51 Shown in Fig.32 All the encoded data values ​​Ecv based on the third encoding related to the presence or absence of the right segment line B505 in the entire target image GS shown in . Fig.52 Shown in Fig.32All the encoded data values ​​Ecv based on the third encoding related to the presence or absence of the rear segmentation line B502 in the entire target image GS shown in . Fig.53 Shown in Fig.32 The entire target image GS shown in Fig.49 The presence or absence of the front segment line B501 shown in FIG. 5 is related to all encoded data values ​​Ecv based on the third encoding.

[0397] That is, all encoded data values ​​Ecv based on the third encoding related to the presence or absence of each of the left segment line B504, the right segment line B505, the rear segment line B502 and the front segment line B501 constitute the third encoded data Ec3.

[0398] Fig.54 The parking space annotation code Ea for the objectified parking space PS included in the target image GS is schematically shown, and the parking space annotation code Ea includes:

[0399] (I) first coded data Ec1 associated with each of the five points BPC, BP1, BP2, BP3 and BP4;

[0400] (II) second coded data Ec2 associated with the connection vector;

[0401] (III) Third encoded data Ec3 related to the presence or absence of the segment line of the parking segment B5.

[0402] Specifically, in the annotation information storage device 2206, the ground truth annotation code Ea of each objectified parking space PS included in each target image GS has been stored to be paired with the corresponding objectified parking space PS and the corresponding target image GS.

[0403] That is, the training process calculates, for each objectified parking space PS in the target image GS input to the parking space training unit 2222, a loss function between (i) the ground truth annotation code Ea of the corresponding objectified parking space PS stored as annotation information in the annotation information storage device 2206 and (ii) the inferred parking space annotation code Ea calculated based on the machine learning model LM for the corresponding objectified parking space PS included in the target image GS.

[0404] Then, the training process updates the parameters of the machine learning model LM using, for example, back propagation to minimize the loss function accordingly. The parking space training unit 2222 is configured to iterate the training process to train the parameters of the machine learning model LM accordingly, thereby improving the accuracy of identifying at least one parking space PS included in the input image.

[0405] Since the training process using the loss function is already known at the time of filing of the corresponding Japanese patent application, further detailed description of the training process is omitted.

[0406] Specifically, the annotation code Ea prepared for each target image GS includes the third encoded data Ec3 related to the presence or absence of the segmentation line of the parking segment B5 of each parking space PS. Therefore, this enables the learning unit 2202 to infer, for each objectified parking space PS, whether the corresponding parking space PS has the parking segment B5 or one or more segmentation line components constituting the parking segment B5.

[0407] Reference Fig.30 The parking space determiner 2203 includes a bird's-eye view image generator 2231 , a parking space detector 2232 , an edge calculator 2233 , and a position corrector 2234 .

[0408] The bird's-eye image generator 2231 is configured to convert an image captured by at least one camera 34 into a bird's-eye image using, for example, a known viewpoint conversion method. The bird's-eye image is used by the processor 21 to detect at least one parking space PS included in the captured image; the detected at least one parking space PS is used by the processor 21 to perform, for example, parking assistance of the own vehicle V.

[0409] The parking space detector 2232 is configured to input the bird's-eye view image into the trained machine learning model LM stored in the learning model storage device 2207, and determine whether one or more parking spaces PS exist in the captured image based on the output data of the trained machine learning model LM.

[0410] In response to determining that there are one or more parking spaces PS in the captured image according to the output data of the trained machine learning model LM, the parking space detector 2232 is configured to infer, i.e., detect, one or more parking spaces PS and their states in the captured image according to the output data of the trained machine learning model LM. The state of each parking space PS includes information on whether the corresponding parking space PS is occupied by a parked vehicle B7 other than the own vehicle V.

[0411] Specifically, the parking space detector 2232 is configured to detect (i.e., infer) position information about the corresponding parking space PS for each parking space PS; the inferred position information about each parking space PS includes: (i) the position of each of the parking space center point BPC, the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3 and the right rear corner point BP4; (ii) the connection vector from the parking space center point BPC to each corner point BP1 to BP4; and (iii) the presence or absence of each segment line component of the parking segment B5 of the corresponding parking space PS.

[0412] The edge calculator 2233 is configured to calculate, in the bird's-eye view image, an inner edge of each segment line component of the parking segment B5 of each actual parking space PS included in the bird's-eye view image.

[0413] The position corrector 2234 is configured to correct the inferred position information about the corresponding parking space PS based on the calculated inner edge of each parking space PS to accordingly determine final position information about the corresponding parking space PS based on the corrected position information about each parking space PS.

[0414] The driving ECU 2 may be configured to perform, based on the final position information about each parking space PS included in the input image captured by at least one camera 34: (i) a task of responding to accelerator misoperation, such as a task of suppressing sudden acceleration and / or a task of notifying one or more passengers of accelerator misoperation; (ii) parking assistance; and / or (iii) automatic parking. Recently, vehicles are required to have higher accuracy in detecting parking spaces PS. In accordance with these requirements, when detecting the corresponding parking space PS from the input image using a machine learning model (e.g., a deep learning model), it is also effective to recognize various items of attribute information about each parking space PS in addition to the position information about each corner point of the corresponding parking space PS.

[0415] In view of the above, the processor 21 of the driving ECU 2 according to the first embodiment is configured to objectify the selected parking space PS as a set of the parking space center BPC and four key points (i.e., four corner points BP1, BP2, BP3, and BP4 positioned around the parking space center BPC) according to the target image GS, in other words, a set of the four corner points BP1, BP2, BP3, and BP4 and the parking space center BPC located inside the four corner points BP1, BP2, BP3, and BP4. Then, the processor 21 learns the recognition rule of the parking space PS according to the input image using the objectified parking space PS.

[0416] Specifically, the processor 21 is configured to train parameters of a machine learning model LM (e.g., a deep learning model) using a parking space annotation code Ea for each training target image GS as ground truth data, the parking space annotation code Ea comprising:

[0417] (I) first encoded data Ec1 of the first type, which includes presence data of each of the five points BPC, BP1, BP2, BP3, and BP4 of the objectified parking space PS;

[0418] (II) first encoded data Ec1 of the second type, which includes horizontal position data of each of the five points BPC, BP1, BP2, BP3, and BP4 of the objectified parking space PS;

[0419] (III) first encoded data Ec1 of a third type, which includes vertical position data of each of five points BPC, BP1, BP2, BP3, and BP4 of the objectified parking space PS;

[0420] (IV) Second coded data Ec2 associated with a connection vector associated with the objectified parking space PS.

[0421] Objectifying multiple groups of five points BPC, BP1, BP2, BP3 and BP4 detected from the target image GS enables (i) each group of five points BPC, BP1, BP2, BP3 and BP4 to be clearly associated with the corresponding objectified parking space PS, and (ii) attribute information to be applied to the corresponding objectified parking space PS.

[0422] Specifically, the processor 21 is configured to train parameters of the machine learning model LM using the parking space annotation code Ea for each training target image GS as ground truth data, wherein the parking space annotation code Ea further includes attribute information, such as third encoded data Ec3 related to the presence or absence of a segmentation line of the parking segment B5 of the objectified parking space PS.

[0423] This enables the processor 21 to determine whether there is a segment line component constituting the corresponding parking segment B5, such as the front segment line B501 or the rear segment line B502, when the processor 21 infers the recognition rule of the parking space PS from the input image. This enables the edge line constituting the actual parking space PS to be calculated from the input image after the detection (i.e., inference) of the parking space PS, so that the position of each inferred corner point can be corrected with high accuracy based on the calculated edge line. Therefore, this improves the parking space inference accuracy of the trained machine learning model LM, thereby correspondingly improving the accuracy of identifying the parking space PS included in the input image. This causes the own vehicle V to be parked in a selected one of the identified parking spaces PS with higher accuracy during parking assistance or automatic parking.

[0424] Fig.55 : is a flowchart schematically showing a position correction routine for each inferred parking space corner point BP according to the first embodiment as an example of a task after detecting (ie, inferring) a parking space PS from an input image. Fig.55 In the figure, the reference symbol S is an abbreviation of step. Specifically, in response to the completion of inferring the parking space PS according to the input image, the processor 21 reads a corresponding set of program instructions from the memory device 22 and executes the program instructions to perform the corresponding Fig.55 The operations in steps S101 to S114 shown in the flowchart of FIG.

[0425] When the position correction routine is started, in step S101, the processor 21 initializes the parking space counter N prepared in advance in the memory device 22 to 1. The parking space counter N is a positive integer variable between 1 and Nmax (including 1 and Nmax). The value Nmax represents the total number of parking spaces PS detected (ie, inferred) by the processor 21.

[0426] Next, the processor 21 selects one of the detected parking spaces PS as the Nth (i.e., first) parking space PS from the detected parking spaces PS in step S102, and initializes the segment line component counter M prepared in advance in the memory device 22 to 1 in step S103. The segment line component counter M is a positive integer that is any one of 1, 2, 3, and 4. The segment line component counter M set to 1 represents the front segment line B501, the segment line component counter M set to 2 represents the rear segment line B502, the segment line component counter M set to 3 represents the left segment line B504, and the segment line component counter M set to 4 represents the right segment line B505.

[0427] After the operation in step S103 , the position correction routine proceeds to step S104 .

[0428] In step S104, the processor 21 uses the segment line component corresponding to the segment line component counter M as a key to refer to the detection information about the selected parking space PS, that is, the inferred information. Specifically, if the segment line component counter M is set to 1 immediately after the operation in step S103, the processor 21 uses the front segment line B501 of the selected parking space PS as a key to refer to the detection information about the selected parking space PS. Next, in step S105, the processor 21 determines whether the segment line component corresponding to the segment line component counter M is included in the detection information about the selected parking space PS, that is, the inferred information.

[0429] Upon determining that the segment line component corresponding to the segment line component counter M is included in the detection information on the selected parking space PS (Yes in step S105 ), the processor 21 is programmed to perform the following operations in steps S106 to S109 .

[0430] Specifically, in step S106, the processor 21 calculates the circumscribed rectangle Rc (see Fig.56). The circumscribed rectangle Rc has a pair of detected corner points BP constituting the segment line component corresponding to the segment line component counter M. Next, in step S107, the processor 21 performs an edge calculation task of calculating the inner edge of the actual segment line component of the parking segment B5 of the selected parking space PS included in the bird's-eye view image using a known filter such as a Sobel filter based on the brightness values ​​of the pixels included in the bird's-eye view image.

[0431] Next, in step S108, the processor 21 applies a known edge line generation task such as Hough transform to the calculated inner edge of the actual segmented line component to calculate the edge line LL of the actual segmented line component accordingly (for example, see Fig.57 ).

[0432] After the operation in step S108 , the processor 21 determines whether the segmented line component counter M is set to 4 in step S109 .

[0433] When it is determined that the segment line component counter M is set to any one of 1 to 3 except 4 (No in step S109), the processor 21 recognizes that there is at least one segment line component included in the detection information, i.e., the inferred information, about the selected parking space P; for at least one segment line component, the edge calculation task in step S107 and the edge line calculation task in step S108 are not performed. Then, the processor 21 increments the segment line component counter M by 1 in step S110, and thereafter returns to the operation in step S104, and iteratively performs the operations in steps S104 to S110 until the determination in step S109 is affirmative.

[0434] Otherwise, when it is determined that the segment line component counter M is set to 4 (Yes in step S109), the processor 21 recognizes that the edge calculation task in step S107 and the edge line calculation task in step S108 have been performed for all segment line components included in the detection information about the selected parking space p, that is, the inference information. Then, the position correction routine proceeds to step S111.

[0435] At this time, if the edge line LL of at least one segmented line component is not calculated because the segmented line component corresponding to the segmented line component counter M is determined in step S105 to be not included in the detection information about the selected parking space PS (No in step S105), the processor 21 calculates at least one straight line connecting the corresponding adjacent pair of parking space corner points BP whose edge lines LL have been calculated in step S111.

[0436] Then, in step S111 , the processor 21 calculates four intersection points among the calculated four edge lines LL, in which at least one straight line may be included.

[0437] Next, in step S112, the processor 21 corrects the parking space corner point BP detected (i.e., inferred) based on the machine learning model LM. After the operation in step S112, the processor 21 determines in step S113 whether the parking space counter N is set to the value Nmax. When it is determined that the parking space counter N is not set to the value Nmax (No in step S113), the processor 21 recognizes that at least one unselected parking space PS remains in the parking space PS detected (i.e., inferred) by the processor 21. Then, the processor 21 increments the parking space counter N by 1 in step S114, and thereafter returns to the operation in step S102, and iteratively performs the operations in steps S102 to S114 until the determination in step S113 is affirmative.

[0438] That is, upon determining that the parking space counter N is set to the value Nmax (Yes in step S113), the processor 21 recognizes that corner position correction for all parking spaces PS detected (ie, inferred) by the processor 21 has been completed, and thus the processor 21 terminates the position correction routine.

[0439] Second embodiment

[0440] Refer to the following Fig.58 and Fig.59 A parking space detection apparatus 2200 according to the second embodiment of the present disclosure is described.

[0441] Therefore, the following mainly describes the differences between the second embodiment and the first embodiment. Similarly, the following third embodiment and each of the subsequent embodiments mainly describe the differences compared to the previous embodiment. For this reason, unless there is a technical contradiction, the structure, function and benefits obtained in each of the subsequent embodiments are substantially the same as the structure, function and benefits obtained in at least one of the previous embodiments.

[0442] In e.g. Fig.58 In the actual parking lot PL shown in FIG. , there may be a situation where a portion of the parking space segment B5 and / or at least one parking space corner point BP may be blocked by at least one parked vehicle B7. If the processor 21 performs training of parameters of the machine learning model LM based on the learning image captured in such a situation, the processor 21 may train the parameters of the machine learning model LM so that a small portion of the parking space segment B5 appearing in the learning image is the identified target parking space PS. This may result in the processor 21 identifying, for example, Fig.18The crosswalk marking B41, the stop line B42, or the broken road marking line B43 on the road Rd shown in FIG. 1 is erroneously detected (ie, inferred) as the parking space PS.

[0443] From this perspective, the parking space annotation generator 2213 of the second embodiment may be configured to generate annotation information, i.e., a parking space annotation code Ea, for each parking space PS included in the target image GS, the annotation information including occlusion information indicating the degree to which the corresponding parking space PS is occluded by an object other than the own vehicle V (e.g., typically, at least one parked vehicle B7). For example, the parking space annotation generator 2213 of the second embodiment may be configured to generate annotation information, i.e., a parking space annotation code Ea, for each parking space PS included in the target image GS. Fig.54 The parking space annotation code Ea of each parking space PS shown in FIG. 1 adds occlusion information indicating the size of the unoccluded area of ​​the corresponding parking space PS or its parking space segment B5. Such occlusion information added to the parking space annotation code Ea for each parking space PS enables the processor 21 to determine whether to train the machine learning model LM for the corresponding parking space PS according to the size of the unoccluded area of ​​the corresponding parking space PS or its parking space segment B5. This makes it possible to reduce such erroneous detection of another object as the parking space PS.

[0444] Alternatively, the parking space annotation generator 2213 of the second embodiment may be configured to generate annotation information for each parking space PS included in the target image GS, i.e., the parking space annotation code Ea, which includes occlusion information indicating the degree to which each parking space corner point BP of the corresponding parking space PS is occluded by, for example, at least one parked vehicle B7. Then, such encoded data added to the parking space annotation code Ea for each objectified parking space PS enables the processor 21 to determine whether to train the machine learning model LM for the corresponding parking space PS. Specifically, the processor 21 may calculate the number of parking space corner points BP of the selected parking space PS that are occluded by, for example, at least one parked vehicle B7, and may determine not to train the machine learning model LM for the selected parking space PS when it is determined that the number of parking space corner points BP of the selected parking space PS that are occluded by at least one parked vehicle B7 is greater than or equal to a non-training threshold value, for example, 3. This makes it possible to reduce such erroneous detection of another object as the parking space PS while minimizing the increase in the load of generating the parking space annotation code Ea.

[0445] Fig.59 2 is a flow chart schematically showing a part of the training routine according to the second embodiment. Specifically, the processor 21 is programmed to execute when starting to train the parameters of the machine learning model LM for each parking space PS using the corresponding parking space annotation code Ea including occlusion information. Fig.59 Part of the training routine shown in .

[0446] Reference Fig.59 In step S201, the processor 21 refers to the parking space annotation code Ea for the selected parking space PS to determine whether the number of one or more obscured corner points of the selected objectified parking space PS is 0. The obscured corner points are one or more parking space corner points BP that are obscured by at least one parked vehicle B7 and are difficult to see. When it is determined that the number of one or more obscured corner points of the selected parking space PS is 0 (yes in step S201), the processor 21 recognizes that the selected parking space PS is not obscured by any parked vehicle B7, and therefore the selected parking space PS is an empty space. Then, the processor 21 determines the parameters for training the machine learning model LM for the selected parking space PS.

[0447] Otherwise, when it is determined that the number of one or more obstructed corner points of the selected parking space PS is not 0 (No in step S201), the processor 21 recognizes that the selected parking space PS is obstructed by at least one parked vehicle B7, so that there is at least one obstructed corner point. Then, the training routine proceeds to step S202.

[0448] In step S202, the processor 21 determines whether the number of one or more occluded corner points of the selected objectified parking space PS is 1. When it is determined that the number of one or more occluded corner points of the selected parking space PS is 1 (yes in step S202), the processor 21 determines the parameters for training the machine learning model LM for the selected parking space PS.

[0449] Otherwise, when it is determined that the number of the one or more obscured corner points of the selected parking space PS is not 1 (No in step S202 ), the training routine proceeds to step S203 .

[0450] In step S203, the processor 21 determines whether the number of one or more occluded corner points of the selected objectified parking space PS is 2. When it is determined that the number of one or more occluded corner points of the selected parking space PS is 2 (yes in step S203), the processor 21 determines the parameters for training the machine learning model LM for the selected parking space PS.

[0451] Otherwise, when it is determined that the number of one or more occluded corner points of the selected parking space PS is not 2 (No in step S203), the processor 21 recognizes that the number of one or more occluded corner points is 3 or more. Then, the processor 21 determines not to train the parameters of the machine learning model LM for the selected parking space PS, that is, the processor 21 determines not to calculate the loss function for the selected parking space PS.

[0452] Third embodiment

[0453] Refer to the following Figure 60 to Figure 62 A parking space detection device 2200 according to a third embodiment of the present disclosure is described. Fig.60 , reference numeral Pc denotes a viewpoint of at least one camera 34, which will be referred to as camera position Pc. In the scenario of parking assistance and / or automatic parking, information on whether each parking space PS is empty is required. That is, the target parking space PS where the own vehicle V is to be parked needs to be an empty space that is not occupied by the parked vehicle B7. For this reason, it is preferred that the parameters of the machine learning model LM are trained for each parking space PS using attribute information including whether the corresponding parking space PS is empty.

[0454] However, if Fig.60 As shown, since the parked vehicle B7 occupying the parking space PS adjacent to the target parking space PS blocks the target parking space PS, there may be a situation where it is difficult to determine whether the target parking space PS is empty. Assume that the processor 21 performs training of parameters of the machine learning model LM based on the learning image captured in such a situation. Under this assumption, the processor 21 may erroneously determine that the target parking space PS is an occupied space during parking assistance and / or automatic parking.

[0455] From this perspective, the parking space annotation generator 2213 of the third embodiment may be configured to generate annotation information, ie, a parking space annotation code Ea, for each parking space PS included in the target image GS, the annotation information including parking space empty information indicating whether each parking space PS included in the target image GS is empty.

[0456] In addition, the parking space annotation generator 2213 of the third embodiment is configured to determine how to train the parameters of the machine learning model LM for each parking space PS included in the target image GS based on the parking space vacancy information and the positional relationship between the camera position Pc of the own vehicle V, the corresponding parking space PS and at least one parked vehicle B7.

[0457] Fig.61 2 is a flowchart schematically illustrating the preprocessing before training the parameters of the machine learning model LM. The processor 21 can be used as an annotation information generator 2201 and / or a learning unit 2202 to perform Fig.61 The preprocessing shown in .

[0458] Reference Fig.61In step S301, the processor 21 initializes the parking space counter N prepared in advance in the memory device 22 to 1. The parking space counter N is a positive integer variable between 1 and Nmax (including 1 and Nmax). The value Nmax represents the total number of parking spaces PS as training candidates included in the target image GS; a parking space annotation code Ea has been prepared for each of the parking spaces PS.

[0459] Next, in step S302, the processor 21 selects one of the training candidate parking spaces PS as the Nth (i.e., first) parking space PS from the training candidate parking spaces PS. After the operation in step S302, in step S303, the processor 21 calculates the amount of occlusion of the selected parking space PS by another object (e.g., typically, at least one parked vehicle B7). How the processor 21 calculates the amount of occlusion of the selected parking space PS will be described later. Next, in step S304, the processor 21 determines whether the amount of occlusion of the selected parking space PS is less than a predetermined threshold amount.

[0460] Refer to the following Fig.62 It is described how the processor 21 calculates the shielding amount of the selected parking space PS.

[0461] As an example, Fig.62 A case is shown where the left-side parallel parking row PSs and the right-side parallel parking row PSs are respectively arranged at both sides of a portion of the public parking passage PSr where the own vehicle V is traveling. The left side and the right side are defined relative to the own vehicle V traveling in a portion of the public parking passage PSr. Each of the left-side parallel parking row PSs and the right-side parallel parking row PSs represents a row of parking spaces PS, which are arranged so that their short sides are continuously aligned in a predetermined direction parallel to the traveling direction of the public parking passage PSr. The public parking passage PSr represents an area through which a vehicle can travel, and faces at least one of the left-side parallel parking row PSs and the right-side parallel parking row PSs. That is, the public parking passage PSr may be defined as an area after removing a plurality of parking rows including the left-side parallel parking row PSs and the right-side parallel parking row PSs from the entire area of ​​the parking lot PL.

[0462] Fig.62 The right-side parallel parking space row PSs shown as an example in FIG. 1 includes the first parking space PS, the second parking space PS, the third parking space PS, and the fourth parking space PS from the near side to the far side of the own vehicle V. The parked vehicle B7 occupies the corresponding first parking space PS to the third parking space PS, and the fourth parking space PS is empty. The empty fourth parking space PS will be referred to as the target parking space PSt. Note that Fig.62An example case is shown for simply describing how the processor 21 calculates the shielding amount of the target parking space PSt as the selected parking space PS, and therefore the present disclosure is not limited to this case. Specifically, the processor 21 may calculate the shielding amount of the selected parking space PS in another case in the same manner as the example case.

[0463] The processor 21 may calculate the shielding amount of the target parking space PSt based on the viewing angle of the own vehicle V relative to the target parking space PSt.

[0464] First, if Fig.62 As shown, in the first case where the entire target parking space PSt is not blocked by the parked vehicle B7 so that the entire target parking space PSt can be seen by at least one camera 34, the value of the viewing angle of at least one camera 34 located at the camera position Pc relative to the target parking space PSt is defined as the existence range angle θe. Fig.62 In the example shown in , the existence range angle θe is defined as the angle formed between a first virtual line extending from the camera position Pc and passing through the right front corner point BP2 and a second virtual line extending from the camera position Pc and passing through the left rear corner point BP3.

[0465] Next, a visual range angle θc is defined in the second case where a portion of the target parking space PSt is blocked by the parked vehicle B7, which represents the visual range of at least one camera 34 located at the camera position Pc relative to the target parking space PSt. Fig.62 In the example shown in , the visible range angle θc is defined as the angle formed between a first virtual line extending from the camera position Pc and passing through the right front corner point BP2 and a third virtual line extending from the camera position Pc and passing through the right front corner of the parked vehicle B7 located in the third parking space PS in the right parallel parking space row PSs. In addition, a blocking angle range θh defined by subtracting the visible range angle θc from the existence range angle θe is defined.

[0466] The processor 21 may be configured to calculate the ratio of the occlusion angle range θh to the existence range angle θe or the percentage of the occlusion angle range θh to the existence range angle θe as the occlusion amount of the target parking space PSt. The predetermined threshold amount used for comparison with the occlusion amount in step S304 may be freely set to any value, such as 0.8 or 80%, according to empirical rules and / or experiments. The threshold amount may be modified or corrected according to training conditions.

[0467] Specifically, when it is determined that the amount of occlusion of the selected parking space PS is greater than or equal to the predetermined threshold amount (No in step S304), the preprocessing proceeds to step S305. In step S305, the processor 21 changes the parking space vacancy information about the selected parking space PS included in the parking space annotation code Ea to "undefined state" in step S305. At this time, when the category of the parking space vacancy information is set to the undefined category, the training of the parameters of the machine learning model LM for the selected parking space PS may be performed, or the training of the parameters of the machine learning model LM for the selected parking space PS may not be performed. Thereafter, the preprocessing proceeds to step S306.

[0468] Otherwise, when it is determined that the shielding amount of the selected parking space PS is less than the predetermined threshold amount (Yes in step S304), the preprocessing proceeds to step S306 while skipping the operation in step S305. This causes the parking space vacancy information about the selected parking space PS included in the parking space annotation code Ea to remain unchanged.

[0469] In step S306, the processor 21 determines whether the parking space counter N is set to the value Nmax. When determining that the parking space counter N is not set to the value Nmax (No in step S306), the processor 21 recognizes that at least one unselected parking space PS remains in the training candidate parking space PS. Then, the processor 21 increments the parking space counter N by 1 in step S307, and thereafter returns to the operation in step S302, and iteratively performs the operations in steps S302 to S307 until the determination in step S306 is affirmative.

[0470] That is, upon determining that the parking space counter N is set to the value Nmax (YES in step S306 ), the processor 21 terminates the pre-processing.

[0471] Fourth embodiment

[0472] Refer to the following Figure 63 to Figure 68 A parking space detection apparatus 2200 according to a fourth embodiment of the present disclosure is described.

[0473] Fig.63 A 5×5 grid GR of a target image GS showing a situation of a lower parallel parking row and an upper parallel parking row including a parking space PS in which the own vehicle V is parked is shown. That is, the parking spaces PS included in the upper parallel parking row are adjacently paired with corresponding parking spaces PS in the lower parallel parking row, while the parking spaces PS of each adjacent pair share a common rear segmentation line B502. Fig.63 The upper row of parallel parking spaces shown in FIG. 1 is actually located in front of the lower row of parallel parking spaces.

[0474] It is assumed that the address of the parking space PS where the own vehicle V is parked will be referred to as id:0, and the addresses of the parking spaces PS arranged in the lower side parallel parking space column at the lower side of the parking space PS with address id:0 will be referred to as id:1, id:2, id:3 and id:4 respectively. The parking space PS with address id:0 is positioned to face two adjacent parking spaces PS with corresponding addresses id:2 and id:3. For ease of description, the left rear corner point BP3 of the parking space corner points of each parking space PS is shown only by a black circle, and the other parking space corner points and the parking space center point BPC are shown by a white circle. In the cell of the 5×5 grid GR surrounded by a white dotted line and referred to as a significant cell GRt, the left rear corner point BP3 of the parking space PS with address id:0 and the left rear corner point BP3 of the parking space PS with address id:2 correspond to the same position.

[0475] Fig.42 The first coded data Ec1 according to the first embodiment shown in FIG. 1 includes:

[0476] (I) first type data including existence data of each of the five points BPC, BP1, BP2, BP3, and BP4 of the objectified parking space PS;

[0477] (II) second type data including horizontal position data of each of the five points BPC, BP1, BP2, BP3, and BP4 of the objectified parking space PS;

[0478] (III) Third type data including vertical position data of each of the five points BPC, BP1, BP2, BP3, and BP4 of the objectified parking space PS.

[0479] That is, the first encoded data Ec1 is generated under the following premise: (i) the existence data of each point BPC, BP1, BP2, BP3 and BP4 is set to the corresponding individual cell in the grid GR; (ii) the horizontal position data of each point BPC, BP1, BP2, BP3 and BP4 is set to the corresponding individual cell in the grid GR; and (iii) the vertical position data of each point BPC, BP1, BP2, BP3 and BP4 is set to the corresponding individual cell in the grid GR.

[0480] Since the salient unit GRt of the fourth embodiment is shared by both the left rear corner point BP3 of the parking space PS with address id:0 and the left rear corner point BP3 of the parking space PS with address id:2, it may be difficult to set (i) the first information item about the left rear corner point BP3 of the parking space PS with address id:0 and (ii) the second information item about the left rear corner point BP3 of the parking space PS with address id:2 to the common salient unit GRt under the above premise. For this reason, information about the midpoint between the left rear corner point BP3 of the parking space PS with address id:0 and the left rear corner point BP3 of the parking space PS with address id:2 is encoded so that a common encoding value can be set to the salient unit GRt. The encoding may cause the encoding data value of the horizontal position or the vertical position of the left rear corner point BP3 of the parking space PS with address id:0 to be different from the inherent encoding value, and similarly, the encoding data value of the horizontal position or the vertical position of the left rear corner point BP3 of the parking space PS with address id:2 to be different from the inherent encoding value. This may result in a decrease in the inference accuracy using the machine learning model LM.

[0481] This situation where the same type of parking space corner points BP of different parking spaces PS are located in a common cell of the target image GS may occur in parking spaces PS with different orientations (see Fig.31 ) in specific parking lots adjacent to each other.

[0482] From this perspective, if Fig.64A As shown, the fourth embodiment uses four anchor indexes (ie, anchor 0, anchor 1, anchor 2, and anchor 3) to define the typical positions of the parking space PS. Fig.31 , the anchor 0 indicated by the upward arrow indicates that the orientation of the parking space direction BD of the parking space PS is upward in the target image GS, and the anchor 1 indicated by the leftward arrow indicates that the orientation of the parking space direction BD of the parking space PS is leftward in the target image GS. Fig.31 , the anchor 2 indicated by the rightward arrow indicates that the parking space direction BD of the parking space PS is oriented rightward in the target image GS, and the anchor 3 indicated by the downward arrow indicates that the parking space direction BD of the parking space PS is oriented downward in the target image GS.

[0483] Specifically, the processor 21 is configured to train parameters of the machine learning model LM using the parking space annotation code Ea for each training target image GS as ground truth data, the first encoded data Ec1 of which additionally includes encoded information indicating each typical orientation of the objectified parking space PS.

[0484] In the actual target image GS, Fig.64BAs shown, the parking space PS is inclined relative to each of the vertical direction and the horizontal direction of the target image GS. Then, the determination of the orientation of the parking space PS uses the circumscribed rectangle Rc of the parking space PS and the reference point PW. The circumscribed rectangle Rc of the parking space PS is a virtual rectangle circumscribed around the four parking space corner points BP and has each side parallel to a corresponding one of the vertical direction and the horizontal direction of the target image GS. The reference point PW of the parking space PS is a point on the circumscribed rectangle Rc. In the fourth embodiment, the reference point PW of the parking space PS is a point on the left corner of the circumscribed rectangle Rc. The orientation of the parking space direction BD of the parking space PS is determined based on which of the four parking space corner points BP of the parking space PS is arranged closest to the reference point PW.

[0485] Specifically, if the left front corner point BP1 is positioned closest to the reference point PW, the orientation of the parking space direction BD of the parking space PS is set to Anchor 0. If the right front corner point BP2 is positioned closest to the reference point PW, the orientation of the parking space direction BD of the parking space PS is set to Anchor 1. If the right rear corner point BP4 is positioned closest to the reference point PW, the orientation of the parking space direction BD of the parking space PS is set to Anchor 2. If the left rear corner point BP3 is positioned closest to the reference point PW, the orientation of the parking space direction BD of the parking space PS is set to Anchor 3.

[0486] Refer to the following Figure 65 to Figure 68 The encoding routine performed by processor 21 is described.

[0487] When to start Fig.65 In the encoding routine shown in FIG. 4 , at step S401, the processor 21 initializes the parking space counter N prepared in advance in the memory device 22 to 1. The parking space counter N is a positive integer variable between 1 and Nmax (including 1 and Nmax). The value Nmax represents the total number of parking spaces PS as training candidates included in the target image GS.

[0488] Next, in step S402 , the processor 21 selects one of the training candidate parking spots PS as the Nth (ie, first) parking spot PS from among the training candidate parking spots PS.

[0489] After selecting the parking space PS in step S402, the processor 21 performs subsequent operations in steps S403 to S408. Specifically, the processor 21 calculates the circumscribed rectangle Rc of the selected parking space PS in step S403, and calculates the reference point PW based on the circumscribed rectangle Rc in step S404. Next, in step S405, the processor 21 calculates the distances between the reference point PW and the corresponding four parking space corner points BP.

[0490] After the operation in step S405, in step S406, the processor 21 performs a matching operation of determining which one of the anchor indexes 0 to 3 matches the selected parking space PS according to the distances calculated for the corresponding four parking space corner points BP. That is, the processor 21 determines one anchor index among the anchor indexes 0 to 3 that matches the selected parking space PS.

[0491] Refer to the following Fig.66 and Fig.67 The matching operation in step S406 is described in detail. Fig.66 and Fig.67 In each of the above, the x-axis direction corresponds to the horizontal direction of the target image GS, where Fig.35 and Fig.36 The horizontal position of any point in the grid GR is calculated in each of the grids, and the y-axis direction corresponds to the vertical direction of the target image GS, where Fig.35 and Fig.36 The vertical position of any point in the grid GR is calculated in each of . Fig.66 In , the two parking spaces PS will be referred to as parking spaces PS1 and PS2, and the circumscribed rectangle Rc of each of the two parking spaces PS1 and PS2 has a first pair of sides parallel to the x-axis direction and a second pair of sides parallel to the y-axis direction. Fig.67 , the two parking spaces PS will be referred to as parking spaces PS3 and PS4, and a circumscribed rectangle Rc of each of the two parking spaces PS3 and PS4 has a first pair of sides parallel to the x-axis direction and a second pair of sides parallel to the y-axis direction.

[0492] exist Fig.66 In the example of FIG. 1 , the left rear corner point BP3 of the parking space PS1 and the left rear corner point BP3 of the parking space PS2 are located in the common salient unit GRt. One of the parking space corner points BP of the parking space PS1 that is positioned closest to the reference point PW is the left front corner point BP1. For this reason, the anchor index of the parking space PS1 is set to anchor 0. On the contrary, one of the parking space corner points BP of the parking space PS2 that is positioned closest to the reference point PW is the right rear corner point BP4. For this reason, the anchor index of the parking space PS2 is set to anchor 2.

[0493] exist Fig.67In the example of FIG. 1 , the left rear corner point BP3 of the parking space PS3 and the left rear corner point BP3 of the parking space PS4 are located in the common salient unit GRt. One of the parking space corner points BP of the parking space PS3 that is located closest to the reference point PW is the left front corner point BP1. For this reason, the anchor index of the parking space PS3 is set to anchor 0. On the contrary, one of the parking space corner points BP of the parking space PS4 that is located closest to the reference point PW is the right front corner point BP2. For this reason, the anchor index of the parking space PS4 is set to anchor 1.

[0494] Reference Fig.65 After the matching operation in step S406, the processor 21 encodes the parking space corner point BP and the parking space center BPC for each of the anchor indexes (i.e., for each of anchor 0, anchor 1, anchor 2, and anchor 3). Specifically, in step S406, the processor 21 generates: (i) the first encoded data Ec1 corresponding to anchor 0; (ii) the first encoded data Ec1 corresponding to anchor 1; (iii) the first encoded data Ec1 corresponding to anchor 2; and (iv) the first encoded data Ec1 corresponding to anchor 3.

[0495] Then, in step S407, the processor 21 generates a parking space annotation code Ea for the objectified parking space PS included in the target image GS, such as Fig.68 As shown, the parking space annotation code Ea includes:

[0496] (I) a first encoded data set, comprising: (i) first encoded data Ec1 with anchor 0 associated with each of five points BPC, BP1, BP2, BP3 and BP4; (ii) first encoded data Ec1 with anchor 1 associated with each of five points BPC, BP1, BP2, BP3 and BP4; (iii) first encoded data Ec1 with anchor 2 associated with each of five points BPC, BP1, BP2, BP3 and BP4; and (iv) first encoded data Ec1 corresponding to anchor 3 associated with each of five points BPC, BP1, BP2, BP3 and BP4;

[0497] (II) second coded data Ec2 associated with the connection vector;

[0498] (III) Third encoded data Ec3 related to the presence or absence of the segment line of the parking segment B5.

[0499] That is, according to the fourth embodiment Fig.68 The parking space annotation code Ea shown in FIG. 1 is configured such that Fig.54 The first encoded data Ec1 shown in is replaced with the first encoded data set.

[0500] Reference Fig.65 , in step S408, the processor 21 determines whether the parking space counter N is set to the value Nmax. When determining that the parking space counter N is not set to the value Nmax (No in step S408), the processor 21 recognizes that at least one unselected parking space PS remains in the training candidate parking space PS. Then, the processor 21 increments the parking space counter N by 1 in step S409, and thereafter returns to the operation in step S402, and iteratively performs the operations in steps S402 to S409 until the determination in step S408 is affirmative.

[0501] That is, upon determining that the parking space counter N is set to the value Nmax (YES in step S408 ), the processor 21 terminates the encoding routine.

[0502] Specifically, the processor 21 of the fourth embodiment is configured to use the parking space annotation code Ea for each training target image GS as ground truth data to train parameters of the machine learning model LM, and the parking space annotation code Ea further includes: (i) first encoded data Ec1 corresponding to anchor 0 (i.e., upward position) associated with each of the five points BPC, BP1, BP2, BP3 and BP4; (ii) first encoded data Ec1 corresponding to anchor 1 (i.e., left position) associated with each of the five points BPC, BP1, BP2, BP3 and BP4; (iii) first encoded data Ec1 corresponding to anchor 2 (i.e., right position) associated with each of the five points BPC, BP1, BP2, BP3 and BP4; and (iv) first encoded data Ec1 corresponding to anchor 3 (i.e., downward position) associated with each of the five points BPC, BP1, BP2 and BP4.

[0503] Therefore, even if parking space corner points BP of the same type (i.e., the same category) of different parking spaces PS are located in a common cell of the target image GS, this configuration enables the processor 21 to infer the recognition rule of the parking space PS from the input image for each of the individual parking space corner points BP of the same type. Therefore, this makes it possible to more accurately detect the parking space corner points BP even if parking space corner points BP of different parking spaces PS are concentrated in the input image.

[0504] Fifth embodiment

[0505] Refer to the following Figure 69 to Figure 75 A parking space detection apparatus 2200 according to a fifth embodiment of the present disclosure is described.

[0506] like Fig.69 As shown in FIG. 1 , at least one of the parking space corner points BP of the parking space PS may be located outside the target image GS. Fig.69 The right rear corner point BP4 of the parking space PS shown by the dotted rectangle in FIG. 1 is located outside the target image GS. In this case, if the right rear corner point BP4 is not encoded so that the training of the parameters of the machine learning model LM for the right rear corner point BP4 is not performed, the accuracy of the inference result of the parking space PS may be reduced.

[0507] From this point of view, the processor 21 is preferably configured to perform an offset task of offsetting the right rear corner point BP4 located outside the target image GS into the target image GS, and thereafter encode the right rear corner point BP4. As the direction in which the right rear corner point BP4 is offset into the target image GS, the direction of the long side of the rectangular parking space PS or the short side of the rectangular parking space PS can generally be used.

[0508] Figure 70 to Figure 73 Each of shows a corresponding example of an offset task for offsetting one of the four parking space corner points BP located outside the target image GS; the region of the target image GS will also be referred to as a training input region AN. Figure 70 to Figure 73 In each of , one parking space corner point BP located outside the target image GS (which will be referred to as an external point) among the four parking space corner points BP is represented by a black circle. Figure 70 to Figure 73 In each of , reference numeral PSe indicates that its offset segment point has been offset to the offset parking space in the target image GS.

[0509] Apart from Fig.70 The offset direction of the external point shown in Fig.71 The external points shown in the figure have different offset directions. Fig.70 How the external points shown in are located outside the target image GS and Fig.71 The external points shown in FIG are located outside the target image GS in the same way. Fig.72 The offset direction of the external point shown in Fig.73 The external points shown in the figure have different offset directions. Fig.72 How the external points shown in are located outside the target image GS and Fig.73 The same is true of how the external points shown in are located outside the target image GS.

[0510] exist Fig.70 and Fig.71 In each of the examples of , the percentage of the portion of the short side of the rectangular parking space PS located outside the target image GS to the total short sides (which will be referred to as the outside percentage) is substantially 30%, and the percentage of the portion of the long side of the rectangular parking space PS located outside the target image GS to the total long sides is substantially 40%. In other words, in Fig.70 and Fig.71The percentage of the remaining portion of the short side of the rectangular parking space PS shown in each of the above (its portion being outside the target image) to the total short side (which will be referred to as the in-image percentage) is 70%, and Fig.70 and Fig.71 The remaining portion of the long side of the rectangular parking space PS shown in each of , which portion is located outside the target image, has an in-image percentage of 60% of the total long side. Fig.70 The offset direction of the external points shown in is set to be along the direction of the long edge, and Fig.71 The offset direction of the outer points shown in is set to be along the direction of the short side.

[0511] Similarly, in Fig.72 and Fig.73 In each of the examples, the percentage of the portion of the short side of the rectangular parking space PS located outside the target image GS to the total short side is substantially 80%, and the percentage of the portion of the long side of the rectangular parking space PS located outside the target image GS to the total long side is substantially 40%. In other words, in Fig.72 and Fig.73 The remaining portion of the short side of the rectangular parking space PS shown in each of the above (part of which is outside the target image) has an image in-image percentage of 20% of the total short side, and Fig.70 and Fig.71 The remaining portion of the long side of the rectangular parking space PS shown in each of , which portion is located outside the target image, has an in-image percentage of 60% of the total long side. Fig.72 The offset direction of the external points shown in is set to be along the direction of the long edge, and Fig.73 The offset direction of the outer points shown in is set to be along the direction of the short side.

[0512] Will Fig.70 and Fig.71 By comparing, we can identify Fig.71 The shape of the offset parking space PSe to be encoded is shown in Fig.70 The shape shown in FIG. 1 can better maintain the original shape of the actual parking space PS. Fig.72 and Fig.73 By comparing, we can identify Fig.72 The shape of the offset parking space PSe to be encoded is shown in Fig.73 The shape shown in can better maintain the original shape of the actual parking space PS.

[0513] That is, the external point of the parking space PS is offset in one of the partial external short side and the partial external long side of the parking space PS (whose external percentage is lower than the external percentage of the other of the partial external short side and the partial external long side of the parking space PS) so that the shape of the offset parking space PSe can maintain the original shape of the parking space PS. In other words, the external point of the parking space PS is offset in one of the partial external short side and the partial external long side of the parking space PS (whose in-image percentage is higher than the in-image percentage of the other of the partial external short side and the partial external long side of the parking space PS) so that the shape of the offset parking space PSe can maintain the original shape of the parking space PS.

[0514] Therefore, offsetting the external points of the parking space PS into the target image in a direction determined to maintain the original shape of the parking space PS (i.e., the rectangular shape) as much as possible can achieve both of the following: (i) improving the inference result of the parking space PS even if the parking space PS has external corner points and (ii) reducing the training of the parking space PS whose external corner points remain intact.

[0515] Fig.74 and Fig.75 Each of the figures shows a corresponding example of an offset task for offsetting two of the four parking space corner points BP of the parking space PS located outside the target image GS. That is, if there are two or fewer external points among the four parking space corner points BP, offsetting the two or fewer external points of the parking space PS into the target image enables encoding the offset parking space PSe for training the parameters of the machine learning model LM. This makes it possible to improve the inference result of the parking space PS even if the parking space PS has two or fewer external corner points. On the contrary, if there are three or more external points among the four parking space corner points BP of the parking space PS, the processor 21 does not encode the parking space PS because the shape of the offset parking space does not maintain the original rectangular shape of the parking space PS.

[0516] Sixth Embodiment

[0517] Refer to the following Figures 76A to 81 A parking space detection apparatus 2200 according to a sixth embodiment of the present disclosure is described.

[0518] like Fig.76A As shown, a plurality of lane lines B433 may be provided in a road Rd having three or more driving lanes in each direction in the road width direction, each of the lane lines B433 having a dotted line. In this case, when focusing on a selected driving lane LNd located in the middle of the road Rd in the road width direction, the selected driving lane LNd is divided by two lane lines B433 located on both sides of the selected driving lane LNd, and the area surrounded by the dotted line X on the selected driving lane LNd is different in appearance from the lane lines B433 on the selected driving lane LNd. Fig.76B The parking space PS shown in FIG. 1 is very similar to the parking space PS shown in FIG. 1 , which is composed only of a pair of left segment lines B503 and right segment lines B503. The selected driving lane LNd located in the middle of the road Rd is a lane located between the outermost lane LNd located adjacent to the vehicle-road edge line B431 and the passing lane LNp located adjacent to the center line B432.

[0519] Therefore, when the own vehicle V is traveling on the road Rd, the area on the selected driving lane LNd surrounded by the dashed line X is mistakenly recognized as Fig.76B The parking space PS shown in the figure may cause the processor 21 to erroneously determine that the own vehicle V is traveling in the parking lot PL. Such erroneous determination may lead to a reduction in the comfort of one or more passengers of the own vehicle V due to sudden, frequent warnings caused by accelerator misoperation. From this perspective, training the parameters of the machine learning model LM based on information about such an area in the road Rd that is different from the parking space PS may cause the occurrence of such erroneous recognition of the area on the road Rd as the parking space PS.

[0520] From this perspective, the sixth embodiment enables the processor 21 to determine whether a parking lot environment exists around the own vehicle V when performing training of parameters of the machine learning model LM. In addition, the processor 21 is configured to infer (i.e., detect) one or more parking spaces PS in the input image based on whether the own vehicle V is located in the parking lot PL.

[0521] Specifically, the parking space annotation generator 2213 of the sixth embodiment is configured to generate annotation information, i.e., a parking space annotation code Ea for each parking space PS included in the target image GS, which annotation information further includes parking lot annotation information indicating whether the own vehicle V is located in the parking lot PL. In addition, the processor 21 determines whether the own vehicle V is located in the parking lot PL when inferring one or more parking spaces PS in the input image, and abandons the detection result of the parking space PS, i.e., the inference result, when determining that the own vehicle V is not located in the parking lot PL. This configuration effectively reduces such misidentification and misdetermination. The parking space annotation generator 2213 of the sixth embodiment can be configured to generate annotation information, i.e., a parking space annotation code Ea for each parking space PS included in the target image GS, which annotation information further includes information indicating whether there is a parking lot environment around the own vehicle V. This enables the processor 21 to determine (i) whether the own vehicle V is located in the parking lot P1, or (ii) whether the own vehicle V is not located, or (iii) whether the own vehicle V is located in an unidentified place when performing training of the parameters of the machine learning model LM.

[0522] Fig.77A22 is a flowchart schematically illustrating a parking lot environment annotation information generation routine executed by the processor 21 acting as the annotation information generator 2201 (ie, the parking space annotation generator 2213).

[0523] Reference Fig.77A When receiving the annotation information input through the annotation input unit 2212, the processor 21 determines whether there is a parking lot environment around the own vehicle V, that is, whether the own vehicle V is located in the parking lot PL based on the annotation information input through the annotation input unit 2212.

[0524] When it is determined that the own vehicle V is located in the parking lot PL (YES in step S501 ), the parking lot environment annotation information generation routine proceeds to step S502 .

[0525] Otherwise, when determining that the own vehicle V is not located in the parking lot PL (No in step S501), the processor 21 determines that there is no parking lot environment around the own vehicle V, that is, the own vehicle V is located outside the parking lot PL. Then, in step S503, the processor 21 adds fourth encoded data Ec4 indicating that there is no parking lot environment around the own vehicle V to the parking space annotation code Ea for each parking space PS included in the target image GS.

[0526] There may be a case where, even if the own vehicle V is actually located in the parking lot PL, it is difficult to accurately determine whether the own vehicle V is located in the parking lot PL based on the target image GS due to, for example, a situation in which the parking space PS is positioned against a wall so that the wall appears only around the parking space PS in the target image GS. From this point of view, in step S502, the processor 21 determines whether there are one or more parked vehicles B7 in the target image GS based on the annotation information input through the annotation input unit 2212.

[0527] When determining that one or more parked vehicles B7 exist in the target image GS (Yes in step S502), the processor 21 determines that the own vehicle V is likely to be located in the parking lot PL, that is, there is a high reliability that the own vehicle V is located in the parking lot PL. Then, in step S504, the processor 21 adds fourth encoded data Ec4 indicating that the own vehicle V is located in the parking lot PL to the parking space annotation code Ea for each parking space PS included in the target image GS.

[0528] Alternatively, in step S502 , the processor 21 determines whether one or more parking spaces PS exist in the target image GS based on the annotation information input through the annotation input unit 2212 in step S502 .

[0529] When determining that one or more parking spaces PS exist in the target image GS (Yes in step S502), the processor 21 determines that the own vehicle V is likely to be located in the parking lot PL, that is, there is a high reliability that the own vehicle V is located in the parking lot PL. Then, in step S504, the processor 21 adds fourth encoded data Ec4 indicating that the own vehicle V is located in the parking lot PL to the parking space annotation code Ea for each parking space PS included in the target image GS.

[0530] Otherwise, when determining that the parked vehicle B7 does not exist in the target image GS (No in step S502), the processor 21 determines that the own vehicle V is unlikely to be located in the parking lot PL, that is, there is a low reliability that the own vehicle V is located in the parking lot PL. Similarly, when determining that the parking space PS does not exist in the target image GS (No in step S502), the processor 21 determines that the own vehicle V is unlikely to be located in the parking lot PL, that is, there is a low reliability that the own vehicle V is located in the parking lot PL. Then, in step S505, the processor 21 adds fourth coded data Ec4 indicating that the own vehicle V is located in an unrecognized place to the parking space annotation code Ea for the objectified parking space PS included in the target image GS.

[0531] As described above, the processor 21 of the sixth embodiment is configured to add fourth encoded data Ec4 to the parking space annotation code Ea for each parking space PS included in the target image GS, and the fourth encoded data Ec4 indicates that (i) there is no parking environment around the own vehicle V, (ii) the own vehicle V is located in the parking lot PL, or (iii) the own vehicle V is located in an unidentified location.

[0532] Fig.77B 2 is a flow chart schematically showing a part of the training routine according to the sixth embodiment. Specifically, the processor 21 is programmed to execute, when starting to train the parameters of the machine learning model LM for each parking space PS using the corresponding parking space annotation code Ea including the fourth encoded data Ec4, Fig.77B Part of the training routine shown in .

[0533] Reference Fig.77B In step S601, the processor 21 refers to the parking space annotation code Ea for the selected parking space PS to determine whether the fourth encoded data Ec4 indicates that the own vehicle V is located in the parking lot PL.

[0534] When it is determined that the fourth encoded data Ec4 indicates that the own vehicle V is not located in the parking lot PL (NO in step S601 ), the training routine proceeds to step S602 .

[0535] Otherwise, when it is determined that the fourth encoded data Ec4 indicates that the own vehicle V is located in the parking lot PL (yes in step S601), in step S603, the processor 21 determines the parameters of the machine learning model LM trained for the selected parking space PS when the own vehicle V is located in the parking lot PL.

[0536] In step S602, the processor 21 refers to the parking space annotation code Ea for the selected parking space PS to determine whether the fourth encoded data Ec4 indicates that the own vehicle V is located in an unrecognized place.

[0537] When it is determined that the fourth encoded data Ec4 indicates that the own vehicle V is located in an unidentified place (yes in step S602), in step S604, the processor 21 determines the parameters of the machine learning model LM trained for the selected parking space PS when the own vehicle V is located in an unidentified place.

[0538] Otherwise, when it is determined that the fourth encoded data Ec4 indicates that the own vehicle V is not located in an unidentified place (No in step S602), in step S605, the processor 21 determines the parameters of the machine learning model LM trained for the selected parking space PS when the own vehicle V is located outside the parking lot PL.

[0539] Fig.78 An example of the fourth coded data Ec4 indicating that the own vehicle V is located in an unrecognized place or the own vehicle V is located outside the parking lot PL is shown. Specifically, the coded values ​​Ecv of all cells in the target image are set to 0. On the contrary, Fig.79 An example of fourth coded data Ec4 indicating that the own vehicle V is located in the parking lot PL is shown. Specifically, the coded values ​​Ecv of all cells in the target image are set to 1.

[0540] Fig.80 A parking space annotation code Ea is schematically shown for each parking space PS included in the target image GS, the parking space annotation code Ea further comprising Fig.78 or Fig.79 The fourth coded data Ec4 indicating whether there is a parking lot environment around the own vehicle V is shown in FIG. Fig.80 The illustration of the second encoded data Ec2 and the third encoded data Ec3 is omitted. Fig.80 The first coded data Ec1 shown in Fig.68 The first encoded data set shown in is replaced by the first encoded data set including first encoded data items Ec1 provided for different orientations of each parking space PS.

[0541] Fig.812 is a flowchart schematically showing a parking space detection routine (ie, parking space inference routine) executed by the processor 21 according to the sixth embodiment.

[0542] Reference Fig.81 , when the parking space detection routine is started, in step S700, the processor 21 detects (ie, infers) a plurality of parking space candidates based on an input image using the machine learning model LM.

[0543] Next, in step S701 , the processor 21 determines whether there is a parking lot environment around the own vehicle V, that is, whether the own vehicle V is located in the parking lot PL.

[0544] Upon determining that a parking lot environment exists around the own vehicle V, that is, the own vehicle V is located in the parking lot PL (YES in step S701 ), in step S702 , the processor 21 adopts all parking space candidates as the parking space PS.

[0545] Otherwise, upon determining that there is no parking lot environment around the own vehicle V, that is, the own vehicle V is not located in the parking lot PL (NO in step S701 ), in step S703 , the processor 21 abandons all parking space candidates.

[0546] Seventh embodiment

[0547] Refer to the following Fig.82 and Fig.83 A parking space detection apparatus 2200 according to a seventh embodiment of the present disclosure is described.

[0548] The seventh embodiment describes the fifth encoding of the wheel stopper information related to the wheel stopper B6 as a part of the attribute information.

[0549] Fig.82 An example of the fifth encoded data Ec5 as a result of the fifth encoding is shown. Specifically, the training process imparts Fig.32 The coded data value Ecv of 0 or 1 associated with the presence or absence of the wheel stopper B6 in each parking space PS included in the target image GS shown in FIG. Fig.33 , Fig.34 , Fig.38 and Fig.82 As can be seen in FIG. 1 , the training process assigns (i) a coded data value Ecv of “1” to the center cell of each selected parking space PS in which the wheel stopper B6 is present, and assigns (ii) a coded data value Ecv of “0” to each of the other cells in the target image GS.

[0550] Fig.83A parking space annotation code Ea is schematically shown for each parking space PS included in the target image GS, the parking space annotation code Ea further comprising Fig.82 The fifth coded data Ec5 shown in FIG. For simplicity of explanation, Fig.83 The illustration of the second to fourth encoded data Ec2 to Ec4 is omitted. Fig.83 The first coded data Ecv1 shown in Fig.68 The first encoded data set shown in is replaced by the first encoded data set including first encoded data items Ec1 provided for different orientations of each parking space PS.

[0551] Eighth Embodiment

[0552] Refer to the following Fig.84 and Fig.85 A parking space detection apparatus 2200 according to an eighth embodiment of the present disclosure is described.

[0553] The eighth embodiment describes the sixth encoding of additional attribute information different from the parking segment information related to the parking segment B5 and the wheel stopper information related to the wheel stopper B6. The additional attribute information may include, for example, the presence or absence of a painted object (e.g., a painted mark indicating double parking, diagonal double parking, parallel parking, an international wheelchair symbol, or an electric vehicle mark) in each parking space PS.

[0554] Fig.84 An example of the sixth encoded data Ec6 as a result of the sixth encoding is shown. Specifically, the training process imparts Fig.32 The coded data value Ecv of 0 or 1 associated with the presence or absence of the additional attribute information in each parking space PS included in the target image GS shown in FIG. Fig.33 , Fig.34 , Fig.38 and Fig.82 As can be seen in , the training process assigns (i) a coded data value Ecv of “1” to the center cell of each selected parking space PS in which attribute information exists, and assigns (ii) a coded data value Ecv of “0” to each of the other cells in the target image GS.

[0555] Fig.85 A parking space annotation code Ea is schematically shown for each parking space PS included in the target image GS, the parking space annotation code Ea further comprising Fig.84 The sixth coded data Ec6 shown in FIG. For simplicity of explanation, Fig.85 The illustration of the second to fifth encoded data Ec2 to Ec5 is omitted. Fig.85 The first coded data Ecv1 shown in Fig.68 The first encoded data set shown in is replaced by the first encoded data set including first encoded data items Ec1 provided for different orientations of each parking space PS.

[0556] Ninth embodiment

[0557] Refer to the following Figures 86 to 118 A parking space detection apparatus 2200 according to a ninth embodiment of the present disclosure is described.

[0558] The processor 21 of the ninth embodiment is configured to determine whether there is a parking lot environment around the own vehicle V, that is, whether the own vehicle V is located in the parking lot PL, based on the detection result or recognition result of the parking space PS. The result of the parking lot environment determination can be used to calculate the driving route of the own vehicle V to the target parking space PS during parking assistance or automatic parking. Alternatively, the parking lot environment determination can be performed in response to receiving a learning image (that is, a target image). In other words, the processor 21 can perform the parking lot environment determination when generating the parking space annotation information.

[0559] Refer to the following Fig.86 and Fig.87 The parking lot environment determination is schematically described.

[0560] First, in Fig.86 In step S801, the processor 21 obtains the parking space list PSs. Fig.87 An example of parking space rows PSs is shown.

[0561] Next, in step S802, the processor 21 generates a parking lot environment determination group. What a parking lot environment determination group is and how the processor 21 generates the parking lot environment determination group will be described later. Next, in step S803, the processor 21 performs parking lot environment determination of whether the own vehicle V is located in the parking lot PL based on the generated parking lot environment determination group.

[0562] The operations in steps S801 to S803 are described in detail below.

[0563] Parking space list acquisition

[0564] The parking space line PSs represents a line of parking spaces PS arranged in a predetermined direction and satisfying predetermined conditions to be described later. Fig.87 As shown, the parking space list PSs represents the alignment of the parking spaces PS divided by the parking segments B5. Each parking segment B5 has any shape. There may be one or more parked vehicles B7 in the corresponding one or more parking spaces PS, or there may be no parked vehicle B7 in the parking space PS. Fig.88As shown, even if there is no parking segment B5, the parking space array PSs can be defined. Specifically, the virtual circumscribed rectangle around each parked vehicle B7 can be defined as a virtual parking space PSv. Each virtual parking space PSv has a pair of long sides corresponding to the total longitudinal dimension of the corresponding parked vehicle V7 and a pair of short sides corresponding to the transverse dimension (i.e., width dimension) of the corresponding parked vehicle V7. The virtual parking space PSv is aligned with its space SP less than a predetermined length, so that the parking space array PSs can be defined.

[0565] like Fig.89 As shown, the combination of the alignment of the parking spaces PS (each of which is surrounded by the corresponding parking segment B5) and the alignment of the virtual parking spaces PSv (each of which is not surrounded by the parking segment B5) can define the parking space column PSs. Unless technically contradictory or otherwise specified, the parking spaces PS (each of which has a corresponding parking segment B5) and the virtual parking spaces PSv (each of which has no parking segment B5) will be collectively referred to as parking spaces PS. Therefore, Fig.90 The parking spaces PS shown in the subsequent figures and the corresponding descriptions described may include: (i) parking spaces, each of which is surrounded by a corresponding parking segment B5; and (ii) virtual parking spaces PSv, each of which is not surrounded by a parking segment B5.

[0566] As described above, the processor 21 obtains the parking space list PSs according to the parking segment B5 and / or the parked vehicle B7.

[0567] Fig.90 Parallel parking spaces PSs are shown. Fig.91 There are diagonal parallel parking spaces listed for PSs. Fig.92 The processor 21 of the ninth embodiment is configured to identify a parking space array PSs for each of the parking types (ie, parallel parking, diagonal parallel parking, and parallel parking). In other words, one parking space array PSs is configured not to include a plurality of types of parking space arrays.

[0568] The following reference Figures 93 to 97 An example of instructing the processor 21 how to acquire the parking space list PSs is described.

[0569] First, the processor 21 selects Fig.93 One of the identified parking spaces PS oriented in the x-direction shown is used as a reference object PSb. Specifically, the processor 21 identifies the orientation of the identified parking space PS to accordingly determine the x-direction corresponding to the orientation direction. One parking space selected from the parking spaces PS constituting the temporary parking space column or the final parking space column will be referred to as an initially selected object PS1. Fig.93, the reference object PSb and the initially selected object PS1 are the same parking space.

[0570] Next, the processor 21 performs a search task of searching for the parking space PS closest to the reference object PSb in the search direction Dt corresponding to the first direction as the positive side of the x-axis direction, thereby finding the nearest parking space PS as the nearest object PSn. Fig.93 in the right direction.

[0571] Thereafter, the search task determines whether the reference object PSb and the nearest object PSn should constitute the parking space line PSs based on the positional relationship between the reference object PSb and the nearest object PSn (see Fig.94 Specifically, when it is determined that the following first to third continuous conditions are satisfied, the processor 21 determines that the reference object PSb and the nearest object PSn have a positional relationship that should constitute the parking space row PSs:

[0572] (I) The first continuity condition is that the interval SP between the reference object PSb and the nearest object PSn in the x-axis direction is less than or equal to a predetermined value, for example, 2 m.

[0573] (II) The second continuity condition is that the positional deviation between the reference object PSb and the nearest object PSn in the y-axis direction perpendicular to the x-axis direction is less than or equal to a predetermined value, for example, 1.8 m.

[0574] (III) The third continuity condition is that the angle Δθ formed between the center line LCb of the reference object PSb and the center line LCn of the nearest object PSn is within a predetermined angle of, for example, 30°.

[0575] exist Fig.94 , the position deviation between the reference object PSb and the nearest object PSn in the y-axis direction represents the offset between the reference object PSb and the nearest object PSn in the y-axis direction. In other words, if each of the reference object PSb and the nearest object PSn has a first end and a second end opposite to each other in the y-axis direction, the position deviation between the reference object PSb and the nearest object PSn in the y-axis direction represents the position difference between the first end of the reference object PSb and the first end of the nearest object PSn in the y-axis direction. The center line LCb of the reference object PSb is a virtual line passing through the center of the reference object PSb and parallel to its long side. The center line LCn of the nearest object PSn is a virtual line passing through the center of the nearest object PSn and parallel to its long side.

[0576] When it is determined that the first to third continuous conditions are satisfied between the reference object PSb and the nearest object PSn, the search task recognizes that the reference object PSb and the nearest object PSn constitute the temporary parking space array PSs, and sets the nearest object PSn as a new reference object PSb, and sets the previous reference object PSb as the initially selected object PS1. This causes the reference object PSb to have been shifted from the initially selected object PSb to the nearest object in the search direction Dt. Fig.95 The schematic diagram shows Fig.94 The parking space PS shown is a parking space PS that has been displaced in the search direction Dt compared to the position of the reference object PSb.

[0577] Then, the search task retrieves the parking space PS closest to the reference object PSb in the search direction Dt, thereby finding the closest parking space PS as the closest object PSn. Thereafter, the search task determines whether the first to third continuous conditions are satisfied between the reference object PSb and the closest object PSn based on the positional relationship between the reference object PSb and the closest object PSn. When it is determined that the first to third continuous conditions are satisfied between the reference object PSb and the closest object PSn, the processor 21 adds the closest object PSn to the temporary parking space list PSs including the initially selected object PS1, thereby identifying that the initially selected object PS1, the reference object PSb, and the nearest object PSn constitute the temporary parking space list PSs.

[0578] As described above, the processor 21 of the ninth embodiment is configured to search for a plurality of parking spaces PS that should constitute the parking space line PSs in the search direction Dt corresponding to the positive side in the x-axis direction, using the initially selected object PS1 as a starting point.

[0579] When it is determined that the nearest object PSn is not found or at least one of the first to third continuous conditions is not satisfied, the processor 21 uses the initially selected object PS1 as the starting point and terminates the above search task in the search direction Dt corresponding to the positive side of the x-axis direction. Then, the processor 21 returns the reference object PSb to the initially selected object PS1, reverses the search direction Dt, and makes the reversed search direction Dt correspond to the second first direction which is the negative side of the x-axis direction opposite to the positive side of the x-axis direction, and as Fig.96 As shown, a search task of searching for the parking space PS closest to the reference object PSb in the search direction Dt is performed, thereby finding the nearest parking space PS as the nearest object PSn. Fig.96 in the left direction.

[0580] Thereafter, when it is determined that the nearest object PSn is not found in the search task in the search direction Dt or at least one of the first to third continuous conditions is not satisfied, the processor 21 terminates the above search task in the search direction Dt, thereby terminating the acquisition of the temporary parking space list PSs including the initially selected object PS1.

[0581] Next, the processor 21 selects a parking space PS that does not belong to one or more identified parking space sequences PSs and is not set as a reference object PSb (i.e., the initially selected object PS1), and starts the search task in a search direction Dt corresponding to the positive side of the x-axis direction using the reference object PSb (i.e., the initially selected object PS1) as a starting point.

[0582] The processor 21 determines, after acquisition of the temporary parking space list PSs, whether the number of parking spaces PS included in the identified temporary parking space list PSs is greater than or equal to a predetermined threshold number.

[0583] Then, when it is determined that the number of parking spaces PS included in the identified temporary parking space list PSs is greater than or equal to the predetermined threshold number, the processor 21 finalizes the identified temporary parking space list PSs as the final parking space list PSs. Otherwise, when it is determined that the number of parking spaces PS included in the identified temporary parking space list PSs is less than the predetermined threshold number, the processor 21 abandons the identified temporary parking space list PSs without finalizing it.

[0584] Specifically, when determining that the number condition indicating the number of parking spaces PS included in the parking space array PSs is greater than or equal to the predetermined threshold number and that the first to third continuous conditions are satisfied, the processor 21 determines that the above predetermined condition is satisfied. The threshold number may be set to, for example, 5, but may be freely set to Q, which is a positive integer greater than or equal to 3. The threshold number may be variably set according to any one of the parking types (i.e., parallel parking, diagonal parallel parking, or parallel parking). For example, for parallel parking, Q is set to 5; for diagonal parallel parking, Q is set to 4; or for parallel parking, Q is set to 3. The value of Q will also be referred to as a parking space array acquisition threshold.

[0585] As described above, the processor 21 of the ninth embodiment is configured to select one of the identified parking spaces PS as a reference object PSb, and retrieve the parking spaces PS in both the first direction and the second direction opposite to the first direction from the reference object PSb. This configuration enables the processor 21 to identify a parking space array PSs including parking spaces PS whose number is greater than or equal to the threshold number; the parking spaces PS satisfy the first to third continuous conditions. Therefore, this enables the processor 21 included in the own vehicle V traveling in the passage of the parking lot PL to reliably identify one or more parking space arrays PSs, thereby determining whether the own vehicle V is located in the parking lot PL with higher accuracy based on the identified one or more parking space arrays PSs.

[0586] Next, refer to the following Fig.98A and Fig.98B Description and Fig.86 The operation in step S801 shown here is an example of a parking space array acquisition routine.

[0587] Reference Fig.98A and Fig.98B , the processor 21 sequentially executes the operations in step S901 to step S905 of the parking space array acquisition routine.

[0588] In step S901, the processor 21 obtains the recognition result of the parking space PS from the input image or the learning image. Next, in step S902, the processor 21 selects one of the recognized parking spaces PS as the initially selected object PS1 from the recognized parking spaces PS. After the operation in step S902, in step S903, the processor 21 sets the initially selected object PS1 as the reference object PSb. Next, in step S904, the processor 21 searches for the parking space PS in the first direction from the reference object PSb to accordingly select the object PSn closest to the reference object PSb as the search result.

[0589] After the operation in step S904, in step S905, the processor 21 determines whether the positional relationship between the reference object PSb and the nearest object PSn satisfies the first to third continuous conditions. When the positional relationship between the reference object PSb and the nearest object PSn does not satisfy at least one of the first to third continuous conditions (No in step S905), the determination in step S905 is negative. Additionally, when the processor 21 does not find any parking space PS in the first direction from the reference object PSb (No in step S905), the determination in step S905 is negative. That is, when the processor 21 finds the nearest object PSn and the positional relationship between the reference object PSb and the nearest object PSn satisfies the first to third continuous conditions, the determination in step S905 is affirmative (Yes in step S905).

[0590] In response to an affirmative determination in step S905 , the parking space row acquisition routine proceeds to step S906 .

[0591] In step S906, the processor 21 adds the nearest object PSn to the temporary parking space list PSs including the reference object PSb, and thereafter determines in step S907 whether there is at least one unselected parking space PS in the search direction Dt. Upon determining that there is at least one unselected parking space PS in the search direction Dt (yes in step S907), the processor 21 shifts the selection of the reference object from the initially selected object PS1 to the nearest object PSn as the reference object PSb. Specifically, in step S908, the processor 21 newly sets the nearest object PSn as the reference object PSb, and returns the previous reference object PSb to the initially selected object PS1. Otherwise, upon determining that there is no unselected parking space PS in the search direction Dt (no in step S907), the parking space list acquisition routine proceeds to step S909. Additionally, when the determination in step S905 is negative, the parking space list acquisition routine proceeds to step S909.

[0592] In step S909, the processor 21 determines whether the search direction Dt is currently set to the second direction, that is, the search direction Dt has been reversed from the first direction to the second direction. When it is determined that the search direction Dt is not currently set to the second direction, that is, the search direction Dt has not been reversed from the first direction to the second direction (No in step S909), the parking space row acquisition routine proceeds to step S910. In step S910, the processor 21 reverses the search direction Dt from the first direction to the second direction, and thereafter returns to the operation in step S903. Then, the processor 21 performs the operations in steps S903 to S909 on the search direction Dt set to the second direction until the determination in step S909 is affirmative.

[0593] When the search task in each of the first direction and the second direction has been completed, the determination in step S909 is affirmative (Yes in step S909). Then, in step S911, the processor 21 determines whether the number of parking spaces PS included in the temporary parking space list PSs (i.e., the number of consecutive parking spaces PS) is greater than or equal to a predetermined threshold number.

[0594] When it is determined that the number of parking spaces PS included in the temporary parking space list PSs (i.e., the number of consecutive parking spaces PS) is greater than or equal to the predetermined threshold number (Yes in step S911), the processor 21 finally acquires the temporary parking space list PSs as the final parking space list PSs in step S912. Otherwise, when it is determined that the number of parking spaces PS included in the temporary parking space list PSs (i.e., the number of consecutive parking spaces PS) is less than the predetermined threshold number (No in step S911), the processor 21 abandons the temporary parking space list PSs as the final parking space list PSs in step S913.

[0595] After the operation in step S912 or step S913 is completed, the parking space row acquisition routine proceeds to step S914. In step S914, the processor 21 determines whether there is at least one parking space PS that does not belong to the one or more final parking space rows PSs finally identified in step S912 and is not set as the reference object PSb. Upon determining that there is at least one parking space PS that does not belong to the one or more final parking space rows PSs finally identified in step S912 and is not set as the reference object PSb (yes in step S914), the processor 21 returns to the operation in step S902. This enables the processor 21 to perform the parking space row acquisition tasks in steps S902 to S914 in each of the first direction and the second direction until the determination in step S914 is negative. Otherwise, upon determining that there is no parking space PS that does not belong to the one or more final parking space lists PSs finally identified in step S912 and is not set as the reference object PSb (No in step S914), the processor 21 terminates the parking space list acquisition routine corresponding to the operation in step S801.

[0596] Generation of parking areas

[0597] Next, refer to the following Figures 99 to 103 Described in Fig.86 The operation of generating at least one parking lot environment determination group in step S802. Hereinafter, at least one parking lot environment determination group will be referred to as at least one parking area PSg. Fig.99As shown, at least one parking area PSg is at least one virtual area including at least one parking space array PSs and a passageway PSr of a parking lot PL. Specifically, when the own vehicle V is traveling in the passageway PSr or is about to travel in the passageway PSr, the processor 21 generally acquires at least one parking area PSg consisting of the passageway PSr and the parking space arrays PSs located on both sides of the passageway PSr. That is, the processor 21 of the ninth embodiment is programmed to acquire at least one parking area PSg based on the parking space arrays PSs identified in step S801, the parking area PSg including (i) the parking space arrays PSs and (ii) an area in which the own vehicle V can travel and faces the parking space arrays PSs. The processor 21 can acquire at least one parking area PSg based on the number of parking space arrays PSs and / or the existence reliability and / or accuracy reliability of each parking space array PSs as a parking area acquisition condition. Then, the processor 21 of the ninth embodiment is programmed to determine whether the own vehicle V is located in the parking lot PL based on the generated at least one parking area PSg. Specifically, the processor 21 of the ninth embodiment is programmed to determine whether the own vehicle V is located inside or outside the parking lot PL based on the positional relationship between the own vehicle V and at least one parking area PSg, such as information about whether the own vehicle V has entered at least one parking area PSg.

[0598] The processor 21 of the ninth embodiment may be programmed to extend each parking space row PSs and at least one parking area PSg when the own vehicle V travels.

[0599] Fig.99 and Fig.100 A first example showing how the processor 21 extends a typical parking area PSg constituted by a passage PSr and parking space rows PSs located on both sides of the passage PSr is shown. Fig.99 The situation where the own vehicle V approaches the parking area PSg is shown, and Fig.100 Shows the own vehicle V ratio Fig.99 The situation shown is closer to the next situation of the parking area PSg. Fig.99 and Fig.100 The processor 21 of the first example shown is programmed to sequentially add a newly identified parking space PS to each of the parking space columns PSs constituting the parking area PSg when the own vehicle V approaches the parking area PSg, the newly identified parking space PS being oriented in the same direction as the alignment of the parking spaces PS included in each of the parking space columns PSs.

[0600] Figure 101 to Figure 103 A second example showing how the processor 21 extends a typical parking area PSg constituted by a passage PSr and parking space rows PSs located on both sides of the passage PSr is shown. Fig.101shows a situation in which the own vehicle V is traveling to partially enter the parking area PSg, and Fig.102 The next situation is shown in which the own vehicle V is traveling to completely enter the parking area PSg. Fig.103 Another situation is shown in which the own vehicle V is traveling to enter deeper into the parking area PSg.

[0601] according to Figure 101 to Figure 103 The processor 21 of the second example shown is programmed to extend the time interval between the start and end of the process as the own vehicle V is traveling behind. Fig.101 The parking area PSg obtained in the case shown; Fig.101 The parking area PSg acquired in the illustrated case is composed of a passageway PSr and parking space rows PSs located on both sides of the passageway PSr, each of the parking space rows PSs having five parking spaces PS.

[0602] The processor 21 is programmed to create a copy of the acquired parking area PSg composed of the passageway PSr and the parking space rows PSs located on both sides of the passageway PSr to continuously join the copied parking area PSg to the acquired parking area PSg in the traveling direction of the own vehicle V. Specifically, the processor 21 is programmed to continuously join the copied parking area PSg having the same size as the acquired parking area PSg to the acquired parking area PSg in the traveling direction of the own vehicle V, thereby extending the parking area PSg twice in the traveling direction. The processor 21 may be programmed to continuously join a plurality of copied parking areas PSg to the acquired parking area PSg in the traveling direction of the own vehicle V, each of the plurality of copied parking areas PSg having the same size as the acquired parking area PSg, thereby extending the parking area PSg in the traveling direction by K times; K is a positive integer of two or more. In response to determining that the parking space rows PSs located on both sides of the passageway each include ten or more parking spaces PS, the processor 21 may be programmed to finally determine the virtual extended parking area PSg.

[0603] Fig.102 It is shown that the parking area PSg extends twice in the direction of travel. Fig.102 It is shown that the parking space rows PSs located on both sides of the passage each include only eight parking spaces PS, so that there are two additional parking spaces PS in each extended parking space row PSs included in the extended parking area PSg (see the double-dashed line). For this reason, the two additional parking spaces PS are subtracted from each extended parking space row PSs included in the extended parking area PSg, thereby correcting the size of each extended parking space row PSs and the size of the extended parking area PSg, so that the corrected parking area PSg including the corrected parking space rows PSs can be created.

[0604] As described above, the processor 21 of the ninth embodiment is configured to extend the acquired parking area PSg in the predetermined direction so that the extended parking area PSg includes the additional parking space PS or the additional parking space row PSs located adjacent to the acquired parking area PSg if there is an additional parking space PS or an additional parking space row PSs located adjacent to the acquired parking area PSg in the predetermined direction. This configuration makes it possible to extend the acquired parking area PSg according to the additionally recognized parking space PS or parking space row PSs as the own vehicle V travels toward the acquired parking area PSg. Therefore, this makes it possible to correctly determine whether the own vehicle V is located inside the parking lot PL or outside the parking lot PL.

[0605] Determine whether the vehicle is in the parking lot

[0606] Next, refer to the following Figures 104 to 113 Described in Fig.86 The operation of determining whether the own vehicle V is located in the parking lot PL in step S803.

[0607] For example, in Figures 104 to 106 , reference symbol “ON” indicates “determination in the parking lot” that the own vehicle V is located in the parking lot PL, and reference symbol “OFF” indicates “determination outside the parking lot” that the own vehicle V is not located in the parking lot PL. The processor 21 of the ninth embodiment is typically programmed to perform determination in the parking lot that the own vehicle V is located in the parking lot PL when determining that the own vehicle V has partially entered the parking area PSg, and to perform determination outside the parking lot that the own vehicle V is not located in the parking lot PL when determining that the own vehicle V has left the parking area PSg.

[0608] Fig.105 An example case is shown in which at least the first parking area PSg and the second parking area PSg are oriented in the driving direction of the own vehicle V and the own vehicle V is planned to travel through the first parking area PSg and the second parking area PSg. In this example case, when the own vehicle V has reached the space between the first parking area PSg and the second parking area PSg, the own vehicle V has already left the first parking area PSg, so that the above determination within the parking lot should have temporarily switched to the determination outside the parking lot. However, in this example case, there is a requirement not to switch the above determination within the parking lot to the determination outside the parking lot in the space between the first parking area PSg and the second parking area PSg. That is, in this example case, it is preferable that the determination within the parking lot is maintained even when the own vehicle V has left the first parking area PSg. This makes it possible to achieve the same Fig.106The same advantageous effects are achieved in the case shown, namely that the first parking area extends in the traveling direction of the own vehicle V, in other words, the first parking area and the second parking area are combined to form an extended parking area.

[0609] Fig.107 A parking area PSg is shown, which includes: (i) a T-shaped passage PSr; (ii) a first parking space row PSs and a second parking space row PSs located on both sides of a leg section of the T-shaped passage PSr in parallel with the driving direction of the own vehicle V; and (iii) a third parking space row PSs located to be separated from the first parking space row PSs and the second parking space row PSs by a head section of the T-shaped passage PSr; the orientation of each of the first parking space row PSs and the second parking space row PSs in parallel with the driving direction of the own vehicle V is orthogonal to the orientation of the third parking space row PSs orthogonal to the driving direction of the own vehicle V. At this time, the own vehicle V travels in the leg section of the T-shaped passage PSr between the first parking space row PSs and the second parking space row PSs.

[0610] exist Fig.107 In the example case shown, the processor 21 can be programmed to perform a determination within the parking lot when the own vehicle V is traveling in the T-shaped passage PSr. Specifically, it is assumed that the head section of the T-shaped passage PSr is divided into a first region R1, a second region R2, and a third region R3. The first region R1 is positioned to intersect with the driving route of the own vehicle V. The second region R2 is located on the left side of the first region R1 and indicates a left turn destination. The third region R3 is located on the right side of the first region R1 and indicates a right turn destination. Because the first region R1 is not a high reliability region sandwiched from both sides by the parking space arrays PSs, when the own vehicle V is traveling in the first region R1, it is preferred to maintain the determination within the parking lot without switching the determination within the parking lot to the determination outside the parking lot.

[0611] Fig.108 A parking area PSg is shown that includes a cross-shaped passageway PSr and four parking space rows PSs arranged such that each section of the cross-shaped passageway PSr is sandwiched by a corresponding pair of parking space rows PSs.

[0612] Fig.109A parking area PSg is shown, which includes: (i) a T-shaped passage PSr; (ii) a first parking space row PSs and a second parking space row PSs located on both sides of a leg section of the T-shaped passage PSr so as to be orthogonal to the driving direction of the own vehicle V; and (iii) a third parking space row PSs located so as to be separated from the first parking space row PSs and the second parking space row PSs by a head section of the T-shaped passage PSr; the orientation of each of the first parking space row PSs and the second parking space row PSs orthogonal to the driving direction of the own vehicle V is orthogonal to the orientation of the third parking space row PSs parallel to the driving direction of the own vehicle V. At this time, the own vehicle V can directly drive through the head section of the T-shaped passage PSr and turn left from the head section of the T-shaped passage PSr to the leg section of the T-shaped passage PSr.

[0613] When the vehicle V Fig.108 The cross-shaped channels PSr and Fig.109 When traveling in each of the illustrated T-shaped corridors PSr, it is preferred to maintain the determination that the vehicle is in the parking lot without switching the determination that the vehicle is in the parking lot to the determination that the vehicle is outside the parking lot.

[0614] Fig.110 Shown with Figure 7 A parking area PSg similar to the parking area PSg shown in FIG. 1 includes: (i) a T-shaped passage PSr; (ii) a first parking space row PSs and a second parking space row PSs located on both sides of a leg section of the T-shaped passage PSr; and (iii) a third parking space row PSs positioned to be separated from the first parking space row PSs and the second parking space row PSs by a head section of the T-shaped passage PSr; the orientation of each of the first parking space row PSs and the second parking space row PSs is orthogonal to the orientation of the third parking space row PSs.

[0615] exist Fig.110 In the case shown, the processor 21 may be programmed to change one of the determination in the parking lot and the determination outside the parking lot to the other one according to the driving direction of the own vehicle V toward the parking area PSg. Fig.110 The parking space row PSs including a plurality of parking spaces PS is located in front of the own vehicle V and intersects with the driving direction of the own vehicle V, that is, perpendicular to the driving direction of the own vehicle V. Such a situation may occur when the own vehicle V is waiting for a traffic light on a road Rd having three or more driving lanes per road. Fig.110 Compared with the first case where the vehicle V enters the head section of the T-shaped channel PSr from the left side, Fig.110The second case where the upper or lower side of the T-shaped channel PSr enters the head section (see Fig.110 The solid line in the figure has a higher reliability that the own vehicle V is located in the parking lot PL.

[0616] To this end, the processor 21 may be programmed to establish Fig.110 As shown, (i) a first entrance / exit determination line PSL located at the top end of the head section of the T-shaped channel PSr indicates the boundary of the parking area PSg at the upper side of the parking area PSg, and (ii) a second entrance / exit determination line PSL located at the bottom end of the head section of the T-shaped channel PSr indicates the boundary of the parking area PSg at the lower side of the parking area PSg. Then, the processor 21 can be programmed to perform a parking determination that the own vehicle V is located in the parking lot PL, that is, to perform an "ON" determination, when it is determined that the own vehicle V has crossed the first entrance / exit determination line PSL or the second entrance / exit determination line PSL to enter the parking area PSg. In contrast, the processor 21 can be programmed to perform an "ON" determination when it is determined that the own vehicle V has not crossed the first entrance / exit determination line PSL or the second entrance / exit determination line PSL to enter the parking area PSg (see Fig.110 ), the determination in the parking lot is not performed, that is, the “ON” determination is not performed.

[0617] That is, the processor 21 of the ninth embodiment is configured to perform determination in the parking lot when determining that the own vehicle V has entered the parking area PSg from a specified direction (i.e., from an upper direction or a lower direction), and is configured not to perform determination in the parking lot when determining that the own vehicle V has entered the parking area PSg from another direction different from the specified direction. Therefore, this makes it possible to perform determination in the parking lot with higher reliability.

[0618] Fig.111A case is shown in which the first parking area PSg1, the second parking area PSg2, and the third parking area PSg3 are arranged in this order to form a substantially L shape as a whole, and the own vehicle V is about to pass through the first parking area PSg1, the second parking area PSg2, and the third parking area PSg3 in this order along the L-shaped planned travel route TP. The first parking area PSg1 includes a first area boundary PSL1 located at the entrance thereof, and the second area boundary PSL2 indicates a boundary between the exit of the first parking area PSg1 and the entrance of the second parking area PSg2. The third area boundary PSL3 indicates a boundary between the exit of the second parking area PSg2 and the entrance of the third parking area PSg3. Because the first parking area PSg1, the second parking area PSg2, and the third parking area PSg3 are continuously arranged, the first area boundary PSL1 serves as an entrance / exit determination line PSL. Therefore, the processor 21 may be configured to maintain determination within the parking lot even when the own vehicle V is passing through the second area boundary PSL2 or the third area boundary PSL3.

[0619] Fig.112 A situation is shown in which the host vehicle V plans to enter a T-shaped passage PSr in which the host vehicle V can travel straight ahead or turn left.

[0620] exist Fig.112 In the embodiment, the rectangular parking spaces PSa and PSb are located on the left side of the head section of the T-shaped passage PSr, so that they are separated from each other by the leg section of the T-shaped passage PSr and their long sides are parallel to the straight-moving direction of the own vehicle V. Additionally, the parallel parking space row PSs is located on the right side of the head section of the T-shaped passage PSr. Therefore, this results in the first parking area PSg1 being composed of the head section of the T-shaped passage PSr and the parking spaces PSa and PSb located on the left side of the head section of the T-shaped passage PSr.

[0621] Additionally, in Fig.112 In the embodiment, the first parking space row PSs and the second parking space row PSs are located on both sides of the leg section of the T-shaped passage PSr, so that the second parking area PSg2 is composed of the leg section of the T-shaped passage PSr and the first parking space row PSs and the second parking space row PSs located on both sides of the leg section of the T-shaped passage PSr.

[0622] That is to say, Fig.112As shown, the first parking area PSg1 and the second parking area PSg2 partially overlap each other. The first parking area PSg1 has an entrance through which the own vehicle V can enter the first parking area PSg1, so that the entrance of the first parking area PSg1 serves as the first area boundary PSL1. The second parking area PSg2 has an exit through which the own vehicle V can leave the second parking area PSg2, so that the exit of the second parking area PSg2 serves as the second area boundary PSL2. That is, each of the first area boundary PSL1 and the second area boundary PSL2 serves as the entrance / exit determination line PSL. This enables the processor 21 to maintain the parking interior determination even if the own vehicle V is crossing the boundary between the overlapping portion of the first parking area PSg1 and the second parking area PSg2.

[0623] Fig.113 A case is shown in which the own vehicle V traveling straight is about to pass through the first parking area PSg1 and the second parking area PSg2 which are arranged continuously in the traveling direction of the own vehicle V in such a manner that their parts overlap. The first parking area PSg1 has an entrance through which the own vehicle V can enter the first parking area PSg1, so that the entrance of the first parking area PSg1 is used as the first area boundary PSL1, that is, the entrance / exit determination line PSL. The first parking area PSg1 also has an exit, and the exit of the first parking area PSg1 is used as the second area boundary PSL2. Because the second area boundary PSL2 is located in the second parking area PSg2, the exit of the first parking area PSg1, that is, the second area boundary PSL2 of the first parking area PSg1 is not used as the entrance / exit determination line PSL. This enables the processor 21 to maintain determination within the parking lot even if the own vehicle V is passing through the second area boundary PSL2 of the first parking area PSg1.

[0624] Accelerator misoperation solution routine

[0625] The processor 21 of the ninth embodiment can be configured to execute an accelerator misoperation resolution routine including a driving power reduction control task and / or an accelerator misoperation notification task of the own vehicle V when it is determined that the own vehicle V is located in the parking lot PL, thereby preventing sudden acceleration of the own vehicle V accordingly when it is determined that the own vehicle V is located in the parking lot PL. Fig.114 and Fig.115 FIG. 2 shows an example of an accelerator misoperation solution routine according to the ninth embodiment. The processor 21 may be programmed to execute the routine every predetermined period, for example, 100 milliseconds, when a predetermined execution condition is satisfied. Fig.114 The accelerator misoperation resolution routine shown in FIG. 1 may include, for example, setting the current shift position of the shift lever to a forward position or a reverse position.

[0626] Reference Fig.114 , in step S1001, when the accelerator maloperation solution routine is started, the processor 21 determines whether the processor 21 performs a determination in the parking lot indicating that the own vehicle V is located in the parking lot PL, that is, performs an "ON" determination. When it is determined that the processor 21 performs a determination in the parking lot so that the own vehicle V is located in the parking lot PL (yes in step S1001), the accelerator maloperation solution routine proceeds to step S1002. Otherwise, when it is determined that the processor 21 performs a determination outside the parking lot indicating that the own vehicle V is not located in the parking lot PL, that is, performs an "ON" determination, the processor 21 temporarily terminates the accelerator maloperation solution routine, while skipping the operations in steps S1002 and S1003.

[0627] In step S1002, the processor 21 determines whether an accelerator misoperation has occurred. Specifically, the processor 21 determines whether all of the following first to fifth conditions are satisfied, and when it is determined that all of the following first to fifth conditions are satisfied, it is determined that an accelerator misoperation has occurred (yes in step S1002). In contrast, when it is determined that at least one of the first to fifth conditions is not satisfied, the processor 21 determines that an accelerator misoperation has not occurred (no in step S1002).

[0628] The first condition is that the speed of the own vehicle V is less than a predetermined threshold speed.

[0629] The second condition is that the operation amount of the accelerator pedal of the own vehicle V is greater than or equal to a predetermined acceleration amount threshold value. ...

Claims

1. A method for learning a recognition rule for at least one parking space around a vehicle, the method comprising: objectifying at least one parking space at least partially included in the learning image into at least one objectified parking space, the at least one objectified parking space comprising: (i) a plurality of corner points of the at least one parking space; (ii) a center point of the at least one parking space; and (iii) attribute information about the at least one parking space; generating annotation data for the objectified at least one parking space; Based on the generated annotation data for the objectified at least one parking space, performing learning of a recognition rule for the at least one parking space according to the learning image; receiving an input image of the surroundings of the vehicle captured by a camera installed in the vehicle; and It is determined whether at least one target parking space exists in the input image according to the learned recognition rule.

2. A method for learning a recognition rule for at least one parking space around a vehicle, the method comprising: objectifying at least one parking space at least partially included in the learning image into at least one objectified parking space, the at least one objectified parking space comprising: (i) a plurality of corner points of the at least one parking space; (ii) a center point of the at least one parking space; and (iii) attribute information about the at least one parking space; generating annotation data for the objectified at least one parking space; Based on the generated annotation data for the objectified at least one parking space, learning of a recognition rule for the at least one parking space is performed according to the learning image.

3. The method according to claim 1 or 2, wherein: The attribute information includes at least one of the following: parking frame information associated with a parking frame of the at least one parking space; wheel stop information about a wheel stop located in the at least one parking space; as well as Additional attribute information associated with the at least one parking space, the additional attribute information being different from the parking frame information and the wheel stopper information.

4. The method according to any one of claims 1 to 3, wherein: The at least one parking space includes a plurality of parking spaces, The method further comprises: calculating the number of one or more corner points of at least one of the parking spaces, the one or more corner points of at least one of the parking spaces being occluded by at least one object other than each of the parking spaces, The learning is performed by not learning a recognition rule for at least one of the parking spaces based on the annotation data generated for the objectified at least one of the parking spaces when it is determined that the number of the one or more corner points of at least one of the parking spaces is greater than or equal to a predetermined threshold number.

5. The method according to any one of claims 1 to 4, wherein: The execution of said learning includes: determining whether one or two of the corner points are outside the learning image; When it is determined that the one or two corner points among the corner points are located outside the learning image, offsetting the one or two corner points among the corner points into the learning image; and After shifting the one or two of the corner points into the learning image, the learning of the recognition rule for the at least one parking space is performed from the learning image based on the generated annotation data for the objectified at least one parking space.

6. The method according to claim 5, wherein: The offset is: offsetting the one or two corner points of the corner points into the learning image along a predetermined offset direction, wherein the predetermined offset direction is selected so that a change in the shape of the at least one parking space before and after the offset is smaller than other offset directions.

7. The method according to any one of claims 1 to 6, further comprising: calculating an amount by which a portion of the at least one parking space is blocked by at least one object other than the at least one parking space as a blocking amount, The learning is performed by setting parking space vacancy information about the at least one parking space to an undefined state when it is determined that the calculated shielding amount of the portion of the at least one parking space is greater than or equal to a threshold amount.

8. The method according to claim 7, wherein: The calculation of the blocking amount includes: calculating the blocking amount based on a viewing angle from the vehicle to the at least one parking space.

9. The method according to any one of claims 1 to 8, wherein: The learning is performed by defining typical positions of the at least one parking space and performing the learning for each of the typical positions.

10. The method according to claim 9, wherein: The learning is performed by encoding each of the center point and the corner point of the at least one parking space for each of the typical orientations to obtain encoded data for each typical orientation, and performing the learning using the encoded data for each typical orientation.

11. The method according to any one of claims 1 to 10, further comprising: identifying a plurality of parking spaces as the at least one parking space from the learning image; selecting a parking space from the identified parking spaces as a reference object; searching for at least one parking space row from the reference object in at least one of a predetermined first direction and a second direction opposite to the first direction, the at least one parking space row including the selected parking space, the selected parking space being included in the plurality of parking spaces and being continuously aligned in at least one of the first direction or the second direction to find the at least one parking space row; as well as Based on the information related to the at least one parking space row, a parking lot environment determination is performed as to whether the vehicle is located in a parking lot.

12. The method according to claim 11, wherein: The execution of the parking lot environment determination includes: acquiring a parking area based on the at least one parking space row, the parking area including the at least one parking space row and a passage arranged to face the parking space row, the vehicle being able to travel in the passage; When it is determined that the vehicle has entered the parking area, determining that the vehicle is located in a parking lot; and When it is determined that the vehicle has left the parking area, it is determined that the vehicle is not located in a parking lot.

13. The method according to any one of claims 1 to 10, further comprising: identifying a plurality of parking spaces as the at least one parking space from the learning image; acquiring at least one parking space row based on the plurality of parking spaces, the at least one parking space row including the selected parking space, the selected parking space being included in the plurality of parking spaces and being continuously aligned in a predetermined direction; acquiring a parking area based on the at least one parking space row, the parking area including the at least one parking space row and a passage arranged to face the parking space row, the vehicle being able to travel in the passage; When it is determined that there is at least one additional parking space or at least one additional parking space row located adjacent to the acquired parking area in the predetermined direction, extending the parking area in the predetermined direction; as well as Based on the information related to the extended parking area, a parking lot environment determination is performed as to whether the vehicle is located in a parking lot.

14. The method according to claim 13, wherein: The at least one parking space array is obtained by: obtaining a first parking space array and a second parking space array as the at least one parking space array; as well as The acquisition of the parking area includes: acquiring the parking area configured such that the first parking space row and the second parking space row are located on both sides of the passage.

15. The method according to claim 13 or 14, wherein: The execution of the parking lot environment determination includes: Upon determining that the vehicle has entered the extended parking area, determining that the vehicle is located in a parking lot; and When it is determined that the vehicle has left the extended parking area, it is determined that the vehicle is not located in a parking lot.

16. A method according to any one of claims 12 to 15, wherein: The parking lot environment determination is performed by: when it is determined that the vehicle has entered the parking area from a specified direction, performing the parking lot environment determination that the vehicle is located in a parking lot, and when it is determined that the vehicle has entered the parking area from another direction different from the specified direction, not performing the parking lot environment determination.

17. The method according to any one of claims 11 to 16, further comprising: When sudden acceleration due to accelerator misoperation is detected when it is determined that the vehicle is located in a parking lot, an accelerator misoperation solution task is performed, the accelerator misoperation solution task including at least one of the following: (i) reducing driving power of the vehicle; and (ii) notifying one or more occupants included in the vehicle of the occurrence of the accelerator maloperation.

18. The method of claim 17, wherein: The accelerator misoperation solution task is executed by determining whether the accelerator misoperation has continued for a predetermined time after the accelerator misoperation solution task is executed, and when it is determined that the accelerator misoperation has continued for the predetermined time, executing the accelerator misoperation solution task again.

19. The method according to claim 17 or 18, wherein: The accelerator misoperation solving task is performed by determining whether there is a new accelerator misoperation after the accelerator misoperation is cancelled, and when it is determined that there is a new accelerator misoperation after the accelerator misoperation is cancelled, executing the accelerator misoperation solving task again.

20. A device for learning a recognition rule of at least one parking space around a vehicle, the device comprising: a memory device storing learning program instructions; as well as A processor configured to execute the learning program instructions to accordingly: objectifying at least one parking space at least partially included in the learning image into at least one objectified parking space, the at least one objectified parking space comprising: (i) a plurality of corner points of the at least one parking space; (ii) a center point of the at least one parking space; and (iii) attribute information about the at least one parking space; generating annotation data for the objectified at least one parking space; Based on the generated annotation data for the objectified at least one parking space, learning of a recognition rule for the at least one parking space is performed according to the learning image.

Images