On-vehicle device, on-vehicle device operation method, and on-vehicle device operation program

Abstract

To promote early and wide spread uses of an advanced driving support vehicle or an automatic driving vehicle.SOLUTION: A control device (2) installed on a vehicle comprises: a processor (21); and a memory (22) which is a non-transitive, tangible recording medium for storing a programming instruction used by the processor. Since the processor executes the programming instruction stored in the memory, the processor converts a parking frame (PS) into an object based on a plurality of corner points (BP) and a center point (BPC) that define a shape of the parking frame that is an area in which the vehicle can be parked, the parking frame being captured by an imaging device (34) that captures a surrounding image of the vehicle, and also, the processor learns the parking frame by adding attribute information of the parking frame.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an in-vehicle device mounted in a vehicle, and an in-vehicle device operation method and an in-vehicle device operation program executed by the in-vehicle device. [Background technology]

[0002] In recent years, research and development of so-called driving assistance technology or automatic driving technology for vehicles such as automobiles has been actively carried out, and various proposals have been made regarding this (see, for example, Patent Document 1, etc.). Driving assistance technology is technology that reduces the burden on the driver and enables comfortable and safe driving, and includes, for example, following distance control, lane keeping assistance control, lane change assistance control, parking assistance control, obstacle warning, collision avoidance assistance control, etc. Automatic driving technology is technology that allows a vehicle to run automatically, that is, autonomously, without the need for a driver's driving operation. A vehicle equipped with such driving assistance technology or automatic driving technology is equipped with sensing devices such as cameras and radars to grasp the situation of the area ahead and the surrounding area. Then, vehicle control such as steering, driving, and braking is automatically performed based on the road conditions, etc. acquired by the traveling vehicle using sensing devices such as cameras and map information, etc. It is possible to improve safety and reliability by using a high-precision map including road data for each lane. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 202397 Summary of the Invention [Problem to be solved by the invention]

[0004] Research and development of these types of driving assistance technology and autonomous driving technology has been accelerating in recent years from various perspectives, such as improving the accuracy of detection or recognition by sensing devices and improving convenience. In other words, by further improving such performance, it is possible to promote the early and widespread dissemination of so-called advanced driving assistance vehicles and autonomous driving vehicles. The present disclosure has been made in consideration of the circumstances exemplified above. [Means for solving the problem]

[0005] The in-vehicle device (2) according to claim 1 is a device mounted on a vehicle (V), A processor (21); A memory (22) which is a non-transitory tangible recording medium for storing program instructions used by the processor; Equipped with The processor executes the program instructions stored in the memory to objectify a parking space (PS), which is an area in which the vehicle can be parked and is captured by an imaging device (34) that captures an image of the surroundings of the vehicle, using a plurality of corner points (BP) and a center point (BPC) that define the shape of the parking space, and learns the parking space by assigning attribute information of the parking space. The in-vehicle device operation method according to claim 24 is a method executed by a processor (21) provided in an in-vehicle device (2) mounted on a vehicle (V), The method includes a procedure or process of objectifying a parking space (PS), which is an area in which the vehicle can be parked and is captured by an imaging device (34) that captures an image of the surroundings of the vehicle, using a plurality of corner points (BP) and a center point (BPC) that define the shape of the parking space, and of learning the parking space by assigning attribute information of the parking space. The in-vehicle device operation program according to claim 25 is a computer-readable program executed by a processor (21) provided in an in-vehicle device (2) mounted in a vehicle (V), The processing executed by the in-vehicle device includes processing for objectifying a parking space (PS), which is an area in which the vehicle can be parked and is captured by an imaging device (34) that captures an image of the surroundings of the vehicle, using a plurality of corner points (BP) and a center point (BPC) that define the shape of the parking space, and for learning the parking space by assigning attribute information of the parking space.

[0006] In addition, in each section of the application documents, each element may be given a reference number in parentheses. In this case, the reference number merely indicates an example of the correspondence between the element and the specific configuration described in the embodiment described later. Therefore, the present disclosure is not limited in any way by the description of the reference number. [Brief description of the drawings]

[0007] [Figure 1] 1 is a plan view showing a schematic configuration of a vehicle equipped with an in-vehicle system including a driving ECU as a control device according to an embodiment. [Diagram 2] FIG. 2 is a schematic block diagram showing the overall configuration of the in-vehicle system shown in FIG. [Diagram 3] FIG. 3 is a diagram showing an outline of target detection using the sonar sensor shown in FIG. 2. [Figure 4] FIG. 3 is a diagram showing an outline of target detection using the sonar sensor shown in FIG. 2. [Diagram 5] 3 is a diagram showing an outline of a target detection range of each of the plurality of sonar sensors shown in FIG. 2. [Figure 6] 3 is a diagram illustrating an outline of parking space detection using the sonar sensor illustrated in FIG. 2. [Figure 7] 3 is a diagram showing an outline of target detection using the radar sensor shown in FIG. 2. [Figure 8] 3 is a diagram showing an outline of target detection using the laser radar sensor shown in FIG. 2. [Figure 9] 9 is a diagram illustrating an outline of the configuration and operation of the laser radar sensor illustrated in FIG. 8. [Figure 10]FIG. 3 is a diagram showing an outline of target detection using the camera shown in FIG. 2. [Figure 11] FIG. 3 is a diagram showing an outline of target detection using the camera shown in FIG. 2. [Figure 12] FIG. 3 is a diagram showing an outline of driver state detection using the driver state monitor shown in FIG. 2. [Figure 13] FIG. 3 is a diagram showing an outline of driver state detection using the driver state monitor shown in FIG. 2. [Figure 14] FIG. 3 is a diagram showing a schematic configuration of the head-up display shown in FIG. 2. [Figure 15] 15 is a schematic diagram showing a road surface ahead of the vehicle and a part of the dashboard seen through the front windshield from the viewpoint of the driver shown in FIG. 14. [Figure 16] 3 is a block diagram showing an example of a schematic functional configuration realized by an operation ECU shown in FIG. 2. [Figure 17] 17 is a block diagram showing an example of a schematic functional configuration realized by a recognition unit shown in FIG. 16. [Figure 18] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 19] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 20] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 21] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 22] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Diagram 23] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 24] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Diagram 25] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 26]2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 27] 2 is a schematic diagram showing a specific example of a target to be recognized in the embodiment. FIG. [Figure 28] FIG. 4 is a schematic diagram showing an example of a control scene using a target recognition result in the embodiment. [Figure 29] FIG. 11 is a schematic diagram showing another example of a control scene using a target recognition result in the embodiment. [Diagram 30] 1 is a block diagram showing a schematic functional configuration of a parking frame detection device according to a first embodiment; [Diagram 31] FIG. 31 is a schematic diagram for explaining annotation information generated by the parking space annotation generating unit shown in FIG. 30. [Diagram 32] 31 is a diagram showing an example of a bird's-eye view image for explaining the operation of the annotation information generating unit and the learning unit shown in FIG. 30. FIG. [Diagram 33] FIG. 33 is an enlarged view of a part of the bird's-eye view image shown in FIG. 32. [Diagram 34] FIG. 34 is a diagram showing an example of encoding regarding the presence of a parking space center point corresponding to FIG. 33; [Diagram 35] FIG. 34 is a diagram for explaining the lateral position of the parking frame center point in the grid shown in FIG. 33. [Diagram 36] FIG. 36 is a diagram showing an example of encoding related to the lateral position of the parking frame center point corresponding to FIG. 35; [Figure 37] FIG. 34 is a diagram for explaining the vertical position of the parking frame center point in the grid shown in FIG. 33. [Figure 38] FIG. 38 is a diagram showing an example of encoding related to the vertical position of the parking frame center point corresponding to FIG. 37 . [Figure 39] FIG. 35 is a diagram showing the encoding regarding the presence of the parking space center point shown in FIG. 34 for the entire target image shown in FIG. 32. [Diagram 40] FIG. 37 is a diagram showing the encoding of the lateral position of the parking space center point shown in FIG. 36 for the entire target image shown in FIG. 32. [Diagram 41] FIG. 39 is a diagram showing encoding related to the vertical direction position of the parking space center point shown in FIG. 38 for the entire target image shown in FIG. 32. [Diagram 42] FIG. 42 is a diagram showing a parking space annotation code including the encoding shown in FIGS. 39 to 41. [Diagram 43] FIG. 43 is an enlarged view of a total of nine grids including the grid indicated by the thick square in FIG. 42 and the eight grids surrounding it, together with the arrangement of parking space corner points, etc. [Diagram 44] FIG. 44 is a diagram showing a connection number table used to encode the connection vector shown in FIG. 43. [Diagram 45] FIG. 45 is a diagram showing the result of encoding the connection vector shown in FIG. 43 using the connection number table shown in FIG. 44. [Figure 46] FIG. 46 is a diagram showing a parking space annotation code including a second encoding formed by the encoding result shown in FIG. 45. [Figure 47] 11 is a diagram for explaining an example of encoding related to frame line information as attribute information. FIG. [Figure 48] 11 is a diagram for explaining an example of encoding related to frame line information as attribute information. FIG. [Figure 49] 11 is a diagram for explaining an example of encoding related to frame line information as attribute information. FIG. [Figure 50] 11 is a diagram for explaining an example of encoding related to frame line information as attribute information. FIG. [Figure 51] 11 is a diagram for explaining an example of encoding related to frame line information as attribute information. FIG. [Figure 52] 11 is a diagram for explaining an example of encoding related to frame line information as attribute information. FIG. [Figure 53] 11 is a diagram for explaining an example of encoding related to frame line information as attribute information. FIG. [Figure 54] FIG. 54 is a diagram showing a parking space annotation code including the third encoding shown in FIGS. 50 to 53. [Figure 55]10 is a flowchart illustrating an example of post-processing of parking frame learning according to the first embodiment. [Figure 56] FIG. 56 is a diagram showing an overview of the circumscribing rectangle calculation process shown in FIG. 55. [Figure 57] FIG. 56 is a diagram showing an overview of the edge calculation process and the straight line calculation process shown in FIG. 55. [Figure 58] FIG. 11 is a diagram showing a situation in which the parking frame line and its corners, that is, the parking frame corner points, are blocked by a parked vehicle in relation to the description of the second embodiment; [Figure 59] 13 is a flowchart showing an outline of a parking frame learning operation according to a second embodiment. [Figure 60] FIG. 13 is a diagram showing an example of an image corresponding to a situation in which a parking space is blocked by a parked vehicle, in relation to the description of the third embodiment; [Figure 61] 13 is a flowchart showing an outline of a parking frame learning operation according to a third embodiment. [Figure 62] FIG. 13 is a conceptual diagram showing an outline of a parking frame learning operation according to a third embodiment; [Figure 63] FIG. 13 is a diagram showing an example of an image corresponding to a situation in which the same type of parking space corner points belonging to different parking spaces are located within the same grid in relation to the fourth embodiment; [Fig. 64A] FIG. 13 is a diagram for explaining the definition of a parking frame direction in the fourth embodiment. [Figure 64B] FIG. 13 is a diagram for explaining the definition of a parking frame direction in the fourth embodiment. [Figure 65] 13 is a flowchart illustrating an overview of a parking frame encoding operation according to the fourth embodiment. [Figure 66] FIG. 13 is a conceptual diagram illustrating an overview of a parking frame encoding operation according to the fourth embodiment. [Figure 67] FIG. 13 is a conceptual diagram illustrating an overview of a parking frame encoding operation according to the fourth embodiment. [Figure 68] FIG. 13 is a diagram showing a parking space annotation code including a parking space encoding result according to the fourth embodiment. [Figure 69] 13 is an image showing an overview of a parking frame encoding operation according to a fifth embodiment. [Figure 70] FIG. 13 is a conceptual diagram illustrating an overview of a parking frame encoding operation according to the fifth embodiment. [Figure 71] FIG. 13 is a conceptual diagram illustrating an overview of a parking frame encoding operation according to the fifth embodiment. [Figure 72] FIG. 13 is a conceptual diagram illustrating an overview of a parking frame encoding operation according to the fifth embodiment. [Figure 73] FIG. 13 is a conceptual diagram illustrating an overview of a parking frame encoding operation according to the fifth embodiment. [Figure 74] FIG. 13 is a conceptual diagram illustrating an overview of a parking frame encoding operation according to the fifth embodiment. [Figure 75] FIG. 13 is a conceptual diagram illustrating an overview of a parking frame encoding operation according to the fifth embodiment. [Figure 76A] FIG. 13 is a schematic diagram for explaining parking lot likeness determination according to the sixth embodiment; [Figure 76B] FIG. 13 is a schematic diagram for explaining parking lot likeness determination according to the sixth embodiment; [Figure 77A] 13 is a flowchart showing an overview of annotations corresponding to a parking lot-likeness determination according to the sixth embodiment. [Fig. 77B] 77B is a flowchart showing an outline of a parking space learning process based on the annotation result shown in FIG. 77A. [Figure 78] FIG. 77C is a diagram showing an encoding result when it is determined that the location is not inside a parking lot in the process shown in FIG. 77B. [Figure 79] FIG. 77C is a diagram showing an encoding result when it is determined to be in a parking lot in the process shown in FIG. 77B. [Figure 80] FIG. 80 shows a parking space annotation code including the fourth encoding shown in FIG. 78 or FIG. 79. [Figure 81] 23 is a flowchart showing an outline of a parking space detection process using a result of a parking lot-likeliness determination in a sixth embodiment. [Figure 82] A figure showing an example of a fifth encoding which is the result of encoding information on wheel stops as an additional attribute in the seventh embodiment. [Figure 83]FIG. 83 shows a parking space annotation code including the fifth encoding shown in FIG. [Figure 84] FIG. 23 is a diagram showing an example of a sixth encoding that is the encoding result of information of additional attributes according to the eighth embodiment. [Figure 85] FIG. 85 is a diagram showing a parking space annotation code including the sixth code shown in FIG. 84. [Figure 86] 13 is a flowchart showing a series of processes for parking lot determination according to the ninth embodiment. [Figure 87] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 88] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 89] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 90] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 91] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 92] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 93] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 94] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 95] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 96] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 97] FIG. 87 is a diagram for explaining an outline of the parking row generation process shown in FIG. 86. [Figure 98A] 87 is a flowchart showing an outline of the parking row generation process shown in FIG. 86. [Figure 98B] 87 is a flowchart showing an outline of the parking row generation process shown in FIG. 86. [Figure 99]FIG. 87 is a diagram for explaining an outline of a first example of a group generation process for parking lot determination shown in FIG. 86. [Figure 100] FIG. 87 is a diagram for explaining an outline of a first example of a group generation process for parking lot determination shown in FIG. 86. [Figure 101] FIG. 87 is a diagram for explaining an outline of a second example of the group generation process for parking lot determination shown in FIG. 86. [Figure 102] FIG. 87 is a diagram for explaining an outline of a second example of the group generation process for parking lot determination shown in FIG. 86. [Figure 103] FIG. 87 is a diagram for explaining an outline of a second example of the group generation process for parking lot determination shown in FIG. 86. [Figure 104] FIG. 87 is a diagram for explaining an outline of the parking lot determination process shown in FIG. 86. [Figure 105] FIG. 87 is a diagram for explaining an outline of the parking lot determination process shown in FIG. 86. [Figure 106] FIG. 87 is a diagram for explaining an outline of the parking lot determination process shown in FIG. 86. [Figure 107] FIG. 87 is a diagram for explaining an outline of the parking lot determination process shown in FIG. 86. [Figure 108] FIG. 87 is a diagram for explaining an outline of the parking lot determination process shown in FIG. 86. [Fig. 109] FIG. 87 is a diagram for explaining an outline of the parking lot determination process shown in FIG. 86. [Figure 110] FIG. 87 is a diagram for explaining an outline of the parking lot determination process shown in FIG. 86. [Figure 111] FIG. 87 is a diagram for explaining an outline of the parking lot determination process shown in FIG. 86. [Figure 112] FIG. 87 is a diagram for explaining an outline of the parking lot determination process shown in FIG. 86. [Figure 113] FIG. 87 is a diagram for explaining an outline of the parking lot determination process shown in FIG. 86. [Fig. 114] 4 is a flowchart showing an outline of a process to be performed when an accelerator is erroneously operated according to the present embodiment. [Figure 115]4 is a flowchart showing an outline of a process to be performed when an accelerator is erroneously operated according to the present embodiment. [Fig. 116] 4 is a flowchart showing an overview of parking assistance processing according to the present embodiment. [Figure 117] FIG. 2 is a diagram showing an overview of parking assistance processing according to the present embodiment. [Figure 118] FIG. 2 is a diagram showing an overview of parking assistance processing according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] (Embodiment) Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings as appropriate. Note that the configuration, functions, and implementation examples of the embodiments can be modified as appropriate. Therefore, multiple embodiments and variations may be described in the specification. In this case, in the following description, the same reference symbols will be given to parts that are identical or equivalent between one embodiment and another embodiment, between one embodiment and its variation, or between one variation and another variation. And, the preceding description of such parts may be appropriately applied to other embodiments or variations that follow, unless there is a technical contradiction or a special additional description.

[0009] (Overall vehicle configuration) Referring to FIG. 1, the system-equipped vehicle V, which is a vehicle to which the present disclosure is applied, is a so-called four-wheeled vehicle, and includes a vehicle body V1 that is substantially rectangular in plan view. Hereinafter, a virtual line that passes through the center of the system-equipped vehicle V in the vehicle width direction and is parallel to the vehicle overall length direction of the system-equipped vehicle V is referred to as a vehicle width center line LC1. Also, a virtual line that passes through the center of the system-equipped vehicle V in the vehicle overall length direction and is parallel to the vehicle width direction of the system-equipped vehicle V is referred to as a vehicle length center line LC2. The vehicle width center line LC1 and the vehicle length center line LC2 pass through a vehicle center point VC. The vehicle center point VC is the center position of the vehicle body V1 in the three-dimensional direction. In FIG. 1, the vehicle width direction is the left-right direction in the figure. The vehicle overall length direction is a direction perpendicular to the vehicle width direction and perpendicular to the vehicle height direction. The vehicle height direction is a direction that defines the vehicle height of the system-equipped vehicle V, and is a direction parallel to the direction of gravity when the system-equipped vehicle V is stably placed on a horizontal plane in a drivable state. Also, for ease of explanation, the "front," "rear," "left," and "right" of the system-equipped vehicle V are defined as shown by the arrows in FIG. 1. In other words, the vehicle overall length direction is synonymous with the front-to-rear direction. Also, the vehicle width direction is synonymous with the left-to-right direction. Note that the vehicle height direction may not be parallel to the direction of gravity depending on the mounting conditions or running conditions of the system-equipped vehicle V. However, in most cases, the vehicle height direction is a direction along the direction of gravity.

[0010] Inside the vehicle body V1, a vehicle compartment V2 is formed, which constitutes a riding space for occupants including the driver D. Wheels V3 are provided at each of the four corners of the vehicle body V1. That is, a left front wheel V3a is provided at the left front part of the vehicle body V1. A right front wheel V3b is provided at the right front part of the vehicle body V1. A left rear wheel V3c is provided at the left rear part of the vehicle body V1. A right rear wheel V3d is provided at the right rear part of the vehicle body V1. The system-equipped vehicle V is not limited to a four-wheeled vehicle, but may be a three-wheeled vehicle, or a six- or eight-wheeled vehicle such as a cargo truck. There is no particular limitation on the position or number of drive wheels.

[0011] A front bumper V12 is attached to a front portion V11, which is the front end of the vehicle body V1. A rear bumper V14 is attached to a rear portion V13, which is the rear end of the vehicle body V1. The side and upper portions of the vehicle body V1 are covered by a vehicle body panel V15. The vehicle body panel V15 includes a door panel V16. In the specific example shown in FIG. 1, a total of four door panels V16 are provided, two on each side. A door mirror V17 is attached to each of the pair of left and right door panels V16 on the front side. In addition, the front side of the vehicle interior V2 is covered by a front windshield V18. The front windshield V18 is formed in a plate shape from translucent glass or synthetic resin. The front windshield V18 is inclined so that the upper end is located further rearward than the lower end when viewed from the side in a line of sight parallel to the vehicle width direction.

[0012] A dashboard V21 is provided at the front end of the vehicle interior V2. A plurality of seats V22 are arranged behind the dashboard V21. Of the plurality of seats V22, a handle V24 is provided in front of a driver's seat V23 in which the driver D sits, for the driver D to grasp and operate when steering. The handle V24 is typically a so-called steering wheel formed in a substantially circular, elliptical, or polygonal ring shape, but may be rod-shaped or control stick-shaped.

[0013] (Overview of the in-vehicle system) The system-equipped vehicle V is equipped with an in-vehicle system 1. The in-vehicle system 1 is configured to function as a driving automation system in the system-equipped vehicle V by being installed in the system-equipped vehicle V. The system-equipped vehicle V may be referred to as the "own vehicle" hereinafter. The "driving automation system" is a system for realizing a driving automation level corresponding to at least one of levels 1 to 5 specified in the standard "SAE J3016" published by SAE International. SAE stands for Society Of Automotive Engineers. Level X in "SAE J3016" is hereinafter simply referred to as "SAE level X". X is any one of 0 to 5. The larger the value of X in SAE level X, or the more dynamic driving tasks the driving automation system is in charge of, i.e., executes, the higher the driving automation level is expressed as being. In addition, a change in the driving automation level to a higher level is referred to as an "increase" in the driving automation level. In contrast, the smaller the value of X, or the fewer dynamic driving tasks that the driving automation system is responsible for, that is, executes, the lower the level of driving automation is expressed. Also, a change to a lower level of driving automation is referred to as a "degradation" of the level of driving automation.

[0014] (Definition of Driving Assistance and Autonomous Driving) The contents of SAE levels 0 to 5 are specifically described below. In the following description, the driver D is a passenger who is in charge of or performs the dynamic driving task. The "dynamic driving task" refers to all operational and tactical functions that need to be performed in real time when operating a vehicle in road traffic, excluding strategic functions. Driving behavior in general can be classified into three types of functions: strategic, tactical, and operational. The "strategic" functions include itinerary planning, route selection, etc., and specifically include deciding or selecting itinerary plans such as "whether to go or not, when, where, and how to go." The "tactical" functions relate to vehicle operation in traffic situations, such as deciding whether and when to overtake or change lanes during the trip, selecting an appropriate speed, and checking mirrors. The "operational" functions relate to instantaneous reactions, such as making minor corrections to steering, braking, accelerator, and acceleration operations to maintain the position of the road lane or to avoid sudden obstacles or dangerous events in the vehicle's path. "OEDR" is an abbreviation for Object and Event Detection and Response, and is also referred to as "detection and response of objects and events." OEDR includes monitoring of the driving environment. Monitoring of the driving environment includes detection, recognition, and classification of objects and events. Monitoring of the driving environment also includes preparation to respond to objects and events as necessary. A "limited domain" is a specific condition in which a driving automation system or its functions are designed to operate, and is also referred to as an operational design domain or ODD. ODD stands for Operational Design Domain. A limited domain includes at least one of multiple constraints, such as geographical, environmental, speed, and time.

[0015] Level 0: Manual driving…Driver D performs all dynamic driving tasks. Level 1: Driving assistance: The driving automation system continuously performs either the longitudinal vehicle motion control subtask or the lateral vehicle motion control subtask of the dynamic driving task in a specific limited area. The longitudinal vehicle motion control subtask is starting, accelerating / decelerating, and stopping. The lateral vehicle motion control subtask is steering. However, the driving automation system does not perform both the longitudinal vehicle motion control subtask and the lateral vehicle motion control subtask simultaneously. Level 2: Advanced driving assistance: The driving automation system continuously performs the longitudinal vehicle motion control subtask and the lateral vehicle motion control subtask of the dynamic driving task in a specific limited area. The driver D is expected to supervise the driving automation system by performing the OEDR, which is a subtask of the dynamic driving task. Level 3: Conditional Autonomous Driving: The driving automation system continuously performs all dynamic driving tasks in specific limited areas. In principle, driver D is not obligated to perform OEDR such as periphery monitoring. Periphery monitoring is the monitoring of the traffic environment around the vehicle. However, if it becomes difficult to continue this driving automation level, the driving automation system will request driver D to take over driving with ample time to spare. Driver D must respond to the request appropriately. Level 4: Highly automated driving: The driving automation system continuously performs all dynamic driving tasks in specific limited areas. If it becomes difficult to maintain the driving automation level in a limited area, the driving automation system will take action. Level 5: Fully automated driving: The driving automation system performs all dynamic driving tasks continuously, without being limited to specific restricted areas. If it becomes difficult to continue with this level of driving automation, the driving automation system will also respond without being limited to specific restricted areas.

[0016] The in-vehicle system 1 is configured to be capable of executing various driving controls during driving of the vehicle and various associated notification operations. Specifically, the in-vehicle system 1 has a configuration as a driving automation system for realizing or implementing driving assistance or automatic driving in the vehicle. "Autonomous driving" refers to a driving automation level corresponding to SAE levels 3 to 5 in which the driving automation system is responsible for, i.e., executes, all dynamic driving tasks. In contrast, "driving assistance" refers to a driving automation level corresponding to SAE levels 1 to 2 in which the driving automation system is responsible for, i.e., executes, some dynamic driving tasks. Hereinafter, the expression "driving assistance" includes "advanced driving assistance" of SAE level 2, unless it is clearly stated as "SAE level 1 driving assistance" or clearly distinguished from "advanced driving assistance" such as "driving assistance or advanced driving assistance".

[0017] In this embodiment, the in-vehicle system 1 is configured to be capable of performing at least SAE level 3 or higher autonomous driving, SAE level 2 advanced driving assistance, and SAE level 1 driving assistance. Therefore, the in-vehicle system 1 according to this embodiment can be called an "autonomous driving system" during autonomous driving, while it can be called a "driving assistance system" during driving assistance or advanced driving assistance. The advanced driving assistance that can be performed by the in-vehicle system 1 according to this embodiment includes so-called "hands-off driving". In hands-off driving, the in-vehicle system 1 automatically executes control of starting, steering, acceleration / deceleration, lane change, and stopping, on the condition that the driver D responds appropriately to an intervention request from the in-vehicle system 1. In hands-off driving, the driver D is not required to be in a hands-on state, but is required to monitor the road conditions, traffic conditions, and obstacle presence conditions around the vehicle. The "hands-on state" is a state in which the driver D can interfere with the steering of the vehicle, i.e., the lateral vehicle motion control subtask. The hands-on state is typically a state in which the driver D is seated in the driver's seat V23 in a position in which he or she can drive the vehicle and can operate the steering wheel V24 with his or her hands. In the hands-on state, the driver D usually holds the steering wheel V24 with his or her hands, but even if the driver D's hands are touching the steering wheel V24 in a state in which the driver D can immediately hold it, this state can be considered to be a "hands-on state." For example, a state in which the driver D is performing a steering operation, that is, actively operating the steering wheel V24, is considered to be a hands-on state. In addition, a state in which the driver D holds the steering wheel V24 against the steering control by the in-vehicle system 1 is also considered to be a hands-on state.

[0018] (In-vehicle system configuration) The overall configuration of the in-vehicle system 1 will be described below with reference to FIG. 1 and FIG. 2. The in-vehicle system 1 includes a driving ECU 2, a driving information input unit 3, an in-vehicle communication device 4, an HD map 5, a navigation device 6, an HMI device 7, lighting devices 8, and a motion control device 9. ECU stands for Electronic Control Unit. HD stands for High Definition. HMI stands for Human Machine Interface. The in-vehicle system 1 is configured by connecting the driving ECU 2, the driving information input unit 3, the in-vehicle communication device 4, the HD map 5, the navigation device 6, the HMI device 7, lighting devices 8, and the motion control device 9 to each other via an in-vehicle communication line 10. The in-vehicle communication line 10 is configured to comply with a predetermined communication standard such as CAN (international registered trademark: international registration number 1048262A). CAN (international registered trademark) stands for Controller Area Network. The in-vehicle communication line 10 may have a sub-network conforming to LIN, FlexRay, etc., in addition to a main network conforming to CAN (international registered trademark). LIN is an abbreviation for Local Interconnect Network.

[0019] (Drive ECU) The driving ECU 2 as a control device according to the present disclosure is configured to control the overall operation of the in-vehicle system 1 by being mounted in the system-mounted vehicle V. That is, the driving ECU 2 has a configuration as a so-called AD / ADAS ECU, which is a controller for driving assistance or automatic driving. AD stands for automatic driving. ADAS stands for Advanced Driver-Assistance Systems. Specifically, the driving ECU 2 has a configuration as an in-vehicle computer including a processor 21 and a memory 22 connected to the processor 21. More specifically, the driving ECU 2 has at least one processor 21 and at least one memory 22.

[0020] The processor 21 has a configuration as a CPU or MPU. CPU stands for Central Processing Unit. MPU stands for Micro Processing Unit. At least one memory 22 includes at least a ROM and a RAM among various non-transient substantial recording media such as ROM, RAM, non-volatile rewritable memory, etc. ROM stands for Read Only Memory. RAM stands for Random Access Memory. A "recording medium" may also be called a "storage medium." A non-volatile rewritable memory is a storage device that can rewrite information while the power is on, but retains information in an unrewritable manner while the power is off, such as a flash ROM or an EEPROM. EEPROM stands for Electronically Erasable and Programmable ROM. Data and program instructions used by the processor 21 are stored in advance in the ROM or non-volatile rewritable memory included in the at least one memory 22. The driving ECU 2 is configured to execute various processes or operations such as vehicle control operations and information notification to occupants by having the processor 21 read and execute program instructions stored in the memory 22.

[0021] (Driving information input section) The driving information input unit 3 is provided to input information required for various processing operations in the driving ECU 2 to the driving ECU 2. Specifically, the driving information input unit 3 includes a sonar sensor 31, a radar sensor 32, a laser radar sensor 33, a camera 34, an operation sensor 35, a behavior sensor 36, a driver state monitor 37, an operation switch 38, and a locator 39. The sonar sensor 31, the radar sensor 32, the laser radar sensor 33, and the camera 34 may be collectively referred to as a "periphery monitoring sensor" or an "ADAS sensor." Each element provided as the driving information input unit 3 will be described below in order.

[0022] (Sonar sensor) The sonar sensor 31 is an ultrasonic distance measuring sensor and is attached to the vehicle body V1. As shown in FIG. 3, the sonar sensor 31 is a so-called ultrasonic sensor, and is configured to detect the target B by irradiating sonar search waves Wsp in an ultrasonic band toward the space outside the vehicle and receiving sonar reflected waves Wsr that are reflected by the target B. Specifically, the sonar sensor 31 is configured to calculate the distance to the target B based on the TOF and the speed of sound. TOF is an abbreviation for Time of Flight. That is, TOF is the time from the transmission time of the sonar search wave Wsp to the reception time of the sonar reflected wave Wsr in the propagation path Ls of the sonar search wave Wsp, which is an ultrasonic wave, and the sonar reflected wave Wsr, and may also be referred to as the propagation time.

[0023] As shown in FIG. 4, when there are a pair of sonar sensors 31, a first sonar sensor 311 and a second sonar sensor 312, which are arranged at different positions in the horizontal direction, it is possible to calculate the relative position of the target B with respect to the vehicle by so-called triangulation. Here, in FIG. 4, the X-axis is set along a straight line connecting the first sonar sensor 311 and the second sonar sensor 312, and the Y-axis is set in a direction perpendicular to the X-axis. Both the X-axis and the Y-axis are assumed to be horizontal, that is, perpendicular to the vehicle height direction. In addition, the first sonar sensor 311 transmits a sonar search wave Wsp, and the reflected wave by the target B is received by the first sonar sensor 311 and the second sonar sensor 312. In this case, the position of the target B in the XY two-dimensional coordinate system is calculated based on the TOF in the first propagation path Ls1 and the TOF in the second propagation path Ls2. The first propagation path Ls1 is a propagation path of an ultrasonic wave that returns to the first sonar sensor 311 via the target B from the first sonar sensor 311. The ultrasonic waves propagating through the first propagation path Ls1 are referred to as "direct waves." The second propagation path Ls2 is a propagation path of ultrasonic waves that travels from the first sonar sensor 311 through the target B to the second sonar sensor 312. The ultrasonic waves propagating through the second propagation path Ls2 are referred to as "indirect waves."

[0024] As shown in FIG. 1, a vehicle body V1 is equipped with a plurality of sonar sensors 31. Specifically, a first front sonar SF1, a second front sonar SF2, a third front sonar SF3, and a fourth front sonar SF4 are mounted on a front bumper V12 as the sonar sensors 31. Similarly, a first rear sonar SR1, a second rear sonar SR2, a third rear sonar SR3, and a fourth rear sonar SR4 are mounted on a rear bumper V14 as the sonar sensors 31. In addition, a first side sonar SS1, a second side sonar SS2, a third side sonar SS3, and a fourth side sonar SS4 are mounted on a side portion of the vehicle body V1 as the sonar sensors 31. When it is not specified that the sonar sensor is one of the first front sonar SF1 to the fourth side sonar SS4, the singular expression "sonar sensor 31" or the expression "plurality of sonar sensors 31" may be used hereinafter. FIG. 5 shows the detection range of each of the plurality of sonar sensors 31. Each of the multiple sonar sensors 31 will be described in turn below with reference to FIGS.

[0025] The first front sonar SF1 is mounted on the front bumper V12 at a position close to the left end in the vehicle width direction so as to transmit a sonar search wave Wsp to the left front of the vehicle. The second front sonar SF2 is mounted on the front bumper V12 at a position close to the right end in the vehicle width direction so as to transmit a sonar search wave Wsp to the right front of the vehicle. The first front sonar SF1 and the second front sonar SF2 are disposed symmetrically across the vehicle width center line LC1. The front corner sonar detection range Rsc, which is the detection range of the first front sonar SF1 and the second front sonar SF2, is set to have a detection distance of, for example, about 60 cm. The "detection distance" is the maximum distance that can be measured from the sonar sensor 31.

[0026] The third front sonar SF3 and the fourth front sonar SF4 are arranged in the vehicle width direction at a position closer to the center of the front bumper V12 in the vehicle width direction. The third front sonar SF3 is arranged between the first front sonar SF1 and the vehicle width center line LC1 in the vehicle width direction so as to transmit a sonar search wave Wsp approximately in front of the vehicle. The fourth front sonar SF4 is arranged between the second front sonar SF2 and the vehicle width center line LC1 in the vehicle width direction so as to transmit a sonar search wave Wsp approximately in front of the vehicle. The third front sonar SF3 and the fourth front sonar SF4 are arranged symmetrically with respect to the vehicle width center line LC1. The front sonar detection range Rsf, which is the detection range of the third front sonar SF3 and the fourth front sonar SF4, is set to a detection distance of, for example, about 1 m.

[0027] The first front sonar SF1 and the third front sonar SF3 mounted on the left side of the vehicle body V1 are arranged at different positions in the horizontal direction. The first front sonar SF1 and the third front sonar SF3, which are adjacent to each other in the vehicle width direction, are arranged in a positional relationship such that the reflected wave of the sonar search wave Wsp emitted by one of them by the target B can be received as a received wave by the other. That is, the first front sonar SF1 is arranged so as to be able to receive both the direct wave corresponding to the sonar search wave Wsp emitted by itself and the indirect wave corresponding to the sonar search wave Wsp emitted by the third front sonar SF3. Similarly, the third front sonar SF3 is arranged so as to be able to receive both the direct wave corresponding to the sonar search wave Wsp emitted by itself and the indirect wave corresponding to the sonar search wave Wsp emitted by the first front sonar SF1.

[0028] Similarly, the third front sonar SF3 and the fourth front sonar SF4, which are mounted toward the center of the vehicle body V1 in the vehicle width direction, are disposed at different positions from each other in the horizontal direction. Moreover, the third front sonar SF3 and the fourth front sonar SF4, which are adjacent to each other in the vehicle width direction, are disposed in a positional relationship such that the one can receive the sonar search wave Wsp emitted by the other sonar and reflected by the target B as a received wave.

[0029] Similarly, the second front sonar SF2 and the fourth front sonar SF4 mounted on the right side of the vehicle body V1 are positioned at different positions from each other in the horizontal direction. The second front sonar SF2 and the fourth front sonar SF4, which are adjacent to each other in the vehicle width direction, are positioned in such a way that the reflected wave of the sonar search wave Wsp emitted by one of them by the target B can be received as a received wave by the other sonar.

[0030] The first rear sonar SR1 is mounted on the rear bumper V14 at a position close to the left end in the vehicle width direction so as to transmit a sonar search wave Wsp to the left rear of the vehicle. The second rear sonar SR2 is mounted on the rear bumper V14 at a position close to the right end in the vehicle width direction so as to transmit a sonar search wave Wsp to the right rear of the vehicle. The first rear sonar SR1 and the second rear sonar SR2 are disposed symmetrically across the vehicle width center line LC1. The rear corner sonar detection range Rsd, which is the detection range of the first rear sonar SR1 and the second rear sonar SR2, is set to a detection distance of, for example, about 60 cm.

[0031] The third rear sonar SR3 and the fourth rear sonar SR4 are arranged in the vehicle width direction at a position closer to the center of the rear bumper V14 in the vehicle width direction. The third rear sonar SR3 is arranged between the first rear sonar SR1 and the vehicle width center line LC1 in the vehicle width direction so as to transmit a sonar search wave Wsp approximately behind the vehicle. The fourth rear sonar SR4 is arranged between the second rear sonar SR2 and the vehicle width center line LC1 in the vehicle width direction so as to transmit a sonar search wave Wsp approximately behind the vehicle. The third rear sonar SR3 and the fourth rear sonar SR4 are arranged symmetrically with respect to the vehicle width center line LC1. The rear sonar detection range Rsr, which is the detection range of the third rear sonar SR3 and the fourth rear sonar SR4, is set to a detection distance of, for example, about 1.5 m.

[0032] The first rear sonar SR1 and the third rear sonar SR3 mounted on the left side of the vehicle body V1 are arranged at different positions in the horizontal direction. In addition, the first rear sonar SR1 and the third rear sonar SR3, which are adjacent to each other in the vehicle width direction, are arranged in a positional relationship such that the reflected wave of the sonar search wave Wsp emitted by one of them by the target B can be received as a received wave by the other. That is, the first rear sonar SR1 is arranged so as to be able to receive both the direct wave corresponding to the sonar search wave Wsp emitted by itself and the indirect wave corresponding to the sonar search wave Wsp emitted by the third rear sonar SR3. Similarly, the third rear sonar SR3 is arranged so as to be able to receive both the direct wave corresponding to the sonar search wave Wsp emitted by itself and the indirect wave corresponding to the sonar search wave Wsp emitted by the first rear sonar SR1.

[0033] Similarly, the third rear sonar SR3 and the fourth rear sonar SR4, which are mounted toward the center of the vehicle body V1 in the vehicle width direction, are disposed at different positions from each other in the horizontal direction. In addition, the third rear sonar SR3 and the fourth rear sonar SR4, which are adjacent to each other in the vehicle width direction, are disposed in a positional relationship such that the reflected wave of the sonar search wave Wsp emitted by one of them by the target B can be received as a received wave by the other.

[0034] Similarly, the second rear sonar SR2 and the fourth rear sonar SR4 mounted on the right side of the vehicle body V1 are disposed at different positions from each other in the horizontal direction. The second rear sonar SR2 and the fourth rear sonar SR4, which are adjacent to each other in the vehicle width direction, are disposed in a positional relationship such that the reflected wave of the sonar search wave Wsp emitted by one of them by the target B can be received as a received wave by the other.

[0035] The first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4 are provided so as to transmit sonar search waves Wsp from the side of the vehicle body V1 to the side of the vehicle. The first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4 are provided so as to be able to receive only direct waves. The side sonar detection range Rss, which is the detection range of the first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4, is set to be, for example, a detection distance of about 2 to 3 m.

[0036] The first side sonar SS1 is disposed between the left door mirror V17 and the first front sonar SF1 in the longitudinal direction so as to transmit sonar search waves Wsp to the left of the vehicle. The second side sonar SS2 is disposed between the right door mirror V17 and the second front sonar SF2 in the longitudinal direction so as to transmit sonar search waves Wsp to the right of the vehicle. The first side sonar SS1 and the second side sonar SS2 are provided symmetrically across the vehicle width center line LC1. The first side sonar SS1 and the second side sonar SS2 may be attached to the vehicle body panel V15, or may be attached to both ends of the front bumper V12 in the vehicle width direction, which are extended rearward along the vehicle overall length.

[0037] The third side sonar SS3 is disposed between the left rear door panel V16 and the first rear sonar SR1 in the front-rear direction so as to transmit sonar search waves Wsp to the left of the vehicle. The fourth side sonar SS4 is disposed between the right rear door panel V16 and the second rear sonar SR2 in the front-rear direction so as to transmit sonar search waves Wsp to the right of the vehicle. The third side sonar SS3 and the fourth side sonar SS4 are provided symmetrically across the vehicle width center line LC1. The third side sonar SS3 and the fourth side sonar SS4 may be attached to the vehicle body panel V15, or may be attached to both ends of the rear bumper V14 in the vehicle width direction, which are extended forward along the vehicle overall length.

[0038] FIG. 6 shows an example of detecting a parking space PS for parallel parking on the left side of the vehicle using the first side sonar SS1, which is one of the multiple sonar sensors 31. The parking space PS is a frame-shaped area, i.e., a rectangular area in a plan view, that defines a space in which the vehicle can be parked, and may also be called a "parking space" or a "parking section". In the figure, the sonar detection point Psr is a point on the surface of the target B that is estimated to have reflected the sonar search wave Wsp emitted from the sonar sensor 31, and may also be called a "distance measurement point". As shown in FIG. 6, the parking space PS can be detected based on the distribution state of the sonar detection points Psr acquired based on the sonar reflection wave Wsr from the target B, which is a parked vehicle lined up along the traveling direction of the vehicle traveling straight. Specifically, for example, in the case of parallel parking shown in FIG. 6, the parking space PS can be detected at a location where the width of the section where the arrangement of the sonar detection points Psr lined up along the traveling direction of the vehicle is interrupted is equal to or greater than a predetermined width that exceeds the overall width of the vehicle.

[0039] (Radar Sensor) 1 and 2 again, the radar sensor 32 is configured to detect the target B by transmitting and receiving radar waves, which are millimeter waves or submillimeter waves. In this embodiment, the radar sensor 32 is mounted on the center of the front part V11 of the vehicle body V1 in the vehicle width direction. As shown in FIG. 7, the radar sensor 32 is a so-called long-distance radar having a detection range of about 10 to 250 m and a radar scanning angle range θr1 centered on the vehicle's forward direction Df, and has a phased array type or beam forming type configuration. The vehicle's forward direction Df is the direction in which a virtual straight line extending the vehicle width center line LC1 forward of the vehicle extends, and corresponds to the traveling direction when the vehicle is traveling forward, that is, when the vehicle is traveling in a shift position other than reverse shift. The vehicle's forward direction Df coincides with the traveling direction when the vehicle is traveling straight, and becomes the tangent direction of the traveling trajectory when the vehicle is traveling on a curve. The radar sensor 32 is configured to detect the target B within the radar detection range Rg1 by emitting radar search waves Wrp while scanning within the radar scan angle range θr1 and receiving radar reflected waves Wrr that are waves reflected by the target B. The radar detection range Rg1 is a sector-shaped range with the radar scan angle range θr1 as its central angle.

[0040] Specifically, the radar sensor 32 is an FMCW radar device equipped with a so-called array antenna, and is configured to be able to detect the distance to the target B, the azimuth angle θa, and the relative speed of the target B with respect to the vehicle, based on the frequency difference and phase difference of the transmitted and received millimeter waves. The azimuth angle θa is the angle between an imaginary line extending forward from the vehicle width center line LC1, which forms the center line of the radar detection range Rg1, and an imaginary line connecting the radar sensor 32 and the target B. The relative speed is the difference between the target moving speed vb and the vehicle speed vm, which is the traveling speed of the vehicle. More specifically, the radar sensor 32 transmits a radar search wave Wrp generated based on a transmission signal of a predetermined modulation frequency, receives a radar reflected wave Wrr, which is a wave reflected by the target B, and acquires a reception signal indicating the frequency characteristic of the radar reflected wave Wrr. The radar sensor 32 generates a beat signal, which is the deviation between the frequency of the transmission signal and the frequency of the reception signal. The radar sensor 32 performs a fast Fourier transform on the beat signal to obtain a frequency spectrum, analyzes the frequency spectrum to obtain a beat frequency, and calculates the distance and relative speed based on the beat frequency. In addition to the long-range radar sensor 32, so-called medium-range and short-range radar sensors 32 may also be provided in the in-vehicle system 1. The detection distance of the medium-range type is about 1 to 100 m, and the detection distance of the short-range type is about 15 cm to 30 m.

[0041] (Laser radar sensor) As shown in FIG. 8, the laser radar sensor 33 is configured to detect the target B by emitting a detection light Lp, which is a laser light in the infrared band, to the outside of the vehicle and receiving a reflected light Lr by the target B. The laser radar sensor 33 may also be called a LIDAR. LIDAR is an abbreviation for Light Detection and Ranging or Laser Imaging Detection and Ranging. In the vehicle-mounted system 1 according to this embodiment, a long-distance laser radar sensor 33 having a configuration as a so-called scanning LIDAR is mounted in the center of the front part V11 of the vehicle body V1 in the vehicle width direction, as shown in FIG. 1. The laser radar sensor 33 is configured to detect the target B within the LIDAR detection range Rg2 by scanning the detection light Lp in the horizontal direction within the LIDAR scan angle range θr2 centered on the vehicle forward direction Df. The LIDAR detection range Rg2 is a sector-shaped range with a radius of 200 m or more and with the LIDAR scan angle range θr2 as a central angle.

[0042] Fig. 9 shows a schematic configuration of the laser radar sensor 33. As shown in Fig. 9, the laser radar sensor 33 includes a light emitting unit 331, a scanning unit 332, and a light receiving unit 333. The light emitting unit 331 is configured to generate detection light Lp. The scanning unit 332 has a configuration as a so-called MEMS mirror. MEMS is an abbreviation for Micro Electro Mechanical Systems. That is, the scanning unit 332 is configured to scan the detection light Lp along the horizontal scanning direction Ds and the vertical scanning direction Dh by electrically controlling the orientation of a reflecting mirror that reflects the detection light Lp emitted from the light emitting unit 331 by a MEMS mechanism.

[0043] The light receiving unit 333 includes a light receiving sensor 334 that is a two-dimensional image sensor. That is, in the light receiving sensor 334, a plurality of light receiving elements 335 are two-dimensionally arranged in the vertical and horizontal directions, that is, in the directions corresponding to the horizontal scanning direction Ds and the vertical scanning direction Dh. The light receiving elements 335 are configured as light detection elements such as APD, SPAD, and photodiodes. APD stands for avalanche photodiode. SPAD stands for Single Photon Avalanche Diode. The laser radar sensor 33 is configured to generate detection point cloud data based on the reflected light Lr received by the light receiving unit 333, and detect the target B based on the detection point cloud data. The detection point cloud data is a two-dimensional array of a plurality of lidar detection points Prr arranged in an image-like manner in a two-dimensional direction corresponding to the horizontal scanning direction Ds and the vertical scanning direction Dh in FIG. 9. The LIDAR detection point Prr is a point on the target B that is estimated to have reflected the detection light Lp corresponding to the reflected light Lr detected by one light receiving element 335, and includes position information and distance measurement information in the above-mentioned two-dimensional array. Therefore, the detection point cloud data may also be called "distance measurement point cloud data." In this way, the laser radar sensor 33 acquires detection point cloud data in which a plurality of LIDAR detection points Prr are two-dimensionally arranged like an image. Therefore, the laser radar sensor 33 can be said to be a type of image sensor.

[0044] (camera) 1 and 2 again, the camera 34 as an image sensor is mounted on the vehicle so as to move with the movement of the vehicle while capturing images of the surroundings of the vehicle. That is, the camera 34 is configured to generate image information corresponding to the captured images of the surroundings of the vehicle. The image information may also be referred to as image data. The camera 34 is a so-called digital camera device and includes an image sensor such as a CCD or a CMOS. CCD stands for Charge Coupled Device. CMOS stands for Complementary Metal Oxide Semiconductor.

[0045] In this embodiment, the system-equipped vehicle V is equipped with multiple cameras 34, namely, a front camera CF, a rear camera CB, a left camera CL, and a right camera CR. When not indicating a specific one of the front camera CF, the rear camera CB, the left camera CL, and the right camera CR, hereinafter, the singular expression "camera 34" or the expression "multiple cameras 34" may be used.

[0046] The front camera CF is disposed at a central position in the vehicle width direction, i.e., on the vehicle width center line LC1 in a plan view, near the upper end of the front windshield V18 at the upper part of the vehicle interior V2 so as to obtain image information corresponding to an image in front of the vehicle. The front camera CF may be mounted on the front part V11 of the vehicle body V1. The rear camera CB is disposed at a central position in the vehicle width direction at the rear part V13 of the vehicle body V1 so as to obtain image information corresponding to an image behind the vehicle. The left camera CL is mounted on the left door mirror V17 so as to obtain image information corresponding to an image to the left of the vehicle. The right camera CR is mounted on the right door mirror V17 so as to obtain image information corresponding to an image to the right of the vehicle.

[0047] Based on an image captured by the camera 34, it is possible to recognize the target B, that is, to determine at least the presence and type of the target B. FIG. 10 shows an overview of an operation for recognizing the target B included in the foreground based on an image of the foreground including the road Rd in front of the vehicle captured by the front camera CF, as an example. Specifically, as shown in FIG. 10, a detection area Aw is set in the captured image. The detection area Aw is all or a part of the captured image area, i.e., the angle of view area of ​​the camera 34. Based on pixel feature parameters in the image within the detection area Aw, a feature point image Gp can be acquired. The pixel feature parameters are feature parameters of pixels constituting an image, such as brightness, contrast, color, etc. The "brightness" may also be called "lightness". The "color" may also be called "saturation". The feature point image Gp is ​​an image in which feature points Pt extracted based on the difference in pixel feature parameters between adjacent pixels, the gradient of change, etc. are arranged in an image shape. The feature points Pt are points that characterize the shape of the target B in the captured image. In other words, the feature points Pt are characteristic points, i.e., pixels, in the image within the detection area Aw. Then, by pattern matching between a feature point group Pg, which is a collection of the feature points Pt in the feature point image Gp, and a pre-stored pattern, it is possible to determine the type of the target B corresponding to the feature point group Pg.

[0048] It should be noted that the method of extracting the feature points Pt is well known at the time of filing the present application. Specifically, as the method of extracting the feature points Pt, for example, well-known methods such as a Sobel filter, a Laplacian filter, and a Canny method can be used. Therefore, in this specification, a detailed description of the method of extracting the feature points Pt is omitted. Also, the "extraction" of the feature points Pt can be expressed as "detection."

[0049] Also, it is possible to estimate the relative position or distance of the recognized target B with respect to the vehicle itself based on an image captured by the camera 34. FIG. 11 shows an overview of calculating an estimated point Pb, which is an estimation result of the three-dimensional position of a feature point Pt corresponding to the target B, by a technique called monocular moving stereo or SFM. SFM is an abbreviation for Structure from Motion. Note that the present disclosure is not limited to such monocular moving stereo. That is, the camera 34 may be a so-called compound eye stereo camera.

[0050] 11, it is assumed that the camera 34 moves in the host vehicle movement direction Dv as the host vehicle travels, and the position of the camera 34 at time t1 is a first camera position Pc1, and the position of the camera 34 at time t2 is a second camera position Pc2. The first captured image A1 is assumed to be an image captured by the camera 34 at the first camera position Pc1. The second captured image A2 is assumed to be an image captured by the camera 34 at the second camera position Pc2.

[0051] The first feature point Pt1 is a feature point Pt extracted from the first captured image A1. The second feature point Pt2 is a feature point Pt extracted from the second captured image A2 and estimated to correspond to the first feature point Pt1 at time t1. That is, the second feature point Pt2 corresponds to a point on the target B corresponding to the first feature point Pt1 at time t1, which is estimated to have moved after (t2-t1) seconds. It is possible to determine that the first feature point Pt1 and the second feature point Pt2 are the same, i.e., corresponding points, by a publicly known or well-known method such as optical flow. Then, the intersection of a first straight line L1 passing through the first camera position Pc1 and the first feature point Pt1 and a second straight line L2 passing through the second camera position Pc2 and the second feature point Pt2 is estimated as an estimated point Pb. Such an estimated point Pb is a point on the target B corresponding to the first feature point Pt1 and the second feature point Pt2 in a three-dimensional coordinate system based on the vehicle. If the estimated point Pb is a stationary point, it satisfies the epipolar constraint condition. The epipolar constraint condition is a condition defined by epipolar geometry, which states that the first camera position Pc1, the second camera position Pc2, and the estimated point Pb exist on the same plane Π at both times t1 and t2.

[0052] (Various sensors) 1 and 2 again, the operation sensor 35 is provided to generate outputs corresponding to various quantities related to the driving operation state of the vehicle by the driver D. The "various quantities related to the driving operation state" are, for example, an accelerator operation amount, a brake operation amount, a shift position, a steering angle, and the like. That is, the operation sensor 35 is a collective name for well-known sensors such as an accelerator pedal sensor, a brake pedal sensor, a shift position sensor, a steering angle sensor, a steering torque sensor, and the like, for the sake of simplicity of illustration and explanation. In addition, the operation sensor 35 may be provided with a so-called steering wheel sensor that detects whether the driver D is gripping the steering wheel V24.

[0053] The behavior sensor 36 is provided to generate outputs corresponding to various quantities related to the driving behavior of the vehicle. The "various quantities related to the driving behavior" include physical quantities related to the behavior or motion state of the vehicle, such as vehicle speed, yaw rate, longitudinal acceleration, lateral acceleration, etc. In other words, the behavior sensor 36 is a collective term for well-known sensors such as a vehicle speed sensor, a yaw rate sensor, an acceleration sensor, etc., for the sake of simplicity of illustration and explanation.

[0054] (Driver status monitor) 1 and 2, and also FIG. 12 and FIG. 13, the driver state monitor 37 is configured to sequentially detect the driver state based on the captured image of the driver D. The driver state includes, for example, whether or not the driver is awake. Specifically, the driver state monitor 37 includes a driver monitor camera provided in the vehicle interior V2 so that at least the head D1 of the driver D seated in the driver's seat V23 in a posture that allows the driver to drive the vehicle appropriately can be captured in the field of view so as to capture the face D2 from the front. The driver monitor camera is configured as a near-infrared camera. The driver state monitor 37 detects the driver state by performing image recognition processing on the captured image by the driver monitor camera.

[0055] The detection contents using the driver state monitor 37 include, for example, the direction of the face D2 of the driver D, the degree of opening of the eyes D3, the position of the pupils D4, and the like, as shown in FIG. 12 and FIG. 13. The direction of the face D2 includes a yaw angle θy and a pitch angle θp. The yaw angle θy represents the rotation angle of the face D2 of the driver D about a vertical axis Dx1 extending in the up-down direction. The yaw angle θy is 0° when the face D2 faces forward. The yaw angle θy is a positive value when the face D2 faces leftward, and a negative value when the face D2 faces rightward. The pitch angle θp represents the rotation angle of the face D2 about a horizontal axis Dx2 extending in the left-right direction. The pitch angle θp is 0° when the face D2 faces horizontally. The pitch angle θp is a positive value when the face D2 faces downward, and a negative value when the face D2 faces upward. FIG. 12 shows an example of a captured image when the driver D is in an awake state. Fig. 13 shows an example of a captured image when the driver D is not awake. As shown in Fig. 12 and Fig. 13, it is possible to detect the driver's state, including whether or not the driver is awake, the line of sight D5, etc., based on the direction of the face D2, the degree of opening of the eyes D3, the position of the pupils D4, etc.

[0056] (Operation switch) 1 and 2 again, the operation switches 38 are various switches operated by the driver D when driving the vehicle, and are provided in various locations in the vehicle compartment V2. The operation switches 38 correspond to switches that are not subject to detection of the operation amount or operation state by the operation sensor 35, and include at least an ignition switch 381, a blinker switch 382, ​​and an AD / ADAS switch 383. The ignition switch 381 is a switch for turning on and off the activation of the system-equipped vehicle V, i.e., the in-vehicle system 1, and may also be called a start switch or a power switch. The blinker switch 382 is a switch for detecting the operation state of a blinker lever. The AD / ADAS switch 383 is a switch for the driver D to input instructions such as start, end, level setting, and function selection related to driving assistance or automatic driving.

[0057] (locator) The locator 39 is configured to acquire highly accurate position information of the vehicle. Specifically, the locator 39 is a so-called composite positioning system, and includes a GNSS receiver 391, an inertial acquisition unit 392, and a locator ECU 393. GNSS is an abbreviation for Global Navigation Satellite System. The "highly accurate position information" is, for example, position information having a position accuracy of SAE level 2 or higher that can be used for advanced driving assistance or automated driving, specifically, an error of less than 10 cm. Note that, as the locator 39, a commercially available system such as the "POSLV" positioning and orientation system for land vehicles manufactured by Applanix, Inc., can be used.

[0058] The GNSS receiver 391 is provided so as to be capable of receiving a positioning signal transmitted from at least one positioning satellite, i.e., an artificial satellite. In this embodiment, the GNSS receiver 391 is configured so as to be capable of receiving a positioning signal from a positioning satellite in at least one of satellite positioning systems such as GPS, QZSS, GLONASS, Galileo, IRNSS, and the Beidou satellite navigation system. GPS is an abbreviation for Global Positioning System. QZSS is an abbreviation for Quasi-Zenith Satellite System. GLONASS is an abbreviation for Global Navigation Satellite System. IRNSS is an abbreviation for Indian Regional Navigation Satellite System.

[0059] The inertia acquisition unit 392 is configured to acquire the acceleration and angular velocity acting on the host vehicle. In this embodiment, the inertia acquisition unit 392 is provided as a three-axis gyro sensor and a three-axis acceleration sensor built into a box-shaped housing of the locator 39. The locator ECU 393 has a configuration as a so-called in-vehicle microcomputer including a CPU, a ROM, a RAM, an input / output interface, and the like. The locator ECU 393 is configured to sequentially determine the position, direction, and the like of the host vehicle based on the positioning signal received by the GNSS receiver 391, the acceleration and angular velocity acquired by the inertia acquisition unit 392, and the like.

[0060] (In-vehicle communication device) The in-vehicle communication device 4 is an in-vehicle communication module also called DCM, and is provided so as to be able to communicate information with base stations around the vehicle by wireless communication conforming to a communication standard such as LTE or 5G. DCM stands for Data Communication Module. LTE stands for Long Term Evolution. 5G stands for 5th Generation. Specifically, for example, the in-vehicle communication device 4 acquires traffic information such as congestion information from a probe server on the cloud and / or a predetermined database. The "traffic congestion information" includes the location and length of the congestion section. Specifically, the traffic congestion information includes the head position of the congestion, the tail position of the congestion, the estimated congestion distance, the estimated congestion time, and the like. The traffic information is also called "road traffic information." In addition, the in-vehicle communication device 4 acquires the latest HD map information from the above-mentioned probe server and stores it in the HD map 5.

[0061] (HD Map) The HD map 5 is mainly composed of a non-volatile rewritable memory so that the HD map information can be rewritably stored and the stored contents can be retained even when the power is cut off. The HD map information can also be called high-precision map data. The HD map information includes map information with higher precision than the map information in the SD map 601 used in the navigation device 6, which has a position error of about several meters. SD stands for Standard. Specifically, the HD map 5 stores information that can be used for advanced driving assistance or automatic driving, such as three-dimensional road shape information, lane number information, and regulation information, in accordance with a predetermined standard such as the ADASIS standard. ADASIS stands for Advanced Driver Assistance Systems Interface Specification.

[0062] (Navigation device) The navigation device 6 is configured to calculate a planned driving route from the current position of the vehicle to the destination. In this embodiment, the navigation device 6 is configured to calculate a planned driving route based on the destination set by the driver D or the like, map information stored in the HD map 5 or SD map 601, position information of the vehicle acquired from the locator 39, and the like. The navigation device 6 is also configured to be able to provide various information including route information, which is the calculation result, to each part such as the driving ECU 2 and the HMI device 7 via the in-vehicle communication line 10. That is, the navigation device 6 is configured to be able to display a navigation screen on the HMI device 7 for map display, route display, and the like.

[0063] (HMI device) The HMI device 7 is a so-called vehicle HMI, and has a configuration for realizing information transmission between the vehicle and the occupants including the driver D. Specifically, the HMI device 7 is configured to present various information related to the vehicle to the occupants at least visually, and to receive input operations by the occupants corresponding to the presented contents. The presented information includes, for example, various kinds of guidance, input operation instructions, input operation content notification, warnings, and the like. Typically, the HMI device 7 has a configuration as a so-called "dashboard HMI" mainly composed of devices attached to the dashboard V21 and the steering wheel V24. However, some of the input / output devices provided in the HMI device 7 may be provided in a place other than the dashboard V21, for example, in the ceiling part of the vehicle interior V2, or in the center console between the driver's seat V23 and the seat V22 adjacent thereto, i.e., the passenger seat, or the like.

[0064] The HMI device 7 includes an HCU 701, a meter panel 702, a main display device 703, a head-up display 704, a speaker 705, and an operation device 706. HCU is an abbreviation for HMI Control Unit. The HCU 701 has a configuration as a so-called in-vehicle microcomputer including a CPU, a ROM, a RAM, an input / output interface, and the like, and is configured to control display output and audio output by the HMI device 7. That is, the HCU 701 is provided to control the operations of the meter panel 702, the main display device 703, the head-up display 704, the speaker 705, and the like.

[0065] The meter panel 702 is provided in front of the driver's seat V23 on the dashboard V21. The meter panel 702 is configured to execute meter display of the vehicle speed, coolant temperature, remaining fuel amount, etc., as well as various information display such as date and time, outside air temperature, mileage, radio station, etc.

[0066] The main display device 703 is a display device also called a CID device, and is disposed in the approximate center of the dashboard V21 in the vehicle width direction so that the display screen can be viewed by the occupant. CID stands for Center Information Display. The main display device 703 can display a navigation screen for map display and route display by the navigation device 6. The main display device 703 can also display information and content different from the navigation screen. Specifically, the main display device 703 is configured to be able to execute a display for setting operations related to driving modes such as "comfort", "normal", "sports", and "circuit". The main display device 703 is also configured to be able to execute a display related to a second task that the driver D can use during automatic driving. The second task is a task other than driving operations that the driver D executes, and includes, for example, reading, operating a mobile communication terminal, watching video content, and the like. The "video content" is, for example, a movie, a concert video, a music video, a television broadcast, and the like. The second task is also called a "non-driving task" or a "secondary activity".

[0067] 1 and 2, and also with reference to FIG. 14 and FIG. 15, the head-up display 704 is provided to display a display image M including characters, figures, and / or symbols in the forward field of vision of the driver D. In this embodiment, the head-up display 704 is configured to display the display image M as a virtual image in front of the driver D using AR technology, thereby enabling the display image M to be superimposed on the foreground including the road surface FR ahead of the vehicle. AR is an abbreviation for Augmented Reality. The "superimposed display" refers to displaying related information in a positional correspondence with a target of interest included in the foreground, for example, displaying related information so as to be superimposed on the target of interest or displaying it in the vicinity of the target of interest. The "target of interest" is, for example, a target object to which attention should be paid or attention should be paid during driving, such as a road marking, a road sign, a vehicle ahead, a pedestrian, etc. The "road marking" may also be referred to as a "road marking". For example, various displays such as a route display, a traveling direction display, and a traffic information display on the road surface FR ahead as an object of attention correspond to "superimposed display".

[0068] As shown in FIG. 14, the head-up display 704 is configured so that the depression angle AD is a positive value of 0 degrees or more. In FIG. 14, the depression angle AD is an angle between a virtual horizontal line passing through the viewpoint EP of the driver D and a virtual straight line connecting the viewpoint EP and the upper end of the projection range AP. The depression angle AD is a positive value when the upper end of the projection range AP is looked down from the viewpoint EP, and a negative value when the upper end is looked up. In addition, the head-up display 704 is configured so that the projection range AP is a horizontally elongated approximately rectangular shape as shown in FIG. 15 by making the horizontal angle of view larger than the vertical angle of view AV. The vertical angle of view AV is an angle between a virtual straight line connecting the viewpoint EP and the upper end of the projection range AP and a virtual straight line connecting the viewpoint EP and the lower end of the projection range AP in FIG. That is, the vertical angle of view AV is an angle range in the vertical direction in which the display image M can be viewed from the viewpoint EP, and may also be referred to as a viewing angle in the vertical direction.

[0069] As shown in FIG. 14, the head-up display 704 includes a projector 741 and a magnifying optical system 742. The projector 741 is configured to generate a display image light LV based on a display image signal generated by the HCU 701 and emit the display image light LV toward the magnifying optical system 742. The magnifying optical system 742 includes a plurality of optical elements including a concave mirror and an actuator for controlling the alignment of the optical elements. The magnifying optical system 742 is configured to adjust the projection state of the display image light LV onto the front windshield V18 by controlling the above alignment with the actuator according to the viewpoint EP detected by the driver state monitor 37. The head-up display 704 projects the display image light LV onto a projection range AP on the front windshield V18 and allows the driver D to visually recognize the reflected light, thereby realizing the display of the display image M.

[0070] FIG. 15 shows an example of a display state of the head-up display 704. The head-up display 704 can display an information display M1 and a graphic display M2. The information display M1 is a display content that displays various information such as the maximum speed on the road Rd on which the vehicle is traveling, the current vehicle speed of the vehicle, the distance to the destination, the estimated time of arrival at the destination, building names, intersection names, etc. The information display M1 is typically a display using characters, road sign-like figures, or symbols. The graphic display M2 is a display content using graphics such as lines and arrows, and is used to display, for example, information related to the traveling direction of the vehicle and lane selection.

[0071] The head-up display 704 can display superimposed content and non-superimposed content. The "superimposed content" is a display image content that is associated with or related to a specific target of attention included in the foreground and is superimposed on the target of attention. In contrast, the "non-superimposed content" is a display image content that is not associated with a specific target of attention included in the foreground and is not superimposed on the specific target of attention. In the display example of FIG. 15, a linear graphic display M2 is superimposed as superimposed content on the lane in which the vehicle is currently traveling, and an information display M1 indicating the maximum speed is displayed as non-superimposed content at an appropriate position that does not overlap with the graphic display M2. In this way, the HMI device 7 has a configuration as a notification unit that notifies a passenger such as the driver D of the vehicle of information.

[0072] 1 and 2 again, speaker 705 is provided to output audio corresponding to the display contents on meter panel 702, main display device 703, and head-up display 704. Note that speaker 705 can also output audio that does not correspond to the display contents on meter panel 702, main display device 703, and head-up display 704, such as music, radio audio, and the like.

[0073] The operation device 706 is an input device that is not included in the operation switch 38 and is not subject to detection of the operation amount or operation state by the operation sensor 35. Specifically, for example, switches provided around the display screen of the main display device 703 and a transparent touch panel provided so as to cover such a display screen correspond to the operation device 706. Also, for example, switches provided on the spokes of the handle V24 of the steering wheel structure correspond to the operation device 706. Furthermore, a pointing device such as a touch panel provided on the center console corresponds to the operation device 706.

[0074] (Lighting device) The lighting device 8 includes a body ECU 801, a headlight 802, and a blinker 803. The body ECU 801 has a configuration as a so-called in-vehicle microcomputer equipped with a CPU, a ROM, a RAM, an input / output interface, etc. The body ECU 801 is configured to control the lighting state of the headlights 802 and the blinkers 803 based on information input from the driving ECU 2 and the driving information input unit 3.

[0075] (Motion Control Device) The motion control device 9 is provided to control the motion, i.e., the driving behavior, of the vehicle based on information input from the driving ECU 2 and the driving information input unit 3. Specifically, the motion control device 9 includes a drive device 91, a shift device 92, a braking device 93, and a steering device 94.

[0076] The drive device 91 includes a drive system ECU 911 and a drive mechanism 912. The drive system ECU 911 has a configuration as a so-called in-vehicle microcomputer including a CPU, ROM, RAM, an input / output interface, etc., and is configured to control the operation of the drive mechanism 912. The drive mechanism 912 that generates the driving force for running the system-equipped vehicle V includes an engine and / or an electric motor. That is, the system-equipped vehicle V may be any of a gasoline engine vehicle, a diesel engine vehicle, a biofuel vehicle, a hydrogen engine vehicle, a hybrid vehicle, a BEV, a fuel cell vehicle, etc. BEV is an abbreviation for Battery Electric Vehicle.

[0077] The shift device 92 includes a shift ECU 921 and a shift mechanism 922. The shift ECU 921 has a configuration as a so-called in-vehicle microcomputer including a CPU, a ROM, a RAM, an input / output interface, etc., and is configured to control the operation of the shift mechanism 922. The shift mechanism 922 is provided between the wheels V3 functioning as drive wheels and the drive device 91, and is configured to switch the shift position, that is, switch between forward running and reverse running, and change the gear ratio during forward running. The shift device 92 may have a configuration as a so-called shift-by-wire.

[0078] The braking device 93 includes a brake ECU 931 and a brake mechanism 932. The brake ECU 931 has a configuration as a so-called on-board microcomputer including a CPU, a ROM, a RAM, an input / output interface, etc., and is configured to control the operation of the brake mechanism 932. The brake mechanism 932 that generates a braking force in the system-equipped vehicle V may include a regenerative brake mechanism in addition to a friction brake mechanism. The braking device 93 may have a so-called brake-by-wire configuration.

[0079] The steering device 94 includes a steering ECU 941 and a steering mechanism 942. The steering ECU 941 has a configuration as a so-called on-board microcomputer including a CPU, a ROM, a RAM, an input / output interface, and the like, and is configured to control the operation of the steering mechanism 942. The steering mechanism 942 is configured to be able to change the orientation of the left front wheel V3a and the right front wheel V3b, which are steered wheels. The steering mechanism 942 may also be able to change the orientation of the left rear wheel V3c and the right rear wheel V3d. That is, the host vehicle may be a so-called four-wheel steering vehicle. The steering device 94 may also have a configuration as a so-called steer-by-wire.

[0080] (Driving control based on target recognition results) 16 and 17 show an example of a functional configuration realized by the driving ECU 2 by the processor 21 executing the program instructions stored in the memory 22. Hereinafter, an overview of the driving control of the host vehicle executed by the driving ECU 2 based on the recognition result of the target B around the host vehicle using the ADAS sensor will be described. Hereinafter, in this specification, "recognition" and "detection" are defined as follows. "Recognition" is a concept including "detection", "identification" and "recognition". "Detection" refers to finding the target B based on images and point cloud data. The "detected" target B is only judged to exist, and the shape, meaning, etc. are not judged. "Identification" refers to classifying the shape, meaning, etc. of the detected target B. The "identified" target B is judged not only to exist, but also to be what it is, for example, whether it is a human, a car, or a building. "Recognition" refers to judging the detected and identified target B as something that should be taken into consideration in the driving control of the host vehicle, such as a pedestrian, an oncoming vehicle, etc. It should be noted that "detection" includes "detection."

[0081] 16, the driving ECU 2 has a recognition unit 2001, an operation determination unit 2002, and a control output unit 2003 as functional configurations realized by the processor 21. The recognition unit 2001 is adapted to execute a recognition operation of a target B around the vehicle based on information input from a surrounding monitoring sensor, an operation sensor 35, a behavior sensor 36, and the like. The operation determination unit 2002 is adapted to determine the content of a control operation required for the vehicle at the present time based on the recognition result of the target B by the recognition unit 2001 and information input from the operation sensor 35, the behavior sensor 36, and the like. The content of such a control operation is, for example, a collision avoidance operation, an emergency stop operation, a warning operation to the driver D, and the like. The control output unit 2003 is adapted to generate a control output according to the control operation content determined by the operation determination unit 2002, and output it to each unit in the in-vehicle system 1. The control output is, for example, a steering amount, a braking amount, a warning message code, and the like.

[0082] 17, the recognition unit 2001 has an input information acquisition function 2101, an input information processing function 2102, a target recognition function 2103, a vehicle lane recognition function 2104, an intersection recognition function 2105, and a surrounding environment recognition function 2106. The input information acquisition function 2101 holds a predetermined amount of information input to the recognition unit 2001 in time series. The input information processing function 2102 executes various processes such as noise removal and coordinate conversion on the information acquired, i.e., held, by the input information acquisition function 2101. The target recognition function 2103 executes a recognition operation of the target B based on the information processed by the input information processing function 2102. The target recognition function 2103 has a lane marking recognition function 2131, a road marking recognition function 2132, a roadside structure recognition function 2133, a traffic light recognition function 2134, a road sign recognition function 2135, a lane recognition function 2136, a pedestrian recognition function 2137, a surrounding vehicle recognition function 2138, and an obstacle recognition function 2139.

[0083] The recognition targets by each recognition function in the recognition unit 2001 are illustrated in FIG. 18, FIG. 19, and the like. Referring to FIG. 18, the recognition targets B include, broadly speaking, a traffic-related three-dimensional object B1, another vehicle B2, a general three-dimensional object B3, and a road marking B4. Referring to FIG. 19, the recognition targets B include a parking frame line B5, a wheel stopper B6, and a parked vehicle B7 as objects related to the detection of a parking space PS, which is an area where a vehicle can be parked, and parking assistance. The parking frame line B5 is paint for defining the shape or range of the parking space PS. The wheel stopper B6 is a block-shaped three-dimensional object installed in the parking space PS. The parked vehicle B7 is a vehicle that is being parked. The traffic-related three-dimensional object B1 is a three-dimensional object for use in road traffic, such as a traffic light B11 and a road sign B12. The other vehicle B2 is a vehicle that may be a following target or an obstacle during the traveling of the host vehicle, and does not include the parked vehicle B7 shown in FIG. 19. The general three-dimensional object B3 is a three-dimensional object other than the traffic-related three-dimensional object B1, the other vehicle B2, the wheel stopper B6, and the parked vehicle B7, and mainly constitutes an "obstacle". Furthermore, for example, it is possible to recognize the lane LN based on the recognition result of the target B using an image. That is, the parking space PS and the lane LN are different from the target B as a direct recognition target by the ADAS sensor, but can be said to be an indirect recognition target based on the recognition result of the target B. The lane LN as such an indirect recognition target may include, for example, the host lane LNm in which the host vehicle is currently traveling, the oncoming lane LNc, and the like. Further details of each recognition target in the recognition unit 2001 will be described later.

[0084] The own lane recognition function 2104 recognizes the position of the own vehicle in the width direction of the road Rd on which the own vehicle is currently traveling, based on the recognition result of the target B by the target recognition function 2103. Hereinafter, the width direction of the road Rd is referred to as the "road width direction". In contrast, the direction perpendicular to the road width direction is referred to as the "road extension direction". The road extension direction is a direction extending along the road Rd, and is also referred to as the "road extension direction" or "road extension direction". When such a road Rd includes multiple lanes LN, the own lane recognition function 2104 recognizes which of the multiple lanes LN lined up in the road width direction the own lane LNm is. The intersection recognition function 2105 recognizes an intersection Xr around the own vehicle, based on the recognition result of the target B by the target recognition function 2103. Specifically, the intersection recognition function 2105 recognizes, for example, the presence or absence of a traffic light B11 and the color of the light, the presence of road markings B4 such as a stop line B42, the positions of the intersection entrance Xr1 and the intersection exit Xr2, the course division, etc., of the intersection Xr that the vehicle is currently approaching. The surrounding environment recognition function 2106 recognizes the driving environment around the vehicle, that is, for example, the presence or absence of obstacles, etc., based on the recognition result of the target B by the target recognition function 2103. These recognition results are provided to the operation determination unit 2002 shown in FIG. 16 to determine the content of the control operation currently required for the vehicle.

[0085] FIG. 18 shows an example of a foreground view from the vehicle in a situation where the vehicle is approaching an intersection Xr that forms a crossroads and has a traffic light B11. The intersection Xr that forms a crossroads is hereinafter simply referred to as a "crossroads". The recognition results of the traffic light B11 and the road sign B12 included in the traffic-related three-dimensional object B1 can be used, for example, for information notification or warning to the driver D by the HMI device 7. In addition, such recognition results can be used for the motion control of the vehicle during automatic driving or driving assistance. In addition to the traffic light B11 and the road sign B12, the traffic-related three-dimensional object B1 includes roadside structures B13 such as guardrails and curbs. The roadside structures B13 are recognized as obstacles when controlling the running of the vehicle. In addition, the traffic-related three-dimensional object B1 includes road studs B14 and poles B15 as shown in FIG. 20. Depending on the height and installation situation, the road studs B14 are recognized as obstacles when controlling the running of the vehicle. The pole B15 is recognized as an obstacle during driving control of the host vehicle.

[0086] 18 again, the general three-dimensional object B3 includes a fallen object B31 on the road, a pedestrian B32, a cyclist B33, a structure B34, etc. The fallen object B31 on the road, the pedestrian B32, and the cyclist B33 are recognized as obstacles when controlling the running of the vehicle. The structure B34 is not recognized as an obstacle because it is usually located outside the road Rd, but it can be a target for superimposed display when notifying the driver D of information by the head-up display 704, for example.

[0087] The road markings B4 include a crosswalk B41, a stop line B42, and a road dividing line B43. As shown in FIG. 21, the road markings B4 also include a character marking B44 and a graphic marking B45. The character markings B44 include, for example, a number indicating a maximum speed, a vehicle traffic division such as "bus only", and a regulatory character such as "stop". The graphic markings B45 include, for example, an arrow indicating a possible travel direction and a graphic for calling attention to the crosswalk B41 ahead. The recognition result of the road markings B4 can be used, for example, to estimate the positions of an intersection entrance Xr1, an intersection exit Xr2, and an intersection center Xrc. The intersection entrance Xr1 is the edge of the intersection of interest on the side where the vehicle enters. The intersection of interest is an intersection Xr that the vehicle is currently approaching and is closest to the vehicle, where the vehicle is scheduled to pass or is highly likely to pass. An intersection Xr that "the vehicle plans to pass through" is an intersection Xr on the planned travel route when the destination and planned travel route of the vehicle are set in the navigation device 6. An intersection Xr that "is likely to be passed through" is an intersection Xr that is estimated with a high probability that the vehicle will pass through, based on the current vehicle speed of the vehicle and the distance to the intersection Xr, when the destination and planned travel route of the vehicle are not set in the navigation device 6. The intersection exit Xr2 is the edge of the intersection of interest on the side from which the vehicle will exit. The intersection center Xrc is the center of the intersection of interest.

[0088] As shown in Figs. 20 to 22, the road division line B43 includes a roadway outer edge line B431, a center line B432, and a lane boundary line B433. The roadway outer edge line B431 is provided to indicate the outer edge line of the roadway in the road width direction. The "roadway" is a part of the road Rd on which vehicles travel. The center line B432 is provided to divide the road Rd, which is a two-way road, i.e., a non-one-way road, by traffic direction. The center line B432 may be a single line, a double line, or a triple line. The lane boundary line B433 is provided to divide multiple lanes LN of the same traffic direction on a road Rd with two or more lanes on one side. The road division line B43 may be painted white or yellow. In the example of Fig. 20, the center line B432 has a triple line pattern with yellow solid lines on both sides of a white solid line, and road studs B14 and poles B15 are provided on the central white solid line. In the example of Fig. 21, the center line B432 has a double line pattern with white solid lines, and the lane boundary line B433 has a yellow solid line portion beyond the white broken line portion. From the recognition results of the roadside structures B13 and road division lines B43, etc., it becomes possible to recognize the driving lane LNd, the oncoming lane LNc, the passing lane LNp, the right-turn lane LNr, the road shoulder LNs, the emergency parking zone EZ, etc.

[0089] The road markings B4 include a guidance strip marking B461, a safety zone marking B462, a no entry sign B463, a no stopping sign B464, and the like, as shown in Figs. 23 to 26. The guidance strip marking B461 shown in Fig. 23 is provided to indicate a guidance strip for guiding vehicles to travel safely and smoothly. Although the guidance strip is a marking for guiding vehicles not to pass through, the entry of vehicles into the guidance strip itself is not prohibited by law. The guidance strip marking B461 is provided, for example, near an intersection Xr, a junction, or a branch. The safety zone marking B462 shown in Fig. 24 is provided to indicate a safety zone. The safety zone is an area provided on the road Rd for the purpose of ensuring the safety of pedestrians B32, etc., and vehicles are not allowed to enter inside the safety zone. The no entry sign B463 shown in Fig. 25 is provided to indicate a no entry area. A no-entry area is an area that is not permitted for vehicle traffic, and vehicles are not permitted to enter inside the area. The no-stop sign B464 shown in Fig. 26 is provided to indicate a no-stop area. A no-stop area is an area that a vehicle is not permitted to enter if it is likely to be forced to stop due to the situation ahead of the path it is about to travel. In other words, a vehicle is not permitted to stop inside the no-stop sign B464 when waiting at a traffic light or in a traffic jam.

[0090] FIG. 27 shows an example of a road sign B12. The road sign B12 is a signboard that is installed beside or above the road Rd and provides necessary information to users through its design. The design includes an external shape, a color, and a symbol. The external shape is a circle, a triangle, an inverted triangle, a square, a diamond, an octagon, etc. The colors are white, black, red, blue, yellow, etc. In FIG. 27, the cross-hatched portion indicates red, the diagonal hatched portion indicates blue, the dot-hatched portion indicates yellow, and the blackened portion indicates black. The symbols include letters, figures, and combinations thereof.

[0091] In FIG. 27, examples of signs in Japan, Europe, and the United States are shown, categorized by meaning. For European examples, examples from Germany and France are used. For example, the design of "No entry" is almost the same in each country. In contrast, the color and symbol of "Stop" are common, but the external shape is almost the same in Europe and the United States, while the Japanese sign is different from Europe and the United States. The design of maximum speed signs is almost the same in Japan and Europe, except for the color of the letters, but the American sign is completely different from Japan and Europe in terms of external shape and color. The design of "Watch out for railroad crossings" is low in common between Japan, Europe, and the United States. The design of "No right turns" is similar in Europe and the United States, but there is no corresponding sign in Japan. However, for example, if a "No going in any direction other than the specified direction" sign, which shows the straight-ahead direction and the left-turn direction with white arrows, is provided at a cross intersection, as shown in FIG. 27, such a sign essentially indicates "No right turns."

[0092] Accurate recognition of the road signs B12 satisfactorily realizes automated driving and advanced driving assistance, but as described above, the design of the road signs B12 may differ greatly from country to country. Therefore, a database for recognizing the meaning indicated by the detected road signs B12 by pattern matching is stored in advance in the memory 22 shown in FIG. 2. Such a database may be country-specific or region-specific, or may be common to the entire world. Note that such a database may be stored in a host computer connected to the vehicle via the in-vehicle communication device 4 in place of or together with the memory 22. V2X is an abbreviation for Vehicle to X.

[0093] FIG. 28 shows an example of a scene in which the recognition result of the target B is used, in which the vehicle is approaching a crossroads. In this scene, for example, an information display M1 such as "Stop line 10m" indicating the distance between the vehicle and the stop line B42 can be superimposed on the stop line B42. In addition, a voice notification such as "10m to the stop line" can be given. This makes it possible to effectively support the driver D using the driving support function to stop at the stop line B42 without fail. Alternatively, by notifying the passengers including the driver D using the automatic driving function of the scheduled occurrence of a stop event at an appropriate timing, it is possible to effectively avoid impairing the comfort of the passengers.

[0094] FIG. 29 shows another example of a scene in which the recognition result of the target B is used, in which the vehicle is approaching a crossroads with a traffic light B11 where the vehicle is scheduled to turn right. In this scene, for example, an information display M1 such as "20m ahead, turn right" indicating the distance to the right turn point can be superimposed on the road surface FR in front of the field of vision of the driver D in the right turn lane LNr, which is the vehicle lane LNm. In addition, a voice notification such as "turn right at the intersection 20m ahead" can be given. Furthermore, an arrow graphic display M2 indicating the planned travel route of the vehicle can be superimposed on the road surface FR so as to pass through the vicinity of the intersection center Xrc from the right turn lane LNr, which is the vehicle lane LNm, and reach the road Rd where the vehicle is to turn right. This allows the vehicle's course guidance and lateral acceleration generation behavior to be properly notified to the driver D and other occupants, thereby making it possible to properly avoid impairing the comfort of the occupants.

[0095] 17 again, the target recognition function 2103 recognizes the type and state of the target B based on the recognition result using the ADAS sensor and the HD map information stored in the HD map 5. The "state" includes the position. The "position" includes the distance from the vehicle, the relative position with respect to the vehicle, the lane position, i.e., one of the multiple lanes LN, and the like.

[0096] The lane marking recognition function 2131 recognizes road markings B43 provided around the vehicle, including on the road surface FR of the road Rd where the vehicle is heading. For example, the lane marking recognition function 2131 recognizes whether the outer roadway line B431, center line B432, lane boundary line B433, etc., as shown in Figs. 20 to 23, are white or yellow, solid or broken, single or multiple lines, etc. Technology for recognizing road markings B43 was well known at the time of filing the present application, and therefore detailed description thereof will be omitted in this specification.

[0097] The road marking recognition function 2132 recognizes the type, meaning, and position of road markings B4 provided around the vehicle, other than the road dividing line B43. Recognition targets by the road marking recognition function 2132 include, for example, a crosswalk B41, a stop line B42, and a graphic marking B45.

[0098] The roadside structure recognition function 2133 recognizes the type and position of the roadside structure B13 provided around the vehicle. At least one of the image recognition result using the camera 34, the recognition result using the laser radar sensor 33, and HD map information is used to recognize the roadside structure B13. The traffic light recognition function 2134 recognizes at least the position and light color of the traffic light B11 provided at the travel destination of the vehicle. The road sign recognition function 2135 recognizes the road signs B12 provided around the vehicle. The lane recognition function 2136 recognizes the lane, that is, recognizes the number and type of the lanes LN on the road Rd on which the vehicle is traveling. At least one of the image recognition result using the camera 34, i.e., the front camera CF, the recognition result using the laser radar sensor 33, and HD map information is used to recognize the lane. Specifically, in lane recognition, the image recognition results obtained using the front camera CF are usually used, but so-called "sensor fusion" that uses other information as well can be performed as necessary.

[0099] The pedestrian recognition function 2137 recognizes a pedestrian B32 existing around the vehicle. The surrounding vehicle recognition function 2138 recognizes another vehicle B2 existing around the vehicle. The obstacle recognition function 2139 recognizes obstacles such as road debris B31 and pedestrians B32 existing around the vehicle. The recognition methods of the traffic light recognition function 2134, the road sign recognition function 2135, the lane recognition function 2136, the pedestrian recognition function 2137, the surrounding vehicle recognition function 2138, and the obstacle recognition function 2139 are well known at the time of filing of the present application. Therefore, the details of these recognition methods will not be described in this specification.

[0100] First embodiment The first embodiment will be described below. The driving ECU 2 as the in-vehicle device according to this embodiment, the method or computer program executed by the driving ECU 2, i.e., the processor 21, and the recording medium on which the computer program is recorded may be collectively referred to as the "present embodiment" below. The same applies to other embodiments described later.

[0101] FIG. 30 shows a schematic functional block configuration of the parking space detection device 2200 according to the present embodiment. The parking space detection device 2200 is configured to detect a parking space PS around the vehicle based on an image captured by the camera 34 as an imaging device during parking assistance operation or automatic parking operation. In addition to "automatic parking" in which the vehicle is automatically parked at a target parking space PS without a driving operation by the driver D, "parking assistance" also includes a case in which some driving operation (e.g., accelerator operation) by the driver D is required while other driving operations (e.g., steering operation) are not required. The parking space detection device 2200 can be realized as at least one functional block in the recognition unit 2001 shown in FIG. 17. Specifically, the parking space detection device 2200 can be realized as, for example, a part of the road surface marking recognition function 2132 or the surrounding environment recognition function 2106 shown in FIG. 17. Alternatively, the parking space detection device 2200 can be realized as, for example, a function spanning the road surface marking recognition function 2132 and the surrounding environment recognition function 2106.

[0102] Hereinafter, the parking space detection device 2200 according to the present embodiment will be described in detail. As shown in Fig. 30, the parking space detection device 2200 includes an annotation information generating unit 2201, a learning unit 2202, and a parking space detection unit 2203. In addition, the parking space detection device 2200 includes an image information storage unit 2204, an image information input unit 2205, an annotation information storage unit 2206, and a learning parameter storage unit 2207.

[0103] The annotation information generating unit 2201 generates annotation information corresponding to the learning image information stored in the image information storage unit 2204, and outputs the annotation information to the annotation information storage unit 2206. The annotation information is information corresponding to so-called "correct answer data" that indicates what the target B included in the learning image input from the image information storage unit 2204 is and where it is shown. The learning unit 2202 executes machine learning of the parking space PS by a so-called deep learning method based on the learning image information stored in the image information storage unit 2204 and the annotation information generated by the annotation information generating unit 2201, and outputs learning parameters as learning results to the learning parameter storage unit 2207. The parking space detection unit 2203 detects the parking space PS from the image captured by the camera 34 input by the image information input unit 2205, using the learning results by the learning unit 2202. That is, the image information input unit 2205 inputs the image captured by the camera 34 to the parking space detection unit 2203.

[0104] The annotation information generating unit 2201 has a bird's-eye image generating unit 2211, an annotation input unit 2212, and a parking space annotation generating unit 2213. The bird's-eye image generating unit 2211 converts the learning image input from the image information storage unit 2204 into a bird's-eye image by a well-known viewpoint conversion method, thereby generating a bird's-eye image for generating annotation information. A "bird's-eye image" is an image in which an object is viewed from above with a line of sight parallel to the direction of gravity, with the viewpoint being above the object. In other words, a bird's-eye image is an image corresponding to a plan view of the object.

[0105] The annotation input unit 2212 is adapted to input annotation information corresponding to the bird's-eye image generated by the bird's-eye image generation unit 2211 to the parking space annotation generation unit 2213. Specifically, for example, the annotation input unit 2212 is adapted to generate annotation information corresponding to the bird's-eye image by image recognition and / or an input operation by an annotation input worker, and input the annotation information to the parking space annotation generation unit 2213. That is, for example, the annotation input unit 2212 is adapted to automatically generate annotation information from the bird's-eye image by an image recognition method, and input the annotation information to the parking space annotation generation unit 2213. Alternatively, for example, the annotation input unit 2212 may have a so-called man-machine interface that accepts a correction input operation by an annotation input worker for the annotation information automatically generated as described above. Alternatively, for example, the annotation input unit 2212 may be provided as a so-called man-machine interface that accepts an input operation of annotation information by an annotation input worker.

[0106] The parking space annotation generating unit 2213 generates annotation information corresponding to the parking space PS included in the bird's-eye image, and outputs the annotation information to the annotation information storage unit 2206. Specifically, the parking space annotation generating unit 2213 has a corner position generating unit 2214 and an attribute generating unit 2215.

[0107] The corner position generating unit 2214 generates position information in the image of a plurality of parking space corner points BP and a parking space center point BPC that define the shape of the parking space PS, as shown in FIG. 31. The plurality of parking space corner points BP include a left front corner point BP1, a right front corner point BP2, a left rear corner point BP3, and a right rear corner point BP4. The definitions of the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, the right rear corner point BP4, and the parking space center point BPC will be described below with reference to FIG. 31. In FIG. 31, for the sake of simplicity of description, a parking space PS surrounded by a rectangular parking frame line B5 formed by a front frame line B501, a rear frame line B502, and a pair of side frame lines B503 is shown. However, as will be described later, the parking space PS is not limited to being surrounded by such a rectangular parking frame line B5. That is, for example, the parking space PS can also be formed by a substantially U-shaped parking frame line B5. Alternatively, for example, the parking frame PS can be formed by a plurality of parallel linear parking frame lines B5. Alternatively, for example, the parking frame PS can be formed by the space between adjacent parked vehicles B7 in the arrangement of a plurality of parked vehicles B7, not by the parking frame lines B5.

[0108] The front frame line B501 and the rear frame line B502 are provided at positions corresponding to the short sides of the rectangular parking stall PS. The front frame line B501 is located forward of the rear frame line B502, that is, on the parking stall direction BD side. The parking stall direction BD is the direction in which the parked vehicle B7 faces when the vehicle is parked in the parking stall PS. The parking stall direction BD can be determined by the structure and arrangement of the parking stall PS. For example, as shown in FIG. 31, when a wheel stopper B6 is provided in the parking stall PS, the parking stall direction BD can be determined so as to face the pair of short sides from the side closer to the wheel stopper B6 to the side farther from the pair of short sides. Alternatively, for example, in the case of parallel parking, the parking stall direction BD can be determined toward the aisle side. Alternatively, for example, in the case of parallel parking, the parking stall direction BD can be determined based on the traveling direction of the vehicle. The pair of side frame lines B503 are provided at positions corresponding to the long sides of the rectangular parking stall PS. When the parking frame direction BD is set as the "forward" direction as a directional reference, the left side of the pair of side border lines B503 is referred to as a left side border line B504, and the right side is referred to as a right side border line B505.

[0109] The left front corner point BP1 is a corner point at the position where the front frame line B501 and the left frame line B504 intersect. The right front corner point BP2 is a corner point at the position where the front frame line B501 and the right frame line B505 intersect. The left rear corner point BP3 is a corner point at the position where the rear frame line B502 and the left frame line B504 intersect. The right rear corner point BP4 is a corner point at the position where the rear frame line B502 and the right frame line B505 intersect. The parking frame center point BPC is the center point of the circumscribed rectangle formed by the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, and the right rear corner point BP4. Specifically, the parking frame center point BPC is the intersection point of the virtual straight line connecting the left front corner point BP1 and the right rear corner point BP4 and the virtual straight line connecting the right front corner point BP2 and the left rear corner point BP3. In addition, as described above, even in cases other than the case where the parking frame PS is surrounded by the rectangular parking frame line B5, each corner point can be defined in the same manner as the case where the parking frame PS is surrounded by the rectangular parking frame line B5. For example, when there is no front frame line B501, the left front corner point BP1 can be set at the tip of the left frame line B504. Similarly, when there is no rear frame line B502, the right rear corner point BP4 can be set at the rear end of the right frame line B505. When there is no parking frame line B5, each corner point can be set by the circumscribing rectangle surrounding the parked vehicle B7, or the virtual rectangular shape of the parking frame PS formed by the space between the adjacent parked vehicles B7.

[0110] The attribute generating unit 2215 generates attribute information of each parking space PS. The "attribute information" includes, for example, frame line information. The frame line information is attribute information related to the parking frame line B5, and includes the presence or absence of the parking frame line B5 and the presence or absence of frame line elements such as the front frame line B501 and the rear frame line B502 that configure the parking frame line B5. In addition, examples of attribute information other than the frame line information will be described in the second embodiment and thereafter to avoid redundant and complicated explanations.

[0111] For example, the bird's-eye image generating unit 2211 generates a target image GS by converting the learning image stored in the image information storage unit 2204 into a bird's-eye image by viewpoint conversion, as shown in FIG. 32. The target image GS is an image to be used for annotation input and learning. FIG. 32 shows an example of a learning image including a plurality of parking spaces PS around the system-equipped vehicle V, i.e., the vehicle itself, which has entered the parking lot PL. FIG. 33 shows a part of FIG. 32, specifically, an enlarged view of the parking space PS located at the left rear of the vehicle itself and the surrounding area where the vehicle itself is about to park. The parking space annotation generating unit 2213 generates annotation information for each parking space PS based on the input information from the annotation input unit 2212. Specifically, the corner position generating unit 2214 generates position information of the parking space corner point BP and the parking space center point BPC in the target image GS for each parking space PS, as shown in FIG. 33. The attribute generating unit 2215 generates attribute information for each parking space PS. The annotation information thus generated or assigned for each parking space PS is stored in the annotation information storage unit 2206.

[0112] The learning unit 2202 has a bird's-eye image generating unit 2221 and a parking space learning unit 2222. The bird's-eye image generating unit 2221 converts the learning image input from the image information storage unit 2204 into a bird's-eye image by a well-known viewpoint conversion method, thereby generating a learning bird's-eye image. The configuration or function of the bird's-eye image generating unit 2221 is similar to that of the bird's-eye image generating unit 2211 in the annotation information generating unit 2201. Therefore, the bird's-eye image generating unit 2221 can be modularized in common with the bird's-eye image generating unit 2211 in the annotation information generating unit 2201. The parking space learning unit 2222 executes learning of the parking space PS by machine learning based on the target image GS generated by the bird's-eye image generating unit 2221 and the corresponding annotation information.

[0113] The details of the learning of the parking space PS will be described below. First, as shown in Fig. 32, the target image GS is divided into a plurality of grids GR by using a plurality of grid lines GL. In Fig. 32, in order to simplify the description, a 10x10 grid GR is formed by nine vertical and nine horizontal grid lines GL, but the present disclosure is not limited thereto.

[0114] FIG. 33 shows a 3×3 portion that is a part of the 10×10 grid GR shown in FIG. 32. In the example of FIG. 33, the parking frame center point BPC exists in the center and upper left grid GR of the 3×3. Therefore, for these grids GR, as shown in FIG. 34, the encoding value Ecv is set to "1" in the first encoding Ec1 that corresponds to "CENTER:Responsible", that is, the existence of the parking frame center point BPC. On the other hand, for the grid GR that does not have the parking frame center point BPC, the encoding value Ecv is set to "0".

[0115] FIG. 35 and FIG. 36 show an encoding example regarding the lateral position of the parking frame center point BPC in the grid GR. In this embodiment, as shown by the dashed arrow in FIG. 35, the lateral position of the parking frame center point BPC in the grid GR is defined based on the left side of the grid GR. The lateral position is defined by setting the position of the left side of the grid GR as "0" and the position of the right side as "1". That is, the lateral position is indicated by a value that indicates the internal division ratio in decimal when the internal division point of the virtual line segment between the left side and the right side of the grid GR that passes through the parking frame center point BPC and is parallel to the grid line GL is set as the parking frame center point BPC. Then, as shown in FIG. 36, in the first encoding Ec1 corresponding to "CENTER:Offset X", that is, the lateral position of the parking frame center point BPC, the encoding value Ecv is set by the lateral position. On the other hand, for the grid GR in which the encoded value Ecv is set to "0" in FIG. 34, the encoded value Ecv at "CENTER:Offset X" is set to "0".

[0116] FIG. 37 and FIG. 38 show an encoding example regarding the vertical position of the parking frame center point BPC in the grid GR. In this embodiment, as shown by the dashed arrow in FIG. 37, the vertical position of the parking frame center point BPC in the grid GR is defined based on the upper side of the grid GR. The vertical position is defined by setting the position of the upper side of the grid GR as "0" and the position of the lower side as "1". That is, the vertical position is indicated by a value that indicates the internal division ratio in decimal when the internal division point of the virtual line segment between the upper side and the lower side of the grid GR that passes through the parking frame center point BPC and is parallel to the grid line GL is set as the parking frame center point BPC. Then, as shown in FIG. 38, in the first encoding Ec1 corresponding to "CENTER:Offset Y", that is, the vertical position of the parking frame center point BPC, the encoding value Ecv is set by the vertical position. On the other hand, for the grid GR in which the encoded value Ecv is set to "0" in FIG. 34, the encoded value Ecv at "CENTER:Offset Y" is set to "0".

[0117] FIG. 39 is a diagram showing the encoding of the presence of the parking space center point BPC shown in FIG. 34 for the entire target image GS shown in FIG. 32. FIG. 40 is a diagram showing the encoding of the horizontal position of the parking space center point BPC shown in FIG. 36 for the entire target image GS shown in FIG. 32. FIG. 41 is a diagram showing the encoding of the vertical position of the parking space center point BPC shown in FIG. 39 for the entire target image GS shown in FIG. 32. Similarly, three types of encoding, presence, horizontal position, and vertical position, are also performed on the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, and the right rear corner point BP4.

[0118] FIG. 42 shows a parking space annotation code Ea having a first encoding Ec1 for a total of five points, including the parking space center point BPC, the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, and the right rear corner point BP4. The parking space learning unit 2222 performs machine learning of the parking space PS by calculating a loss function using the parking space annotation code Ea. Since machine learning using a loss function in deep learning or the like is already a well-known technique at the time of filing of this application, detailed description thereof will be omitted in this specification. Thus, in this embodiment, the parking space learning unit 2222 learns the parking space PS so that a total of three channel outputs, namely, the presence, lateral position, and longitudinal position, are obtained for a total of five points, including the parking space center point BPC and the four parking space corner points BP.

[0119] Using Fig. 42 to Fig. 46, the encoding of the connection vector connecting the parking space center point BPC and each of the four parking space corner points BP will be described. Fig. 43 shows an enlarged view of a total of nine grids GR in a 3x3 arrangement including the grid GR shown in Fig. 42 by a thick square and the eight grids GR surrounding it, together with the arrangement of the parking space corner points BP and the like. The bird's-eye view image, the grid GR, and the arrangement of the parking space center point BPC and the parking space corner points BP shown in Fig. 43 are the same as those in Fig. 33. The grid GR shown in Fig. 43 by a thick square is the same as the grid GR shown in Fig. 42 by a thick square. Fig. 44 shows an example of a connection number table Tb used for such encoding. Fig. 45 and Fig. 46 show the contents of the second encoding Ec2, which is the encoding result.

[0120] In FIG. 43, the connection vector starting from the parking space center point BPC and heading toward the left front corner point BP1 is indicated by a dashed arrow. In encoding the connection vector shown in FIG. 43, if we focus on the central grid GR where the parking space center point BPC is located, indicated by a thick line, the left front corner point BP1, which is the end point of the connection vector, is located in the grid GR adjacent to the upper side. Here, when encoding the connection vector, the connection number table Tb shown in FIG. 44 is applied. The connection number table Tb has a 3×3 arrangement corresponding to the arrangement of the above 3×3 grid GR, and each position is assigned a connection number Tbn from 0 to 8 from the upper left to the lower right. That is, the connection numbers Tbn in the connection number table Tb are "0", "1", and "2" in the top row from left to right, "3", "4", and "5" in the middle row from left to right, and "6", "7", and "8" in the bottom row from left to right. Therefore, when such a connection number table Tb is applied, the connection vector shown in Figure 43 can be represented using connection number 1, since it is a vector going from connection number 4 to connection number 1, as shown by the arrow in Figure 44.

[0121] As shown in Fig. 45, the second encoding Ec2 is such that an encoding value Ecv of "0" or "1" is assigned to each of the connection numbers Tbn=0 to 8. Therefore, in the examples of Fig. 42 and Fig. 43, the second encoding Ec2 is generated as follows.

[0122] First, for the connection vector starting from the parking space center point BPC and heading towards the left front corner point BP1, as shown in FIG. 45, the grid GR indicated by the thick square is encoded so that the connection number Tbn=1 is "1" and the other connection numbers Tbn are "0". Such encoding is performed for each of the 10×10 grids GR corresponding to the target image GS. This allows the generation of the second encoding Ec2 for the connection vector starting from the parking space center point BPC and heading towards the left front corner point BP1, which is arranged in accordance with the arrangement of the 10×10 grids GR corresponding to the target image GS by the nine layers of encoding shown in FIG. 45.

[0123] Similarly, referring to FIG. 43 and FIG. 44, the right front corner point BP2 exists in the grid GR corresponding to the connection number Tbn=5, which is adjacent to the grid GR shown by the thick square. Therefore, for the grid GR shown by the thick square, the connection vector starting from the parking space center point BPC and heading toward the right front corner point BP2 is encoded so that the connection number Tbn=5 is "1" and the other connection numbers Tbn are "0". Such encoding is performed for each of the 10×10 grids GR corresponding to the target image GS. As a result, the 9-layer encoding shown in FIG. 45 can generate the second encoding Ec2 for the connection vector starting from the parking space center point BPC and heading toward the right front corner point BP2, which is arranged in accordance with the arrangement of the 10×10 grids GR corresponding to the target image GS. In this way, a total of four second encodings Ec2 corresponding to the left front corner point BP1, the right front corner point BP2, the left rear corner point BP3, and the right rear corner point BP4 can be generated, as shown in FIG. 46.

[0124] The third encoding Ec3, which is encoding related to frame information as attribute information, will be described with reference to Figs. 47 to 54. Fig. 47 shows a 3×3 portion of the 10×10 grid GR shown in Fig. 32, similar to Fig. 33. The bird's-eye image, grid GR, and the arrangement of the parking frame center point BPC and the parking frame corner point BP shown in Fig. 47 are the same as those in Figs. 33 and 43. In this embodiment, the frame information is the presence or absence of frame elements such as the front frame line B501 and the rear frame line B502 that constitute the parking frame line B5.

[0125] In the example of FIG. 47, the parking frame center point BPC exists in the upper left and center grids GR. Therefore, the encoding value Ecv in the third encoding Ec3 related to frame information is assigned to the upper left and center grids GR where the parking frame center point BPC exists. FIG. 48 shows the encoding result of the presence or absence of the left frame line B504 corresponding to FIG. 47. FIG. 49 shows the encoding result of the presence or absence of the front frame line B501 corresponding to FIG. 47. The thick squares in FIG. 48 and FIG. 49 indicate the encoding corresponding to the upper left and center grids GR where the parking frame center point BPC exists in FIG. 47. As shown in FIG. 48 and FIG. 49, the frame information is assigned to the position corresponding to the grid GR where the parking frame center point BPC exists, and the encoding value Ecv for other positions is uniformly set to "0".

[0126] The parking frame line B5 shown in FIG. 47 is composed of only a pair of left and right side frame lines B503. Therefore, in the example of FIG. 47, the left frame line B504 exists, so the corresponding encoded value Ecv is set to "1" as shown in FIG. 48. On the other hand, in this example, the front frame line B501 does not exist, so the corresponding encoded value Ecv is set to "0" as shown in FIG. 49. FIG. 50 is a diagram showing the encoding corresponding to the left frame line B504 shown in FIG. 48 for the entire target image GS shown in FIG. 32. FIG. 51 is a diagram showing the encoding corresponding to the right frame line B505 for the entire target image GS shown in FIG. 32. FIG. 52 is a diagram showing the encoding corresponding to the rear frame line B502 for the entire target image GS shown in FIG. 32. FIG. 53 is a diagram showing the encoding corresponding to the front frame line B501 shown in FIG. 49 for the entire target image GS shown in FIG. 32.

[0127] 54 shows a parking space annotation code Ea having a first encoding Ec1, a second encoding Ec2, and a third encoding Ec3. The parking space learning unit 2222 uses the parking space annotation code Ea to calculate a loss function, thereby performing deep learning of the parking space PS. This makes it possible to infer the presence or absence of the parking space line B5 or its frame line element for each object of each parking space PS.

[0128] Referring again to FIG. 30, the parking frame detection unit 2203 includes a bird's-eye image generation unit 2231, a parking frame candidate detection unit 2232, an edge calculation unit 2233, and a position correction unit 2234. The bird's-eye image generation unit 2231 converts the captured image by the camera 34 input by the image information input unit 2205 using a well-known viewpoint conversion method, thereby detecting the parking frame PS in the captured image and generating a bird's-eye image for use in parking assistance and the like. The parking frame candidate detection unit 2232 detects the parking frame PS candidate and its state in the captured image using the learning result by the learning unit 2202. The "state" includes whether the parking frame PS is vacant, that is, whether it is not occupied by another parked vehicle B7 different from the host vehicle. The edge calculation unit 2233 calculates an edge corresponding to the parking frame line B5. The "edge" is a virtual line constituting the outer edge of the rectangular parking frame PS, and is generated at a position corresponding to the edge in the line width direction of the parking frame line B5. The position correction unit 2234 corrects the position of the parking space PS based on the detection result of the parking space PS candidate by the parking space candidate detection unit 2232 and the edge calculation result by the edge calculation unit 2233, thereby determining the final position of the detected parking space PS.

[0129] By the way, it is possible to use the detection result of the parking space PS to suppress sudden acceleration when the accelerator is erroneously operated, to notify the occupant, and to perform parking assistance or automatic parking. In recent years, in order to provide such functions with high convenience, it is required to improve the detection accuracy of the parking space PS. In this regard, it is effective to be able to recognize not only the position information of each parking space PS but also various attribute information when detecting the parking space PS from an image using deep learning.

[0130] Therefore, in this embodiment, the parking space PS is objectified and learned by a plurality of key points, i.e., four parking space corner points BP and the parking space center point BPC inside the parking space PS, which define the shape of the parking space PS. Specifically, deep learning is performed using a parking space annotation code Ea, which is an encoding having annotation information on the existence, lateral position, longitudinal position, and connection vector of a plurality of parking space corner points BP and parking space center points BPC in each grid GR. This makes it possible to determine which parking space PS the detected parking space corner point BP corresponds to. In addition, this objectification makes it possible to give attribute information to each parking space PS.

[0131] Then, in this embodiment, attribute information is given to the objectified parking space PS, and the parking space PS is learned.Therefore, for example, by inferring whether or not there is a frame line element in the parking space line B5, straight line calculation is possible by post-processing, and the line and vertex position of the corresponding part can be calculated with high accuracy.Therefore, according to this embodiment, learning accuracy is improved, and thus the recognition accuracy of the parking space PS is improved.Specifically, for example, in parking assistance or automatic parking, the parking position can be realized with high accuracy.

[0132] Figure 55 shows an example of the position correction process of the parking frame corner point BP as post-processing. Each process in the flowchart shown in Figure 55 is realized by the processor 21 shown in Figure 2 reading and executing a program from the memory 22. In addition, in the flowchart shown in the figure, "S" is an abbreviation of "step". Below, referring to Figure 55, a specific example of such post-processing will be described.

[0133] First, in step 101, the processor 21 initializes the parking space counter N to 1. The value of the parking space counter N is a positive integer from 1 to Nmax. Nmax is the total number of parking space PS that are inferred, that is, detected. Next, in step 102, the processor 21 selects the Nth parking space PS, which is one of the detected parking space PS. Then, in step 103, the processor 21 initializes the frame line element counter M to 1. The value of the frame line element counter M is a positive integer from 1 to 4, where "1" corresponds to the front frame line B501, "2" corresponds to the rear frame line B502, "3" corresponds to the left frame line B504, and "4" corresponds to the right frame line B505. After the process of step 103, the processor 21 advances the process to step 104.

[0134] In step 104, the processor 21 acquires detection information on the frame line element corresponding to the frame line element counter M. Specifically, immediately after the process of step 103 is executed and the frame line element counter M is 1, the processor 21 acquires detection information on the front frame line B501 of the selected parking space PS. Then, in step 105, the processor 21 determines whether or not such a frame line element exists. If a corresponding frame line element exists (i.e., step 105=YES), the processor 21 advances the process to steps 106 to 109.

[0135] In step 106, the processor 21 calculates a circumscribing rectangle Rc shown in FIG. 56. The circumscribing rectangle Rc is a rectangle with a parking space corner point BP constituting a corresponding frame line element as a diagonal point. In step 107, the processor 21 executes an edge calculation process. The edge calculation process can be performed by using a known filter such as a Sobel filter in the circumscribing rectangle Rc. In step 108, the processor 21 applies a known process such as a Hough transform to the edge calculation result to obtain an edge line LL shown in FIG. 57. In step 109, the processor 21 determines whether the value of the frame line element counter M is 4 or not.

[0136] When the value of the frame line element counter M is 3 or less (i.e., step 109=NO), there is a frame line element for which the edge calculation process in step 107 and the straight line calculation process in step 108 have not been processed. In this case, the processor 21 executes the process of step 110 and then returns the process to step 104. In step 110, the processor 21 increments the value of the frame line element counter M by 1. In addition, even if the determination result in step 105 is "NO", the processor 21 executes the process of step 110 and then returns the process to step 104. This enables the edge calculation process and the straight line calculation process for the unprocessed frame line element. On the other hand, when the value of the frame line element counter M is 4 (i.e., step 109=YES), the edge calculation process and the straight line calculation process, or the determination of whether or not to execute these processes, have been performed for all four frame line elements in the selected parking stall PS. In this case, the processor 21 advances the process to step 111 and thereafter.

[0137] In step 111, the processor 21 calculates the intersection of the calculated edge line LL. It is possible that the edge line LL has not been calculated because it is determined that there is no corresponding frame line element for some of the four frame line elements (i.e., step 105=NO). In this case, in step 111, the processor 21 substitutes a straight line using the parking space corner point BP constituting the other edge line LL that has already been calculated, and calculates the intersection of the substitute straight line and the other edge line LL that has already been calculated. Next, in step 112, the processor 21 corrects the position of the parking space corner point BP using the intersection calculated in step 111. Then, in step 113, the processor 21 determines whether the value of the parking space counter N is Nmax. If the value of the parking space counter N is less than Nmax (i.e., step 113=NO), it means that an unselected parking space PS exists. Therefore, in this case, the processor 21 executes the process of step 114, and then returns the process to step 102. In step 114, the processor 21 increments the value of the parking space counter N by 1. On the other hand, if the value of the parking frame counter N is Nmax (that is, step 113=YES), the corner point position correction is completed for all the detected parking frame PS. Therefore, in this case, the processor 21 ends the post-processing.

[0138] Second Embodiment A second embodiment of the present disclosure will be described below. In the following description of the second embodiment, differences from the first embodiment will be mainly described. The same applies to the third and subsequent embodiments described below. That is, in the description of the subsequent embodiments, differences from the other preceding embodiments will be mainly described. Therefore, the subsequent embodiments will have the same configurations, functions, and effects as the other preceding embodiments, unless otherwise noted or unless a technical contradiction occurs.

[0139] In the actual parking lot PL, as shown in Fig. 58, the parking frame line B5 and its corner, which is the parking frame corner point BP, are often covered by the parked vehicle B7. If the image taken in such a situation is learned as it is, even if the parking frame line B5 is slightly reflected in the image, it will be learned as the parking frame PS. Then, for example, as shown in Fig. 18, the parking frame PS may be erroneously detected based on the crosswalk B41, the stop line B42, the broken road division line B43, etc. on the road Rd.

[0140] Therefore, in this embodiment, the occlusion state of the parking space PS by another target B, that is, typically the parked vehicle B7, is included in the annotation information for learning the parking space PS. For example, at the time of annotation, a value indicating the size of the unoccluded portion of the parking space PS or the parking frame line B5 is given to each parking space PS, and it is possible to determine whether or not to exclude the corresponding parking space PS from the learning target according to the value. This makes it possible to suppress the above-mentioned erroneous detection. Alternatively, for example, an annotation regarding the occlusion state of the parking space corner point BP is given, and it is possible to determine whether or not to exclude the corresponding parking space PS from the learning target according to the annotation. Specifically, for example, the number of parking space corner points BP occluded by the parked vehicle B7 is calculated, and when the number is equal to or greater than the learning exclusion threshold (for example, 3), it is possible to exclude the corresponding parking space PS from the learning target. This makes it possible to suppress the above-mentioned erroneous detection well while suppressing the increase in annotation load as much as possible.

[0141] FIG. 59 shows an outline of the parking space learning operation according to the present embodiment. The process shown in the flowchart of FIG. 59 is executed for each of the annotated parking space PS that is a learning target candidate. Referring to FIG. 59 below, first, in step 201, the processor 21 judges whether the number of hidden points is 0 or not. The "hidden points" are the number of the parking space corner points BP that are hidden by the parked vehicle B7 and cannot be seen among the four parking space corner points BP corresponding to the corresponding parking space PS. When the number of hidden points is 0 (i.e., step 201=YES), the parking space PS is in a state where it is not hidden by the parked vehicle B7, that is, in a state where it is vacant. Therefore, in this case, the processor 21 includes the parking space PS as a learning target. On the other hand, when the number of hidden points is not 0 (i.e., step 201=NO), the parking space PS is hidden by the parked vehicle B7, and the number of hidden points is at least 1 or more. Therefore, in this case, the processor 21 advances the process to step 202.

[0142] In step 202, the processor 21 judges whether the number of hidden points is one. If the number of hidden points is one (i.e., step 202=YES), the processor 21 includes the parking space PS in the learning target. On the other hand, if the number of hidden points is not one (i.e., step 202=NO), the processor 21 advances the process to step 203. In step 203, the processor 21 judges whether the number of hidden points is two. If the number of hidden points is two (i.e., step 203=YES), the processor 21 includes the parking space PS in the learning target. On the other hand, if the number of hidden points is not two (i.e., step 203=NO), the number of hidden points is three or more. Therefore, in this case, the processor 21 excludes the parking space PS from the learning target. That is, the processor 21 does not use the parking space PS in the calculation of the loss function.

[0143] Third embodiment Hereinafter, the third embodiment of the present disclosure will be described with reference to Figs. 60 to 62. In Fig. 60, the camera position Pc indicates the viewpoint position of the camera 34. In the scene of parking assistance or automatic parking, information on the vacancy state of the parking space PS is required. That is, the target or target parking space PS where the vehicle is going to park needs to be in an "vacant" state that is not occupied by the parked vehicle B7. For this reason, it is preferable to learn the parking space PS by adding attribute information on the state of whether it is vacant or not.

[0144] However, as shown in FIG. 60, a situation may occur in which it is difficult to determine the vacancy of a parking space PS that is originally "vacant" because it is blocked by a parked vehicle B7 that occupies an adjacent "unvacant" parking space PS. If the state of the parking space PS is learned directly from an image corresponding to such a situation, there is a possibility that erroneous determination of the vacancy state occurs during parking assistance or automatic parking. Therefore, in this embodiment, the learning mode of the vacancy state of the parking space PS is determined by considering the positional relationship between the camera position Pc of the vehicle and the parking space PS or the parked vehicle B7.

[0145] FIG. 61 shows a flowchart corresponding to this embodiment. The process shown in the flowchart of FIG. 61 is executed for each of the annotated parking space PS that is a learning target candidate. Note that, in this embodiment, annotation information includes information on whether each parking space PS is vacant or not, in addition to the position information of the parking space corner point BP in the first embodiment and the like. Each process shown in FIG. 61 can be performed by the annotation information generating unit 2201 and / or the learning unit 2202 shown in FIG. 30.

[0146] Referring to FIG. 61 below, first, in step 301, the processor 21 initializes the parking space counter N to 1. The value of the parking space counter N is a positive integer from 1 to Nmax. Nmax is the total number of annotated parking spaces PS that are candidates for learning. Next, in step 302, the processor 21 selects the Nth parking space PS, which is one of the candidates for learning. Here, since the process proceeds to step 302 immediately after the execution of step 301, the first parking space PS is selected in step 302. Next, in step 303, the processor 21 calculates the "hidden amount", which is the amount of the parking space PS that is blocked by another target, that is, typically the parked vehicle B7. A method for calculating the hidden amount will be described later. Then, in step 304, the processor 21 determines whether the hidden amount is less than a predetermined threshold value.

[0147] A method for calculating the amount of concealment will be described with reference to FIG. 62. FIG. 62 shows, as an example, a situation in which the vehicle is traveling in a parking lot passage PSr, and there are parking rows PSs for parallel parking on the left and right of the vehicle. The parking rows PSs are a plurality of parking frames PS arranged in a certain direction. The parking lot passage PSr is a passage in which the vehicle can travel in the parking lot PL, and is an area facing the parking rows PSs. That is, the parking lot passage PSr can be said to be an area excluding the parking rows PSs from the entire area of ​​the parking lot PL. In addition, in the specific example of FIG. 62, in the parking row PSs on the right side based on the traveling direction of the vehicle, the first to third cars from the front are in an "unvacant" state occupied by the parked vehicle B7, and the fourth car is in an "vacant" state. The fourth "vacant" parking frame PS is hereinafter referred to as the target parking frame PSt. Note that FIG. 62 is merely an example for simply explaining a method for calculating the amount of concealment, and the present disclosure is not limited to such a situation. That is, even in situations other than those shown in FIG. 62, it is possible to calculate the occlusion amount in a similar manner by referring to the following explanation.

[0148] The "hidden amount" of the target parking space PSt can be calculated based on the viewing angle of the parking space PS seen from the vehicle. That is, first, as shown in FIG. 62, when there is no occlusion by the parked vehicle B7 and the whole of the target parking space PSt is visible from the vehicle, the viewing angle of the target parking space PSt centered on the camera position Pc of the vehicle is set as the existence range angle θe. In the example of FIG. 62, the existence range angle θe is the angle between the imaginary line starting from the camera position Pc and passing through the right front corner point BP2, and the imaginary line starting from the camera position Pc and passing through the left rear corner point BP3. Next, as shown in FIG. 62, when a part of the target parking space PSt is actually occluded, the range that is not occluded and can be seen from the camera position Pc is set as the visible range angle θc, and the occluded range is set as the occluded range angle θh. In the example of FIG. 62, the visible range angle θc is the angle between a virtual line starting from the camera position Pc and passing through the right front corner point BP2, and a virtual line starting from the camera position Pc and passing through the right front end of the third parked vehicle B7 from the front. The hidden range angle θh is obtained by subtracting the visible range angle θc from the existence range angle θe. The hidden amount can be calculated as the value of θh / θe or as a percentage. The threshold value can be set to any value based on experience or experiment, such as 0.8 or 80%, but can also be modified or corrected depending on the learning situation.

[0149] If the amount of hiding is equal to or greater than the threshold (i.e., step 304=NO), the processor 21 executes the process of step 305 and then proceeds to step 306. In step 305, the processor 21 sets the vacant state of the corresponding parking space PS to "unknown" in learning. In this case, the parking space PS may be learned as having an "unknown class" for its vacant state. Alternatively, in this case, the parking space PS may be excluded from the learning target. On the other hand, if the amount of hiding is less than the threshold (i.e., step 304=YES), the processor 21 skips the process of step 305 and proceeds to step 306. In this case, the information regarding the vacant state included in the original annotation information is maintained as it is.

[0150] In step 306, the processor 21 determines whether the value of the parking space counter N is Nmax. If the value of the parking space counter N is less than Nmax (i.e., step 306 = NO), there will be an unselected parking space PS. Therefore, in this case, after the processor 21 executes the process of step 307, the process returns to step 302. In step 307, the processor 21 increments the value of the parking space counter N by 1. Then, in step 302, the K-th parking space PS is selected. 1 < K ≤ Nmax. On the other hand, if the value of the parking space counter N is Nmax (i.e., step 306 = YES), the processor 21 ends this process.

[0151] (Fourth Embodiment) Hereinafter, a fourth embodiment of the present disclosure will be described. FIG. 63 shows a situation where the parallel parking row in which the host vehicle is parking and the parallel parking row behind it are connected and exist. That is, the upper parallel parking row in the figure and the lower parallel parking row in the figure have a so-called "back-to-back" positional relationship, and their rear frame lines B502 are shared. In the figure, the upper side of the upper parallel parking row is the front, and the lower side of the lower parallel parking row in the figure is the front. Here, the parking space PS in which the host vehicle is parking is set as id:0, and each parking space PS in the parallel parking row behind it is set as id:1 to 4 as shown in the figure. The parking space PS of id:0 is located between the parking space PS of id:2 and the parking space PS of id:3 in the left-right direction, that is, the vehicle width direction. For the sake of convenience of explanation, among the parking space corner points BP belonging to each parking space PS, only the left rear corner point BP3 is shown as a black circle, and the other parking space corner points BP and the parking space center point BPC are shown as white circles. At this time, in the attention grid GRt which is the grid GR surrounded by a white broken line in the figure, the left rear corner point BP3 in the parking space PS of id:0 and the left rear corner point BP3 in the parking space PS of id:2 coexist.

[0152] Here, the first encoding Ec1 in the first embodiment shown in FIG. 42 has three types of encodings, that is, the presence, the horizontal position, and the vertical position, in the grid GR, for a total of five points, including four parking space corner points BP and one parking space center point BPC, as shown in FIG. 38 to FIG. 42. That is, the first encoding Ec1 shown in FIG. 42 and the like is generated on the premise that the same type of parking space corner point BP (for example, the left rear corner point BP3) is one in one grid GR. Therefore, for the example of FIG. 63, according to the above first embodiment, it is not possible to give information on both the left rear corner point BP3 in the parking space PS of id:0 and the left rear corner point BP3 in the parking space PS of id:2 to the attention grid GRt. For this reason, in this case, for example, the midpoint of both is encoded as a common left rear corner point BP3. Then, the encoded values ​​of the horizontal position and the vertical position may deviate from the original values, and the accuracy of the inference may decrease.

[0153] In this regard, the situation where a plurality of the same type of parking space corner points BP belonging to different parking space PS are located in the same grid GR as described above occurs when parking space PS with different orientations are adjacent to each other. Therefore, in this embodiment, first, as shown in FIG. 64A, the orientation of the parking space PS is defined using four types of anchor indexes, anchor0 to anchor3. The arrow in the figure indicates the parking space direction BD shown in FIG. 31, that is, the standard orientation of the vehicle when the vehicle is parked in the parking space PS corresponding to anchor0 to anchor3. In the following, in order to avoid complication of the illustration, the parking space direction BD shown in FIG. 31, which is the orientation of the parking space PS, is indicated by an arrow in the parking space PS. As shown in FIG. 64A, the parking space PS of anchor0 is "upward" in the image, the parking space PS of anchor1 is "leftward" in the image, the parking space PS of anchor2 is "rightward" in the image, and the parking space PS of anchor3 is "downward" in the image. Then, in this embodiment, learn parking space PS for each direction of parking space PS.That is, in this embodiment, learn parking space PS based on the information that encodes parking space corner point BP and parking space center point BPC for each direction of parking space PS.

[0154] In addition, in an actual image, as shown in FIG. 64B, the parking space PS may be inclined with respect to the vertical and horizontal directions of the image. Therefore, when determining the orientation of the parking space PS, a circumscribing rectangle Rc and a reference point Pw are used. Specifically, the circumscribing rectangle Rc is a virtual rectangle that is circumscribing four parking space corner points BP and each side is parallel to the vertical and horizontal directions of the image. The reference point Pw is a point on the circumscribing rectangle Rc, and is the upper left corner point in this embodiment. Then, the orientation of the parking space PS is determined according to which of the four parking space corner points BP is closest to the reference point Pw. That is, for example, it is possible to define as follows. When the left front corner point BP1 is closest to the reference point Pw, the orientation of the parking space PS is anchor0. When the right front corner point BP2 is closest to the reference point Pw, the orientation of the parking space PS is anchor1. When the right rear corner point BP4 is closest to the reference point Pw, the orientation of the parking space PS is anchor2. When the left rear corner point BP3 is closest to the reference point Pw, the orientation of the parking space PS is set as anchor3.

[0155] With reference to Fig. 65 to Fig. 68, an example of a parking space PS encoding method in this embodiment will be described. Referring to the flowchart shown in Fig. 65, first, in step 401, the processor 21 initializes a parking space counter N to 1. The value of the parking space counter N is a positive integer from 1 to Nmax. Nmax is the total number of annotated parking space PS that are candidates for learning. Next, in step 402, the processor 21 selects the Nth parking space PS. Here, since the process proceeds to step 402 immediately after the execution of step 401, the first parking space PS is selected in step 402.

[0156] After selecting the parking space PS in step 402, the processor 21 executes the process of step 403 to step 408 in order. In step 403, the processor 21 calculates a circumscribing rectangle Rc. In step 404, the processor 21 calculates a reference point Pw based on the circumscribing rectangle Rc calculated in step 403. In step 405, the processor 21 calculates the distance between the reference point Pw calculated in step 404 and each of the four parking space corner points BP.

[0157] In step 406, the processor 21 performs matching between the parking space PS and the anchor index indicating the orientation of the parking space PS based on the distance calculated in step 406. That is, the anchor index in the parking space PS is determined. The details of the matching of the anchor index in step 406 will be described with reference to Figs. 66 and 67. In Figs. 66 and 67, the x-axis direction indicates the horizontal direction of the image, that is, the calculation direction of the horizontal position of the point in the grid GR in Figs. 35 and 36. In addition, the y-axis direction indicates the vertical direction of the image, that is, the calculation direction of the vertical position of the point in the grid GR in Figs. 37 and 38. In Figs. 66 and 67, the circumscribed rectangle Rc is a rectangle having sides parallel to the x-axis and y-axis.

[0158] In the example of FIG. 66, the left rear corner point BP3 of the parking space PS at the upper right and the left rear corner point BP3 of the parking space PS at the lower left are located within the target grid GRt. Here, in the parking space PS at the upper right, the parking space corner point BP that is closest to the reference point Pw is the left front corner point BP1. Therefore, the anchor index of this parking space PS is anchor0. On the other hand, in the parking space PS at the lower left, the parking space corner point BP that is closest to the reference point Pw is the right rear corner point BP4. Therefore, the anchor index of this parking space PS is anchor2.

[0159] In the example of FIG. 67, the left rear corner point BP3 of the lower right parking space PS and the left rear corner point BP3 of the upper left parking space PS are located within the target grid GRt. Here, in the lower right parking space PS, the parking space corner point BP closest to the reference point Pw is the left front corner point BP1. Therefore, the anchor index of this parking space PS is anchor0. On the other hand, in the upper left parking space PS, the parking space corner point BP closest to the reference point Pw is the right front corner point BP2. Therefore, the anchor index of this parking space PS is anchor1.

[0160] Referring again to FIG. 65, in step 407, the processor 21 encodes the parking space corner point BP and the parking space center point BPC for each orientation of the parking space PS, that is, for each anchor index, based on the matching result of the anchor index in step 406. Specifically, the processor 21 generates a first encoding Ec1 corresponding to each of anchor0 to anchor3, as shown in FIG. 68. In the figure, Ec1_anchor0 is the first encoding Ec1 corresponding to anchor0. Ec1_anchor1 is the first encoding Ec1 corresponding to anchor1. Ec1_anchor2 is the first encoding Ec1 corresponding to anchor2. Ec1_anchor3 is the first encoding Ec1 corresponding to anchor3. That is, the parking space annotation code Ea shown in FIG. 68 is generated by generating the first encoding Ec1 in the parking space annotation code Ea shown in FIG. 54 for each of anchor0 to anchor3.

[0161] Referring back to FIG. 65, at step 408, the processor 21 determines whether the value of the parking frame counter N is Nmax. If the value of the parking frame counter N is less than Nmax (i.e., step 408 = NO), there will be an unselected parking frame PS. Therefore, in this case, after the processor 21 executes the process of step 409, the process returns to step 402. At step 409, the processor 21 increments the value of the parking frame counter N by 1. Then, at step 402, the K-th parking frame PS is selected. Here, 1 < K ≤ Nmax. On the other hand, if the value of the parking frame counter N is Nmax (i.e., step 408 = YES), the processor 21 ends this process.

[0162] As described above, in this embodiment, the parking frame PS is learned based on Ec1_anchor0 to Ec1_anchor3 shown in FIG. 68, which is information encoding the parking frame corner point BP and the parking frame center point BPC for each orientation of the parking frame PS. Thereby, even if there are multiple parking frame corner points BP of the same type, i.e., the same class, within the same grid GR, they can be independently inferred by learning as separate anchor indices. Therefore, according to this embodiment, it is possible to detect key points at the correct positions even in a scene where the detection targets are dense.

[0163] (Fifth Embodiment) Hereinafter, the fifth embodiment of the present disclosure will be described. As shown in FIG. 69, a part of the parking space corner point BP, which is a key point constituting the parking space PS, may "protrude" outside the target image GS. Specifically, in the parking space PS shown by a dashed rectangle in FIG. 69, only one of the four parking space corner points BP, the right rear corner point BP4, is outside the image. In this case, if the right rear corner point BP4 outside the image is not encoded and is excluded from the learning target, there is a concern that the detection result will become unstable. Therefore, as shown by the arrow in FIG. 69, it is preferable to offset the right rear corner point BP4 outside the image into the image, encode it, and learn it. In this case, it is typically convenient to set the offset direction to the long side direction or the short side direction of the rectangular parking space PS.

[0164] 70 to 73 show an example of offset processing when one of the four parking space corner points BP is outside the target image GS, that is, outside the learning input area AN. In the figures, the parking space corner points BP outside the image are shown with black circles. It is assumed that such parking space corner points BP outside the image are virtually annotated at the corresponding positions outside the image, that is, the positions of the black circles in the figures. In FIG. 70 and FIG. 71, the manner of "protruding" of the parking space PS outside the image is the same, but the offset direction is different. The same is true for FIG. 72 and FIG. 73. In the examples of FIG. 70 and FIG. 71, the "protruding" rate of the short side is 30%, and the "protruding" rate of the long side is 40%. In other words, the in-image ratio of the short side is 70%, and the in-image ratio of the long side is 60%. In FIG. 70, the offset direction is the long side direction. On the other hand, in FIG. 71, the offset direction is the short side direction. On the other hand, in the examples of Figures 72 and 73, the "protrusion" rate of the short side is 80%, and the "protrusion" rate of the long side is 40%. In other words, the ratio of the short side to the image is 20%, and the ratio of the long side to the image is 60%. In Figure 72, the offset direction is the long side direction. On the other hand, in Figure 73, the offset direction is the short side direction.

[0165] 70 and 71, FIG. 71 shows that the shape of the encoded parking space PSe, which is a quadrangle formed by the parking space corner point BP after offset, retains the original shape of the actual parking space PS. On the other hand, FIG. 72 shows that the shape of the encoded parking space PSe formed by the parking space corner point BP after offset retains the original shape of the actual parking space PS. That is, the shape after offset retains the original shape when offsetting along the side with a smaller "protrusion" rate, that is, the side with a larger in-image ratio. In this way, the offset direction is set to the direction that retains the original shape, that is, the direction that maintains the rectangular shape of the parking space PS, and it is possible to eliminate the concern that the detection result becomes unstable while suppressing the contradictory learning with the parking space PS without "protrusion" of the parking space corner point BP.

[0166] 74 and 75 show an example of offset processing when two of four parking space corner points BP are outside the image.In this way, for parking space PS that has two or less parking space corner points BP outside the image, offset the parking space corner points BP into the imaging range and learn, so that it is possible to eliminate the concern that detection results become unstable.However, when three of four parking space corner points BP are outside the image, it is difficult to make the shape after offset a rectangle, so it is not encoded.

[0167] Sixth Embodiment Hereinafter, the sixth embodiment of the present disclosure will be described. As shown in FIG. 76A, for example, in a road Rd with three or more lanes on each side, a plurality of dashed lane boundary lines B433 may be provided in the road width direction. In this case, the area surrounded by the dashed line in the figure, which is sandwiched between the lane boundary lines B433 on both sides of the driving lane LNd at the middle position in the road width direction, is very similar in appearance to the parking stall PS formed only by a pair of left and right side frame lines B503 as shown in FIG. 76B. Here, the "driving lane LNd at the middle position in the road width direction" is the driving lane LNd located between the outermost driving lane LNd in the road width direction adjacent to the outer roadway line B431 and the overtaking lane LNp adjacent to the center line B432.

[0168] Therefore, if the area surrounded by the dashed line in FIG. 76A is erroneously recognized as a parking space PS as shown in FIG. 76B while the vehicle is traveling on the road Rd, the vehicle may be erroneously determined as traveling in a parking lot PL. If such an erroneous determination occurs, there is a concern that the convenience of the vehicle occupants may be reduced, such as an unintentional warning associated with the processing at the time of erroneous operation of the accelerator. In this regard, even if the above-mentioned area on the road Rd is learned as "not a parking space PS," it is difficult to distinguish it in the image recognition processing during actual traveling.

[0169] Therefore, in this embodiment, a "determination of parking lot resemblance by DNN" is introduced to determine whether or not the image is in the parking lot PL from the entire image. DNN is an abbreviation of Deep Neural Network, which is a type of the above-mentioned "deep learning". In this embodiment, the parking space PS is detected according to the result of the determination of parking lot resemblance. Specifically, for example, learning is performed by adding an annotation regarding whether or not the image is in the parking lot PL to the image. In the scene of detecting the parking space PS using the learning result, if it is determined that the vehicle is not in the parking lot PL, all the detection results of the parking space PS are rejected, that is, discarded. This can effectively suppress the occurrence of problems due to the above-mentioned erroneous recognition or erroneous detection. In addition, the number of parked vehicles B7 and parking space PS may be saved as annotations, and used to determine whether or not the vehicle is in the parking lot PL or is unknown during learning.

[0170] FIG. 77A shows an annotation generation process related to parking lot-likeness, which is executed by the annotation information generation unit 2201 shown in FIG. 30, that is, the parking space annotation generation unit 2213. Referring to FIG. 77A, first, in step 501, the processor 21 judges whether or not the vehicle is in the parking lot PL, that is, whether or not the vehicle is in the parking lot at the time of acquiring the corresponding learning image information, based on the annotation information inputted in the annotation input unit 2212 shown in FIG. 30. If the vehicle is not in the parking lot PL (i.e., step 501=NO), the processor 21 judges that the vehicle is not in the parking lot. That is, the processor 21 judges that the vehicle is not in the parking lot PL at the time of acquiring the corresponding learning image information. Note that the "non-parking lot determination" may also be referred to as the "outside parking lot determination". On the other hand, if the vehicle is in the parking lot PL (i.e., step 501=YES), the processor 21 advances the process to step 502.

[0171] Even if the vehicle is in the parking lot PL, it may be difficult to correctly determine whether the vehicle is in the parking lot PL from the image, for example, when the parking space PS is located near a wall and only the wall is reflected around the parking space. In this case, in step 502, the processor 21 determines whether there is one or more parked vehicles B7 or parking spaces PS in the image based on the annotation information. If the determination result in step 502 is "YES", the possibility or reliability that the vehicle is in the parking lot PL at the time of acquiring the corresponding learning image information is high. In this case, the processor 21 determines that the vehicle is in the parking lot. That is, the processor 21 determines that the vehicle is in the parking lot PL at the time of acquiring the corresponding learning image information. In contrast, if the determination result in step 502 is "NO", the possibility or reliability is low. In this case, the processor 21 determines that the vehicle is in the parking lot as "unknown". In other words, in this case, the processor 21 changes the annotation content from "inside the parking lot" to "unknown". In this manner, in this embodiment, the annotation content is changed depending on the number of parked vehicles B7 and parking stalls PS included in the image.

[0172] FIG. 77B shows an overview of the learning process using annotation information related to the parking lot-likeness, which is executed by the learning unit 2202 shown in FIG. 30, that is, the parking space learning unit 2222. That is, FIG. 77B shows a part of the entire learning process based on the annotation information generated by the process shown in FIG. 77A. Referring to FIG. 77B, first, in step 601, the processor 21 judges whether the annotation content is "inside a parking lot". If the judgment result in step 601 is "YES", the processor 21 executes the learning process of the parking space PS as being "inside a parking lot". On the other hand, if the judgment result in step 601 is "NO", the processor 21 advances the process to step 602.

[0173] In step 602, the processor 21 judges whether the annotation content is "unknown". If the judgment result in step 602 is "YES", the processor 21 executes learning processing of the parking space PS as "unknown". On the other hand, if the judgment result in step 602 is "NO", the processor 21 executes learning processing of the parking space PS as "not in a parking lot".

[0174] FIG. 78 and FIG. 79 show the fourth encoding Ec4 corresponding to the parking lot-likeness. Here, FIG. 78 shows the encoding result when it is determined to be "not in a parking lot" or "unknown". Meanwhile, FIG. 79 shows the encoding result when it is determined to be "in a parking lot". FIG. 80 shows the parking space annotation code Ea including the fourth encoding Ec4 shown in FIG. 78 or FIG. 79. In addition, for the sake of simplicity, the encoding that may be included in the parking space annotation code Ea other than the first encoding Ec1 and the fourth encoding Ec4, such as the second encoding Ec2, is omitted from the illustration. In addition, the first encoding Ec1 shown in FIG. 80 may be provided in multiple numbers for each direction of the parking space PS, as shown in FIG. 68.

[0175] FIG. 81 shows an example in which the parking space likelihood determination result is used in the detection of the parking space PS by the parking space detection unit 2203 shown in FIG. 30. Referring to FIG. 81, in step 701, the processor 21 determines whether the host vehicle is in the parking space PL. If the host vehicle is in the parking space PL (i.e., step 701=YES), the processor 21 adopts all the detected, i.e., recognized, parking space candidates as parking space PS. On the other hand, if the host vehicle is not in the parking space PL (i.e., step 701=NO), the processor 21 rejects, i.e., discards, all the information of all the detected parking space candidates.

[0176] Seventh embodiment Hereinafter, a seventh embodiment of the present disclosure will be described. In this embodiment, the attribute information further includes information on the wheel stop B6 on the inside of the parking space PS. FIG. 82 shows an example of the fifth encoding Ec5 which is an encoding result of such information. Specifically, FIG. 82 shows an encoding example in the case where the wheel stop B6 exists in each of the multiple parking space PS in the target image GS shown in FIG. 32. As is clear from FIG. 33, FIG. 34, FIG. 38, and FIG. 82, in the fifth encoding Ec5, the encoding value Ecv is set to "1" at the position of the grid GR corresponding to the parking space center point BPC in the parking space PS where the wheel stop B6 exists. FIG. 83 shows the parking space annotation code Ea including the fifth encoding Ec5 shown in FIG. 82. For the sake of simplicity, the encodings that may be included in the parking space annotation code Ea other than the first encoding Ec1 and the fifth encoding Ec5, for example, the second encoding Ec2 to the fourth encoding Ec4, are omitted from the illustration. In addition, the first encode Ec1 shown in FIG. 83 may be provided in a plurality of ways for each direction of the parking space PS as shown in FIG.

[0177] Eighth embodiment Hereinafter, an eighth embodiment of the present disclosure will be described. In this embodiment, the attribute information further includes information of additional attributes different from the parking frame line B5 and the wheel stopper B6. For example, the type of parallel parking / diagonal parallel parking / parallel parking, the type of paint inside the frame such as a wheelchair mark or an EV mark, or the presence or absence of such paint can be used as the additional attribute. EV is an abbreviation for Electric Vehicle. FIG. 84 shows an example of a sixth encoding Ec6 which is an encoding result of such information. In the sixth encoding Ec6, similar to the fifth encoding Ec5 shown in FIG. 82, the encoding value Ecv is set to "1" at the position of the grid GR corresponding to the parking frame center point BPC in the parking frame PS where the additional attribute exists.

[0178] FIG. 85 shows the parking space annotation code Ea including the sixth encoding Ec6 shown in FIG. 84. For the sake of simplicity, the encodings that may be included in the parking space annotation code Ea other than the first encoding Ec1 and the sixth encoding Ec6, such as the second encoding Ec2 to the fifth encoding Ec5, are omitted. Also, the first encoding Ec1 shown in FIG. 85 may be provided in multiple numbers for each direction of the parking space PS, as shown in FIG. 68.

[0179] Ninth embodiment Hereinafter, a ninth embodiment of the present disclosure will be described. In this embodiment, a parking lot determination, that is, a parking lot inside / outside determination, is performed based on the result of recognition or detection of a parking space PS, as to whether the vehicle is in a parking lot PL or not. Such a determination result can be used, for example, in a process when an accelerator is erroneously operated, or in the calculation of a driving route to a target parking space PS in parking assistance or automatic parking. Alternatively, such a determination is performed, for example, when learning image information is acquired using the vehicle in the first embodiment. That is, the parking lot determination result can be used to generate annotation information.

[0180] An overview of parking lot determination will be described with reference to Fig. 86 and Fig. 87. First, in step 801, the processor 21 generates a parking row PSs as shown in Fig. 87. Next, in step 802, the processor 21 generates a group for parking lot determination. The group for parking lot determination and its generation will be described later. Next, in step 803, the processor 21 performs parking lot determination as to whether or not the host vehicle is in a parking lot PL, based on the group for parking lot determination generated in step 802. The details of these will be described in order below.

[0181] (Parking queue generation) The parking row PSs is a plurality of parking stalls PS arranged in a certain direction under a predetermined condition. The predetermined condition will be described later. Typically, the parking row PSs is an arrangement of parking stalls PS partitioned by parking frame lines B5 as shown in FIG. 87. The shape of the parking frame lines B5 and the presence or absence of a parked vehicle B7 are not important. The parking row PSs can also be defined in a parking lot PL without parking frame lines B5 as shown in FIG. 88. That is, a rectangular area circumscribing the parked vehicle B7 is defined as a virtual parking stall PSv. The virtual parking stall PSv can be defined by a rectangle having a long side with the same dimension as the vehicle overall length dimension of the parked vehicle B7 and a short side with the same dimension as the vehicle width dimension. The parking row PSs can be formed by arranging a plurality of virtual parking stalls PSv so that the interval SP between the virtual parking stalls PSv is less than a predetermined value. The parking row PSs can also be formed by an arrangement of a parking stall PS with a parking frame line B5 and a virtual parking stall PSv without a parking frame line B5 as shown in FIG. 89. Hereinafter, unless otherwise noted or technically inconsistent, the parking space PS with the parking frame line B5 and the virtual parking space PSv without the parking frame line B5 are collectively referred to as "parking space PS". Therefore, in the drawings after FIG. 90 and the corresponding description, the parking space PS includes the case where the parking space PS is formed by the parking frame line B5 and the case where the parking space PSv is formed. In this way, in this embodiment, the parking space PS is recognized by the parking frame line B5 or the parked vehicle B7.

[0182] FIG. 90 shows parking rows PSs in a parallel parking configuration. FIG. 91 shows parking rows PSs in a diagonal parallel parking configuration. FIG. 92 shows parking rows PSs in a parallel parking configuration. In this manner, parking rows PSs are generated for each of the configurations of parallel parking / diagonal parallel parking / parallel parking. That is, in this embodiment, one parking row PSs does not include an arrangement of parking stalls PS in a plurality of configurations. Hereinafter, an example of the generation logic of parking rows PSs according to this embodiment will be described with reference to FIG. 93 to FIG. 97.

[0183] First, as shown in FIG. 93, the processor 21 selects any one of the recognized parking frames PS arranged in the x-axis direction as a reference target PSb. That is, the x-axis direction can be set by recognizing the arrangement direction of the parking frames PS. The parking frame PS that is first selected as the reference target PSb among the parking frames PS that later constitute a provisional or formal parking row PSs is called an initial selection target PS1. In FIG. 93, the reference target PSb and the initial selection target PS1 are coincident. Next, the processor 21 searches for the parking frame PS that is closest to the reference target PSb with the search direction Dt as the first direction, that is, the positive direction of the x-axis. The parking frame PS found by the search is called the nearest target PSn.

[0184] Next, as shown in FIG. 94, the processor 21 determines whether the reference object PSb and the nearest object PSn should constitute the parking row PSs based on the positional relationship between them. Specifically, when the following successive conditions are satisfied, the processor 21 determines that the reference object PSb and the nearest object PSn have a predetermined positional relationship that should constitute the parking row PSs. In FIG. 94, the position difference Δy is the offset amount between the reference object PSb and the nearest object PSn in the y-axis direction, that is, the position difference between one end in the y-axis direction. The y-axis is perpendicular to the x-axis. The reference object center line LCb is a virtual straight line that passes through the center of the reference object PSb and is parallel to the long side direction. Furthermore, the target object center line LCn is a virtual straight line that passes through the center of the nearest object PSn and is parallel to the long side direction. The interval SP in the x-axis direction is less than a specified value (e.g., 2 m). The position difference Δy in the y-axis direction is less than a specified value (e.g., 1.8 m). The angle Δθ between the reference target center line LCb and the target target center line LCn is within a specified value (e.g. 30 degrees).

[0185] When the positional relationship between the reference object PSb and the nearest object PSn satisfies the above-mentioned successive condition, the processor 21 generates a tentative parking row PSs from both of them and selects the nearest object PSn as a new reference object PSb. As a result, the reference object PSb shifts in the search direction Dt. The state after the shift is shown in FIG. 95. Then, as shown in FIG. 95, the processor 21 searches for the nearest object PSn based on the new reference object PSb, and determines whether the above-mentioned successive condition is satisfied for the found nearest object PSn. If the condition is satisfied, the processor 21 adds the found nearest object PSn to the tentative parking row PSs including the initially selected object PS1. In this way, in this embodiment, the initially selected object PS1 is used as a starting point and the parking stall PS that should constitute the parking row PSs is searched in the search direction Dt, which is the positive direction of the x-axis.

[0186] When the nearest target PSn is not found or the above-mentioned successive condition is no longer satisfied, the processor 21 temporarily ends the search in the search direction Dt, which is the positive direction of the x-axis, starting from the initially selected target PS1. Then, as shown in FIG. 96, the processor 21 returns the reference target PSb to the initially selected target PS1, and inverts the search direction Dt to the second direction, i.e., the negative direction of the x-axis, and performs a similar search. Then, the search in the search direction Dt, which is the negative direction of the x-axis, ends, and the generation of the tentative parking row PSs including the current initially selected target PS1 ends. After that, the processor 21 sets a parking stall PS that has not yet been selected as the reference target PSb as a new initially selected target PS1, and starts a search in the first direction, i.e., the positive direction of the x-axis, starting from the new initially selected target PS1, as shown in FIG. 97.

[0187] Here, if the generated temporary parking row PSs includes a predetermined number or more of parking spaces PS, the processor 21 confirms the generated temporary parking row PSs as a formal parking row PSs, and if the generated temporary parking row PSs includes less than the predetermined number of parking spaces PS, the processor 21 does not generate the parking row PSs and discards the temporary parking row PSs. That is, the above-mentioned "predetermined condition" for forming the parking row PSs is to satisfy both the above-mentioned continuity condition and the number condition. The predetermined number can be, for example, 5, but the present disclosure is not limited to this. That is, for example, the predetermined number can be Q. Q is a positive integer of 3 or more. In addition, the predetermined number can be set to a different value for each parking mode. Specifically, for example, Q=5 for parallel parking mode, Q=4 for diagonal parallel parking mode, and Q=3 for parallel parking mode. The "predetermined number" can also be referred to as a "threshold number". The value of Q can also be referred to as a parking row generation threshold.

[0188] In this way, in this embodiment, one of the parking frames PS arranged in a row is selected as a reference target PSb, and the parking frame PS is searched for in the first direction and the opposite second direction from the reference target PSb. As a result, a parking row PSs is obtained in which a predetermined number or more of parking frames PS that satisfy the above-mentioned consecutive conditions are consecutive in a predetermined direction along the first direction. Therefore, according to this embodiment, it is possible to effectively detect the parking row PSs from the vehicle traveling in the passage in the parking lot PL. In addition, it is possible to accurately determine whether the vehicle is in the parking lot PL based on the detection result of the parking row PSs.

[0189] FIG. 98A and FIG. 98B show an example of the process of step 801 shown in FIG. 86, that is, the generation process of the parking row PSs in this embodiment. Referring to FIG. 98A and FIG. 98B, first, the processor 21 executes the processes of steps 901 to 905 in order. In step 901, the processor 21 acquires the recognition result of the parking space PS. In step 902, the processor 21 selects the initial selection object PS1. In step 903, the processor 21 sets the reference object PSb to the initial selection object PS1. In step 904, the processor 21 searches for the parking space PS in the first direction from the reference object PSb, and selects the nearest object PSn, which is the search result. In step 905, the processor 21 judges whether or not the successive condition is satisfied for the reference object PSb and the nearest object PSn. In addition to the case where the successive condition is not satisfied, the judgment result in step 905 is also set to "NO" when the nearest object PSn is not found as a result of the search in step 904. That is, if the nearest target PSn is found and the successive condition is met, the determination result in step 905 becomes "YES".

[0190] If the determination result in step 905 is "YES", the processor 21 advances the process to step 906 and step 907. In step 906, the processor 21 adds the nearest target PSn selected in step 904 to the tentative parking row PSs. In step 907, the processor 21 determines whether or not an unselected parking space PS exists in the search direction Dt. If an unselected parking space PS exists in the search direction Dt (i.e., step 907=YES), the processor 21 executes the process of step 908, and then returns the process to step 904. In step 908, the processor 21 shifts the reference target PSb to the search direction Dt. That is, the processor 21 sets the nearest target PSn selected in step 904 as a new reference target PSb. On the other hand, if an unselected parking space PS does not exist in the search direction Dt (i.e., step 907=NO), the processor 21 advances the process to step 909. Also, if the determination result in step 905 is “NO”, the processor 21 advances the process to step 909 .

[0191] In step 909, the processor 21 determines whether the search direction Dt has been reversed, i.e., is currently the second direction. If the current search direction Dt is the first direction (i.e., step 909=NO), the processor 21 executes the process of step 910 and then returns the process to step 903. In step 910, the processor 21 reverses the search direction Dt from the first direction to the second direction. As a result, the processes of steps 903 to 908 are executed with the search direction Dt set as the second direction.

[0192] When the search in both the first direction and the second direction is completed, the determination result in step 909 becomes "YES". In this case, the processor 21 advances the process to step 911. In step 911, the processor 21 determines whether or not the continuous number, i.e., the number of parking spaces PS included in the temporary parking row PSs, is equal to or greater than the parking row generation threshold. If it is equal to or greater than the parking row generation threshold (i.e., step 911=YES), the processor 21 advances the process to step 912. In step 912, the processor 21 confirms the temporary parking row PSs as the official parking row PSs. On the other hand, if it is less than the parking row generation threshold (i.e., step 911=NO), the processor 21 advances the process to step 913. In step 913, the processor 21 discards the temporary parking row PSs.

[0193] After executing the process of step 912 or step 913, the processor 21 advances the process to step 914. In step 914, the processor 21 determines whether or not there is a parking space PS that has not been selected as the reference object PSb. If there is a parking space PS that has not been selected as the reference object PSb (i.e., step 914=YES), the processor 21 returns the process to step 902. As a result, the two-way search and the generation of the parking space PS are performed again using the new initially selected object PS1. On the other hand, if all the recognized parking space PS have been selected as the reference object PSb at least once (i.e., step 914=NO), the processor 21 ends the parking line generation process.

[0194] (Parking area generation) Hereinafter, the parking lot determination group generation process shown in FIG. 86 will be described with reference to FIG. 99 and the like. The "parking lot determination group" is hereinafter abbreviated as "parking area PSg". As shown in FIG. 99, the parking area PSg includes at least one parking row PSs and a parking lot passage PSr. Typically, the parking area PSg can be formed so that the parking row PSs is located on both sides of the parking lot passage PSr through which the vehicle is traveling or will travel. That is, in this embodiment, based on the acquired parking row PSs, the parking area PSg is generated, that is, acquired, including the parking row PSs and the parking lot passage PSr that is an area through which the vehicle can travel and faces the parking row PSs. For example, the number of parking stalls PS constituting the parking row PSs, the presence reliability, the accuracy reliability, and the like may be taken into consideration as the generation condition of the parking area PSg. Then, in this embodiment, based on the acquisition result of the parking area PSg, it is determined whether the vehicle is present in the parking lot PL. Specifically, for example, it is possible to determine whether the vehicle is in a parking lot or not based on the positional relationship between the vehicle and the parking area PSg, such as whether the vehicle has entered inside the parking area PSg.

[0195] In this embodiment, the parking row PSs and the parking area PSg are expanded as the vehicle travels. Figures 99 and 100 show an example of the expansion of the parking area PSg. Figure 100 shows a state in which the vehicle has further advanced toward the inside of the parking area PSg from the situation shown in Figure 99. In the first example shown in Figures 99 and 100, the newly recognized parking stalls PS arranged in the same direction as the arrangement direction of the parking stalls PS in the parking row PSs included in the acquired parking area PSg are sequentially added to the acquired parking row PSs and the parking area PSg as the vehicle travels.

[0196] Meanwhile, Fig. 101 to Fig. 103 show another example of the enlargement of the parking area PSg. Fig. 102 shows a state where the vehicle has further advanced toward the inside of the parking area PSg from the situation shown in Fig. 101. Fig. 103 shows a state where the vehicle has further advanced toward the inside of the parking area PSg from the situation shown in Fig. 102. In this example, the following procedure is executed when the parking area PSg having one parking row PSs on each side, each of which is made up of five parking spaces PS, which has already been acquired in the state shown in Fig. 101, is enlarged in the traveling direction of the vehicle.

[0197] First, as shown in FIG. 102, the acquired parking row PSs and parking area PSg are duplicated in the traveling direction of the vehicle. That is, assuming that the same parking row PSs and parking area PSg as the acquired parking row PSs and parking area PSg are consecutive in the traveling direction of the vehicle, the parking area PSg is provisionally enlarged in the traveling direction by an integer multiple (i.e., in this example, double) with the same size. If there are consecutive parking stalls PS for 10 or more vehicles on the left and right of the vehicle, the situation provisionally enlarged in this manner may be confirmed, that is, confirmed as it is. In contrast, FIG. 102 shows a case where there are only eight parking stalls PS on the left and right of the vehicle. In this case, the parking row PSs and parking area PSg newly generated by copying have two extra parking stalls PS, as shown by the two-dot chain line in the figure. Therefore, it is possible to generate the corrected parking row PSs and parking area PSg shown in FIG. 103 by subtracting the parking stalls PS for the extra vehicles by correction.

[0198] In this way, in the present embodiment, when the parking space PS or the parking row PSs adjacent to the parking row PSs included in the acquired parking area PSg in a predetermined direction is present outside the parking area PSg, the parking area PSg is expanded in a predetermined direction to include the parking space PS or the parking row PSs. This allows the parking row PSs and the parking area PSg to be expanded sequentially based on the parking space PS or the parking row PSs additionally recognized as the vehicle moves forward. Therefore, according to the present embodiment, it is possible to accurately perform the parking lot determination or the non-parking lot determination.

[0199] (Parking lot determination) The parking lot determination will be described with reference to Fig. 104 etc. In Fig. 104 etc., "ON" indicates that the parking lot determination is "ON", i.e., a determination in the parking lot, and "OFF" indicates that the parking lot determination is "OFF", i.e., a determination not in the parking lot. Typically, as shown in Fig. 104, this embodiment determines that the vehicle is in the parking lot when it enters inside the parking area PSg, and determines that the vehicle is not in the parking lot when it exits outside the parking area PSg.

[0200] Incidentally, as shown in FIG. 105, there may be a case where a plurality of parking areas PSg are arranged in the traveling direction of the vehicle, and the vehicle is scheduled to pass through both of them. In this case, when the vehicle reaches an area between two parking areas PSg, it is considered to have left the first of the two parking areas PSg. Strictly speaking, the vehicle should be temporarily switched from the parking lot determination to the non-parking lot determination. However, there is a demand for control similar to that in the parking lot even in such an area between two parking areas PSg. Therefore, in this case, it is preferable to maintain the parking lot determination even after leaving the first parking area PSg. This provides an effect similar to that of expanding or copying the parking area PSg in the traveling direction of the vehicle, as shown in FIG. 106.

[0201] FIG. 107 shows a scene in which the parking lot passage PSr is T-shaped, the vehicle is currently traveling between the parking rows PSs located on the left and right of the vehicle, and another parking row PSs arranged in a direction perpendicular to the traveling direction is present in the traveling direction. In such a scene, it is possible to maintain the parking lot in-parking lot determination in the entire T-shaped parking lot passage PSr. That is, the part of the T-shaped parking lot passage PSr that is perpendicular to or intersects with the traveling direction of the vehicle is divided into a first region R1 to a third region R3. The first region R1 is an intersection region in the T-shaped parking lot passage PSr. The second region R2 is an area turned left from the first region R1. The third region R3 is an area turned right from the first region R1. In this case, although the first region R1 is not a high reliability region sandwiched between the parking rows PSs on both sides, it is preferable to determine that the first region R1 is not a non-parking lot in ... 109 shows an example in which the vehicle is passing through a T-shaped passage PSr in a parking lot where the vehicle can go straight or turn left. In these examples, it is preferable to determine that the entire cross-shaped or T-shaped passage PSr in the parking lot is inside the parking lot.

[0202] FIG. 110 shows an example of changing the parking lot determination mode according to the approach direction of the vehicle in the same scene as FIG. 107. When the vehicle is about to enter the parking area PSg in the manner shown by the broken line in FIG. 110, there are a plurality of parking stalls PS arranged in a manner perpendicular or intersecting with the traveling direction of the vehicle in front of the vehicle. This state may occur, for example, when the vehicle is waiting for a traffic light while traveling on a road Rd with three or more lanes on each side. For this reason, in FIG. 110, the reliability that the vehicle is in the parking lot PL is higher when the vehicle enters from the upper or lower side than when the vehicle enters from the left side in the figure. Therefore, in this specific example, as shown in FIG. 110, the entrance / exit determination line PSL is set at the upper and lower boundaries of the parking area PSg. Then, in this specific example, the parking lot in-determination is turned ON only when the vehicle passes through the entrance / exit determination line PSL and enters the parking area PSg. On the other hand, in this specific example, when the vehicle enters the parking area PSg in the manner shown by the dashed line in Fig. 110, the vehicle does not pass the entrance / exit determination line PSL, so the in-parking lot determination is not turned on. In this way, this specific example determines that the vehicle is in the parking lot when the vehicle enters the parking area PSg from a specific direction, but does not determine that the vehicle is in the parking lot when the vehicle enters from a direction different from the specific direction. This makes it possible to perform a highly reliable in-parking lot determination.

[0203] FIG. 111 shows a scene in which the first parking area PSg1, the second parking area PSg2, and the third parking area PSg3 are arranged in this order in a substantially L-shape, and the vehicle is about to pass through them in this order along the L-shaped planned travel route TP. The first area boundary PSL1 is the near side boundary of the first parking area PSg1 located at the nearest side. The second area boundary PSL2 is the boundary between the first parking area PSg1 and the second parking area PSg2. The third area boundary line PSL3 is the boundary between the second parking area PSg2 and the third parking area PSg3. In this case, since the first parking area PSg1, the second parking area PSg2, and the third parking area PSg3 are continuous, the first area boundary PSL1 functions as the entrance / exit determination line PSL. Then, when passing through the second area boundary PSL2 and the third area boundary line PSL3, the parking lot in-carriage determination is maintained.

[0204] FIG. 112 shows a situation in which a vehicle is about to turn left while entering an arrangement area of ​​parking stalls PS having a T-shaped parking lot passage PSr that allows a vehicle to go straight or turn left. This area has a first parking area PSg1 located in front of the vehicle and a second parking area PSg2 located to the left of the first parking area PSg1 based on the traveling direction of the vehicle. The first parking area PSg1 has a parking lot passage PSr along the current straight-line direction of the vehicle. In the first parking area PSg1, on the right side of the parking lot passage PSr, multiple parking stalls PS are arranged in a side-by-side parking manner along the straight-line direction of the vehicle, and on the right side, there is a parking stall PS with a longitudinal direction along the straight-line direction of the vehicle. The second parking area PSg2 has a parking lot passage PSr along the traveling direction of the vehicle after turning right, and multiple parking stalls PS are arranged in a side-by-side parking manner on both sides of the parking lot passage PSr.

[0205] In this case, as shown in Fig. 112, the first parking area PSg1 and the second parking area PSg2 partially overlap. Therefore, in this case, the first area boundary PSL1 set at the front boundary of the first parking area PSg1 and the second area boundary PSL2 set at the rear boundary of the second parking area PSg2 function as the entry / exit determination line PSL. In other words, the boundaries of the first parking area PSg1 and the second parking area PSg2 in the overlapping portion between the first parking area PSg1 and the second parking area PSg2 do not function as the entry / exit determination line PSL, and the in-parking lot determination is maintained when passing through these areas.

[0206] FIG. 113 shows a situation in which the vehicle is passing through the first parking area PSg1 and the second parking area PSg2 while moving straight ahead. In this situation, the first parking area PSg1 and the second parking area PSg2 partially overlap. In this case, of the first area boundary PSL1, which is the boundary on the front side of the first parking area PSg1, and the second area boundary PSL2, which is the boundary on the back side, only the first area boundary PSL1 functions as the entrance / exit determination line PSL. Since the second area boundary PSL2 is located in the overlapping portion between the first parking area PSg1 and the second parking area PSg2, it does not function as the entrance / exit determination line PSL, and the in-parking lot determination is maintained when passing through.

[0207] (Processing when accelerator is operated incorrectly) In this embodiment, when it is determined that the vehicle is in a parking lot PL, it is possible to execute processing at the time of accelerator erroneous operation, including suppression control of the vehicle's driving force to suppress sudden acceleration due to an accelerator erroneous operation and / or notification to the occupants. Figs. 114 and 115 show an example of the processing at the time of accelerator erroneous operation according to this embodiment. The routine of the processing at the time of accelerator erroneous operation shown in Fig. 114 is executed at predetermined time intervals (e.g., 100 msec) while a predetermined execution condition is satisfied. The execution condition includes, for example, that the shift position of the vehicle is in a forward or reverse driving position.

[0208] 114, in step 1001, the processor 21 determines whether the parking lot determination is "ON". If the parking lot determination is "ON" (i.e., step 1001=YES), the processor 21 advances the process to step 1002. On the other hand, if the parking lot determination is "OFF" (i.e., step 1001=NO), the processor 21 skips the processes of steps 1002 and 1003, and temporarily ends the process at the time of erroneous accelerator operation.

[0209] In step 1002, the processor 21 determines whether or not an erroneous accelerator operation has occurred. Specifically, the processor 21 determines that an erroneous accelerator operation has occurred when all of the following conditions (1) to (5) are met. In contrast, the processor 21 determines that an erroneous accelerator operation has not occurred when at least one of the following conditions (1) to (5) is not met. (1) The vehicle speed is less than a predetermined vehicle speed threshold. (2) The accelerator operation amount is equal to or greater than a predetermined operation amount threshold. (3) The accelerator operation speed is equal to or greater than a predetermined operation speed threshold. (4) The brakes were not operated. (5) The blinker switch 382 is not operated.

[0210] If an accelerator erroneous operation has occurred (i.e., step 1002=YES), the processor 21 advances the process to step 1003. In step 1003, the processor 21 executes driving force suppression control. The details of the driving force suppression control will be described later. On the other hand, if an accelerator erroneous operation has not occurred (i.e., step 1002=NO), the processor 21 skips the process of step 1003 and temporarily terminates the accelerator erroneous operation process.

[0211] The processing content of step 1003 in FIG. 114 will be described below with reference to FIG. 115. First, in step 1101, the processor 21 executes driving force suppression control and issues a warning, i.e., a strong notification, to the occupant that an erroneous accelerator operation has occurred. Next, in step 1102, the processor 21 determines whether or not the accelerator operation amount has become less than a predetermined suppression control stop threshold after the driving force suppression control and warning are executed in step 1101. If the accelerator operation amount has become less than the predetermined suppression control stop threshold (i.e., step 1102=YES), the processor 21 advances the processing to step 1103. In step 1103, the processor 21 temporarily ends the driving force suppression control and warning in step 1101.

[0212] If the accelerator operation amount is equal to or greater than the predetermined suppression control stop even after the driving force suppression control and warning are started in step 1101 (i.e., step 1102=NO), the processor 21 advances the process to step 1104. In step 1104, the processor 21 determines whether or not a predetermined time has elapsed after the driving force suppression control and warning are started in step 1101. Before the predetermined time has elapsed (i.e., step 1104=NO), the processor 21 returns the process to step 1101. In contrast, if the predetermined time has elapsed after the driving force suppression control and warning are started in step 1101 (i.e., step 1104=YES), there is a possibility that the accelerator operation mode that was the basis of the accelerator erroneous operation determination determined in step 1002 was not an erroneous operation but was intentional. Therefore, in this case, the processor 21 advances the process to step 1105. In step 1105, the processor 21 executes the driving force suppression control and the warning in a weaker manner than the driving force suppression control and warning in step 1101 and notifies the occupant. Thereafter, the processor 21 returns the process to step 1102.

[0213] After the driving force suppression control and the warning are temporarily ended in step 1103, processor 21 advances the process to step 1106. In step 1106, processor 21 determines whether or not a predetermined time has elapsed since the driving force suppression control and the warning were temporarily ended in step 1103. If the predetermined time has not yet elapsed (i.e., step 1106=NO), processor 21 advances the process to step 1107. On the other hand, if the predetermined time has elapsed (i.e., step 1106=YES), processor 21 temporarily ends the process of step 1003 in FIG. 114.

[0214] In step 1107, the processor 21 determines whether the accelerator operation amount exceeds a predetermined suppression control resumption threshold. If the accelerator operation amount is equal to or less than the predetermined suppression control resumption threshold (i.e., step 1107=NO), the processor 21 returns the process to step 1106. In contrast, if the accelerator operation amount exceeds the predetermined suppression control resumption threshold (i.e., step 1107=YES), the processor 21 advances the process to step 1108. In step 1108, the processor 21 executes driving force suppression control and notifies the occupant in a weaker manner than the driving force suppression control and warning in step 1101. Note that the manner of the driving force suppression control and warning in step 1108 may be the same as or different from that in step 1105. Thereafter, the processor 21 returns the process to step 1102.

[0215] In this way, in this embodiment, if it is determined that the accelerator erroneous operation continues for a predetermined period of time or more even after the accelerator erroneous operation process has been executed, the accelerator erroneous operation process can be executed again. Also, in this embodiment, if it is determined that the accelerator erroneous operation is repeated after the accelerator erroneous operation process has been executed to resolve the accelerator erroneous operation, the accelerator erroneous operation process can be executed again. Therefore, this embodiment allows for fine-grained control.

[0216] (Warehouse receipt support processing) 116 to 118, the driving control of the host vehicle to a parking position during parking assistance or automatic parking will be described. That is, in this embodiment, when it is determined that the host vehicle is in a parking lot PL, an entrance route TPp is generated, which is a driving route to a parking space PS where the host vehicle can be parked. In this embodiment, the parking assistance control will be mainly described, but automatic parking control can also be executed in the same manner.

[0217] FIG. 116 shows an outline of parking assistance control. The routine shown in FIG. 116 is started by the processor 21 at a predetermined timing when it is determined that the vehicle is in the parking lot PL, that is, when the parking lot determination is "ON". When the routine is started, first, in step 1201, the processor 21 waits for an operation by the driver D. When there is an operation by the driver D, the processor 21 advances the process to step 1202. In step 1202, the processor 21 determines whether or not there is an operation to start search travel. Search travel is low-speed travel in the parking lot PL to search for a parking available frame PSc, which is an empty parking frame PS for parking the vehicle. Before the search travel start operation (i.e., step 1202=NO), the processor 21 returns the process to step 1201. On the other hand, when there is an operation to start search travel (i.e., step 1202=YES), the processor 21 advances the process to step 1203 and thereafter.

[0218] In step 1203, the processor 21 executes search travel. In search travel, as shown in Fig. 117, the processor 21 calculates a search travel path TPs, which is a travel path of the host vehicle during search travel, so as to fall within the entrance support possible range Rpp. The entrance support possible range Rpp is a strip-shaped range or area along the detected parking row PSs, the distance from which is within a predetermined range from the parking row PSs. The entrance support possible range Rpp is set within a distance range within which the host vehicle can be guided without difficulty if a parking space PSc is present in the parking row PSs.

[0219] Next, in step 1204, the processor 21 determines whether or not a vacant space, i.e., a parking space PSc, has been found by the search travel. Until a parking space PSc has been found (i.e., step 1204=NO), the processor 21 returns the process to step 1203. That is, the processor 21 continues the search travel. On the other hand, when a parking space PSc has been found (i.e., step 1204=YES), the processor 21 advances the process to steps 1205 and 1206. In step 1205, the processor 21 generates an entrance route TPp for causing the host vehicle to enter the parking space PSc, as shown in FIG. 118. In step 1206, the processor 21 executes entrance assistance based on the entrance route TPp generated in step 1205.

[0220] (Modification) The present disclosure is not limited to the above embodiment. Therefore, the above embodiment can be modified as appropriate. Representative modified examples will be described below. In the following description of the modified examples, differences from the above embodiment will be mainly described. In addition, the same reference numerals are used for parts that are the same or equivalent to each other in the above embodiment and the modified examples. Therefore, in the following description of the modified examples, the description of the above embodiment can be appropriately used for components having the same reference numerals as the above embodiment, unless there is a technical contradiction or special additional explanation.

[0221] The present disclosure is not limited to the specific device configurations shown in the above embodiments. That is, for example, the shape and structure of the vehicle body V1 in the system-equipped vehicle V are not limited to a box shape, i.e., a substantially rectangular shape in a plan view. In addition, the vehicle body panel V15 does not need to be provided on the upper side of the vehicle interior V2. Alternatively, the vehicle body panel V15 covering the upper side of the vehicle interior V2 may be detachable. There are also no particular limitations on the use of the system-equipped vehicle V, the positions of the driver's seat V23 and the steering wheel V24, the number of occupants, and the like.

[0222] The definition, level, or category of driving automation is not limited to those specified in "SAE J3016". Specifically, the "automated driving" in the above embodiment is a driving automation level corresponding to levels 3 to 5 in "SAE J3016" where the driving automation system is responsible for, i.e., executes, all dynamic driving tasks. For this reason, the definition of "automated driving" in the above embodiment naturally includes the fact that the driver D is not required to monitor the surroundings. However, the present disclosure is not limited to such an aspect. That is, for example, depending on the definition of "automated driving", not only "automated driving without the obligation to monitor the surroundings" but also "automated driving with the obligation to monitor the surroundings" can be conceived. Specifically, for example, the hands-off driving in the above embodiment can be interpreted as "automated driving with the obligation to monitor the surroundings". In this case, "automated driving" is a concept that includes so-called "partially automated driving" in which the driver D is responsible for, i.e., executes, some dynamic driving tasks such as the obligation to monitor the surroundings. "Partially automated driving" can be evaluated as being substantially synonymous with "advanced driving assistance". In addition, in the road traffic system of each country, the type and conditions of automated driving (for example, feasible roads, driving speed range, lane change availability, etc.) may be appropriately considered according to domestic circumstances, etc. For this reason, the present disclosure may be implemented with specifications that are compatible with the road traffic system of each country.

[0223] The configuration of the in-vehicle system 1 is not limited to the above example. For example, there is no particular limitation on the number and mounting positions of so-called ADAS sensors such as the sonar sensor 31, the radar sensor 32, and the camera 34. That is, for example, by reducing the number of radar sensors 32 and laser radar sensors 33, which are relatively expensive at the time of filing the present application, as much as possible, it is possible to promote the early spread of autonomous vehicles. Therefore, for example, the radar sensor 32 or the laser radar sensor 33 may be omitted.

[0224] Locator 39 is not limited to a configuration having inertial acquisition unit 392 built therein. Specifically, inertial acquisition unit 392 may receive output information from an angular velocity sensor and an acceleration sensor provided outside locator 39 as behavior sensor 36. Locator 39 may be integrated with HD map 5. Locator 39 is not limited to "POSLV" manufactured by Applanix.

[0225] The navigation device 6 may be connected to the HMI device 7 so as to be capable of communicating information via a sub-communication line different from the in-vehicle communication line 10. Alternatively, the navigation device 6 may have a display screen dedicated to displaying a navigation screen, separate from the HMI device 7. Alternatively, the navigation device 6 may be provided as a part of the HMI device 7. Specifically, for example, the navigation device 6 may be integrated with the main display device 703.

[0226] The HMI device 7 is not limited to a configuration including a meter panel 702, a main display device 703, and a head-up display 704. That is, for example, the meter panel 702 and the main display device 703 may be integrated into one display device. In this case, the meter panel 702 may be provided as a part of a display area in one display device that is a liquid crystal or organic EL display. That is, the meter panel 702 may be realized by displaying images of bezels, pointers, scales, etc. corresponding to a tachometer, a speedometer, a water temperature gauge, etc. Also, the HMI device 7 does not have to include the head-up display 704.

[0227] In the above embodiment, each ECU has a configuration as a so-called in-vehicle microcomputer including a CPU, an MPU, and the like. However, the present disclosure is not limited to such a configuration. That is, all or a part of the ECU may be configured to include a digital circuit configured to enable the above-mentioned operation, such as an ASIC or an FPGA. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array. That is, in the ECU, an in-vehicle microcomputer portion and a digital circuit portion may coexist.

[0228] The program according to the present disclosure, which enables the execution of various operations, procedures, or processes described in the above embodiment, may be downloaded or upgraded via V2X communication by the in-vehicle communication device 4. Alternatively, such a program may be downloaded or upgraded via a terminal device provided in a vehicle manufacturing plant, a maintenance plant, a dealer, or the like. Such a program may be stored on a memory card, an optical disk, a magnetic disk, or the like.

[0229] In this way, each of the above functional configurations and methods may be realized by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, each of the above functional configurations and methods may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, each of the above functional configurations and methods may be realized by one or more dedicated computers configured by combining a processor and a memory programmed to execute one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transitive tangible recording medium as instructions to be executed by a computer. In other words, each of the above functional configurations and methods may be expressed as a computer program including a procedure for realizing the same, or as a non-transitive tangible recording medium storing the program.

[0230] The present disclosure is not limited to the specific operation and processing modes shown in the above embodiment. That is, for example, this specification is described on the premise of so-called "vehicles driving on the left side" that conforms to the road traffic laws of Japan, but the present disclosure is not limited to this. That is, in the case of so-called "vehicles driving on the right side" that conforms to the road traffic laws of the United States, etc., the drawings such as FIG. 10 may be appropriately reversed left and right, and the explanation in the specification may be changed accordingly. The same applies to the meanings corresponding to the colors and line types of the road dividing line B43.

[0231] The image information storage unit 2204 may store the target image GS that has already been converted into a bird's-eye image. In this case, the bird's-eye image generation unit 2211 in the annotation information generation unit 2201 and the bird's-eye image generation unit 2221 in the learning unit 2202 may be omitted.

[0232] In each embodiment described above, the parking space annotation code Ea is generated by the parking space learning unit 2222. However, the present disclosure is not limited to this aspect. That is, for example, the parking space annotation code Ea may be generated by the parking space annotation generating unit 2213.

[0233] In the ninth embodiment, the parking area PSg is generated under the condition that the parking row PSs exists on both sides of the parking lot passage PSr. This is based on the fact that the situation in which the parking row PSs exists on both sides of the parking lot passage PSr has the highest reliability of being in the parking lot PL, as described above. However, the present disclosure is not limited to such an embodiment. That is, for example, in the examples of Figures 105 and 106, even if there is no parking row PSs on the lower right, it is possible to generate a parking area PSg similar to the case in which there is a parking row PSs on the lower right.

[0234] Similar expressions such as "obtain," "calculate," "estimate," "detect," "sensing," and "determine" may be substituted for each other as appropriate to the extent that they are not technically inconsistent. Furthermore, "information" and "signal" may be substituted for each other as appropriate to the extent that they are not technically inconsistent. Furthermore, "above the threshold" and "exceed the threshold" may be substituted for each other as appropriate to the extent that they are not technically inconsistent. Similarly, "below the threshold" and "below the threshold" may be substituted for each other as appropriate to the extent that they are not technically inconsistent.

[0235] Needless to say, the elements constituting the above-mentioned embodiments are not necessarily essential, except when expressly stated as essential or when clearly considered essential in principle. In addition, when the numbers, amounts, ranges, etc. of the components are mentioned, the present disclosure is not limited to the specific values, except when expressly stated as essential or when clearly limited to a specific value in principle. Similarly, when the shapes, directions, positional relationships, etc. of the components are mentioned, the present disclosure is not limited to the shapes, directions, positional relationships, etc., except when expressly stated as essential or when clearly limited to a specific shape, direction, positional relationship, etc. in principle.

[0236] The modified examples are not limited to the above examples. For example, multiple embodiments may be combined with each other as long as there is no technical contradiction. Similarly, multiple specific examples may be combined with each other as long as there is no technical contradiction. Furthermore, multiple modified examples may be combined with each other as long as there is no technical contradiction. Furthermore, all or part of the above embodiment and all or part of the modified examples may be combined with each other as long as there is no technical contradiction.

[0237] (Disclosure Perspective) As is apparent from the above description of the embodiments and modifications, at least the following disclosure items are disclosed in this specification.

[0238] [Perspective A1-1] An in-vehicle device (2) mounted on a vehicle (V), A processor (21); A memory (22) which is a non-transitory tangible recording medium for storing program instructions used by the processor; Equipped with The processor executes the program instructions stored in the memory to objectify a parking space (PS), which is an area in which the vehicle can be parked and is captured by an imaging device (34) that captures an image of the surroundings of the vehicle, using a plurality of corner points (BP) and a center point (BPC) that define the shape of the parking space, and learns the parking space by assigning attribute information of the parking space. In-vehicle device. [Perspective A1-2] The attribute information includes information of the frame line (B5) of the parking frame, The in-vehicle device according to aspect A1-1. [Perspective A1-3] The attribute information includes information on a wheel stopper (B6) on the inside of the parking frame, The in-vehicle device according to aspect A1-1 or A1-2. [Point of View A1-4] The attribute information includes information of additional attributes different from the information of the frame line (B5) of the parking space and the wheel stopper (B6) on the inside of the parking space, The in-vehicle device according to any one of the aspects A1-1 to A1-3. [Point of View A1-5] The processor calculates the number of the corner points occluded by other targets (B7), and if the number is equal to or greater than a learning exclusion threshold, excludes the corresponding parking space from the learning target. The in-vehicle device according to any one of the aspects A1-1 to A1-4. [Point of View A1-6] The processor, for the parking frame in which the number of corner points outside the imaging range of the imaging device is two or less, offsets the corner points within the imaging range and learns the corner points. The in-vehicle device according to any one of the aspects A1-1 to A1-5. [Point of View A1-7] The processor sets the offset direction to a direction that maintains the shape of the parking space. The in-vehicle device according to aspect A1-6. [Point of View A1-8] The processor calculates an obscuration amount, which is an area obscured by other targets (B7) in the parking space, and when the obscuration amount is equal to or greater than a threshold, the vacancy state of the corresponding parking space is determined to be unknown in learning the parking space. The in-vehicle device according to any one of the aspects A1-1 to A1-7. [Point of View A1-9] The processor calculates the occlusion amount based on a viewing angle of the parking frame as seen from the vehicle. The in-vehicle device according to aspect A1-8. [Point of View A1-10] The processor defines a parking frame direction (anchor0 to anchor3) which is a direction of the parking frame, and learns the parking frame for each parking frame direction. The in-vehicle device according to any one of the aspects A1-1 to A1-9. [Point of View A1-11] The processor learns the parking space based on information (Ec1_anchor0 to Ec1_anchor3) obtained by encoding the corner points and the center point for each parking space direction. The in-vehicle device according to aspect A1-10. [Point of View A1-12] The processor, The parking frame is recognized by the parking frame line (B5) or the parked vehicle (B7) captured by the imaging device, Select one of the recognized parking spaces as a reference object (PSb), and search for the parking spaces from the reference object in a predetermined first direction and / or a second direction opposite to the first direction, thereby acquiring a parking row (PSs) in which the parking spaces are consecutive in a predetermined direction along the first direction by a predetermined number or more; Based on the acquired parking row, it is determined whether the vehicle is present in a parking lot (PL); The in-vehicle device according to any one of the aspects A1-1 to A1-11. [Point of View A1-13] The processor, Acquire a parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. The in-vehicle device according to aspect A1-12. [Point of View A1-14] The processor generates the parking area such that the parking rows are located on both sides of the aisle. The in-vehicle device according to aspect A1-13. [Point of View A1-15] The processor, The parking frame is recognized by the parking frame line (B5) or the parked vehicle (B7) captured by the imaging device, Acquire a parking row (PSs) in which the recognized parking frames are consecutive in a predetermined direction for a predetermined number or more; Acquire a parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the parking frame or the parking row adjacent to the parking row included in the acquired parking area in the predetermined direction is outside the parking area, the parking area is expanded in the predetermined direction to include the parking frame or the parking row; determining whether the vehicle is in a parking lot (PL) based on the parking area; The in-vehicle device according to any one of the aspects A1-1 to A1-11. [Point of View A1-16] The processor generates the parking area such that the parking rows are located on both sides of the aisle. The in-vehicle device according to aspect A1-15. [Point of View A1-17] The processor, When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. The in-vehicle device according to aspect A1-15 or A1-16. [Point of View A1-18] the processor determines that the vehicle is in the parking lot when the vehicle enters the parking area from a specific direction, and does not determine that the vehicle is in the parking lot when the vehicle enters the parking area from a direction different from the specific direction. The in-vehicle device according to any one of the aspects A1-13 to A1-17. [Point of View A1-19] When the processor determines that the vehicle is not present in the parking lot, the processor discards the recognized parking space information. The in-vehicle device according to any one of Aspects A1-12 to A1-18. [Point of View A1-20] The processor generates the parking row based on a positional relationship between the parking frames. The in-vehicle device according to any one of the aspects A1-12 to A1-19. [Point of View A1-21] When the processor determines that the vehicle is in the parking lot, the processor is capable of executing a process for erroneous accelerator operation, including suppressing control of a driving force of the vehicle to suppress sudden acceleration due to an erroneous accelerator operation and / or notifying an occupant. The in-vehicle device according to any one of the aspects A1-12 to A1-20. [Point of View A1-22] when it is determined that the accelerator erroneous operation continues for a predetermined period of time or more after the accelerator erroneous operation process is executed, the processor enables the accelerator erroneous operation process to be executed again. The in-vehicle device according to aspect A1-21. [Point of View A1-23] When the processor determines that the accelerator erroneous operation is being performed again after the accelerator erroneous operation has been eliminated by executing the accelerator erroneous operation process, the processor enables the accelerator erroneous operation process to be executed again. The in-vehicle device according to aspect A1-21 or A1-22. [Point of View A1-24] When the processor determines that the vehicle is present in the parking lot, the processor generates a driving route (TPp) to the parking space where the vehicle can be parked. The in-vehicle device according to any one of Aspects A1-12 to A1-23.

[0239] [Perspective A2-1] A method for operating an in-vehicle device, the method being executed by a processor (21) provided in an in-vehicle device (2) mounted on a vehicle (V), comprising: A parking space (PS) that is an area in which the vehicle can be parked and is captured by an imaging device (34) that captures an image of the periphery of the vehicle is objectified by a plurality of corner points (BP) and a center point (BPC) that define the shape of the parking space, and attribute information of the parking space is assigned to the parking space, thereby learning the parking space. How to operate onboard equipment. [Perspective A2-2] The attribute information includes information of the frame line (B5) of the parking frame, The in-vehicle device operating method according to aspect A2-1. [Perspective A2-3] The attribute information includes information on a wheel stopper (B6) on the inside of the parking frame, The in-vehicle device operation method according to aspect A2-1 or A2-2. [Point of View A2-4] The attribute information includes information of additional attributes different from the information of the frame line (B5) of the parking space and the wheel stopper (B6) on the inside of the parking space, The in-vehicle device operation method according to any one of Aspects A2-1 to A2-3. [Point of View A2-5] Calculate the number of the corner points occluded by other targets (B7), and if the number is equal to or greater than a learning exclusion threshold, exclude the corresponding parking space from the learning target. The in-vehicle device operation method according to any one of Aspects A2-1 to A2-4. [Point of View A2-6] For the parking space having two or less corner points outside the imaging range of the imaging device, the corner points are offset to within the imaging range and learned. The in-vehicle device operation method according to any one of Aspects A2-1 to A2-5. [Point of View A2-7] The offset direction is a direction that maintains the shape of the parking space. A method for operating an in-vehicle device according to aspect A2-6. [Point of View A2-8] Calculate the amount of obscuration, which is the range of the parking space obscured by other objects (B7), and when the amount of obscuration is equal to or greater than a threshold, determine the vacancy status of the corresponding parking space in learning as unknown. The in-vehicle device operation method according to any one of the aspects A2-1 to A2-7. [Point of View A2-9] Calculating the amount of obscuration based on a viewing angle of the parking frame as seen from the vehicle. A method for operating an in-vehicle device according to aspect A2-8. [Point of View A2-10] A parking frame direction (anchor 0 to anchor 3) which is the orientation of the parking frame is defined, and the parking frame is learned for each parking frame direction. The in-vehicle device operation method according to any one of the aspects A2-1 to A2-9. [Point of View A2-11] The parking frame is learned based on information (Ec1_anchor0 to Ec1_anchor3) obtained by encoding the corner points and the center point for each parking frame direction. A method for operating an in-vehicle device according to aspect A2-10. [Point of View A2-12] The parking frame is recognized by the parking frame line (B5) or the parked vehicle (B7). Select one of the recognized parking spaces as a reference object (PSb), and search for the parking spaces from the reference object in a predetermined first direction and / or a second direction opposite to the first direction, thereby acquiring a parking row (PSs) in which the parking spaces are consecutive in a predetermined direction along the first direction by a predetermined number or more; Based on the acquired parking row, it is determined whether the vehicle is present in a parking lot (PL); The in-vehicle device operation method according to any one of the aspects A2-1 to A2-11. [Point of View A2-13] Based on the acquired parking row, a parking area (PSg) is acquired, the parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. A method for operating an in-vehicle device according to aspect A2-12. [Point of View A2-14] generating the parking area such that the parking rows are located on both sides of the passage; A method for operating an in-vehicle device according to aspect A2-13. [Point of View A2-15] The parking frame is recognized by the parking frame line (B5) or the parked vehicle (B7). Acquire a parking row (PSs) in which the recognized parking frames are consecutive in a predetermined direction for a predetermined number or more; Based on the acquired parking row, a parking area (PSg) is acquired, the parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the parking frame or the parking row adjacent to the parking row included in the acquired parking area in the predetermined direction is outside the parking area, the parking area is expanded in the predetermined direction to include the parking frame or the parking row; determining whether the vehicle is in a parking lot (PL) based on the parking area; The in-vehicle device operation method according to any one of the aspects A2-1 to A2-11. [Point of View A2-16] generating the parking area such that the parking rows are located on both sides of the passage; A method for operating an in-vehicle device according to aspect A2-15. [Point of View A2-17] When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. The in-vehicle device operation method according to aspect A2-15 or A2-16. [Point of View A2-18] When the vehicle enters the parking area from a specific direction, it is determined that the vehicle is in the parking lot, whereas when the vehicle enters the parking area from a direction different from the specific direction, it is not determined that the vehicle is in the parking lot. The in-vehicle device operation method according to any one of A2-13 to A2-17. [Point of View A2-19] If it is determined that the vehicle is not present in the parking lot, the recognized parking space information is rejected. The in-vehicle device operation method according to any one of A2-12 to A2-18. [Point of View A2-20] generating the parking row based on a positional relationship between the parking frames; The in-vehicle device operation method according to any one of the aspects A2-12 to A2-19. [Point of View A2-21] When it is determined that the vehicle is present in the parking lot, it is possible to execute a process for erroneous accelerator operation, including suppression control of the driving force of the vehicle to suppress sudden acceleration due to erroneous accelerator operation and / or notification to an occupant. The in-vehicle device operation method according to any one of A2-12 to A2-20. [Point of View A2-22] when it is determined that the accelerator erroneous operation continues for a predetermined period of time or more even after the accelerator erroneous operation process is executed, the accelerator erroneous operation process is enabled to be executed again; A method for operating an in-vehicle device according to aspect A2-21. [Point of View A2-23] when it is determined that the accelerator erroneous operation is performed again after the accelerator erroneous operation has been eliminated by executing the accelerator erroneous operation process, the accelerator erroneous operation process can be executed again. The in-vehicle device operating method according to aspect A2-21 or A2-22. [Point of View A2-24] When it is determined that the vehicle is present in the parking lot, a driving route (TPp) to the parking space where the vehicle can be parked is generated. The in-vehicle device operation method according to any one of A2-12 to A2-23.

[0240] [Perspective A3-1] An in-vehicle device operation program executed by a processor (21) provided in an in-vehicle device (2) mounted on a vehicle (V), The process executed by the processor includes a process of objectifying a parking space (PS), which is an area in which the vehicle can be parked and is captured by an imaging device (34) that captures an image of the surroundings of the vehicle, by a plurality of corner points (BP) and a center point (BPC) that define the shape of the parking space, and learning the parking space by adding attribute information of the parking space. In-vehicle device operation program. [Perspective A3-2] The attribute information includes information of the frame line (B5) of the parking frame, The in-vehicle device operation program according to aspect A3-1. [Perspective A3-3] The attribute information includes information on a wheel stopper (B6) on the inside of the parking frame, An in-vehicle device operation program according to aspect A3-1 or A3-2. [Point of View A3-4] The attribute information includes information of additional attributes different from the information of the frame line (B5) of the parking space and the wheel stopper (B6) on the inside of the parking space, The in-vehicle device operation program according to any one of Aspects A3-1 to A3-3. [Point of View A3-5] The process executed by the processor includes a process of calculating the number of the corner points occluded by other targets (B7), and excluding the corresponding parking space from the learning target when the number is equal to or greater than a learning exclusion threshold. The in-vehicle device operation program according to any one of Aspects A3-1 to A3-4. [Point of View A3-6] In the process of learning the parking space, for the parking space having two or less corner points outside the imaging range of the imaging device, the corner points are offset into the imaging range and learned. The in-vehicle device operation program according to any one of Aspects A3-1 to A3-5. [Point of View A3-7] The offset direction is a direction that maintains the shape of the parking space. An in-vehicle device operation program according to aspect A3-6. [Point of View A3-8] The process executed by the processor includes a process of calculating a hidden amount, which is a range blocked by another target (B7) in the parking frame, When the hidden amount is equal to or greater than a threshold value, the vacancy status of the corresponding parking frame is set as unknown in learning the parking frame. The in-vehicle device operation program according to any one of Aspects A3-1 to A3-7. [Point of View A3-9] The process of calculating the occlusion amount is a process of calculating the occlusion amount based on a viewing angle of the parking frame as seen from the vehicle. An in-vehicle device operation program according to aspect A3-8. [Point of View A3-10] In the process of learning the parking frame, a parking frame direction (anchor 0 to anchor 3) which is the orientation of the parking frame is defined, and the parking frame is learned for each parking frame direction. The in-vehicle device operation program according to any one of Aspects A3-1 to A3-9. [Point of View A3-11] In the process of learning the parking space, the parking space is learned based on information (Ec1_anchor0 to Ec1_anchor3) obtained by encoding the corner point and the center point for each parking space direction. An in-vehicle device operation program according to aspect A3-10. [Point of View A3-12] The process executed by the processor includes: The parking frame is recognized by the parking frame line (B5) or the parked vehicle (B7). A process of selecting one of the recognized parking spaces as a reference object (PSb), searching for the parking spaces from the reference object in a predetermined first direction and / or a second direction opposite to the first direction, thereby acquiring a parking row (PSs) in which the parking spaces are consecutive in a predetermined direction along the first direction by a predetermined number or more; A process of determining whether the vehicle is present in a parking lot (PL) based on the acquired parking row information; Further comprising: The in-vehicle device operation program according to any one of Aspects A3-1 to A3-11. [Point of View A3-13] The process executed by the processor further includes a process of acquiring a parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, the area being drivable by the vehicle, based on the acquired parking row; In the process of determining whether the vehicle is present in the parking lot, When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot. If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. An in-vehicle device operation program according to aspect A3-12. [Point of View A3-14] In the process of generating the parking area, the parking area is generated so that the parking rows are located on both sides of the passage. The in-vehicle device operation program according to aspect A3-13. [Point of View A3-15] The process executed by the processor includes: A process of recognizing the parking frame by a parking frame line (B5) or a parked vehicle (B7); A process of acquiring a parking row (PSs) in which the recognized parking frames are consecutive in a predetermined direction for a predetermined number or more; A process of acquiring a parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel, based on the acquired parking row; When the parking frame or the parking row adjacent to the parking row included in the acquired parking area in the predetermined direction is outside the parking area, a process of expanding the parking area in the predetermined direction to include the parking frame or the parking row; A process of determining whether the vehicle is in a parking lot (PL) based on the parking area; Including, The in-vehicle device operation program according to any one of Aspects A3-1 to A3-11. [Point of View A3-16] In the process of generating the parking area, the parking area is generated so that the parking rows are located on both sides of the passage. The in-vehicle device operation program according to aspect A3-15. [Point of View A3-17] In the process of determining whether the vehicle is present in the parking lot, When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. An in-vehicle device operation program according to aspect A3-15 or A3-16. [Point of View A3-18] In the process of determining whether the vehicle is present in the parking lot, When the vehicle enters the parking area from a specific direction, it is determined that the vehicle is in the parking lot, whereas when the vehicle enters the parking area from a direction different from the specific direction, it is not determined that the vehicle is in the parking lot. The in-vehicle device operation program according to any one of A3-13 to A3-17. [Point of View A3-19] The process executed by the processor further includes a process of rejecting the recognized parking space information when it is determined that the vehicle is not present in the parking lot. The in-vehicle device operation program according to any one of A3-12 to A3-18. [Point of View A3-20] In the process of generating the parking row, the parking row is generated based on a positional relationship between the parking spaces. The in-vehicle device operation program according to any one of A3-12 to A3-19. [Point of View A3-21] The in-vehicle device operation program according to any one of A3-12 to A3-20. [Point of View A3-22] The process executed by the processor further includes a process for enabling the accelerator erroneous operation process to be executed again when it is determined that the accelerator erroneous operation has continued for a predetermined time or more after the accelerator erroneous operation process has been executed. The in-vehicle device operation program according to aspect A3-21. [Point of View A3-23] The processing executed by the processor further includes processing for enabling the accelerator erroneous operation processing to be executed again when it is determined that the accelerator erroneous operation is being performed again after the accelerator erroneous operation has been eliminated by execution of the accelerator erroneous operation processing. An in-vehicle device operation program according to aspect A3-21 or A3-22. [Point of View A3-24] The process executed by the processor further includes a process of generating a driving route (TPp) to the parking space where the vehicle can be parked when it is determined that the vehicle is present in the parking lot. The in-vehicle device operation program according to any one of A3-12 to A3-23.

[0241] [Perspective B1-1] An in-vehicle device (2) mounted on a vehicle (V), A processor (21); A memory (22) which is a non-transitory tangible recording medium for storing program instructions used by the processor; Equipped with The processor executes the program instructions stored in the memory to A parking space (PS) that is an area in which the vehicle can be parked is recognized based on a parking frame line (B5) or a parked vehicle (B7) included in a target (B) captured by an imaging device (34) that captures an image of the periphery of the vehicle; Select one of the recognized parking spaces as a reference object (PSb), and search for the parking spaces from the reference object in a predetermined first direction and / or a second direction opposite to the first direction, thereby acquiring a parking row (PSs) in which the parking spaces are consecutive in a predetermined direction along the first direction by a predetermined number or more; Based on the acquired parking row, it is determined whether the vehicle is present in a parking lot (PL); In-vehicle device. [Perspective B1-2] The processor, Based on the acquired parking row, a parking area (PSg) is acquired, the parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. The in-vehicle device according to aspect B1-1. [Perspective B1-3] The processor generates the parking area such that the parking rows are located on both sides of the aisle. The in-vehicle device according to aspect B1-2. [Point of View B1-4] the processor determines that the vehicle is in the parking lot when the vehicle enters the parking area from a specific direction, and does not determine that the vehicle is in the parking lot when the vehicle enters the parking area from a direction different from the specific direction. The in-vehicle device according to aspect B1-2 or B1-3. [Point of View B1-5] When the processor determines that the vehicle is not present in the parking lot, the processor discards the recognized parking space information. The in-vehicle device according to any one of the aspects B1-1 to B1-4. [Point of View B1-6] The processor generates the parking row based on a positional relationship between the parking frames. The in-vehicle device according to any one of the aspects B1-1 to B1-5. [Point of View B1-7] When the processor determines that the vehicle is in the parking lot, the processor is capable of executing a process for erroneous accelerator operation, including suppressing control of a driving force of the vehicle to suppress sudden acceleration due to an erroneous accelerator operation and / or notifying an occupant. The in-vehicle device according to any one of the aspects B1-1 to B1-6. [Point of View B1-8] when it is determined that the accelerator erroneous operation continues for a predetermined period of time or more after the accelerator erroneous operation process is executed, the processor enables the accelerator erroneous operation process to be executed again. The in-vehicle device according to aspect B1-7. [Point of View B1-9] When the processor determines that the accelerator erroneous operation is being performed again after the accelerator erroneous operation has been eliminated by executing the accelerator erroneous operation process, the processor enables the accelerator erroneous operation process to be executed again. The in-vehicle device according to aspect B1-7 or B1-8. [Point of View B1-10] When the processor determines that the vehicle is present in the parking lot, the processor generates a driving route (TPp) to the parking space where the vehicle can be parked. The in-vehicle device according to any one of the aspects B1-1 to B1-9.

[0242] [Perspective B2-1] A method for operating an in-vehicle device, the method being executed by a processor (21) provided in an in-vehicle device (2) mounted on a vehicle (V), comprising: A parking space (PS) that is an area in which the vehicle can be parked is recognized based on a parking frame line (B5) or a parked vehicle (B7) included in a target (B) captured by an imaging device (34) that captures an image of the periphery of the vehicle; Select one of the recognized parking spaces as a reference object (PSb), and search for the parking spaces from the reference object in a predetermined first direction and / or a second direction opposite to the first direction, thereby acquiring a parking row (PSs) in which the parking spaces are consecutive in a predetermined direction along the first direction by a predetermined number or more; Based on the acquired parking row, it is determined whether the vehicle is present in a parking lot (PL); How to operate onboard equipment. [Perspective B2-2] Based on the acquired parking row, a parking area (PSg) is acquired, the parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. A method for operating an in-vehicle device according to aspect B2-1. [Perspective B2-3] generating the parking area such that the parking rows are located on both sides of the passage; A method for operating an in-vehicle device according to aspect B2-2. [Point of View B2-4] When the vehicle enters the parking area from a specific direction, it is determined that the vehicle is in the parking lot, whereas when the vehicle enters the parking area from a direction different from the specific direction, it is not determined that the vehicle is in the parking lot. The in-vehicle device operation method according to aspect B2-2 or B2-3. [Point of View B2-5] If it is determined that the vehicle is not present in the parking lot, the recognized parking space information is rejected. The in-vehicle device operation method according to any one of the aspects B2-1 to B2-4. [Point of View B2-6] generating the parking row based on a positional relationship between the parking frames; The in-vehicle device operation method according to any one of the aspects B2-1 to B2-5. [Point of View B2-7] When it is determined that the vehicle is present in the parking lot, it is possible to execute a process for erroneous accelerator operation, including suppression control of the driving force of the vehicle to suppress sudden acceleration due to erroneous accelerator operation and / or notification to an occupant. The in-vehicle device operation method according to any one of the aspects B2-1 to B2-6. [Point of View B2-8] when it is determined that the accelerator erroneous operation continues for a predetermined period of time or more even after the accelerator erroneous operation process is executed, the accelerator erroneous operation process is enabled to be executed again; A method for operating an in-vehicle device according to aspect B2-7. [Point of View B2-9] when it is determined that the accelerator erroneous operation is performed again after the accelerator erroneous operation has been eliminated by executing the accelerator erroneous operation process, the accelerator erroneous operation process can be executed again. A method for operating an in-vehicle device according to aspect B2-7 or B2-8. [Point of View B2-10] When it is determined that the vehicle is present in the parking lot, a driving route (TPp) to the parking space where the vehicle can be parked is generated. The in-vehicle device operation method according to any one of the aspects B2-1 to B2-9.

[0243] [Perspective B3-1] An in-vehicle device operation program executed by a processor (21) provided in an in-vehicle device (2) mounted on a vehicle (V), The process executed by the processor includes: A process of recognizing a parking space (PS) that is an area in which the vehicle can be parked, based on a parking frame line (B5) or a parked vehicle (B7) included in a target (B) captured by an imaging device (34) that captures an image of the periphery of the vehicle; A process of selecting one of the recognized parking spaces as a reference object (PSb), searching for the parking spaces from the reference object in a predetermined first direction and / or a second direction opposite to the first direction, thereby acquiring a parking row (PSs) in which the parking spaces are consecutive in a predetermined direction along the first direction by a predetermined number or more; A process of determining whether the vehicle is present in a parking lot (PL) based on the acquired parking row information; Including, In-vehicle device operation program. [Point of View B3-2] The process executed by the processor further includes a process of acquiring a parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, the area being drivable by the vehicle, based on the acquired parking row; In the process of determining whether the vehicle is present in the parking lot, When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. An in-vehicle device operation program according to aspect B3-1. [Perspective B3-3] In the process of generating the parking area, the parking area is generated so that the parking rows are located on both sides of the passage. An in-vehicle device operation program according to aspect B3-2. [Point of View B3-4] In the process of determining whether the vehicle is present in the parking lot, When the vehicle enters the parking area from a specific direction, it is determined that the vehicle is in the parking lot, whereas when the vehicle enters the parking area from a direction different from the specific direction, it is not determined that the vehicle is in the parking lot. An in-vehicle device operation program according to aspect B3-2 or B3-3. [Point of View B3-5] The process executed by the processor further includes a process of rejecting the recognized parking space information when it is determined that the vehicle is not present in the parking lot. The in-vehicle device operation program according to any one of the aspects B3-1 to B3-4. [Point of View B3-6] In the process of generating the parking row, the parking row is generated based on a positional relationship between the parking spaces. The in-vehicle device operation program according to any one of the aspects B3-1 to B3-5. [Point of View B3-7] The process executed by the processor further includes a process for enabling execution of a process for erroneous accelerator operation, including suppression control of the driving force of the vehicle to suppress sudden acceleration due to erroneous accelerator operation and / or notification to an occupant, when it is determined that the vehicle is present in the parking lot. The in-vehicle device operation program according to any one of the aspects B3-1 to B3-6. [Point of View B3-8] The process executed by the processor further includes a process for enabling the accelerator erroneous operation process to be executed again when it is determined that the accelerator erroneous operation has continued for a predetermined time or more after the accelerator erroneous operation process has been executed. An in-vehicle device operation program according to aspect B3-7. [Point of View B3-9] The processing executed by the processor further includes processing for enabling the accelerator erroneous operation processing to be executed again when it is determined that the accelerator erroneous operation is being performed again after the accelerator erroneous operation has been eliminated by execution of the accelerator erroneous operation processing. An in-vehicle device operation program according to aspect B3-7 or B3-8. [Point of View B3-10] The process executed by the processor further includes a process of generating a driving route (TPp) to the parking space where the vehicle can be parked when it is determined that the vehicle is present in the parking lot. The in-vehicle device operation program according to any one of the aspects B3-1 to B3-9.

[0244] [Point of View C1-1] An in-vehicle device (2) mounted on a vehicle (V), A processor (21); A memory (22) which is a non-transitory tangible recording medium for storing program instructions used by the processor; Equipped with The processor executes the program instructions stored in the memory to A parking space (PS) that is an area in which the vehicle can be parked is recognized based on a parking frame line (B5) or a parked vehicle (B7) included in a target (B) captured by an imaging device (34) that captures an image of the periphery of the vehicle; Acquire a parking row (PSs) in which the recognized parking frames are consecutive in a predetermined direction for a predetermined number or more; Based on the acquired parking row, a parking area (PSg) is acquired, the parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the parking frame or the parking row adjacent to the parking row included in the acquired parking area in the predetermined direction is outside the parking area, the parking area is expanded in the predetermined direction to include the parking frame or the parking row; determining whether the vehicle is in a parking lot (PL) based on the parking area; In-vehicle device. [Point of View C1-2] The processor generates the parking area such that the parking rows are located on both sides of the aisle. The in-vehicle device according to aspect C1-1. [Point of View C1-3] The processor, When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. The in-vehicle device according to aspect C1-1 or C1-2. [Point of View C1-4] the processor determines that the vehicle is in the parking lot when the vehicle enters the parking area from a specific direction, and does not determine that the vehicle is in the parking lot when the vehicle enters the parking area from a direction different from the specific direction. The in-vehicle device according to any one of the aspects C1-1 to C1-3. [Point of View C1-5] When the processor determines that the vehicle is not present in the parking lot, the processor discards the recognized parking space information. The in-vehicle device according to any one of the aspects C1-1 to C1-4. [Point of View C1-6] The processor generates the parking row based on a positional relationship between the parking frames. The in-vehicle device according to any one of the aspects C1-1 to C1-5. [Point of View C1-7] When the processor determines that the vehicle is in the parking lot, the processor is capable of executing a process for erroneous accelerator operation, including suppressing control of a driving force of the vehicle to suppress sudden acceleration due to an erroneous accelerator operation and / or notifying an occupant. The in-vehicle device according to any one of the aspects C1-1 to C1-6. [Point of View C1-8] when it is determined that the accelerator erroneous operation continues for a predetermined period of time or more after the accelerator erroneous operation process is executed, the processor enables the accelerator erroneous operation process to be executed again. The in-vehicle device according to aspect C1-7. [Point of View C1-9] When the processor determines that the accelerator erroneous operation is being performed again after the accelerator erroneous operation has been eliminated by executing the accelerator erroneous operation process, the processor enables the accelerator erroneous operation process to be executed again. The in-vehicle device according to aspect C1-7 or C1-8. [Point of View C1-10] When the processor determines that the vehicle is present in the parking lot, the processor generates a driving route (TPp) to the parking space where the vehicle can be parked. The in-vehicle device according to any one of the aspects C1-1 to C1-9.

[0245] [Point of View C2-1] A method for operating an in-vehicle device, the method being executed by a processor (21) provided in an in-vehicle device (2) mounted on a vehicle (V), comprising: A parking space (PS) that is an area in which the vehicle can be parked is recognized based on a parking frame line (B5) or a parked vehicle (B7) included in a target (B) captured by an imaging device (34) that captures an image of the periphery of the vehicle; Acquire a parking row (PSs) in which the recognized parking frames are consecutive in a predetermined direction for a predetermined number or more; Based on the acquired parking row, a parking area (PSg) is acquired, the parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the parking frame or the parking row adjacent to the parking row included in the acquired parking area in the predetermined direction is outside the parking area, the parking area is expanded in the predetermined direction to include the parking frame or the parking row; determining whether the vehicle is in a parking lot (PL) based on the parking area; How to operate onboard equipment. [Point of View C2-2] generating the parking area such that the parking rows are located on both sides of the passage; A method for operating an in-vehicle device according to aspect C2-1. [Point of View C2-3] When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. A method for operating an in-vehicle device according to aspect C2-1 or C2-2. [Point of View C2-4] When the vehicle enters the parking area from a specific direction, it is determined that the vehicle is in the parking lot, whereas when the vehicle enters the parking area from a direction different from the specific direction, it is not determined that the vehicle is in the parking lot. The in-vehicle device operation method according to any one of the aspects C2-1 to C2-3. [Point of View C2-5] If it is determined that the vehicle is not present in the parking lot, the recognized parking space information is rejected. The in-vehicle device operation method according to any one of the aspects C2-1 to C2-4. [Point of View C2-6] generating the parking row based on a positional relationship between the parking frames; The in-vehicle device operation method according to any one of the aspects C2-1 to C2-5. [Point of View C2-7] When it is determined that the vehicle is present in the parking lot, it is possible to execute a process for erroneous accelerator operation, including suppression control of the driving force of the vehicle to suppress sudden acceleration due to erroneous accelerator operation and / or notification to an occupant. The in-vehicle device operation method according to any one of the aspects C2-1 to C2-6. [Point of View C2-8] when it is determined that the accelerator erroneous operation continues for a predetermined period of time or more even after the accelerator erroneous operation process is executed, the accelerator erroneous operation process is enabled to be executed again; A method for operating an in-vehicle device according to aspect C2-7. [Point of View C2-9] when it is determined that the accelerator erroneous operation is performed again after the accelerator erroneous operation has been eliminated by executing the accelerator erroneous operation process, the accelerator erroneous operation process can be executed again. A method for operating an in-vehicle device according to aspect C2-7 or C2-8. [Point of View C2-10] When it is determined that the vehicle is present in the parking lot, a driving route (TPp) to the parking space where the vehicle can be parked is generated. The in-vehicle device operation method according to any one of the aspects C2-1 to C2-9.

[0246] [Point of View C3-1] An in-vehicle device operation program executed by a processor (21) provided in an in-vehicle device (2) mounted on a vehicle (V), The process executed by the processor includes: A process of recognizing a parking space (PS) that is an area in which the vehicle can be parked, based on a parking frame line (B5) or a parked vehicle (B7) included in a target (B) captured by an imaging device (34) that captures an image of the periphery of the vehicle; A process of acquiring a parking row (PSs) in which the recognized parking frames are consecutive in a predetermined direction for a predetermined number or more; A process of acquiring a parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel, based on the acquired parking row; When the parking frame or the parking row adjacent to the parking row included in the acquired parking area in the predetermined direction is outside the parking area, a process of expanding the parking area in the predetermined direction to include the parking frame or the parking row; A process of determining whether the vehicle is in a parking lot (PL) based on the parking area; Including, In-vehicle device operation program. [Point of View C3-2] In the process of generating the parking area, the parking area is generated so that the parking rows are located on both sides of the passage. An in-vehicle device operation program according to aspect C3-1. [Point of View C3-3] In the process of determining whether the vehicle is present in the parking lot, When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. An in-vehicle device operation program according to aspect C3-1 or C3-2. [Point of View C3-4] In the process of determining whether the vehicle is present in the parking lot, When the vehicle enters the parking area from a specific direction, it is determined that the vehicle is in the parking lot, whereas when the vehicle enters the parking area from a direction different from the specific direction, it is not determined that the vehicle is in the parking lot. The in-vehicle device operation program according to any one of the aspects C3-1 to C3-3. [Point of View C3-5] The process executed by the processor further includes a process of rejecting the recognized parking space information when it is determined that the vehicle is not present in the parking lot. The in-vehicle device operation program according to any one of the aspects C3-1 to C3-4. [Point of View C3-6] In the process of generating the parking row, the parking row is generated based on a positional relationship between the parking spaces. The in-vehicle device operation program according to any one of the aspects C3-1 to C3-5. [Point of View C3-7] The process executed by the processor further includes a process for enabling execution of a process for erroneous accelerator operation, including suppression control of the driving force of the vehicle to suppress sudden acceleration due to erroneous accelerator operation and / or notification to an occupant, when it is determined that the vehicle is present in the parking lot. The in-vehicle device operation program according to any one of the aspects C3-1 to C3-6. [Point of View C3-8] The process executed by the processor further includes a process for enabling the accelerator erroneous operation process to be executed again when it is determined that the accelerator erroneous operation has continued for a predetermined time or more after the accelerator erroneous operation process has been executed. An in-vehicle device operation program according to aspect C3-7. [Point of View C3-9] The processing executed by the processor further includes processing for enabling the accelerator erroneous operation processing to be executed again when it is determined that the accelerator erroneous operation is being performed again after the accelerator erroneous operation has been eliminated by execution of the accelerator erroneous operation processing. An in-vehicle device operation program according to aspect C3-7 or C3-8. [Point of View C3-10] The process executed by the processor further includes a process of generating a driving route (TPp) to the parking space where the vehicle can be parked when it is determined that the vehicle is present in the parking lot. The in-vehicle device operation program according to any one of the aspects C3-1 to C3-9.

[0247] [Point of View D] A method executed by a processor (21) provided in an in-vehicle device (2) mounted on a host vehicle (V), comprising: The processor, Acquiring information about an image including a target (B) captured by an imaging device (34) that captures an image of the surroundings of the vehicle; Image-recognizing the target based on the image information; A method for learning a parking space by objectifying a parking space (PS), which is included in the target object and is an area in which the vehicle can be parked, using a plurality of corner points (BP) and a center point (BPC) that define its shape, and by assigning attribute information of the parking space.

[0248] [Viewpoint E] A method executed by a processor (21) provided in an in-vehicle device (2) mounted on a host vehicle (V), comprising: The processor, Acquiring information about an image including a target (B) captured by an imaging device (34) that captures an image of the surroundings of the vehicle; Based on the information of the image, a parking frame (PS) is recognized as an area in which the vehicle can be parked by a parking frame line (B5) or a parked vehicle (B7) included in the target object; Select one of the recognized parking spaces as a reference object (PSb), and search for the parking spaces from the reference object in a predetermined first direction and / or a second direction opposite to the first direction, thereby acquiring a parking row (PSs) in which the parking spaces are consecutive in a predetermined direction along the first direction by a predetermined number or more; The method determines whether the vehicle is present in a parking lot (PL) based on the parking row acquisition result.

[0249] [Point of View F] A method executed by a processor (21) provided in an in-vehicle device (2) mounted on a host vehicle (V), comprising: The processor, Acquiring information about an image including a target (B) captured by an imaging device (34) that captures an image of the surroundings of the vehicle; Based on the information of the image, a parking frame (PS) is recognized as an area in which the vehicle can be parked by a parking frame line (B5) or a parked vehicle (B7) included in the target object; Acquire a parking row (PSs) in which the recognized parking frames are consecutive in a predetermined direction for a predetermined number or more; Based on the acquired parking row, a parking area (PSg) is acquired, the parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the parking frame or the parking row adjacent to the parking row included in the acquired parking area in the predetermined direction is outside the parking area, the parking area is expanded in the predetermined direction to include the parking frame or the parking row; The method for determining whether the vehicle is in a parking lot (PL) based on the parking area. [Explanation of symbols]

[0250] 2 Driving ECU (on-board device) 21 Processors 22 Memory 34 Camera (imaging device) B Target B5 Parking frame line BP Parking space corner point BPC Parking space center point PS Parking space V Vehicle

Claims

1. An in-vehicle device (2) mounted on a vehicle (V), A processor (21); A memory (22) which is a non-transitory tangible recording medium for storing program instructions used by the processor; Equipped with The processor executes the program instructions stored in the memory to objectify a parking space (PS), which is an area in which the vehicle can be parked and is captured by an imaging device (34) that captures an image of the surroundings of the vehicle, using a plurality of corner points (BP) and a center point (BPC) that define the shape of the parking space, and learns the parking space by assigning attribute information of the parking space. In-vehicle device.

2. The attribute information includes information of the frame line (B5) of the parking frame, The in-vehicle device according to claim 1 .

3. The attribute information further includes information of a wheel stopper (B6) on the inside of the parking frame, The vehicle-mounted device according to claim 2 .

4. The attribute information further includes information of additional attributes different from the frame line and the ring clasp. The vehicle-mounted device according to claim 3.

5. The processor calculates the number of the corner points occluded by other targets (B7), and if the number is equal to or greater than a learning exclusion threshold, excludes the corresponding parking space from learning targets. The in-vehicle device according to claim 1 .

6. The processor offsets the corner points of the parking frame that are outside the imaging range of the imaging device to within the imaging range and learns the corner points. The in-vehicle device according to claim 1 .

7. The processor sets the offset direction to a direction that maintains the shape of the parking space. The vehicle-mounted device according to claim 6.

8. The processor calculates a hidden amount, which is a range of the parking space blocked by another target (B7), and when the hidden amount is equal to or greater than a threshold, the vacancy state of the corresponding parking space is unknown in learning the parking space. The in-vehicle device according to claim 1 .

9. The processor calculates the occlusion amount based on a viewing angle of the parking frame as seen from the vehicle. The in-vehicle device according to claim 8.

10. The processor defines a parking frame direction (anchor0 to anchor3) which is a direction of the parking frame, and learns the parking frame for each parking frame direction. The in-vehicle device according to claim 1 .

11. The processor learns the parking frame based on information (Ec1_anchor0 to Ec1_anchor3) obtained by encoding the corner points and the center point for each parking frame direction. The in-vehicle device according to claim 10.

12. The processor, The parking frame is recognized by the parking frame line (B5) or the parked vehicle (B7) captured by the imaging device, Select one of the recognized parking spaces as a reference object (PSb), and search for the parking spaces from the reference object in a predetermined first direction and / or a second direction opposite to the first direction, thereby acquiring a parking row (PSs) in which the parking spaces are consecutive in a predetermined direction along the first direction by a predetermined number or more; Based on the acquired parking row, it is determined whether the vehicle is present in a parking lot (PL). The in-vehicle device according to claim 1 .

13. The processor, The parking frame is recognized by the parking frame line (B5) or the parked vehicle (B7) captured by the imaging device, Acquire a parking row (PSs) in which the recognized plurality of parking frames are consecutive in a predetermined direction for a predetermined number or more; Acquire a parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the parking space or the parking row adjacent to the parking row included in the acquired parking area in the predetermined direction is outside the parking area, the parking area is expanded in the predetermined direction to include the parking space or the parking row; determining whether the vehicle is in a parking lot (PL) based on the parking area; The in-vehicle device according to claim 1 .

14. The processor generates the parking area such that the parking rows are located on both sides of the aisle. The in-vehicle device according to claim 13.

15. The processor generates the parking row based on a positional relationship between the parking frames. The in-vehicle device according to claim 12 or 13.

16. When the processor determines that the vehicle is not present in the parking lot, the processor discards the recognized parking space information. The in-vehicle device according to claim 12 or 13.

17. The processor, Acquire a parking area (PSg) including the parking row and an aisle (PSr) facing the parking row, which is an area in which the vehicle can travel; When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. The in-vehicle device according to claim 12.

18. The processor, When the vehicle enters inside the parking area, it is determined that the vehicle is present in the parking lot; If the vehicle exits outside the parking area, it is determined that the vehicle is not present in the parking lot. The in-vehicle device according to claim 13.

19. the processor determines that the vehicle is in the parking lot when the vehicle enters the parking area from a specific direction, and does not determine that the vehicle is in the parking lot when the vehicle enters the parking area from a direction different from the specific direction.

19. The vehicle-mounted device according to claim 17 or 18.

20. When the processor determines that the vehicle is in the parking lot, the processor is capable of executing a process for erroneous accelerator operation, including suppressing control of a driving force of the vehicle to suppress sudden acceleration due to an erroneous accelerator operation and / or notifying an occupant. The in-vehicle device according to claim 12 or 13.

21. when it is determined that the accelerator erroneous operation continues for a predetermined period of time or more after the accelerator erroneous operation process is executed, the processor enables the accelerator erroneous operation process to be executed again. The in-vehicle device according to claim 20.

22. When the processor determines that the accelerator erroneous operation is being performed again after the accelerator erroneous operation has been eliminated by executing the accelerator erroneous operation process, the processor enables the accelerator erroneous operation process to be executed again. The in-vehicle device according to claim 20.

23. When the processor determines that the vehicle is present in the parking lot, the processor generates a driving route (TPp) to the parking space where the vehicle can be parked. The in-vehicle device according to claim 12 or 13.

24. A method for operating an in-vehicle device, the method being executed by a processor (21) provided in an in-vehicle device (2) mounted on a vehicle (V), comprising the steps of: A parking space (PS), which is an area in which the vehicle can be parked and is captured by an imaging device (34) that captures an image of the periphery of the vehicle, is objectified by a plurality of corner points (BP) and a center point (BPC) that define the shape of the parking space, and attribute information of the parking space is assigned to the parking space, thereby learning the parking space. How to operate onboard equipment.

25. An in-vehicle device operation program executed by a processor (21) provided in an in-vehicle device (2) mounted on a vehicle (V), The process executed by the processor includes a process of objectifying a parking space (PS), which is an area in which the vehicle can be parked and is captured by an imaging device (34) that captures an image of the surroundings of the vehicle, by a plurality of corner points (BP) and a center point (BPC) that define the shape of the parking space, and learning the parking space by adding attribute information of the parking space. In-vehicle device operation program.

Images