Parking support device and parking support system

Abstract

To park a vehicle safely in a target parking space when blind spots for a driver of the vehicle exist around the target parking space.SOLUTION: A parking support device executes: a blind spot area identification process for identifying a blind spot area, which is a blind spot from the viewpoint of a vehicle, and a non-recognized blind spot area, which is a blind spot area not recognized by a driver of the vehicle; a danger determination process for determining whether or not a dangerous object that could be an obstacle for the vehicle exists based on information surrounding a target parking space obtained from a camera mounted on an unmanned aerial vehicle; and a display process for superimposing and displaying information on dangerous objects in the blind spot area on the display so that the information overlaps with the scenery seen by the driver and displaying information on dangerous objects in the non-recognized blind spot area by creating a sub-window area within the display.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present disclosure relates to a parking assistance device and a parking assistance system. [Background technology]

[0002] In recent years, parking assistance systems have been developed that detect vacant parking spaces among a plurality of parking spaces in a parking facility and guide a vehicle to the vacant parking space.

[0003] Recently, parking assistance systems have become known that use unmanned aerial vehicles such as drones to take pictures of the inside of a parking facility and guide a vehicle to a target parking space within the parking facility that is detected based on the taken images (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-214952 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, while the system described in Patent Document 1 is capable of guiding a target vehicle to a target parking space, it does not provide sufficient measures to safely park the vehicle when a blind spot for the driver of the vehicle exists around the target parking space.

[0006] The present disclosure has been made in consideration of the above circumstances, and aims to provide a parking assistance device and a parking assistance system that can safely park a vehicle when a blind spot for the driver of the vehicle exists around a target parking space. [Means for solving the problem]

[0007] In order to solve the above problem, according to one aspect of the present disclosure, there is provided a parking assistance device that assists in parking a vehicle, comprising one or more processors and one or more memories communicatively connected to the one or more processors, wherein the one or more processors perform a blind spot area identification process that identifies blind spot areas that are blind spots as seen from the vehicle and unrecognized blind spot areas that are blind spot areas not recognized by the driver of the vehicle, a danger determination processing unit that determines the presence or absence of hazardous objects that may pose an obstacle to the vehicle based on information about the surroundings of a target parking space obtained from an aircraft-mounted camera mounted on an unmanned aircraft, and a display process that superimposes information about hazardous objects present in the blind spot areas on a display unit so that it overlaps with the scenery as seen by the driver, and displays information about hazardous objects present in the unrecognized blind spot areas by providing a sub-window area within the display unit.

[0008] In addition, in order to solve the above problem, according to another aspect of the present disclosure, there is provided a parking assistance system in which a control device mounted on a vehicle and an aircraft control device mounted on an unmanned aircraft communicate with each other to assist in parking the vehicle, the parking assistance system comprising: an ambient environment detection processing unit that detects the ambient environment of a target parking space; a blind spot area identification processing unit that identifies blind spot areas that are blind spots as seen from the vehicle and unrecognized blind spot areas that are blind spot areas not recognized by the driver of the vehicle; a danger determination processing unit that determines the presence or absence of dangerous objects that may pose an obstacle to the vehicle based on information about the surroundings of the target parking space obtained from an aircraft-mounted camera mounted on the unmanned aircraft; and a display processing unit that superimposes information about dangerous objects present in the blind spot areas on a display unit provided on the vehicle so that it overlaps with the scenery as seen by the driver, and displays information about dangerous objects present in the unrecognized blind spot areas by providing a sub-window area within the display unit.

[0009] In addition, in order to solve the above problem, according to another aspect of the present disclosure, there is provided a parking assistance system in which a parking assistance device mounted on a vehicle communicates with an aircraft control device mounted on an unmanned aircraft and an information processing device that stores information within a motor pool, and assists a driver in parking the vehicle, the parking assistance system comprising: an ambient environment detection processing unit that detects the ambient environment of a target parking space; a blind spot area identification processing unit that identifies blind spot areas that are blind spots as seen from the vehicle and unrecognized blind spot areas that are blind spot areas not recognized by the driver of the vehicle, based on information about the ambient environment of the target parking space and map data around the target parking space acquired from the information processing device; a danger determination processing unit that determines the presence or absence of a hazardous object that may be an obstacle to the vehicle, based on information about the surroundings of the target parking space acquired from an aircraft-mounted camera mounted on the unmanned aircraft; and a display processing unit that superimposes information about hazardous objects present in the blind spot areas on a display unit provided on the vehicle so as to overlap with the scenery as seen by the driver, and displays information about hazardous objects present in the unrecognized blind spot areas by providing a sub-window area within the display unit. Effect of the Invention

[0010] As described above, according to the present disclosure, it is possible to provide a parking assistance device and a parking assistance system that can safely park a vehicle when a blind spot for the driver of the vehicle exists around a target parking space. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a parking assistance system according to a first embodiment of the present disclosure. [Diagram 2] A block diagram showing an example of the configuration of an unmanned aerial vehicle according to this embodiment. [Diagram 3] 4 is a flowchart showing an example of parking assistance control for an unmanned aerial vehicle by the parking assistance system according to the present embodiment. [Figure 4]4 is a flowchart showing an example of parking assistance control for an unmanned aerial vehicle by the parking assistance system according to the present embodiment. [Diagram 5] 1 is a schematic diagram showing an example of the operation of an unmanned aerial vehicle. [Figure 6] 1 is a schematic diagram showing an example of the operation of an unmanned aerial vehicle. [Figure 7] 1 is a schematic diagram showing an example of the configuration of a vehicle according to an embodiment of the present invention; [Figure 8] 1 is a block diagram showing an example of the configuration of a parking assistance device according to an embodiment of the present invention; [Figure 9] 10 is a flowchart illustrating an example of a process for generating a learned model by the parking assistance device according to the present embodiment. [Figure 10] 4 is a flowchart showing an example of parking assistance control for a vehicle performed by the parking assistance device according to the present embodiment. [Figure 11] 4 is a flowchart showing an example of parking assistance control for a vehicle performed by the parking assistance device according to the present embodiment. [Figure 12] 5 is a flowchart showing an example of a target parking space setting process performed by the parking assistance device according to the present embodiment. [Figure 13] 5 is a flowchart showing an example of a blind spot area identification process performed by the parking assistance device according to the present embodiment. [Figure 14] 5 is a flowchart showing an example of a risk determination process and a display process performed by the parking assistance device according to the present embodiment. [Figure 15] 5A to 5C are diagrams showing specific examples of display modes by the parking assistance device according to the embodiment. [Figure 16] 5A to 5C are diagrams showing specific examples of display modes by the parking assistance device according to the embodiment. [Figure 17] 5A to 5C are diagrams showing specific examples of display modes by the parking assistance device according to the embodiment. [Figure 18] 5A to 5C are diagrams showing specific examples of warning displays by the parking assistance device according to the embodiment. [Figure 19] FIG. 4 is a schematic diagram illustrating a configuration example of a parking assistance system according to a second embodiment of the present disclosure. [Figure 20]1 is a block diagram showing an example of the configuration of an information processing device according to an embodiment of the present invention; [Figure 21] 5 is a flowchart showing an example of parking assistance control by the information processing device according to the present embodiment. [Figure 22] 4 is a flowchart showing an example of parking assistance control for a vehicle performed by the parking assistance device according to the present embodiment. [Figure 23] 13 is a flowchart showing an example of a target parking space setting process according to the first modified example. [Figure 24] 11 is a flowchart showing an example of a blind spot area identification process according to Modification 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the dimensions and scale of each part in the drawings may differ from the actual dimensions. The drawings may be shown in schematic form to facilitate understanding. Furthermore, the scope of the present disclosure is not limited to the following exemplary embodiments unless otherwise specified to limit the present disclosure.

[0013] <<1. First embodiment>> First, a parking assistance system according to a first embodiment of the present disclosure will be described. FIG. 1 is a diagram showing a schematic diagram of an example of a parking assistance system 1 and a motor pool MP. The parking assistance system 1 has a parking assistance device 11 provided in a vehicle 10, and an unmanned aerial vehicle 20. The parking assistance system 1 according to this embodiment is a system that assists a driver in parking the vehicle 10 in a parking space in the motor pool MP. When the parking assistance device 11 enters the motor pool MP, it connects to the unmanned aerial vehicle 20 so that they can communicate with each other. The unmanned aerial vehicle 20 waits at a predetermined waiting position P until it receives a flight start command from the vehicle 10.

[0014] The unmanned aerial vehicle 20 is not limited to a mode in which it waits at a predetermined waiting position P. For example, the unmanned aerial vehicle 20 may wait at a waiting position provided on the vehicle 10, and fly out from the waiting position of the vehicle 10 upon receiving a flight start command from the vehicle 10.

[0015] <1-1. Unmanned aerial vehicles> (composition) The unmanned aerial vehicle 20 has an aircraft control device 21 and an aircraft-mounted camera 22. The unmanned aerial vehicle 20 is a moving object that is controlled by a computer and can fly automatically toward a predetermined target position. Representative examples of the unmanned aerial vehicle 20 include a so-called drone or a small helicopter, but are not limited to these examples. The unmanned aerial vehicle 20 shown in FIG. 1 is a so-called drone, and has, for example, a motor that rotates each of four propellers.

[0016] The aircraft-mounted camera 22 is driven by the aircraft control device 21 and generates image data of the captured image of the shooting range. The aircraft control device 21 has one or more processors. The aircraft control device 21 controls the flight of the unmanned aircraft 20 and controls the shooting by the aircraft-mounted camera 22.

[0017] 2 is a block diagram showing an example of the configuration of the unmanned aerial vehicle 20. The unmanned aerial vehicle 20 has an aerial vehicle control device 21, an aerial vehicle-mounted camera 22, a position detection sensor 23, an aerial vehicle status sensor 24, and a voice input device 25.

[0018] The aircraft-mounted camera 22 is a camera equipped with an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and generates image data of the shooting range. The aircraft-mounted camera 22 is driven by the aircraft control device 21 to generate image data, and outputs the image data to the aircraft control device 21. The aircraft-mounted camera 22 may be a so-called monocular camera, or may be a pair of left and right stereo cameras.

[0019] The position detection sensor 23 receives satellite signals from positioning satellites of the Global Navigation Satellite System (GNSS), such as Global Positioning System (GPS) satellites. The position detection sensor 23 transmits the position information of the unmanned aerial vehicle 20 contained in the received satellite signals to the aircraft control device 21. Note that the position detection sensor 23 may include an antenna that receives satellite signals from other satellite systems that identify the position of the unmanned aerial vehicle 20, in addition to the GPS sensor.

[0020] The flying object status sensor 24 is composed of one or more sensors that detect the acceleration in three axial directions along the forward / backward, left / right, and height directions of the unmanned flying object 20, and the angular velocity around the three axes. The flying object status sensor 24 is sometimes called an inertial measurement unit. The flying object status sensor 24 transmits a sensor signal indicating the measured status to the flying object control device 21.

[0021] The aircraft control device 21 includes a processing device 121, a communication device 122, and a storage device 123. The communication device 122 is an interface for communicating with the parking assistance device 11 of the vehicle 10 via wireless communication means. The processing device 121 includes one or more processors such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0022] The processing device 121 functions as a device that controls the operation of the unmanned aerial vehicle 20 by one or more processors executing a computer program. The computer program is a computer program for causing the processor to execute the operations to be performed by the processing device 121, which will be described later. The computer program executed by the processor may be recorded on a recording medium that functions as a storage unit (memory) provided in the processing device 121, or may be recorded on a recording medium built into the processing device 121 or any recording medium that can be externally attached to the processing device 121.

[0023] Examples of recording media for recording computer programs include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs (Compact Disk Read Only Memory), DVDs (Digital Versatile Disks), and Blu-ray (registered trademark); magneto-optical media such as floptical disks; memory elements such as RAMs (Random Access Memory) and ROMs (Read Only Memory); and flash memories such as USB (Universal Serial Bus) memories and SSDs (Solid State Drives), as well as other media capable of storing programs.

[0024] The storage device 123 includes one or more memories, and stores computer programs executed by the processing device 121, various parameters used in the arithmetic processing, and information on the results of the arithmetic processing. A part of the storage device 123 is used as a work area for the processing device 121.

[0025] The storage device 123 may be a magnetic medium such as a hard disk, a floppy disk, or a magnetic tape, an optical recording medium such as a CD-ROM, a DVD, or a Blu-ray (registered trademark), a magneto-optical medium such as a floppy disk, a memory element such as a RAM or a ROM, a flash memory such as a USB memory or an SSD, or other recording medium.

[0026] (Functional configuration of processing device) The processing device 121 functionally includes a communication control unit 121A, a flight control unit 121B, and an ambient environment detection processing unit 121C. The functions of these units are realized by the execution of a computer program by the processor.

[0027] The communication control unit 121A communicates with the parking assistance device 11 of the vehicle 10 and transmits and receives predetermined information. The flight control unit 121B controls the flight of the unmanned aerial vehicle 20. Specifically, the flight control unit 121B flies the unmanned aerial vehicle 20 in accordance with a flight start command transmitted from the parking assistance device 11 of the vehicle 10. The flight control unit 121B also executes a process of moving the unmanned aerial vehicle 20 to a position where it can photograph the surroundings of the target parking space acquired together with the flight start command.

[0028] The flight control unit 121B according to this embodiment executes control to move the unmanned aerial vehicle 20 to a specified location based on position information (e.g., latitude and longitude) of the location specified by the parking assistance device 11 and information on the current location of the unmanned aerial vehicle 20 (e.g., latitude and longitude). Specifically, the flight control unit 121B obtains a vector from the current location of the unmanned aerial vehicle 20 to the specified location based on the position information of the specified location specified by the parking assistance device 11 and information on the current location of the unmanned aerial vehicle 20. The flight control unit 121B controls the output of the motor that drives the propeller based on the obtained vector, and causes the unmanned aerial vehicle 20 to fly toward the target parking space.

[0029] The flight control unit 121B also executes a process of returning the unmanned aerial vehicle 20 to a predetermined waiting position P. When the waiting position of the unmanned aerial vehicle 20 is set on the vehicle 10, the flight control unit 121B may move the unmanned aerial vehicle 20 to the vicinity of the vehicle 10 based on the position information of the vehicle 10 transmitted from the parking assistance device 11, and land the unmanned aerial vehicle 20 at the waiting position set on the vehicle 10.

[0030] The flight control unit 121B controls the height, direction of movement, and speed of movement of the unmanned aerial vehicle 20 by individually controlling the output of each motor corresponding to each of the multiple propellers. While controlling the flight of the unmanned aerial vehicle 20, the flight control unit 121B acquires acceleration and angular velocity information transmitted from the aircraft status sensor 24, and controls the attitude of the unmanned aerial vehicle 20. The unmanned aerial vehicle 20 is also equipped with a distance sensor (not shown) capable of measuring at least the distance to the ground vertically below the unmanned aerial vehicle 20. The flight control unit 121B controls the flight of the unmanned aerial vehicle 20 while maintaining the unmanned aerial vehicle 20 at a predetermined height.

[0031] The surrounding environment detection processing unit 121C detects the surrounding environment of the target parking space based on information about the surroundings of the target parking space acquired from the aircraft-mounted camera 22 mounted on the unmanned aerial vehicle 20. Specifically, the surrounding environment detection processing unit 121C acquires image data of the surroundings of the target parking space from the aircraft-mounted camera 22, and performs mapping processing to map the surroundings of the target parking space based on the image data.

[0032] The mapping process by the surrounding environment detection processing unit 121C may be performed by a conventionally known image processing technique, and is not particularly limited. In the mapping process, the surrounding environment detection processing unit 121C generates an environmental map of the surroundings of the target parking space. The environmental map includes information on the positions (arrangement) and heights of objects existing around the target parking space. The "environmental map" according to this embodiment is map data that represents the positions and shapes of various objects existing around the parking space, and the same applies in the following explanation.

[0033] (Parking assistance control for unmanned aerial vehicles) 3 and 4 are flowcharts showing an example of parking assist control by the processing device 121 of the unmanned aerial vehicle 20. Hereinafter, an example of parking assist control of the unmanned aerial vehicle 20 will be described with reference to FIGS.

[0034] The processing device 121 of the flying object control device 21 causes the unmanned flying object 20 to wait at a predetermined waiting position P until it receives a flight start command from the parking assistance device 11. In other words, the processing device 121 stops all functions except for the function of receiving the flight start command from the parking assistance device 11.

[0035] 5 and 6 are schematic diagrams showing an example of the motor pool MP. The flight control unit 121B receives position information of the target parking space SP and a flight start command from the parking assistance device 11 (step S1).

[0036] Next, the flight control unit 121B executes a process to move the unmanned aerial vehicle 20 into the air above the target parking space SP based on the position information of the target parking space SP obtained from the parking assistance device 11 and the information on the current position of the unmanned aerial vehicle 20 detected by the position detection sensor 23 (step S2).

[0037] When the unmanned aerial vehicle 20 moves to the sky above the target parking space SP, the processing device 121 causes the camera 22 mounted on the aerial vehicle to capture images of the surroundings of the target parking space SP at a predetermined calculation cycle and acquires image data (step S3), as shown in Fig. 5. In the processing of step S3, the unmanned aerial vehicle 20 flies around the target parking space SP, and the camera 22 mounted on the aerial vehicle captures images of the surroundings of the target parking space SP.

[0038] Next, the surrounding environment detection processing unit 121C executes a surrounding environment detection process to map the surroundings E1 of the target parking space SP based on the image data acquired in the previous step S3 (step S4), and transmits information on the mapping result to the parking assistance device 11 (step S5). Note that the range to be mapped is not limited to the range E1 shown in FIG.

[0039] Next, the flight control unit 121B receives information on the blind spot area and the unrecognized blind spot area and a movement command from the parking assistance device 11 (step S6). The information on the blind spot area and the unrecognized blind spot area includes information on the position and range of the blind spot area and the unrecognized blind spot area.

[0040] The above-mentioned "blind spot area" means an absolute blind spot that exists from the viewpoint of the driver of the vehicle 10. Also, the above-mentioned "unrecognized blind spot area" means a blind spot that is not within the field of view of the driver of the vehicle 10 or is not recognized by the driver of the vehicle 10. The meanings of the blind spot area and the unrecognized blind spot area are the same in the following explanations.

[0041] Next, the flight control unit 121B executes a process to move the unmanned aerial vehicle 20 to a position where the camera 22 mounted on the aircraft can photograph the blind spot areas and unrecognized blind spot areas based on information regarding the blind spot areas and unrecognized blind spot areas and information regarding the current position of the unmanned aerial vehicle 20 detected by the position detection sensor 23 (step S7).

[0042] When the unmanned aerial vehicle 20 moves to a position where the aerial vehicle-mounted camera 22 can capture the blind spot area and the unrecognized blind spot area, the processing device 121 causes the aerial vehicle-mounted camera 22 to capture the blind spot area and the unrecognized blind spot area from the sky (step S8) and acquires image data. The processing device 121 transmits the image data to the parking assistance device 11 (step S9).

[0043] Next, the flight control unit 121B executes a process to move the unmanned aerial vehicle 20 to a position where the aircraft-mounted camera 22 can overlook the entire target parking space SP, including the blind spot area and the unrecognized blind spot area (hereinafter referred to as an overlooking position) (step S10). For example, the flight control unit 121B increases the altitude of the unmanned aerial vehicle 20 and expands the shooting range of the aircraft-mounted camera 22 so that the entire periphery of the target parking space SP is included in the shooting range of the aircraft-mounted camera 22.

[0044] Next, the processing device 121 judges whether or not the environment around the target parking space SP is likely to change based on the information about the surroundings of the target parking space SP (step S11). If the processing device 121 judges that the environment around the target parking space SP is likely to change (YES in step S11), it transmits information about environmental change factors that may change the environment to the parking assistance device 11 (step S12). On the other hand, if the processing device 121 judges that the environment around the target parking space SP is not likely to change (NO in step S11), it executes the processing of step S13 described later.

[0045] As a specific example of the processing of steps S11 and S12, as shown in Figs. 5 and 6, when another vehicle is parked near the target parking space SP and the engine of the other vehicle is started or is in an idling state, the voice input device 25 of the unmanned aerial vehicle 20 detects the sound of the other vehicle. The processing device 121 detects that the other vehicle (environmental change factor) is emitting a sound based on the information of the sound detected by the voice input device 25. The processing device 121 identifies the other vehicle in which an occupant is present from image data obtained by photographing the surroundings of the target parking space SP by the camera 22 mounted on the aircraft. In this case, the processing device 121 determines that the other vehicle (environmental change factor) may move, and transmits information capable of identifying the other vehicle that may move to the parking assistance device 11.

[0046] In addition, if the processing device 121 detects another vehicle, pedestrian, etc. approaching the target parking space SP, it determines that the environment around the target parking space SP may change, and transmits information about the environmental change factor to the parking assistance device 11, treating the other vehicle or pedestrian, etc. as an environmental change factor.

[0047] Next, the processing device 121 determines whether or not a return command has been received from the parking assistance device 11 to return the unmanned aerial vehicle 20 to a predetermined waiting position P (step S13). If the processing device 121 determines that a return command has been received from the parking assistance device 11 (YES in step S13), the flight control unit 121B executes control to return the unmanned aerial vehicle 20 to the waiting position P (step S14). On the other hand, if the processing device 121 determines that a return command has not been received from the parking assistance device 11 (NO in step S13), it repeatedly executes the processing of the previous steps S8 to S13 at a predetermined calculation cycle until a return command is received.

[0048] <1-2.Vehicles> (composition) Fig. 7 is a schematic diagram showing a configuration example of a vehicle 10 equipped with a parking assistance device 11. The vehicle 10 shown in Fig. 7 is configured as a four-wheel drive vehicle in which driving torque output from a driving force source 9 that generates driving torque for the vehicle 10 is transmitted to a left front wheel 3LF, a right front wheel 3RF, a left rear wheel 3LR, and a right rear wheel 3RR (hereinafter collectively referred to as "wheels 3" unless a distinction is required). The driving force source 9 may be an internal combustion engine such as a gasoline engine or a diesel engine, may be a driving motor, or may be equipped with both an internal combustion engine and a driving motor.

[0049] The vehicle 10 may be an electric vehicle equipped with two drive motors, for example, a front-wheel drive motor and a rear-wheel drive motor, or an electric vehicle equipped with drive motors corresponding to each of the wheels 3. If the vehicle 10 is an electric vehicle or a hybrid electric vehicle, the vehicle 10 is equipped with a secondary battery that stores power supplied to the drive motors, and a generator such as a motor or fuel cell that generates power to charge the battery.

[0050] The vehicle 10 is equipped with a driving force source 9, an electric steering device 15, and brake devices 17LF, 17RF, 17LR, 17RR (hereinafter collectively referred to as "brake devices 17" unless a distinction is required) as devices used to control the operation of the vehicle 10. The driving force source 9 outputs a driving torque that is transmitted to the front wheel drive shaft 5F and the rear wheel drive shaft 5R via a transmission (not shown) and a front wheel differential mechanism 7F and a rear wheel differential mechanism 7R. The operation of the driving force source 9 and the transmission is controlled by a vehicle control unit 41 that includes one or more electronic control units (ECUs: Electronic Control Units).

[0051] The front-wheel drive shaft 5F is provided with an electric steering device 15. The electric steering device 15 includes an electric motor (not shown) and a gear mechanism, and is controlled by a vehicle control unit 41 to adjust the steering angles of the left front wheel 3LF and the right front wheel 3RF. During manual driving, the vehicle control unit 41 controls the electric steering device 15 based on the steering angle of the steering wheel 13 by the driver. During automatic driving control, the vehicle control unit 41 controls the electric steering device 15 based on a set driving trajectory.

[0052] Brake devices 17LF, 17RF, 17LR, and 17RR apply braking forces to the front, rear, left, and right drive wheels 3LF, 3RF, 3LR, and 3RR, respectively. The brake devices 17 are configured as hydraulic brake devices, for example, and generate a predetermined braking force by controlling the hydraulic pressure supplied to each brake device 17 by the vehicle control unit 41. When the vehicle 10 is an electric vehicle or a hybrid electric vehicle, the brake devices 17 are used in combination with regenerative braking by the drive motor.

[0053] The vehicle control unit 41 includes one or more electronic control devices that control the driving of the driving force source 9 that outputs the driving torque of the vehicle 10, the electric steering device 15 that controls the steering wheel 13 or the steering angle of the steering wheels, and the brake device 17 that controls the braking force of the vehicle 10. The vehicle control unit 41 may have a function of controlling the driving of a transmission that changes the speed of the output output from the driving force source 9 and transmits it to the wheels 3. The vehicle control unit 41 is configured to be able to acquire information transmitted from the parking assistance device 11, and is configured to be able to execute automatic driving control of the vehicle 10.

[0054] The vehicle 10 also includes front photographing cameras 31LF, 31RF, a rear photographing camera 31R, side cameras 31LS, 31RS, an interior photographing camera 33, a vehicle state sensor 35, a navigation system 39, and an image display device 43.

[0055] The front photographing cameras 31LF, 31RF, the rear photographing camera 31R, and the side cameras 31LS, 31RS constitute an ambient environment sensor 31 for acquiring information on the ambient environment of the vehicle 10. The front photographing cameras 31LF, 31RF and the rear photographing camera 31R photograph the front or rear of the vehicle 10 and generate image data. The side cameras 31LS, 31RS are provided on the side mirrors 11L, 11R, photograph the left rear or right rear of the vehicle 10, and generate image data.

[0056] The front photographing cameras 31LF, 31RF, the rear photographing camera 31R, and the side cameras 31LS, 31RS are equipped with imaging elements such as CCD or CMOS, and transmit the generated image data to the parking assistance device 11. In the vehicle 10 shown in Fig. 7, the front photographing cameras 31LF, 31RF are configured as stereo cameras including a pair of left and right cameras, and the rear photographing camera 31R is configured as a so-called monocular camera, but each may be either a stereo camera or a monocular camera.

[0057] In addition to the front photographing cameras 31LF, 31RF, the rear photographing camera 31R, and the side cameras 31LS, 31RS, the vehicle 10 may be equipped with one or more sensors selected from a radar sensor such as a LiDAR (Light Detection And Ranging) and a millimeter wave radar, and an ultrasonic sensor, as an ambient environment sensor 31 for acquiring information on the ambient environment of the vehicle 10. The ambient environment sensor 31 may also be configured to have an anemometer for measuring wind pressure.

[0058] The interior camera 33 captures images of the interior of the vehicle 10 and generates image data. The interior camera 33 includes an imaging element such as a CCD or a CMOS, and transmits the generated image data to the parking assistance device 11. In this embodiment, the interior camera 33 is disposed so as to be able to capture an image of at least the driver. Only one interior camera 33 may be installed, or multiple interior cameras 33 may be installed.

[0059] The vehicle state sensor 35 is composed of at least one sensor that detects the operation state and behavior of the vehicle 10. The vehicle state sensor 35 includes at least one of a steering angle sensor, an accelerator position sensor, a brake stroke sensor, a brake pressure sensor, and an engine speed sensor, and detects the operation state of the vehicle 10, such as the steering angle of the steering wheel or steering wheels, the accelerator opening, the brake operation amount, and the engine speed. The vehicle state sensor 35 also includes at least one of a vehicle speed sensor, an acceleration sensor, and an angular velocity sensor, and detects the behavior of the vehicle, such as the vehicle speed, the longitudinal acceleration, the lateral acceleration, and the yaw rate. The vehicle state sensor 35 transmits a sensor signal including the detected information to the parking assistance device 11.

[0060] The position detection sensor 37 receives satellite signals from GNSS positioning satellites, such as GPS satellites. The position detection sensor 37 transmits position information of the vehicle 10 contained in the received satellite signals to the parking assistance device 11. Note that the position detection sensor 37 may be provided with an antenna that receives satellite signals from other satellite systems that identify the position of the vehicle 10, in addition to the GPS sensor.

[0061] The navigation system 39 is a known navigation system that sets a driving route to a destination set by the driver of the vehicle 10 and notifies the driver of the driving route. The navigation system 39 notifies information for guiding the vehicle 10 to the destination, for example, by image display and audio output. In this embodiment, the images and text information displayed by the navigation system 39 are displayed by an image display device 43. However, the display by the navigation system 39 may be performed by a display device other than the image display device 43.

[0062] The image display device 43 is driven by the parking assistance device 11 and displays various pieces of information visible to the driver of the vehicle 10. The image display device 43 may be a display device made of an optical panel such as a liquid crystal panel installed on the dashboard or instrument panel, or may be a head-up display (HUD) that displays on a display unit 44 described later.

[0063] The parking assistance device 11 functions as a device that assists the driver in parking the vehicle by executing a computer program by one or more processors. The computer program is a computer program for causing the processor to execute the operations to be performed by the parking assistance device 11, which will be described later. The computer program executed by the processor may be recorded on a recording medium that functions as a storage unit (memory) provided in the parking assistance device 11, or may be recorded on a recording medium built into the parking assistance device 11 or any recording medium that can be externally attached to the parking assistance device 11.

[0064] Examples of recording media for recording computer programs include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs, DVDs, and Blu-ray (registered trademark); magneto-optical media such as floptical disks; memory elements such as RAMs and ROMs; flash memories such as USB memories and SSDs; and other media capable of storing programs.

[0065] Fig. 8 is a block diagram showing an example of the configuration of the parking assistance device 11 according to this embodiment. As shown in Fig. 8, the vehicle 10 includes the parking assistance device 11, a surrounding environment sensor 31, an interior camera 33, a vehicle state sensor 35, a position detection sensor 37, a navigation system 39, a vehicle control unit 41, an image display device 43, and a display unit 44.

[0066] The display unit 44 is a display surface capable of displaying an image that can be superimposed on the scenery outside the vehicle as seen by the driver. For example, the display unit 44 is a window glass, a rear window glass, a side window glass, or the like of the vehicle 10.

[0067] The parking assistance device 11 is connected to an ambient environment sensor 31, an in-vehicle camera 33, a vehicle state sensor 35, a position detection sensor 37, and a navigation system 39 directly or via a communication means such as a controller area network (CAN) or a local internet (LIN). A vehicle control unit 41 and an image display device 43 are also connected to the parking assistance device 11. The parking assistance device 11 is not limited to an electronic control device mounted on the vehicle 10, and may be a terminal device such as a smartphone or a wearable device.

[0068] The parking assistance device 11 has a control device 111, a communication device 112, a storage device 113, a map data storage device 114, and a learning data storage device 115. The control device 111 has one or more processors such as a CPU and various peripheral components. A part or all of the control device 111 may be configured with updatable firmware or the like, or may be a program module or the like executed by a command from the CPU or the like.

[0069] The communication device 112 is an interface for communicating with the flying object control device 21 via wireless communication means. The parking assistance device 11 transmits and receives information to and from the flying object control device 21 via the communication device 112.

[0070] The storage device 113 is configured by one or more storage elements such as RAM or ROM connected to the control device 111 so as to be able to communicate with the control device 111, or storage media such as a hard disc drive (HDD), a compact disc (CD), a DVD, an SSD, a USB flash, or a storage device. However, the type and number of the storage devices 113 are not particularly limited. The storage device 113 stores information such as computer programs executed by the control device 111, various parameters used in arithmetic processing, detection data, and arithmetic results. A part of the storage device 113 is used as a work area for the control device 111.

[0071] The map data storage device 114 is configured by a storage element such as a RAM or a ROM communicably connected to the control device 111, or a storage medium such as an HDD, a CD, a DVD, an SSD, a USB flash, or a storage device. The map data stored in the map data storage device 114 is configured to be associated with the position of the vehicle 10 based on position information detected by the position detection sensor 37. For example, the map data is associated with latitude and longitude information, and the control device 111 can identify the position of the vehicle 10 on the map data based on the latitude and longitude information of the vehicle 10 detected by the position detection sensor 37.

[0072] The learning data storage device 115 is configured with an updatable storage medium such as a RAM or HDD, or a CD, DVD, SSD, USB flash, or storage device, and stores the learned model LD. The learned model LD is, for example, learning data obtained as a result of the control device 111 learning at least one of the line of sight and the face direction of the driver of the vehicle 10 during parking operation.

[0073] (Functional configuration of the control device) The control device 111 functionally includes a communication control unit 111A, a target parking space setting unit 111B (hereinafter referred to as the setting unit 111B), a flight instruction unit 111C, a blind spot area identification processing unit 111D, a risk judgment processing unit 111E, a surrounding environment detection unit 111F, a mapping processing unit 111G, a driving state detection unit 111H, a driver detection unit 111I, a display processing unit 111J, and a learning unit 111K. The functions of each of these units are realized by the execution of a computer program by a processor. Note that some of the communication control unit 111A, the setting unit 111B, the flight instruction unit 111C, the blind spot area identification processing unit 111D, the risk judgment processing unit 111E, the surrounding environment detection unit 111F, the mapping processing unit 111G, the driving state detection unit 111H, the driver detection unit 111I, the display processing unit 111J, and the learning unit 111K may be configured by hardware such as an analog circuit.

[0074] The communication control unit 111A communicates with the aircraft control device 21 and transmits and receives various information. The setting unit 111B sets the parking space designated by the driver of the vehicle 10 as the target parking space SP. For example, when the image display device 43 is configured using a touch panel that accepts input operations by the driver of the vehicle 10, the driver performs an operation to select an arbitrary parking space in the planar data of the motor pool MP displayed on the image display device 43 as the target parking space SP, and the setting unit 111B sets the parking space as the target parking space SP. When the parking position of the vehicle 10 is designated in advance, the setting unit 111B may detect that the vehicle 10 has entered the motor pool MP and set the designated parking position as the target parking space SP.

[0075] The flight instruction unit 111C transmits to the aircraft control device 21 position information of the target parking space SP specified by the driver of the vehicle 10 and command information to move the unmanned aircraft 20 to the target parking space.

[0076] The blind spot area identification processing unit 111D executes a blind spot area identification process that identifies a blind spot area that is a blind spot as seen by the driver of the vehicle 10 and an unrecognized blind spot area that is a blind spot area that is not within the field of view of the driver of the vehicle 10 or a blind spot area that is not recognized by the driver of the vehicle 10. In the blind spot area identification process, the blind spot area identification processing unit 111D identifies an unrecognized blind spot area around the target parking space SP based on the learned model LD.

[0077] The danger judgment processing unit 111E performs danger judgment processing to determine the presence or absence of dangerous objects that could pose an obstacle to the vehicle 10 based on information about the surroundings of the target parking space SP obtained from the aircraft-mounted camera 22 mounted on the unmanned airborne vehicle 20.

[0078] The surrounding environment detection unit 111F detects information on the surrounding environment of the target parking space SP. The information on the surrounding environment of the target parking space SP includes, for example, information on obstacles existing around the target parking space SP and obstructions that cause blind spots around the target parking space SP. Specifically, the surrounding environment detection unit 111F detects obstacles or obstructions existing around the target parking space SP based on detection data acquired from the surrounding environment sensor 31. In addition, the surrounding environment detection unit 111F calculates the position of the obstacle or obstruction as seen from the vehicle 10, the distance from the vehicle 10 to the obstacle or obstruction, and the like.

[0079] The obstacles or obstructions detected by the surrounding environment detection unit 111F include other vehicles (parked vehicles), buildings, utility poles, street lights, traffic signs, natural objects, and any other detection targets existing around the target parking space SP. The surrounding environment detection unit 111F may also detect white lines that define the parking space together with the obstacles or obstructions.

[0080] The mapping processing unit 111G executes a mapping process to map the surroundings of the target parking space SP based on the detection result detected by the surrounding environment detection unit 111F. The mapping process by the mapping processing unit 111G may be executed by a conventionally known image processing technique, and is not particularly limited. In the mapping process, the mapping processing unit 111G generates an environmental map of the surroundings of the target parking space SP. The environmental map includes information on the position (arrangement), width, and depth of obstacles or obstructions existing around the target parking space SP.

[0081] The mapping processing unit 111G generates a three-dimensional environmental map (hereinafter referred to as map data) by combining information on the mapping result obtained by mapping the periphery of the target parking space SP by the surrounding environment detection processing unit 121C of the unmanned aerial vehicle 20 and information on the mapping result obtained by mapping the periphery of the target parking space SP. The map data includes information on the position (arrangement), height, width, and depth of obstacles or obstructions that exist around the target parking space SP.

[0082] The map data is three-dimensional map data that indicates the positions and shapes of various objects such as obstacles and obstructions that exist around the target parking space SP, and is expressed, for example, in a three-dimensional spatial coordinate system of X-, Y-, and Z-axes.

[0083] The driving state detection unit 111H detects information on the operation state and behavior of the vehicle 10 based on the detection data acquired from the vehicle state sensor 35. The driving state detection unit 111H acquires information on the operation state of the vehicle 10, such as the steering angle of the steering wheel or steering wheels, the accelerator opening, the amount of brake operation, or the engine speed, as well as information on the behavior of the vehicle 10, such as the vehicle speed, the longitudinal acceleration, the lateral acceleration, and the yaw rate.

[0084] The driver detection unit 111I detects the state of the driver of the vehicle 10. Specifically, the driver detection unit 111I detects the line of sight and the face direction of the driver of the vehicle 10 during a parking operation, based on detection data transmitted from the interior photographing camera 33.

[0085] The display processing unit 111J executes display processing to superimpose information about dangerous objects present in blind spot areas on the display unit 44 so that the information overlaps with the scenery as seen by the driver of the vehicle 10, and to display information about dangerous objects present in unrecognized blind spot areas by providing a sub-window area within the display unit 44.

[0086] The above-mentioned "dangerous object" is, for example, a person or other moving object approaching the target parking space SP, or a person or other moving object approaching the vehicle 10, and the same applies in the following explanation.

[0087] The display processing unit 111J executes a process to display information alerting the driver of the vehicle 10 on the display unit 44 as information visible to the driver of the vehicle 10, based on information on environmental change factors that may change the environment surrounding the target parking space SP.

[0088] The learning unit 111K constructs a trained model using a known machine learning model with the data set as training data. The machine learning model may be, for example, a random forest, or a computational model using a support vector machine, a neural network such as a nearest neighbor method or deep learning, or a Bayesian network.

[0089] (How to generate a trained model) Fig. 9 is a flowchart showing an example of a process for generating a trained model LD stored in the training data storage device 115. The control device 111 according to the present embodiment executes a learning process for learning at least one of the driver's habits of line of sight or face direction during parking operation based on an image of the driver of the vehicle 10 captured by the in-vehicle camera 33 mounted on the vehicle 10, and generates a trained model LD. Hereinafter, a method for generating the trained model LD will be described with reference to Fig. 9 as appropriate.

[0090] The driver detection unit 111I executes a process of identifying the driver of the vehicle 10 (step S21). For example, the driver detection unit 111I extracts features of the driver's face based on image data acquired from the in-vehicle camera 33, and identifies the driver. The driver detection unit 111I assigns identification information to each identified driver, or identifies identification information already assigned to the driver, and stores it in the storage device 113. In other words, the driver detection unit 111I records information that identifies which driver's correction during parking operation the learned model LD is a model that has learned. The driver detection unit 111I may identify the driver based on information input by a user such as the driver via an input device such as a touch panel or a microphone, and assign or identify the driver's identification information.

[0091] The surrounding environment detection unit 111F detects information on the surrounding environment of the parking space of the vehicle 10 based on the detection data acquired from the surrounding environment sensor 31. The mapping processing unit 111G executes a mapping process for mapping the surroundings of the parking space based on the information on the detection result detected by the surrounding environment detection unit 111F (step S22).

[0092] Next, the driver detection unit 111I detects, in chronological order, information on at least one of the line of sight and the face direction of the driver who is parking in the parking space, based on the image data acquired from the interior camera 33 (step S23).

[0093] Next, the driving state detection unit 111H detects, in chronological order, information on the operation state and behavior of the vehicle 10 attempting to park in the parking space, based on the detection data acquired from the vehicle state sensor 35 (step S24).

[0094] Next, the learning unit 111K generates one data set by chronologically associating the information on the mapping result in which the periphery of the parking space is mapped, the information on at least one of the line of sight or the face direction of the driver while parking in the parking space, and the information on the operation state and behavior of the vehicle 10 while parking in the parking space (step S25). At this time, the learning unit 111K stores the generated data set in the learning data storage device 115 in association with the driver's identification information.

[0095] Next, the learning unit 111K determines whether or not a predetermined number of data sets have been generated (step S26). The learning unit 111K may determine whether or not a predetermined number of data sets have been generated when the vehicle 10 is parked in the same parking space, or may determine whether or not a predetermined number of data sets have been generated when the vehicle 10 is parked regardless of whether the parking space is the same. If the learning unit 111K determines that the predetermined number of data sets have not been generated (NO in step S26), the learning unit 111K repeats the processing of steps S21 to S26 until the predetermined number of data sets have been generated, and generates multiple data sets.

[0096] The multiple data sets do not need to be generated in one driving cycle from when the system of the vehicle 10 is started to when it is stopped, but are generated from information collected when the same driver performs an operation to park the vehicle 10 in a predetermined parking space in multiple driving cycles. At this time, the learning unit 111K associates, in time series, the information on the mapping result, the information on the operation state and behavior of the vehicle 10, and at least one of the information on the driver's line of sight or face direction for each driver, and stores them in the learning data storage device 115 together with the driver's identification information.

[0097] On the other hand, when it is determined that a predetermined number of data sets have been generated (YES in step S26), the learning unit 111K constructs a trained model associated with the driver by training a known machine learning model using each of the generated data sets as training data (step S27). Note that the learning unit 111K may repeat the process of training the trained model with a new data set and reconstructing (updating) the machine learning model after first constructing the trained model, thereby updating the trained model at an appropriate timing.

[0098] (Vehicle parking assistance control) Figures 10 and 11 are flowcharts showing an example of parking assist control by the control device 111 of the vehicle 10. Hereinafter, the parking assist control of the vehicle 10 will be described with reference to Figures 10 to 11 as appropriate. Note that the processing of the flowcharts shown in Figures 10 to 11 is executed when the vehicle 10 approaches within a predetermined distance from a predetermined motor pool MP.

[0099] When the control device 111 of the parking assistance device 11 mounted on the vehicle 10 detects that the vehicle 10 approaches within a predetermined distance from a predetermined motor pool MP (step S31), the flight instruction unit 111C determines whether or not the flight of the unmanned aerial vehicle 20 is possible (step S32). Specifically, the flight instruction unit 111C determines that the flight of the unmanned aerial vehicle 20 is possible when, for example, the wind pressure or wind speed measured by an anemometer is less than a predetermined reference value. The flight instruction unit 111C may communicate with the aircraft control device 21 of the unmanned aerial vehicle 20, obtain information such as the state of charge (SOC) of the battery of the unmanned aerial vehicle 20, and determine whether or not the flight of the unmanned aerial vehicle 20 is possible. When the flight instruction unit 111C does not determine that the flight of the unmanned aerial vehicle 20 is possible (NO in step S32), it ends the parking assistance control because it is difficult to fly the unmanned aerial vehicle 20.

[0100] On the other hand, if the flight instruction unit 111C determines that the unmanned aerial vehicle 20 can fly (YES in step S32), the setting unit 111B sets the target parking space SP (step S33). In this embodiment, the setting unit 111B sets the parking space designated by the driver of the vehicle 10 as the target parking space SP.

[0101] 12 is a flowchart showing an example of a target parking space setting process by the setting unit 111B. The setting unit 111B causes the image display device 43 to display parking spaces selectable by the driver of the vehicle 10 from among parking spaces whose position information has been recorded in advance in the map data storage device 114 (step S331). The parking spaces are areas where no parked vehicles exist and where the vehicle 10 can be parked. Information on the parking area may be information recognized based on image data output from the front-facing photographing cameras 31LF, RF, etc. of the vehicle 10.

[0102] Next, the setting unit 111B determines whether or not any parking space has been selected by the driver of the vehicle 10 (step S332). For example, the setting unit 111B determines whether or not the driver has touched any of the parking spaces displayed on the touch panel type image display device 43. If any parking space has been selected by the driver (YES in step S332), the setting unit 111B sets the selected parking space as the target parking space SP (step S333). For example, the setting unit 111B records the latitude and longitude information of the selected parking space as the position information of the target parking space SP.

[0103] On the other hand, when the setting unit 111B does not determine that any parking space has been selected by the driver of the vehicle 10 (NO in step S332), it determines whether or not the driver has specified any parking space (step S334). For example, the setting unit 111B determines whether or not the driver has touched any parking space on the motor pool MP displayed on the touch panel type image display device 43. When an arbitrary parking space has been specified by the driver (YES in step S334), the setting unit 111B sets the specified parking space as the target parking space SP (step S335). For example, the setting unit 111B records the latitude and longitude information of the specified parking space as the position information of the target parking space SP.

[0104] On the other hand, when the setting unit 111B does not determine that any parking space has been specified by the driver (NO in step S334), it ends the target parking space setting process without setting the target parking space SP.

[0105] Returning to FIG. 10, after the target parking space setting process is completed, the flight instruction unit 111C judges whether or not the target parking space SP is set (step S34). The flight instruction unit 111C judges whether or not the position information of the target parking space SP is recorded by the setting unit 111B. If the flight instruction unit 111C judges that the target parking space SP is not set (NO in step S34), the control device 111 ends the parking assistance control by the parking assistance device 11.

[0106] On the other hand, if the flight instruction unit 111C determines that the target parking space SP has been set (YES in step S34), the flight instruction unit 111C transmits a flight start command together with the position information of the target parking space SP to the aircraft control device 21 (step S35). As a result, the aircraft control device 21 starts moving the unmanned aircraft 20 toward the target parking space SP.

[0107] Next, the surrounding environment detection unit 111F detects the surrounding environment of the target parking space SP based on the detection data acquired from the surrounding environment sensor 31 (step S36). Specifically, as the vehicle 10 approaches the target parking space SP, the control device 111 acquires image data of the surroundings of the target parking space SP from the surrounding environment sensor 31 at a predetermined calculation cycle as shown in Fig. 6, for example, and detects obstacles and obstructions existing around the target parking space SP, as well as white lines defining each parking space, and the like.

[0108] Next, the mapping processing unit 111G executes a mapping process for mapping the surroundings of the target parking space SP based on the information of the detection result detected by the surrounding environment detection unit 111F (step S37).

[0109] Next, the mapping processing unit 111G receives mapping result information generated by the surrounding environment detection processing unit 121C in the previous step S4 from the aircraft control device 21, and generates map data by matching the mapping result information mapping the area around the target parking space SP with the mapping result information generated by the surrounding environment detection processing unit 121C of the unmanned aircraft 20 (step S38).

[0110] Next, the blind spot area identification processing unit 111D executes a blind spot area identification process (step S39). Details of step S39 will be described below with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the blind spot area identification process.

[0111] The blind spot area identification processing unit 111D identifies a blind spot area existing around the target parking space SP based on the generated map data and the position information of the vehicle 10 (step S391). Specifically, the blind spot area identification processing unit 111D identifies, as the blind spot area, an absolute blind spot that exists around the target parking space SP and cannot be seen from the position of the driver of the vehicle 10.

[0112] Next, the blind spot area identification processing unit 111D inputs information on the current situation in which the vehicle 10 is placed, the position information of the vehicle 10, and information on the operation state and behavior of the vehicle 10 as input data to the learned model LD, and obtains at least one of information on the driver's line of sight or face direction assumed from the operation during the past parking operation as output data (step S392). The information on the current situation in which the vehicle 10 is placed includes information on obstacles and obstructions existing around the vehicle 10 and white lines defining each parking space. In addition, the information on the current situation in which the vehicle 10 is placed may include information on the brightness and weather of the surroundings. These pieces of information can be obtained by communication with a sensor mounted on the vehicle 10 or an external system such as a telematics service.

[0113] Next, the blind spot area identification processing unit 111D identifies an unrecognized blind spot area based on at least one of the information on the line of sight or the face direction of the driver of the vehicle 10 obtained in the previous step S392 (step S393). Specifically, the blind spot area identification processing unit 111D identifies as an unrecognized blind spot area a blind spot that is not within the field of view of the driver of the vehicle 10 among the blind spot areas identified in the previous step S391. That is, among the areas identified as blind spot areas in the previous step S391, a blind spot to which the driver directs his / her gaze is identified as a blind spot area, and a blind spot to which the driver does not direct his / her gaze is identified as an unrecognized blind spot area. In addition, the blind spot area identification processing unit 111D identifies as an unrecognized blind spot area an area not recognized by the driver of the vehicle 10, that is, an area to which the driver does not direct his / her gaze, in addition to the blind spot areas identified in the previous step S391.

[0114] Returning to FIG. 11, the flight instruction unit 111C transmits information on the identified blind spot area and unrecognized blind spot area and a movement command to the aircraft control device 21 (step S40). In step S40, the flight instruction unit 111C transmits, as information on the blind spot area and unrecognized blind spot area, for example, information on the position and range of the blind spot area and unrecognized blind spot area to the aircraft control device 21. As a result, the aircraft control device 21 starts photographing the blind spot area and unrecognized blind spot area with the aircraft-mounted camera 22, and executes a process of transmitting image data of the photographed blind spot area and unrecognized blind spot area to the control device 111.

[0115] After transmitting image data of the captured blind spot areas and unrecognized blind spot areas to the control device 111, the aircraft control device 21 moves the unmanned aircraft 20 to a position where it can capture an overhead image of the surrounding area centered on the target parking space SP, and performs a process of monitoring for the presence of environmental change factors that may alter the environment around the target parking space SP.

[0116] Next, the danger determination processing unit 111E receives image data of the blind spot area and the unrecognized blind spot area captured by the aircraft-mounted camera 22 of the unmanned aircraft 20 from the aircraft control device 21 (step S41).

[0117] Next, the control device 111 executes a risk determination process and a display process (step S42). Details of step S42 will be described below with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the risk determination process and the display process.

[0118] The danger determination processing unit 111E determines whether or not a dangerous object has been detected in each of the blind spot area and the unrecognized blind spot area based on image data of the blind spot area and the unrecognized blind spot area captured by the aircraft-mounted camera 22 (step S421).

[0119] When the danger determination processing unit 111E determines that a danger object is detected in each of the blind spot area and the unrecognized blind spot area (YES in step S421), the display processing unit 111J executes a display process to display information about the danger object (step S422). In the process of step S422, the display processing unit 111J superimposes information about the danger object present in the blind spot area on the display unit 44 so as to overlap with the scenery seen by the driver of the vehicle 10, and also displays information about the danger object present in the unrecognized blind spot area in a sub-window area provided in the display unit 44.

[0120] Fig. 15 is a diagram showing a specific example of the display by the processing of step 422. Fig. 15 shows an example of the display displayed on the rear window 181 of the vehicle 10. When the danger determination processing unit 111E determines that a dangerous object (a person H1 approaching the target parking space SP) has been detected in the blind spot area, as shown in Fig. 14, the outline of the person H1 is displayed (AR display) superimposed on the real space around the vehicle 10 as information on the dangerous object existing in the blind spot area. Instead of the outline of the person H1, an image of the person H1 photographed by the aircraft-mounted camera 22 may be displayed.

[0121] Furthermore, when the danger determination processing unit 111E determines that a dangerous object (a person H2 approaching the vehicle 10) is detected in the unrecognized blind spot area, an image of the person H2 present in the unrecognized blind spot area photographed by the aircraft-mounted camera 22 is displayed in a sub-window area W in the rear window 181, as shown in Fig. 15. Instead of the image of the person H2 photographed by the aircraft-mounted camera 22, the outline of the person H2 may be displayed.

[0122] The display processing unit 111J may provide a sub-window area W at a position corresponding to the direction of the unrecognized blind spot area in which a dangerous object (person H2) is detected as seen from the driver. For example, when the unrecognized blind spot area in which a dangerous object is detected is located to the left front as seen from the driver, the display processing unit 111J provides a sub-window area W at the upper left of the rear window 181 as shown in Fig. 15, and when the unrecognized blind spot area is located to the left or rear left as seen from the driver, the display processing unit 111J provides a sub-window area W at the lower left of the rear window 181.

[0123] Returning to FIG. 14, if the danger judgment processing unit 111E does not determine that a dangerous object is detected in either the blind spot area or the unrecognized blind spot area (NO in step S421), it determines whether or not a dangerous object is detected in the blind spot area while no dangerous object is detected in the unrecognized blind spot area (step S423).

[0124] If the danger judgment processing unit 111E determines that a dangerous object is not detected in the unrecognized blind spot area, but that a dangerous object is detected in the blind spot area (YES in step S423), the display processing unit 111J superimposes information about the dangerous object present in the blind spot area on the display unit 44 so that it overlaps with the scenery as seen by the driver of the vehicle 10 (step S424).

[0125] Fig. 16 is a diagram showing a specific example of the display by the processing of step S424. Fig. 16 shows an example of the display displayed on the rear window 181 of the vehicle 10, similar to Fig. 15. When the danger determination processing unit 111E determines that a dangerous object (a person H1 approaching the target parking space SP) has been detected in the blind spot area, the outline of the person H1 is displayed (AR display) superimposed on the real space around the vehicle 10 as information on the dangerous object existing in the blind spot area. In the situation shown in Fig. 16, since no dangerous object has been detected in the unrecognized blind spot area, the sub-window area W is not provided.

[0126] Returning to FIG. 14, if the danger judgment processing unit 111E does not determine that a dangerous object is detected in the blind spot area while no dangerous object is detected in the unrecognized blind spot area (NO in step S423), it then determines whether a dangerous object is detected in the unrecognized blind spot area while no dangerous object is detected in the blind spot area (step S425).

[0127] If the danger judgment processing unit 111E determines that a dangerous object has been detected in the unrecognized blind spot area, but that no dangerous object has been detected in the blind spot area (YES in step S425), the display processing unit 111J creates a sub-window area in the display unit 44 and displays information about the dangerous object present in the unrecognized blind spot area (step S426).

[0128] 17 is a diagram showing a specific example of the display by the process of step S426. When the danger determination processing unit 111E determines that a dangerous object (person H2 approaching the vehicle 10) is detected in the unrecognized blind spot area, an image of the person H2 present in the unrecognized blind spot area is displayed in a sub-window area W in the rear window 181. Instead of an image of the person H2 captured by the aircraft-mounted camera 22, an outline of the person H2 may be displayed. The display processing unit 111J may provide the sub-window area W at a position corresponding to the direction of the unrecognized blind spot area in which the dangerous object (person H2) is detected as seen from the driver.

[0129] On the other hand, when the risk determination processing unit 111E detects a dangerous object in the unrecognized blind spot area but does not determine that a dangerous object is not detected in the blind spot area (NO in step S425), the display processing unit 111J omits the display process and ends the risk determination process and the display process. The display processing unit 111J may display text or the like indicating that no dangerous object exists around the display unit 44 in the driver's line of sight.

[0130] Returning to FIG. 11, after the risk determination process and the display process are executed, the risk determination processing unit 111E determines whether or not the information on the environmental change factor is received from the flying object control device 21 of the unmanned flying object 20 (step S43). If it is determined that the risk determination processing unit 111E has received the information on the environmental change factor (YES in step S43), the display processing unit 111J executes a warning display process to alert the driver based on the acquired information on the environmental change factor (step S44). For example, the display processing unit 111J displays on the display unit 44 of the vehicle 10 information that there is another vehicle that may be transmitting and information on the position of the other vehicle, or that there is another vehicle or person approaching the target parking space SP or the vehicle 10 and the direction in which the other vehicle or person is approaching, as information that can be seen by the driver. On the other hand, if the risk determination processing unit 111E determines that the information on the environmental change factor is received (NO in step S43), the processing of step S45 described later is executed.

[0131] 18 is a diagram showing a specific example of a warning display by the processing of step S44. When another vehicle 30 that has started its engine is parked near the target parking space SP as an environmental change factor, the display processing unit 111J generates information to alert the driver of the vehicle 10 based on information capable of identifying the other vehicle 30 (environmental change factor) acquired from the aircraft control device 21. For example, the display processing unit 111J displays a colored frame E as the information in AR on the rear window 181, superimposed on the other vehicle 30.

[0132] 11, the control device 111 determines whether parking of the vehicle 10 in the target parking space SP is complete (step S45). If the control device 111 determines that parking of the vehicle 10 is not complete (NO in step S45), it repeatedly executes the processes of the previous steps S39 to S45 at a predetermined calculation cycle until parking of the vehicle 10 is complete.

[0133] On the other hand, if the control device 111 determines that parking of the vehicle 10 is complete (YES in step S45), it transmits a return command to the aircraft control device 21 to return the unmanned aircraft 20 to the waiting position P (step S46).

[0134] As described above, the parking assistance device 11 and parking assistance system 1 of this embodiment execute a blind spot area identification process that identifies blind spot areas that are blind spots as seen from the vehicle 10 and unrecognized blind spot areas that are blind spot areas not recognized by the driver of the vehicle 10, a danger determination processing unit that determines the presence or absence of dangerous objects that may be an obstacle to the vehicle 10 based on information about the surroundings of the target parking space SP obtained from an aircraft-mounted camera mounted on the unmanned aircraft, and a display process that superimposes information about dangerous objects present in the blind spot areas on the display unit so as to overlap with the scenery as seen by the driver, and displays information about dangerous objects present in the unrecognized blind spot areas by providing a sub-window area within the display unit.

[0135] According to the above aspect, when a dangerous object exists not only in a blind spot area but also in a blind spot area (unrecognized blind spot area) whose existence the driver is not aware of, the driver of the vehicle 10 can recognize the presence of the dangerous object. This enables the driver to perform a driving operation to avoid a collision with the dangerous object during parking operation of the vehicle 10, and the vehicle can be safely parked in the target parking space SP.

[0136] In addition, the parking assistance device 11 and the parking assistance system 1 according to the present embodiment identify an unrecognized blind spot area that is expected according to the surrounding situation of the target parking space where the vehicle is to be parked, based on learning data that is learned by associating the surrounding situation during the parking operation of the vehicle 10 with at least one of the driver's habits of line of sight or face direction for each driver. This allows the driver to learn at least one of the habits of line of sight or face direction during the parking operation, and to notify the driver of a dangerous object that exists in an area that is difficult for the driver to recognize. This makes it possible to perform a driving operation that avoids a collision with a dangerous object during the parking operation of the vehicle 10, and to safely park the vehicle in the target parking space SP.

[0137] <<2. Second embodiment>> Next, a parking assistance system according to a second embodiment of the present disclosure will be described. The parking assistance system according to this embodiment includes an external information processing device that assumes part of the functions of the parking assistance device and the aircraft control device in the parking assistance system according to the first embodiment.

[0138] 19 is a schematic diagram showing a configuration example of a parking assistance system 100 according to a second embodiment of the present disclosure. The parking assistance system 100 includes a parking assistance device 11 mounted on a vehicle 10, an unmanned aerial vehicle 20, and an information processing device 40. In the following description of the second embodiment, the description of the same configuration and control method as in the first embodiment will be omitted or simplified.

[0139] The parking assistance device 11, the aircraft control device 21 mounted on the unmanned aircraft 20, and the information processing device 40 are connected to each other so as to be able to communicate with each other via a communication network N. The communication network N is a network connected by wireless communication means, such as mobile communication or wi-fi.

[0140] The information processing device 40 is typically a management server, but is not limited to this, and may be a computer different from the management server. The information processing device 40 functions as a device that assists the driver in parking the vehicle in cooperation with the parking assistance device 11 by executing a computer program by one or more processors. The computer program is a computer program for causing the processor to execute the operation to be performed by the information processing device 40, which will be described later. The computer program executed by the processor may be recorded on a recording medium that functions as a storage unit (memory) provided in the information processing device 40, or may be recorded on a recording medium built into the information processing device 40 or any recording medium that can be externally attached to the information processing device 40.

[0141] Examples of recording media for recording computer programs include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs, DVDs, and Blu-ray (registered trademark); magneto-optical media such as floptical disks; memory elements such as RAMs and ROMs; flash memories such as USB memories and SSDs; and other media capable of storing programs.

[0142] <2-1. Configuration of information processing device> Fig. 20 is a block diagram showing a configuration example of the information processing device 40. The information processing device 40 includes a processing device 141, a communication device 142, a storage device 143, and an in-venue data storage device 144, as shown in Fig. 20.

[0143] Parked vehicle detection sensors 145a, 145b, ..., 145n that detect parked vehicles parked in each parking space in the motor pool MP are connected to the information processing device 40. The parked vehicle detection sensors 145a, 145b, ..., 145n may be object detection sensors such as ultrasonic sensors provided in each parking space, or may be sensors such as cameras or LiDAR (Light Detection and Ranging) that monitor multiple parking spaces.

[0144] The processing device 141 has one or more processors such as a CPU, etc. The processing device 141 executes a computer program stored in the storage device 143 to control all or part of the operations in the information processing device 40.

[0145] The communication device 142 is an interface for communicating with the parking assistance device 11 and the aircraft control device 21 via the communication network N.

[0146] The storage device 143 includes one or more memories, and stores computer programs executed by the processing device 141, various parameters used in arithmetic processing, and information on calculation results. A part of the storage device 143 is used as a work area for the processing device 141. The storage device 143 may be a magnetic medium such as a hard disk, a floppy disk, or a magnetic tape, an optical recording medium such as a CD-ROM, a DVD, or a Blu-ray (registered trademark), a magneto-optical medium such as a floptical disk, a memory element such as a RAM or a ROM, a flash memory such as a USB memory or an SSD, or other recording medium. A part of the storage device 143 is used as a work area for the processing device 141.

[0147] The on-site data storage device 144 is composed of a storage element such as a RAM or a ROM communicatively connected to the processing device 141, or a storage medium such as an HDD, a CD, a DVD, an SSD, a USB flash, or a storage device.

[0148] The on-site data storage device 144 stores map data in which the surrounding environment of each of the multiple parking spaces in the motor pool MP is recorded. Specifically, the on-site data storage device 144 stores map data in which the surrounding environment of each of the multiple parking spaces in the motor pool MP is pre-mapped, which remains unchanged even if the arrangement of vehicles in the motor pool MP changes. The map data is map data generated by inputting information on the surrounding environment of each parking space in advance. However, the map data may also be map data generated based on information on the surrounding environment collected by the unmanned aerial vehicle 20 when an unspecified vehicle parks in a parking space using the parking assistance function.

[0149] <2-2. Parking assistance control by information processing device> 21 is a flowchart showing an example of parking assist control by the processing device 141 of the information processing device 40. Hereinafter, the parking assist control by the processing device 141 will be described with reference to FIG.

[0150] The processing device 141 starts communication with the parking assistance device 11 of the vehicle 10 that has entered the motor pool MP, and receives a transmission request from the parking assistance device 11 requesting transmission of information about the target parking space SP (step S101).

[0151] Next, the processing device 141 searches for a parking space and determines a target parking space SP (step S102). Specifically, the processing device 141 acquires sensor signals output from the parked vehicle detection sensors 145a, 145b, . . . 145n and detects a parking space (vacant space) where no parked vehicle exists. The processing device 141 also determines a target parking space SP for the vehicle 10 from among the vacant spaces using an arbitrary algorithm. For example, the processing device 141 may determine the vacant space closest to the entrance of the motor pool MP as the target parking space SP, or may prioritize a vacant space that is used less frequently as the target parking space SP. The processing device 141 may acquire information capable of identifying the size of the vehicle 10 from the parking assistance device 11 and use the information when determining the target parking space SP.

[0152] If the parking space for the vehicle 10 has been determined in advance, the process of step S102 may be omitted.

[0153] Next, the processing device 141 reads out information on map data in which the periphery of the determined target parking space SP has been mapped in advance from the in-park data storage device 144 (step S103).

[0154] Next, the processing device 141 transmits information on the determined target parking space SP and map data around the target parking space SP to the parking assistance device 11 (step S104). The processing device 141 may also transmit information on parked vehicles existing in parking spaces adjacent to the target parking space SP or in surrounding parking spaces.

[0155] <2-3. Vehicle parking assistance control> Fig. 22 is a flowchart showing an example of parking assist control by the control device 111 of the parking assist device according to this embodiment. The flowchart shown in Fig. 22 can be replaced with the flowchart of the parking assist control of the first embodiment shown in Fig. 10, and the processes of steps S201 and S202 are executed instead of steps S33 to S34. Hereinafter, the parking assist control of the vehicle 10 according to this embodiment will be described with reference to Fig. 22 as needed.

[0156] The control device 111 executes the processes of steps S31 and S32 in the same manner as in the first embodiment. In this embodiment, the control device 111 starts communication with the information processing device 40 when it detects the entry of the vehicle 10 into the motor pool MP in step S31.

[0157] Next, the blind spot area identification processing unit 111D transmits a transmission request to request the transmission of information on the target parking space SP to the information processing device 40 (step S201).

[0158] Next, the blind spot area identification processing unit 111D acquires information on the target parking space SP determined by the information processing device 40 together with map data around the target parking space SP from the information processing device 40 (step S202).

[0159] Thereafter, the control device 111 executes the processes of steps S35 to S46 in the same manner as in the first embodiment. However, in this embodiment, since the control device 111 has acquired map data of the surroundings of the target parking space SP from the information processing device 40, it is not necessary for the unmanned aerial vehicle 20 to execute the mapping process, and the result of the mapping process acquired from the information processing device 40 and the result of the mapping process (step S37) by the control device 111 are combined to create map data (step S38).

[0160] <2-4. Parking assistance control for unmanned aerial vehicles> The processing device 121 of the unmanned aerial vehicle 20 in this embodiment performs parking assistance control in a similar procedure to the processing of steps S1 to S14, except that the processing of steps S3 to S5 in the parking assistance control in the first embodiment shown in Figures 3 and 4 is omitted.

[0161] According to the parking assistance system 100 of this embodiment, the parking assistance device 11 mounted on the vehicle 10 communicates with the aircraft control device 21 mounted on the unmanned aircraft 20 and an information processing device 40 that stores information within the motor pool MP, and assists the driver in parking the vehicle 10. The parking assistance system 100 executes an ambient environment detection process for detecting the ambient environment of the target parking space SP, a blind spot area identification process for identifying blind spot areas that are blind spots as seen from the vehicle 10 and unrecognized blind spot areas that are not recognized by the driver of the vehicle 10 based on information about the ambient environment of the target parking space SP and map data about the area around the target parking space SP acquired from the information processing device 40, a danger determination process for determining the presence or absence of dangerous objects that may pose an obstacle to the vehicle 10 based on information about the area around the target parking space SP acquired from the aircraft-mounted camera 22 mounted on the unmanned aircraft 20, and a display process for superimposing information about dangerous objects present in the blind spot areas on a display unit 44 provided on the vehicle 10 so as to overlap with the scenery as seen by the driver, and for displaying information about dangerous objects present in the unrecognized blind spot areas by providing a sub-window area W within the display unit 44.

[0162] According to the above-mentioned embodiment, when a dangerous object exists not only in a blind spot area but also in a blind spot area (unrecognized blind spot area) whose existence the driver is not aware of, the driver of the vehicle 10 can recognize the presence of the dangerous object. This enables the driver to perform a driving operation to avoid a collision with the dangerous object during parking operation of the vehicle 10, and the vehicle can be safely parked in the target parking space SP.

[0163] Moreover, according to the parking assistance system 100 of this embodiment, the information processing device 40 that manages the motor pool MP holds map data of the surroundings of each parking space in advance, and transmits it to the parking assistance device 11 in response to a transmission request from the parking assistance device 11. This eliminates the need to perform mapping processing of the surroundings of the target parking space SP every time the vehicle 10 parks, and reduces the processing load on the aircraft control device 21 of the unmanned aircraft 20 and shortens the time required for parking assistance. Furthermore, if the map data is stored in advance, the accuracy of the position and size of the surrounding environment other than the presence or absence of parked vehicles and objects whose positions may change is high, and highly accurate parking assistance can be performed.

[0164] Moreover, according to the parking assistance system 100 of this embodiment, the information processing device 40 has information on vacant spaces in the motor pool MP, and is configured to determine the target parking space SP of the vehicle 10. Therefore, the driver does not need to specify the target parking space SP, and the information processing device 40 can quickly provide the target parking space SP and map data around it to the parking assistance device 11 of the vehicle 10.

[0165] <<3. Modifications>> Although the embodiment of the present disclosure has been described above, the present disclosure is not limited to the above-described embodiment and may be modified in various ways. The above-described embodiment may adopt the following exemplary aspects.

[0166] (Variation 1) In the first embodiment, the target parking space SP is specified by the driver of the vehicle 10, and in the second embodiment, the target parking space SP is determined by the information processing device 40, but this is not limited to this and the target parking space SP may be set by other methods.

[0167] Fig. 23 is a flowchart showing an example of the target parking space setting process according to the modified example 1. The target parking space setting process according to the modified example 1 can be applied to the second embodiment that includes an information processing device 40 that holds map data around each of the parking spaces of the motor pool MP. The target parking space setting process according to the modified example 1 will be described below with reference to Fig. 23 as needed.

[0168] The setting unit 111B acquires map data around each of the multiple parking spaces in the motor pool MP from the information processing device 40 (step S341). The acquired map data does not include objects whose presence or absence or location may change, such as parked vehicles, but is map data in which the positions of objects whose presence is constant are recorded with high accuracy.

[0169] Next, the setting unit 111B acquires map data from the aircraft control device 21 of the unmanned aircraft 20, which maps the surrounding environment detected by the aircraft control device 21 for each parking space (step S342). The acquired map data is map data that reflects the presence and location of parked vehicles, etc. in real time. At this time, if the motor pool MP is large, the aircraft control device 21 may share the information on the vacant space held by the information processing device 40, and transmit only the map data around the vacant space to the parking assistance device 11.

[0170] Next, the setting unit 111B combines the map data acquired from the information processing device 40 and the map data acquired from the aircraft control device 21 to generate map data of the periphery of each parking space (step S343).

[0171] Next, the setting unit 111B inputs map data (surrounding environment data) for each parking space as input data into the learned model LD, and based on the obtained output results, sets a parking space with relatively few unrecognized blind spots as the target parking space SP (step S344).

[0172] By setting the target parking space SP in this manner, a parking space in which the blind spot area (unrecognized blind spot area) that is not recognized by the driver of the vehicle 10 during parking is reduced is set as the target parking space SP. Therefore, it becomes possible to perform driving operations that avoid collision with dangerous objects during parking of the vehicle 10, and the vehicle can be safely parked in the target parking space SP.

[0173] (Variation 2) In each of the above embodiments, the unrecognized blind spot area for each driver is identified using the learned model LD, but this is not limited to this, and the unrecognized blind spot area may be identified based on the driver's line of sight or facial direction detected in real time without using the learned model LD.

[0174] Fig. 24 is a flowchart showing an example of a blind spot area identification process according to Modification 2. The blind spot area identification process according to Modification 2 can be applied to both the first embodiment and the second embodiment. Hereinafter, the blind spot area identification process according to Modification 2 will be described with reference to Fig. 24 as necessary.

[0175] The blind spot area identification processing unit 111D identifies a blind spot area existing around the target parking space SP based on the generated map data and the position information of the vehicle 10 (step S395). Specifically, the blind spot area identification processing unit 111D identifies, as the blind spot area, an absolute blind spot that exists around the target parking space SP and cannot be seen from the position of the driver of the vehicle 10.

[0176] Next, the driver detection unit 111I acquires image data of the driver parking in the target parking space SP from the in-vehicle camera 33 (step S396). Next, the driver detection unit 111I detects the line of sight or the direction of the face of the driver based on the image data (step S397).

[0177] Next, the blind spot identification processing unit 111D identifies an unrecognized blind spot area that is not recognized by the driver based on information on the detected driver's line of sight or facial direction, information on the surrounding situation in which the vehicle 10 is located, and position information of the vehicle 10 (step S398).

[0178] By executing the danger judgment processing and display processing shown in FIG. 14, etc. based on the blind spot areas and unrecognized blind spot areas identified in this manner, even if the driver shows gaze or facial movements that differ from the driver's past habits, it is possible to perform driving operations that avoid collisions with dangerous objects while parking the vehicle 10, and the vehicle can be safely parked in the target parking space SP.

[0179] <<4. Supplementary Information>> The parking assistance device and parking assistance system illustrated in the above embodiments are typically applied to passenger cars, but the parking assistance device and parking assistance system of the present disclosure may be applied to moving bodies other than passenger cars, and the applications of the present disclosure are not particularly limited.

[0180] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting, that is, the present invention may have other effects in addition to or in place of the above effects that would be apparent to a person skilled in the art from the description of the present specification.

[0181] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the accompanying drawings, the technology of the present disclosure is not limited to the above embodiment. It is clear that a person having ordinary knowledge in the technical field to which the present disclosure belongs can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0182] For example, in each of the above embodiments, an example has been described in which the technology of the present disclosure is applied to a situation in which a vehicle is parked in a parking space in a motor pool, but the present disclosure is not limited to such an example. The technology of the present disclosure may be applied to a situation in which a vehicle is parked in a specific parking space such as a home. In addition, in a configuration in which an unmanned aerial vehicle is mounted on a vehicle and can fly, the technology of the present disclosure can be applied even in a situation in which a vehicle is parked in any parking space.

[0183] In addition, some of the functions of the parking assistance device, the aircraft control device, and the information processing device described in the above embodiments may be provided in other components.

[0184] In the above embodiment, the parking assistance device is an electronic control device mounted on a vehicle, but the technology of the present disclosure is not limited to this example. For example, the parking assistance device may be a mobile terminal that is configured to be able to communicate with a device other than the vehicle and issues a drive command to an arbitrary display device. Examples of such a mobile terminal include a laptop computer, a mobile phone, a smartphone, and a tablet terminal.

[0185] In addition, the technology disclosed herein can also be realized as a vehicle equipped with the parking assistance device described in the above embodiments, a parking assistance processing method using the parking assistance device, a computer program that causes a computer to function as the above parking assistance device, and a non-transitory tangible recording medium on which the computer program is recorded. [Explanation of symbols]

[0186] 10. Vehicle 11...Parking assistance device 20...Unmanned aerial vehicle 21...Aircraft control device 40...Information processing device

Claims

1. A parking assistance device that assists in parking a vehicle, one or more processors; and one or more memories communicatively coupled to the one or more processors; The one or more processors: a blind spot area identification process for identifying a blind spot area that is a blind spot as seen from the vehicle and an unrecognized blind spot area that is a blind spot area that is not recognized by a driver of the vehicle; A danger determination processing unit that determines the presence or absence of a danger object that may be an obstacle to the vehicle based on information about the surroundings of the target parking space acquired from an aircraft-mounted camera mounted on the unmanned aircraft; a display process for displaying information on dangerous objects present in the blind spot area on a display unit so as to overlap with a view seen by the driver, and displaying information on dangerous objects present in the unrecognized blind spot area in a sub-window area provided within the display unit; A parking assistance device that performs the following:

2. The one or more processors: In the blind spot area identification process, 2. The parking assistance device according to claim 1, wherein the unrecognized blind spot area is identified based on learning data obtained by learning at least one of the driver's line of sight and face direction during parking operations.

3. The one or more processors: In the blind spot area identification process, Detecting a line of sight or a face direction of the driver based on an image captured by an in-vehicle camera of the driver during a parking operation in the target parking space; The parking assistance device according to claim 1 , wherein the unrecognized blind spot area is identified based on the detected information on the line of sight or face direction.

4. A parking assistance system in which a control device mounted on a vehicle and an aircraft control device mounted on an unmanned aircraft communicate with each other to assist in parking the vehicle, A surrounding environment detection processing unit that detects the surrounding environment of the target parking space; a blind spot area identification processing unit that identifies a blind spot area that is a blind spot as seen from the vehicle and an unrecognized blind spot area that is a blind spot area that is not recognized by a driver of the vehicle; A danger determination processing unit that determines the presence or absence of a danger object that may be an obstacle to the vehicle based on information about the surroundings of the target parking space acquired from an aircraft-mounted camera mounted on the unmanned aircraft; a display processing unit that displays information about hazardous objects present in the blind spot area in a superimposed manner on a display unit provided in the vehicle so as to overlap with the scenery seen by the driver, and that displays information about hazardous objects present in the unrecognized blind spot area in a sub-window area within the display unit.

5. A parking assistance system in which a parking assistance device mounted on a vehicle communicates with an aircraft control device mounted on an unmanned aircraft and an information processing device that stores information within a motor pool, and assists a driver in parking the vehicle, A surrounding environment detection processing unit that detects the surrounding environment of the target parking space; a blind spot area identification processing unit that identifies a blind spot area that is a blind spot as seen from the vehicle and an unrecognized blind spot area that is a blind spot area that is not recognized by a driver of the vehicle, based on information on the surrounding environment of the target parking space and map data around the target parking space acquired from the information processing device; A danger determination processing unit that determines the presence or absence of a danger object that may be an obstacle to the vehicle based on information about the surroundings of the target parking space acquired from an aircraft-mounted camera mounted on the unmanned aircraft; a display processing unit that displays information about hazardous objects present in the blind spot area in a superimposed manner on a display unit provided in the vehicle so as to overlap with the scenery seen by the driver, and that displays information about hazardous objects present in the unrecognized blind spot area in a sub-window area within the display unit.

Images