Vehicle control system

Abstract

To provide a technique capable of controlling a drone by appropriately determining the necessity of a flight and the possibility of the flight.SOLUTION: In a vehicle control system 1, a drone control device is provided in a drone 20 and provides an image, which is picked up by an imaging section of the drone 20, to a vehicle 10. The drone control device controls a flight of the drone 20. A vehicle control device is provided in a vehicle 10, where the drone 20 can perform landing and take-off, and controls a travel of the vehicle 10 based on the picked-up image acquired from the drone control device. In a case where a condition requiring imaging from the sky is satisfied, the vehicle control device instructs the take-off from the vehicle 10 to the drone control device. In a case where a condition which makes the flight impossible is satisfied during the flight of the drone 20, the drone control device causes the drone 20 to land on the vehicle 10 and provides the image, which is picked up from an upper side of the vehicle 10, to the vehicle 10.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a vehicle control system.

Background Art

[0002] In recent years, the application of unmanned aerial vehicles called drones has been progressing. Patent Document 1 discloses a vehicle control device that controls platoon driving in which a plurality of vehicles drive while forming a platoon. Based on imaging results obtained by drones flying around the plurality of vehicles, the vehicle control device acquires the situation of vehicles around the plurality of vehicles and controls the plurality of vehicles.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a technique for controlling the driving of a vehicle using an image captured by a drone from above, it is required to appropriately determine the necessity and feasibility of flight and fly the drone.

[0005] An object of the present invention is to provide a technique capable of appropriately determining the necessity and feasibility of flight and controlling a drone.

Means for Solving the Problems

[0006] To solve the above problems, a vehicle control system according to an aspect of the present invention is provided on a drone, provides an image captured by an imaging unit of the drone to a vehicle, and controls the flight of the drone. The vehicle control system is provided on the vehicle on which the drone can take off and land, When the drone is flying and when the drone has landed on the vehicle, the captured image acquired from the drone control device and information around the vehicle detected by in-vehicle sensors of the vehicle A vehicle control device that controls the running of the vehicle based on the above, is provided. When conditions that require imaging from above are satisfied, the vehicle control device instructs the drone control device to take off from the vehicle. When conditions that make flight impossible are satisfied during the flight of the drone, the drone control device lands the drone on the vehicle and provides the vehicle with an image captured from above the vehicle. When the drone has landed on the vehicle, the imaging unit images an area that cannot be detected by the in-vehicle sensors and an area close to the vehicle.

Advantages of the Invention

[0007] According to the present invention, it is possible to appropriately determine the necessity and feasibility of flight and control the drone.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0009] FIG. 1 shows the configuration of a vehicle control system 1 according to an embodiment. The vehicle control system 1 includes a vehicle 10 and a drone 20. In FIG. 1, the size of the drone 20 is greatly exaggerated. The vehicle 10 is, for example, an autonomous vehicle. In the illustrated example, the vehicle 10 is traveling between the vehicle 100a in front and the vehicle 100b behind. The drone 20 is configured to be wirelessly communicable with a predetermined vehicle 10.

[0010] The drone 20 is a flying object that drives a plurality of propellers by a motor to perform autonomous flight, and can fly up and down, forward and backward, left and right by controlling the rotation speed of each propeller. The drone 20 flies above the vehicle 10 by automatic control and monitors the periphery of the vehicle 10 with the fifth camera 24e. The drone 20 captures the imaging range A5 shown by the dashed line in FIG. 1 with the fifth camera 24e and transmits the data of the captured image to the vehicle 10. The drone 20 can change the imaging position and range according to an instruction transmitted from the vehicle 10. As the vehicle 10 moves, the drone 20 follows the vehicle 10.

[0011] The vehicle 10 performs automatic driving based on the information around the vehicle 10 detected by an in-vehicle sensor (not shown) and the captured image data transmitted from the drone 20. Since the captured image by the drone 20 may include information on areas that cannot be acquired by the in-vehicle sensors of the vehicle 10, the vehicle 10 can perform automatic driving with high precision.

[0012] The drone 20 determines whether flight can continue during flight. If flight continuation is impossible, it lands on the vehicle 10 and provides the vehicle 10 with an image captured from above the vehicle 10. In this case, the drone 20 may land on the moving vehicle 10 or may land when the vehicle 10 stops. The drone 20 determines that flight continuation is impossible, for example, when there are obstacles such as roadside trees, pedestrian overpasses, and tunnels in the traveling direction of the vehicle 10 that obstruct the flight of the drone 20. Details of the conditions for determining that flight continuation is impossible will be described later.

[0013] FIG. 2 is a front view showing a state where the drone 20 has landed on the vehicle 10 in the vehicle control system 1 of FIG. 1. FIG. 3 is a side view of the vehicle control system 1 of FIG. 2.

[0014] The vehicle 10 is configured such that the drone 20 can take off and land. FIGS. 2 and 3 show an example where the drone 20 has landed on the roof of the vehicle 10, but it may land not only on the roof but also on the bonnet, rear panel, side, luggage compartment, trunk, etc. of the vehicle 10. The legs of the drone 20 are fixed to the vehicle 10.

[0015] The drone 20 is provided with a first camera 24a, a second camera 24b, a third camera 24c, and a fourth camera 24d. The first camera 24a to the fifth camera 24e are collectively referred to as the camera 24.

[0016] The first camera 24a is provided at the tip of a first support 22a that extends substantially horizontally to the right of the vehicle 10 from the main body portion of the drone 20, and images an imaging range A1 shown by a broken line in FIG. 2, that is, the right side of the vehicle 10.

[0017] The second camera 24b is provided at the tip of a second support 22b that extends substantially horizontally to the left of the vehicle 10 from the main body portion of the drone 20, and images an imaging range A2 shown by a broken line in FIG. 2, that is, the left side of the vehicle 10.

[0018] The third camera 24c is provided at the tip of a third support 22c that extends substantially horizontally forward of the vehicle 10 from the main body portion of the drone 20, and images an imaging range A3 shown by a broken line in FIG. 3, that is, the front of the vehicle 10.

[0019] The fourth camera 24d is provided at the tip of a fourth support 22d that extends substantially horizontally rearward of the vehicle 10 from the main body portion of the drone 20, and images an imaging range A4 shown by a broken line in FIG. 3, that is, the rear of the vehicle 10.

[0020] The first camera 24a to the fourth camera 24d mainly image areas that cannot be detected by the in-vehicle sensors of the vehicle 10. The orientation of each of the first camera 24a to the fourth camera 24d may be changeable according to the instruction of the vehicle 10. The drone 20 transmits the data of the captured images of the first camera 24a to the fourth camera 24d to the vehicle 10. When the drone 20 lands on the vehicle 10, the drone 20 also transmits information indicating that it has landed to the vehicle 10.

[0021] Even when the drone 20 has landed, the vehicle 10 performs automatic driving based on the surrounding information detected by in-vehicle sensors and the captured image data transmitted from the drone 20. Since the captured image by the drone 20 includes information on the area close to the vehicle 10, the vehicle 10 can perform automatic driving with high precision.

[0022] In this way, even when the drone 20 cannot continue flying, the vehicle 10 lands the drone 20 and provides the vehicle 10 with the image captured from above the vehicle 10, so that the monitoring of the area around the vehicle 10 by the drone 20 can be continued.

[0023] FIG. 4 shows the functional configuration of the vehicle 10 in FIG. 1. The vehicle 10 includes an in-vehicle sensor 30, a navigation device 32, a communication device 34, and a vehicle control device 36.

[0024] The in-vehicle sensor 30 periodically detects information on obstacles such as pedestrians around the vehicle 10 and information such as white lines on the road, and outputs the detection result to the vehicle control device 36. The in-vehicle sensor 30 includes, for example, at least one of a camera, a lidar (laser imaging detection and ranging), and a radar.

[0025] The navigation device 32 receives signals from GPS satellites by a GPS receiver (not shown) and derives the position and orientation of the vehicle 10. The position includes latitude and longitude. The navigation device 32 outputs information on the position and orientation of the vehicle 10 and route information from the current position to the destination to the vehicle control device 36.

[0026] The communication device 34 performs wireless communication with the drone 20. The communication device 34 may communicate with the drone 20 via a network such as the Internet, or may communicate with the drone 20 using a short-range wireless communication technology such as Wi-Fi or Bluetooth (registered trademark). The communication device 34 receives the captured image data etc. transmitted from the drone 20 and supplies it to the vehicle control device 36.

[0027] The vehicle control device 36 includes a first acquisition unit 38, a second acquisition unit 40, and a control unit 42. The configuration of the vehicle control device 36 can be realized hardware-wise by the CPU, memory, and other LSIs of any computer, and software-wise by a program loaded into the memory or the like. Here, however, functional blocks realized by their cooperation are depicted. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

[0028] The first acquisition unit 38 acquires the information output from the in-vehicle sensor 30 and supplies it to the control unit 42. The second acquisition unit 40 acquires the imaging image data by the drone 20 output from the communication device 34 and supplies it to the control unit 42.

[0029] The control unit 42 controls a drive device such as a drive motor (not shown), a braking device, a steering device, etc. according to the information supplied from the first acquisition unit 38 and the second acquisition unit 40, the information on the position and orientation of the vehicle 10 supplied from the navigation device 32, and the route information, and automatically runs the vehicle 10. The control unit 42 moves the vehicle 10 when there is no obstacle in the traveling direction based on the information on the obstacles around the vehicle 10 detected by at least one of the in-vehicle sensor 30 and the camera 24 of the drone 20. For such control, known autonomous driving technologies can be used.

[0030] The control unit 42 determines whether imaging from above is necessary, and if imaging from above is necessary, gives a flight instruction to the drone 20. Specifically, when the drone 20 has landed on the vehicle 10 and the conditions for imaging from above are satisfied, the control unit 42 instructs the drone 20 to take off from the vehicle 10 via the communication device 34. In this case, the control unit 42 sets the flight instruction flag to "1" and transmits it to the drone 20.

[0031] The conditions that require imaging from above are, for example, the need for information on the distance of the vehicle 10, the occurrence of an area that cannot be detected by the in-vehicle sensor 30, the failure of the in-vehicle sensor 30, or the need for close-up imaging. The conditions that require imaging from above can also be said to be the conditions that require taking off for imaging.

[0032] For example, when most of the area of the image in front of the vehicle obtained due to traffic congestion or the like is occupied by the vehicle in front and the road conditions ahead cannot be fully grasped, the control unit 42 determines that information on the distance in front of the vehicle is necessary.

[0033] For example, when part of the information detected by the in-vehicle sensor 30 is affected by the influence of sunlight such as backlight or the influence of dirt or water droplets on the surface of the in-vehicle sensor 30, the control unit 42 determines that an area that cannot be detected by the in-vehicle sensor 30 has occurred.

[0034] For example, when the control unit 42 determines based on the information obtained by the in-vehicle sensor 30 that the road surface may be rough, there may be falling objects, steps, ditches, etc. on the road, and there may be people, especially children, the elderly, or disabled people, and these detailed information or degrees cannot be recognized due to the resolution of the in-vehicle sensor 30, the control unit 42 determines that close-up imaging is necessary.

[0035] When the conditions that require imaging from above are satisfied, the control unit 42 also transmits the information on the position where imaging is required to the drone 20. When the control unit 42 determines that close-up imaging is necessary, it also transmits the information requesting close-up imaging to the drone 20.

[0036] When the conditions that require imaging from above are not satisfied, the control unit 42 instructs the drone 20 to land on the vehicle 10 via the communication device 34. In this case, the control unit 42 sets the flight instruction flag to "0" and transmits it to the drone 20.

[0037] FIG. 5 shows the functional configuration of the drone 20 in FIG. 1. The drone 20 includes an external sensor 50, an imaging unit 52, a GPS receiving unit 54, a communication device 56, a drone control device 58, and a driving device 60.

[0038] The external sensor 50 includes sensors that detect information for controlling the attitude of the airframe of the drone 20, such as, for example, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, and an atmospheric pressure sensor. The external sensor 50 supplies the detected information to the drone control device 58.

[0039] The imaging unit 52 includes a plurality of cameras 24 and sequentially supplies time-series images captured periodically to the drone control device 58. The imaging unit 52 may include an ultrasonic sensor, a millimeter-wave radar, and the like.

[0040] The GPS receiver 54 receives signals from GPS satellites, derives the position and orientation of the drone 20, and supplies them to the drone control device 58. The communication device 56 performs wireless communication with the communication device 34 of the vehicle 10.

[0041] The drive device 60 includes a motor (not shown) that drives a plurality of propellers respectively and a motor drive circuit (not shown). By rotating the propellers, lift is generated. The drive device 60 is controlled by the drone control device 58. Each part of the drone 20 such as the drive device 60 operates with electric power supplied from a battery (not shown).

[0042] The drone control device 58 sequentially provides the time-series images captured by the imaging unit 52 to the vehicle 10 via the communication device 56. Also, the drone control device 58 controls the flight of the drone 20 by controlling the drive device 60.

[0043] The drone control device 58 has an acquisition unit 62 and a control unit 64. The configuration of the drone control device 58 can be realized in terms of hardware by a CPU, a memory, and other LSIs of an arbitrary computer, and can be realized in terms of software by a program loaded into the memory, etc. Here, however, functional blocks realized by their cooperation are depicted. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

[0044] The acquisition unit 62 acquires the information supplied from the external sensor 50 and the image data supplied from the imaging unit 52, and outputs the acquired information to the control unit 64. The acquisition unit 62 supplies the image data supplied from the imaging unit 52 to the communication device 56, and the communication device 56 transmits the image data to the vehicle 10. For example, when the drone 20 is flying, the communication device 56 may transmit the imaging image data by the fifth camera 24e, and when the drone 20 is landing on the vehicle 10, the imaging image data by the first camera 24a to the fourth camera 24d may be transmitted. When the drone 20 is flying, the communication device 56 may also transmit the imaging image data by the first camera 24a to the fourth camera 24d.

[0045] When the control unit 64 is instructed to fly from the vehicle 10, that is, when the communication device 56 receives the flight instruction flag set to "1" from the vehicle 10, if the non-flight conditions are not satisfied, the control unit 64 controls the drive device 60 so that the drone 20 takes off from the vehicle 10. Based on the captured image, the information for attitude control, and the position information, the control unit 64 controls the drive device 60 to fly while avoiding collisions with obstacles and trailing the vehicle 10 around the vehicle 10.

[0046] The control unit 64 controls the drive device 60 to fly so that an image of a position where imaging is required can be captured. Thereby, when the in-vehicle sensor 30 cannot sufficiently grasp the road conditions ahead of the vehicle due to traffic jams or the like, the road conditions in the distance ahead of the vehicle can be acquired. When an area that cannot be detected by the in-vehicle sensor 30 occurs, the information of that area can be acquired. When the in-vehicle sensor 30 fails, the information of the area that the failed sensor should acquire can be acquired.

[0047] When the control unit 64 receives the information requested for copying by the communication device 56, the control unit 64 controls the drive device 60 to fly to a position where copying is possible. Thereby, since an image of an object that cannot be recognized in detail due to the resolution of the in-vehicle sensor 30 can be acquired with high resolution, the detailed information of the object can also be acquired.

[0048] When the control unit 64 is instructed to fly from the vehicle 10, it does not allow the drone 20 to take off from the vehicle 10 while the non - flyable conditions are satisfied.

[0049] When a non - flyable condition is satisfied during the flight of the drone 20, the control unit 64 controls the drive device 60 so that the drone 20 lands on the vehicle 10, and provides the vehicle 10 with an image captured from above the vehicle 10 by the landed drone 20.

[0050] The non - flyable conditions are, for example, that the remaining battery level is less than a predetermined value, the imaging unit 52 has detected a predetermined flight obstacle, or the flight attitude is unstable. The predetermined flight obstacle is an obstacle that is difficult for the drone 20 to avoid while following the vehicle 10, and examples include a relatively large pedestrian overpass, street trees existing at a relatively long distance, a tunnel, etc. The control unit 64 can detect a flight obstacle from the captured image using a known image recognition technique.

[0051] The control unit 64 can identify that the flight attitude is unstable based on the information output from the external sensor 50. For example, the flight attitude can become unstable due to deteriorated weather such as rainfall, snowfall, strong wind, etc.

[0052] When the control unit 64 lands the drone 20 because a non - flyable condition is satisfied during the flight of the drone 20, if the non - flyable condition is no longer satisfied and no landing instruction is received from the vehicle 10, the control unit 64 controls the drive device 60 so that the drone 20 takes off from the vehicle 10 again.

[0053] When the control unit 64 is instructed to land from the vehicle 10 during the flight of the drone 20, that is, when the communication device 56 receives a flight instruction flag set to "0" from the vehicle 10, the control unit 64 controls the drive device 60 so that the drone 20 lands on the vehicle 10.

[0054] The battery of the drone 20 that has landed on the vehicle 10 may be charged by the power supplied from the vehicle 10. For power supply to the drone 20, non-contact power supply or power supply via contacts may be used.

[0055] Next, the overall operation of the vehicle control system 1 will be described. FIG. 6 is a flowchart showing the processing of the vehicle control device 36 in FIG. 4. When remote information is required (Y in S10), the vehicle control device 36 sets the flight instruction flag to "1" (S20), transmits the flight instruction flag to the drone 20, and returns to S10. When remote information is not required (N in S10), if there is an area that cannot be detected by the in-vehicle sensor 30 (Y in S12), the process proceeds to S20.

[0056] If there is no area that cannot be detected by the in-vehicle sensor 30 (N in S12), if the in-vehicle sensor 30 is malfunctioning (Y in S14), the process proceeds to S20. If the in-vehicle sensor 30 is not malfunctioning (N in S14), if copying is required (Y in S16), the process proceeds to S20. If copying is not required (N in S16), the vehicle control device 36 sets the flight instruction flag to "0" (S18), transmits the flight instruction flag to the drone 20, and returns to S10.

[0057] FIG. 7 is a flowchart showing the processing of the drone control device 58 in FIG. 5. If the flight instruction flag is not "1" (N in S30), the drone control device 58 lands the drone 20 on the vehicle 10 and monitors the vicinity of the vehicle 10 (S40), and returns to S30. In S40, if it has already landed, the landing continues. If the flight instruction flag is "1" (Y in S30), if the remaining battery level is not equal to or higher than a predetermined value (N in S32), the process proceeds to S40.

[0058] If the remaining battery level is equal to or higher than a predetermined value (Y in S32), if a flight obstacle is detected (Y in S34), the process proceeds to S40. If no flight obstacle is detected (N in S34), if the flight attitude is unstable (Y in S36), the process proceeds to S40. If the flight attitude is not unstable (N in S36), the drone control device 58 starts or continues the flight of the drone 20 and monitors from above (S38), and returns to S30.

[0059] According to the embodiment, when the drone 20 cannot continue flying due to an obstacle or the like, the drone 20 is landed on the vehicle 10, and the image captured from above the vehicle 10 is provided to the vehicle 10, so that the monitoring of the periphery of the vehicle 10 by the drone 20 can be continued. It is also possible to prevent the drone 20 whose flight has been obstructed from moving away from the vehicle 10 that continues to travel. In this way, it is possible to appropriately determine the necessity and feasibility of the flight of the drone 20 and control the drone 20.

[0060] As described above, the present invention has been described based on the embodiment. It should be understood by those skilled in the art that the embodiment is merely an example, and various modifications are possible for the combination of each component and each processing process, and such modifications are also within the scope of the present invention.

Description of Reference Numerals

[0061] 1... Vehicle control system, 10... Vehicle, 20... Drone, 24... Camera, 24a... First camera, 24b... Second camera, 24c... Third camera, 24d... Fourth camera, 24e... Fifth camera, 30... On-vehicle sensor, 32... Navigation device, 34... Communication device, 36... Vehicle control device, 38... First acquisition unit, 40... Second acquisition unit, 42... Control unit, 50... External sensor, 52... Imaging unit, 54... GPS receiver, 56... Communication device, 58... Drone control device, 60... Driving device, 62... Acquisition unit, 64... Control unit.

Claims

1. A drone control device provided on a drone, which provides an image captured by an imaging unit of the drone to a vehicle and controls the flight of the drone; A vehicle control device provided on the vehicle on which the drone can take off and land, which controls the running of the vehicle based on the captured image obtained from the drone control device and the information around the vehicle detected by an in-vehicle sensor of the vehicle when the drone is flying and when the drone has landed on the vehicle; Comprising: When a condition requiring imaging from above is satisfied, the vehicle control device instructs the drone control device to take off from the vehicle; When a non-flight condition is satisfied during the flight of the drone, the drone control device lands the drone on the vehicle and provides the vehicle with an image captured from above the vehicle; When the drone has landed on the vehicle, the imaging unit images an area that cannot be detected by the in-vehicle sensor and an area close to the vehicle; A vehicle control system characterized by the above.

2. The imaging unit is: A first camera provided at the tip of a first support extending rightward from the main body of the drone in a state where the drone has landed on the vehicle, for imaging the right side of the vehicle; A second camera provided at the tip of a second support extending leftward from the main body of the drone in a state where the drone has landed on the vehicle, for imaging the left side of the vehicle; A third camera provided at the tip of a third support extending forward from the main body of the drone in a state where the drone has landed on the vehicle, for imaging the front of the vehicle; A fourth camera provided at the tip of a fourth support extending rearward from the main body of the drone in a state where the drone has landed on the vehicle, for imaging the rear of the vehicle; A fifth camera for imaging the periphery of the vehicle during the flight of the drone; Having: The drone is: When the drone is flying, it has a communication device that transmits imaging image data by the fifth camera to the vehicle, and when the drone has landed on the vehicle, it transmits imaging image data by the first camera, the second camera, the third camera, and the fourth camera to the vehicle. The vehicle control system according to Claim 1, characterized by the above.

3. When an area that cannot be detected by the in-vehicle sensor occurs, the vehicle control device determines that the conditions for imaging from above are satisfied, transmits information on the position where imaging is required to the drone, and the drone control device controls the drone to fly so as to be able to image an image of the position where imaging is required. The vehicle control system according to claim 1 or 2, characterized by the above.

4. When the vehicle control device determines that close-up imaging of an object is necessary based on the information acquired by the in-vehicle sensor, it determines that the conditions for imaging from above are satisfied, and transmits information on the position where imaging is required and information requesting close-up imaging to the drone, and when the drone control device receives the information requesting close-up imaging, it controls the drone to fly to a position where close-up imaging is possible. The vehicle control system according to any one of claims 1 to 3, characterized by the above.

Images