Control device, control method, and control program

Abstract

To provide a controller, control method, and control program making it possible to perform display according to a travel situation of a vehicle.SOLUTION: A controller includes an acquisition unit 31 that acquires outside world information, which signifies an ambient situation of a vehicle 1, from an outside world sensor including an imaging unit (camera 111) which takes an ambience of the vehicle 1, an image processing unit 32 capable of generating a first bird's-eye view image of the vehicle 1 viewed from the sky behind a vehicle center and encompasses an advancing direction of the vehicle in an angle of view, and a second bird's-eye view image of the vehicle 1 viewed from the sky in a vertical direction of the vehicle 1, on the basis of the acquired outside world information, a control unit 33 that implements control of moving the vehicle 1 to a target position, and a display control unit 34 that displays an image, which is generated by the image processing unit 32, on a predetermined display unit (touch panel 22). When the distance to the target position is equal to or larger than a predetermined value, the display control unit 34 displays the first bird's-eye view image on the display unit. When the distance to the target position falls below the predetermined value, the display control unit displays the second bird's-eye view image on the display unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control device, a control method, and a control program. [Background technology]

[0002] In recent years, efforts to provide access to sustainable transport systems that take into consideration vulnerable transport participants have been gaining momentum. As part of these efforts, research and development is being conducted on driver assistance technologies and autonomous driving technologies for automobiles and other vehicles in order to further improve road safety and convenience.

[0003] For example, conventionally, a system has been known that provides driving assistance by displaying a camera image on a display. As an example of such a system, Patent Document 1 listed below discloses a technology for providing driving assistance by displaying a surrounding image including the vehicle on a display using a function of a so-called around view monitor, for example, when the vehicle is traveling at a low speed (or when parking). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2016 / 0021312 Summary of the Invention [Problem to be solved by the invention]

[0005] However, as in the prior art, simply displaying an image of the vehicle's surroundings may be insufficient depending on the driving situation. For example, when the vehicle is not traveling at a low speed but traveling at a normal speed, a user such as a driver may want to check the situation in the vehicle's traveling direction (for example, a distant situation not included in the peripheral image) from the image of the vehicle's surroundings, and a display that corresponds to the vehicle's driving situation is desired.

[0006] The present invention provides a control device, a control method, and a control program that enable display according to the driving conditions of a vehicle. [Means for solving the problem]

[0007] One aspect of the present invention is A control device that moves a vehicle to a predetermined target position, an acquisition unit that acquires external environment information indicating a surrounding situation of the vehicle from an external environment sensor including an imaging device that captures an image of the surroundings of the vehicle; Based on the acquired external information, A first bird's-eye view image and a second bird's-eye view image of the vehicle are generated, each of which is obtained by viewing the vehicle from a different angle of view, and the first bird's-eye view image is compared with the second bird's-eye view image, The direction of travel of the vehicle Wider range of I put it in the frame Bird's-eye view an image processing unit capable of generating a control unit that controls the vehicle to move to the target position; a display control unit that displays the image generated by the image processing unit on a predetermined display unit; a storage unit for recording route data that is a route history when the vehicle moves to the target position; Equipped with The control unit performing control to move the vehicle to the target position based on the route data stored in the storage unit; The display control unit When the distance to the target position is equal to or greater than a predetermined value, the first overhead image is displayed on the display unit; When the distance to the target position is less than the predetermined value, the second overhead image is displayed on the display unit; When the route data is recorded in the storage unit, the route data is referenced and the display of the first overhead image and the second overhead image is switched in accordance with the distance to the target position. .

[0008] Another aspect of the present invention is A computer that moves a vehicle to a predetermined target position acquiring external environment information indicating a surrounding situation of the vehicle from an external environment sensor including an imaging device that captures an image of the surroundings of the vehicle; Based on the acquired external information, A first bird's-eye view image and a second bird's-eye view image of the vehicle can be generated, each of which is obtained by viewing the vehicle from a different angle of view, and the first bird's-eye view image has a different image quality than the second bird's-eye view image. The direction of travel of the vehicle Wider range of I put it in the frame Bird's-eye view Generate performing control to move the vehicle to the target position; Displaying the generated image on a predetermined display unit; execute a process to record route data that is a route history when the vehicle moves to the target position; In the process of moving the vehicle to the target position, the vehicle is moved to the target position based on the recorded route data; In the process of displaying the generated image on the display unit, When the distance to the target position is equal to or greater than a predetermined value, the first overhead image is displayed on the display unit; When the distance to the target position is less than the predetermined value, the second overhead image is displayed on the display unit; When the route data is recorded, the route data is referenced and the display of the first overhead image and the second overhead image is switched in accordance with the distance to the target position. .

[0009] Another aspect of the present invention is A computer that moves a vehicle to a predetermined target position acquiring external environment information indicating a surrounding situation of the vehicle from an external environment sensor including an imaging device that captures an image of the surroundings of the vehicle; Based on the acquired external information, A first bird's-eye view image and a second bird's-eye view image of the vehicle can be generated, each of which is obtained by viewing the vehicle from a different angle of view, and the first bird's-eye view image has a different image quality than the second bird's-eye view image. The direction of travel of the vehicle Wider range of I put it in the frame Bird's-eye view Generate performing control to move the vehicle to the target position; Displaying the generated image on a predetermined display unit; Execute a process to record route data that is a route history when the vehicle moves to the target position; In the process of moving the vehicle to the target position, the vehicle is moved to the target position based on the recorded route data; In the process of displaying the generated image on the display unit, When the distance to the target position is equal to or greater than a predetermined value, the first overhead image is displayed on the display unit; When the distance to the target position is less than the predetermined value, the second overhead image is displayed on the display unit; When the route data is recorded, the route data is referenced and the display is switched between the first overhead image and the second overhead image in accordance with the distance to the target position.

[0010] Another aspect of the present invention is A control device that moves a vehicle to a predetermined target position, an acquisition unit that acquires external environment information indicating a surrounding situation of the vehicle from an external environment sensor including an imaging device that captures an image of the surroundings of the vehicle; an image processing unit capable of generating an overhead image of the vehicle from above and behind the center of the vehicle, the overhead image including the traveling direction of the vehicle in an angle of view based on the acquired external environment information; a control unit that controls the vehicle to move to the target position; a display control unit that causes the overhead image generated by the image processing unit to be displayed on a predetermined display unit, The display control unit causes the display unit to display the overhead image when the vehicle moves backward and exits to a predetermined location. [Effects of the Invention]

[0011] The present invention enables display according to the vehicle's driving situation, which in turn improves traffic safety and contributes to the development of a sustainable transportation system. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a schematic configuration of a vehicle 1 equipped with a control device 30 according to an embodiment. [Figure 2] 2 is a diagram showing an example of a route along a driveway 201 to a parking position 211. FIG. [Figure 3] FIG. 10 is a diagram showing an example of a first overhead image. [Figure 4] FIG. 10 is a diagram showing an example of a second overhead image. [Figure 5] FIG. 10 is a diagram illustrating an example of a superimposed image. [Figure 6] 1 is a flowchart (part 1) illustrating an example of processing executed by a control device 30 according to an embodiment. [Figure 7] 10 is a flowchart (part 2) illustrating an example of processing executed by the control device 30 according to an embodiment. [Figure 8] 10 is a flowchart (part 3) illustrating an example of processing executed by the control device 30 according to an embodiment. [Figure 9] FIG. 10 is a diagram for explaining an example of another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] An embodiment of a vehicle control device of the present invention will be described below with reference to the drawings. The following embodiment does not limit the present invention, and not all of the elements described in the following embodiment are necessarily essential to the present invention. Furthermore, two or more elements described in the following embodiment may be arbitrarily combined without departing from the spirit of the present invention. Note that, below, identical or similar elements are denoted by identical or similar reference numerals, and their description may be omitted or simplified.

[0014] [Vehicles equipped with control devices] First, the vehicle of this embodiment will be described. Fig. 1 is a block diagram showing the configuration of a vehicle 1 equipped with a control device 30 of this embodiment. The vehicle 1 is an automobile equipped with a drive source (not shown) and wheels (not shown) including drive wheels driven by the power of the drive source and steerable wheels. As an example, the vehicle 1 can be a four-wheeled automobile having a pair of front wheels and a pair of rear wheels on the left and right.

[0015] The drive source of vehicle 1 may be an electric motor, an internal combustion engine such as a gasoline engine or a diesel engine, or a combination of an electric motor and an internal combustion engine. The drive source of vehicle 1 may drive a pair of left and right front wheels, a pair of left and right rear wheels, or a pair of left and right front and rear wheels, i.e., four wheels. Either one of the front wheels or the rear wheels of vehicle 1 may be steerable wheels, or both may be steerable wheels.

[0016] The vehicle 1 is capable of autonomous driving and driving assistance, which automatically controls driving operations to drive the vehicle. Autonomous driving, as defined here, refers to a system of the vehicle that recognizes or monitors the driving environment and surrounding conditions, as well as all driving operations such as starting, accelerating / decelerating, steering, and stopping. Driving assistance refers to a system of the vehicle that performs some of the driving operations such as starting, accelerating / decelerating, steering, and stopping, such as an automatic parking system (APS), a lane keep assist system (LKAS), and adaptive cruise control (ACC). In the following embodiment, an example will be described in which the vehicle 1 is moved to a parking position designated by a user as a target position. Note that there may be multiple levels of driving control in autonomous driving and driving assistance, as is conventionally known, and may be defined, for example, by levels 0 to 5 established by the Society of Automotive Engineers (SAE) in the United States. The higher the level number of the driving control level, the lighter the operational burden on the driver (in other words, the higher the level number, the higher the degree of automation). The specific contents of levels 0 to 5 are already known, so a description thereof will be omitted here.

[0017] The vehicle 1 includes a sensor group 10, a navigation device 20, a control device 30, an electric power steering (EPS) system 40, a driving force control system 50, a braking force control system 60, a communication unit 70, and an operation input unit 80.

[0018] The sensor group 10 includes an external sensor 11 that acquires information about the surroundings of the vehicle 1, and a vehicle sensor 12 that acquires information about the vehicle 1. Information acquired by each sensor included in the sensor group 10 (in other words, detected values) is output to the control device 30.

[0019] The external sensor 11 includes, for example, a camera 111, a sonar 112, and a radar 113. The camera 111 is a digital camera that captures images of the surroundings of the vehicle 1, including the vehicle 1, and outputs image data of the obtained surrounding images to the control device 30. In this embodiment, the vehicle 1 has a front camera 111a, a rear camera 111b, a left side camera 111c, and a right side camera 111d in order to acquire surrounding images in all directions of the vehicle 1.

[0020] The front camera 111a is mounted, for example, on the upper part of the windshield or the front bumper inside the vehicle cabin, and captures an image of the area in front of the vehicle 1. The rear camera 111b is mounted, for example, on the rear bumper, and captures an image of the area behind the vehicle 1. The left side camera 111c is mounted, for example, on the left side mirror, and captures an image of the area to the left of the vehicle 1. The right side camera 111d is mounted, for example, on the right side mirror, and captures an image of the area to the right of the vehicle 1. Each of the cameras 111a to 111d can be, for example, a digital camera using an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The front camera 111a, rear camera 111b, left side camera 111c, and right side camera 111d are examples of imaging devices of the present invention. In the following description, the front camera 111a, rear camera 111b, left side camera 111c, and right side camera 111d will be simply referred to as "camera 111" unless they are particularly distinguished from one another.

[0021] The sonar 112 emits sound waves around the vehicle 1 (for example, in front of, behind, and to the sides of the vehicle 1) and receives reflected sound from objects around the vehicle 1, thereby detecting the distance to the objects, direction, etc. The detected information is transmitted to the control device 30 at predetermined intervals. The radar 113 emits radio waves around the vehicle 1, including in front of the vehicle 1, and receives reflected waves from objects around the vehicle 1, thereby detecting the distance to the objects, direction, etc. The detected information is transmitted to the control device 30 at predetermined intervals. For example, a millimeter wave radar can be used as the radar 113.

[0022] The external sensor 11 may be configured to include a LiDAR (Light Detection and Ranging) instead of or in addition to the sonar 112 or the radar 113. In this case, the LiDAR emits laser light to the periphery of the vehicle 1 including the area ahead of the vehicle 1, and receives reflected light from an object present around the vehicle 1 to detect the distance and direction to the object.

[0023] The vehicle sensor 12 includes, for example, a wheel sensor 121, a vehicle speed sensor 122, an inertial measurement unit (IMU) 123, an occupant camera 124, an operation detection unit 125, and a steering touch sensor 126.

[0024] The wheel sensor 121 detects the rotation angle of one or more wheels of the vehicle 1. As an example, the wheel sensor 121 detects the rotation angle of each of the left rear wheel and the right rear wheel. As the wheel sensor 121, for example, an angle sensor or a displacement sensor can be adopted.

[0025] The vehicle speed sensor 122 detects the vehicle speed, which is the running speed (in other words, the moving speed of the vehicle body) of the vehicle 1. For example, the vehicle speed sensor 122 detects the vehicle speed based on the number of rotations of a countershaft (not shown) provided in the vehicle 1.

[0026] The inertial measurement unit 123 detects angular velocities in the pitch, roll, and yaw directions of the vehicle 1, and accelerations in the front-to-rear, left-to-right, and up-to-down directions of the vehicle 1. Note that instead of the inertial measurement unit 123, the vehicle sensor 12 may be configured to include an acceleration sensor that detects acceleration in a predetermined direction of the vehicle 1 and a gyro sensor that detects angular velocity in a predetermined direction of the vehicle 1.

[0027] The occupant camera 124 is a digital camera that captures an image of the interior of the vehicle 1 and outputs image data of the obtained interior image to the control device 30. For example, the occupant camera 124 can be a so-called "driver monitor camera" that is provided so as to be able to capture an image of the head of the driver sitting in the driver's seat of the vehicle 1 from the front (in other words, to be able to capture an image of the face). As with the camera 111, a digital camera using an imaging element such as a CCD or CMOS can be used as the occupant camera 124. Note that in this embodiment, the image data of the interior image obtained by the occupant camera 124 capturing an image of the interior of the vehicle serves as information that can identify the direction of the driver's line of sight.

[0028] The operation detection unit 125 detects operations performed using the operation input unit 80, which is operable by occupants including the driver. In this embodiment, the operation input unit 80 includes, for example, an operation switch (not shown) that accepts an operation to switch the autonomous driving or autonomous parking on (in other words, activated) and off (in other words, not activated). In this case, the operation detection unit 125 can detect an operation to turn the autonomous driving or autonomous parking on or off.

[0029] The steering touch sensor 126 detects whether the steering wheel 46 of the vehicle 1 is being properly gripped. For example, the steering touch sensor 126 is realized by a capacitance sensor or the like. In this case, the capacitance sensor is provided at a portion where the driver touches the steering wheel 46 when the steering wheel 46 is being properly gripped.

[0030] The navigation device 20 includes, for example, a GNSS (Global Navigation Satellite System) receiver 21, a touch panel 22, and a speaker 23. The navigation device 20 also has a storage unit (not shown) configured with a flash memory or the like. The storage unit of the navigation device 20 stores a map information database (DB) 24 and the like.

[0031] The GNSS receiver 21 identifies the current position of the vehicle 1 (for example, the latitude and longitude of the location where the vehicle 1 is located) based on the signals received from the GNSS satellites. Note that the navigation device 20 may acquire, for example, detection results from the vehicle sensors 12 (for example, the wheel sensors 121 and the vehicle speed sensor 122) via the control device 30, and identify or complement the current position of the vehicle 1 by an INS (Inertial Navigation System) that uses the detection values ​​of the vehicle sensors 12.

[0032] The touch panel 22 functions as an input device that receives various types of information input to the control device 30, and as a display device controlled by the control device 30. The touch panel 22 is configured by combining a display device such as a liquid crystal display or an OLED (Organic Light Emitting Diode) with a pointing device (e.g., a touchpad). The speaker 23 is configured to be able to output audio to an occupant of the vehicle 1 (e.g., the driver). The touch panel 22 is an example of a "predetermined display unit" in the present invention.

[0033] For example, the navigation device 20 searches for a route from the current position of the vehicle 1 to a destination set by the driver using the touch panel 22 by referring to the map information database 24. Then, the navigation device 20 provides route guidance using the touch panel 22 and the speaker 23 based on the searched route. The navigation device 20 may also cause the touch panel 22 to display a predetermined display in accordance with an instruction from the control device 30. Specific displays will be described later. Furthermore, the navigation device 20 may output predetermined information to the control device 30, such as information indicating the identified current position of the vehicle 1 or information indicating an operation received via the touch panel 22.

[0034] The control device 30 is a computer that has, for example, a processor that performs various calculations, a memory unit 35 having a non-transitory storage medium that stores various information, an input / output unit that controls the input and output of data between the inside and outside of the control device 30, etc. (not shown), and controls the entire vehicle 1.

[0035] The storage unit 35 stores various data, programs to be executed by the control device 30, and the like. For example, the storage unit 35 stores route data 35a when the vehicle 1 moves to a predetermined target position. The route data 35a is information indicating the route history when the vehicle 1 moves to the target position, and the route data is accumulated as the vehicle 1 moves. This route data includes information on the route as well as information on plants, trees, obstacles, and the like present on the route. In particular, in this embodiment, when the vehicle 1 moves through a location not included in the map information database 24 (e.g., private property), the control device 30 refers to the route data 35a and executes the image display process described below. Note that the predetermined target position is assumed to be, for example, a parking position designated by the user. In the following description, the parking position designated by the user will be described as the target position.

[0036] The control device 30 is realized by one ECU (Electronic Control Unit) or by multiple ECUs working together. Note that the control device 30 performs driving assistance such as controlling the vehicle on behalf of the driver, and therefore can also be called a control device in a so-called advanced driving assistance system (ADAS ECU). Specific configurations and specific examples of control of the control device 30 will be described later, so explanations thereof will be omitted here.

[0037] The EPS system 40 includes, for example, a steering angle sensor 41, a torque sensor 42, an EPS motor 43, a resolver 44, and an EPS ECU 45.

[0038] The steering angle sensor 41 detects the steering angle θst of the steering wheel 46 and outputs information indicating the detected steering angle θst to the EPS ECU 45. The torque sensor 42 detects the steering torque TQ, which is the torque applied to the steering wheel 46 of the vehicle 1, and outputs information indicating the detected steering torque TQ to the EPS ECU 45.

[0039] The EPS motor 43 applies a driving force or a reaction force to a steering column 47 connected to the steering wheel 46 in accordance with instructions from the EPS ECU 45, thereby assisting the driver in operating the steering wheel 46. The resolver 44 detects a rotation angle θm of the EPS motor 43, and outputs information indicating the detected rotation angle θm to the EPS ECU 45.

[0040] The EPS ECU 45 is a computer that includes, for example, a processor that performs various calculations, a storage unit that has a non-transitory storage medium that stores various information, and an input / output unit that controls input and output of data between the inside and outside of the EPS ECU 45 (none of which are shown), and controls the EPS system 40 (e.g., the EPS motor 43), and is realized by one or more ECUs. For example, the EPS ECU 45 controls the EPS system 40 (e.g., the EPS motor 43) based on the steering angle θst detected by the steering angle sensor 41, the steering torque TQ detected by the torque sensor 42, the rotation angle θm detected by the resolver 44, etc.

[0041] Furthermore, the EPS system 40 (for example, the EPS ECU 45) may output information indicating the steering angle θst detected by the steering angle sensor 41, the steering torque TQ detected by the torque sensor 42, the rotation angle θm detected by the resolver 44, etc. to the control device 30. Furthermore, the EPS system 40 (for example, the EPS ECU 45) may output information indicating the steering speed ω of the steering wheel 46 to the control device 30. In this case, the steering speed ω can be obtained, for example, by differentiating the steering angle θst with respect to time.

[0042] The driving force control system 50 includes a driving ECU 51 and is configured to control the driving force of the vehicle 1. The driving ECU 51 is a computer that controls the driving force control system 50 and includes, for example, a processor that performs various calculations, a storage unit having a non-transitory storage medium that stores various information, and an input / output unit that controls input and output of data between the inside and outside of the driving ECU 51 (all of which are not shown). The driving ECU 51 is implemented by one or more ECUs. For example, the driving ECU 51 controls the driving force output from a driving source of the vehicle 1 based on the operation amount of an accelerator pedal 52 provided in the vehicle 1 and the detection value of a shift position sensor 53 that detects the shift position Ps of a shift device (e.g., a shift lever or a shift switch) (not shown). Note that the driving source is an internal combustion engine or a motor, as described above, and the driving ECU 51 controls the output of the internal combustion engine or the motor based on the operation amount of the accelerator pedal 52 and the shift position Ps. The driving ECU 51 can also control the driving force control system 50 (e.g., the driving source) in accordance with instructions from the control device 30.

[0043] The braking force control system 60 includes a braking ECU 61 and is configured to be able to control the braking force of the vehicle 1. The braking ECU 61 is a computer that controls the braking force control system 60 and includes, for example, a processor that performs various calculations, a storage unit having a non-transitory storage medium that stores various information, and an input / output unit that controls input and output of data between the inside and outside of the braking ECU 61 (all of which are not shown). The braking ECU 61 is realized by one or more ECUs. For example, the braking ECU 61 controls the braking force of the vehicle 1 by controlling a brake device (not shown) provided in the vehicle 1 based on operation of a brake pedal 62 provided in the vehicle 1. Here, the brake device includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, and an electric motor that generates hydraulic pressure in the cylinder. The braking ECU 61 controls the electric motor of the brake device so as to generate a braking force corresponding to operation of the brake pedal 62. The braking ECU 61 can also control the braking force control system 60 according to instructions from the control device 30.

[0044] The communication unit 70 is a communication interface that communicates with the external device 2 in accordance with control instructions from the control device 30. That is, the control device 30 can communicate with the external device 2 via the communication unit 70. Examples of the external device 2 include a driver's terminal device (e.g., a smartphone) and a server device managed by the manufacturer of the vehicle 1. Note that communication between the vehicle 1 and the external device 2 can be achieved using, for example, a mobile communication network such as a cellular line, Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0045] [Control device] Next, a specific description will be given of the control device 30. The control device 30 executes, for example, various programs stored in a storage unit 35 of the control device 30. In this embodiment, the control device 30 automatically moves the vehicle 1 to a parking position, which is a predetermined target position designated by a user including the driver or other occupants of the vehicle 1 (hereinafter also referred to as "user"), and executes control to display an image according to the traveling situation of the vehicle 1 on a display unit such as the touch panel 22 during the movement.

[0046] Here, an example of a route for moving the vehicle 1 to a parking position in this embodiment will be described. FIG. 2 is a diagram showing an example of the route. As indicated by the arrow in FIG. 2, the vehicle 1 may travel on an ordinary road 200, which is a public road (including a highway, etc.), enter a user's private property 210, and move to a parking position 211, such as a garage, provided on the private property 210 as a target position. The section from when the vehicle 1 enters the private property 210 to the parking position 211 is a driveway 201, which indicates a road within the private property 210. Note that an obstacle 202 on the driveway 201 is, for example, a permanent trash can or plants. The control device 30 moves the vehicle 1 along the driveway 201 toward the parking position 211 based on external environment information acquired by the camera 111, etc., and route data 35a.

[0047] In autonomous driving, as is conventionally known, the vehicle 1 recognizes the surrounding conditions of the vehicle 1 using a camera 111 or the like mounted on the vehicle 1 and drives accordingly. In addition, at that time, a surrounding image showing the surrounding conditions of the vehicle 1 may be displayed on a display unit such as the touch panel 22 while driving. Meanwhile, some users may want to grasp the conditions in the traveling direction (for example, a distant condition not included in the surrounding image) rather than the surrounding conditions of the vehicle 1 while driving. In such cases, it is preferable that an image according to the driving conditions of the vehicle 1 is displayed on a display unit such as the touch panel 22. Therefore, in this embodiment, a configuration is made such that an image according to the driving conditions is displayed on a display unit such as the touch panel 22.

[0048] Specifically, the control device 30 executes, as an example of a program stored in the storage unit 35, an image display processing program that displays an image during autonomous driving when the start of autonomous driving is detected via operation of an operation switch on the operation input unit 80 or operation of the touch panel 22, etc. The control device 30 includes, as functional units realized by executing the program, an acquisition unit 31, an image processing unit 32, a control unit 33, and a display control unit 34. Note that, hereinafter, the processes described as being performed by the acquisition unit 31, the image processing unit 32, the control unit 33, and the display control unit 34 are processes realized by the control device 30.

[0049] The acquisition unit 31 acquires external environment information indicating the surrounding conditions of the vehicle 1. For example, the acquisition unit 31 performs sensor fusion processing on detection values ​​from some or all of the camera 111, sonar 112, and radar 113 included in the external environment sensor 11, recognizes the surrounding conditions of the vehicle 1 based on the processing results, and acquires external environment information indicating the recognition results. More specifically, the acquisition unit 31 recognizes the position, type, speed, acceleration, etc. of objects present around the vehicle 1. At this time, the acquisition unit 31 recognizes the position of the object as a position on absolute coordinates with a representative point of the vehicle 1 (e.g., the center of gravity or the center of the drive shaft) as the origin. This makes it possible to recognize the relative position between the vehicle 1 and the objects present around it. Furthermore, on the absolute coordinates, the position of the object may be represented using a representative point such as the center of gravity or a corner of the object, or may be represented as an area. Note that the acquisition unit 31 acquires the external environment information not only while the vehicle is traveling on the driveway 201 described above, but also while the vehicle is traveling on the general road 200 (i.e., it is constantly acquired while the vehicle is traveling).

[0050] The acquisition unit 31 can recognize, as an object present in the vicinity of the vehicle 1, for example, an obstacle 202 present in the vicinity of the vehicle 1. Possible examples of the obstacle 202 include a trash can permanently placed on the road, an object that has fallen onto the road, and other traffic participants (for example, other vehicles or pedestrians).

[0051] The acquisition unit 31 also recognizes, for example, the position and posture of the vehicle 1 while it is traveling. For example, the acquisition unit 31 recognizes the position of the vehicle 1 by acquiring road dividing lines around the vehicle 1 and dividing lines indicating parking spaces in parking positions from images captured by the camera 111. The acquisition unit 31 also recognizes the position and posture of the vehicle 1 based on, for example, the distance from the road dividing lines and dividing lines in the parking positions, and the angle between the vehicle 1 and the road dividing lines and the dividing lines in the parking positions. Note that the acquisition unit 31 may recognize the position and posture of the vehicle 1 by recognizing not only road dividing lines and dividing lines in parking positions, but also boundaries with the road and boundaries with the parking positions, including road shoulders, curbs, block walls, guardrails, etc.

[0052] Image processing unit 32 performs image processing based on the external environment information acquired by control device 30 using the function of acquisition unit 31. Specifically, image processing unit 32 generates a composite image (hereinafter also simply referred to as a "composite image") by combining the images captured by front camera 111a, rear camera 111b, left side camera 111c, and right side camera 111d based on the imaging data acquired by each of these cameras 111a to 111d. Then, based on the generated composite image, image processing unit 32 generates a first overhead image of vehicle 1 viewed from above and behind the center of vehicle 1, with the traveling direction of vehicle 1 included in the angle of view, and a second overhead image of vehicle 1 viewed from above in the vertical direction of vehicle 1.

[0053] Here, the "first bird's-eye image of vehicle 1 viewed from above behind the center of the vehicle, with the traveling direction included in the angle of view" refers to, for example, when vehicle 1 is moving forward, an image of vehicle 1 viewed from above at least behind the center of the vehicle (in other words, diagonally behind vehicle 1), with the traveling direction being the traveling direction and the angle of view including the traveling direction. FIG. 3 is a diagram showing an example of the first bird's-eye image of vehicle 1 viewed from above behind the vehicle. As shown in FIG. 3, by displaying the first bird's-eye image of vehicle 1 viewed from above behind vehicle 1 on touch panel 22, the user can grasp not only the situation around vehicle 1 but also the situation in the traveling direction.

[0054] Furthermore, the "second bird's-eye image of the vehicle viewed from above in the vertical direction of the vehicle" refers to an image of the vehicle 1 viewed from above (i.e., directly above the vehicle) with the vehicle 1 at the center. FIG. 4 is a diagram showing an example of the second bird's-eye image of the vehicle viewed from above in the vertical direction. As shown in FIG. 4, by displaying the second bird's-eye image of the vehicle 1 viewed from above in the vertical direction on the touch panel 22, the user can grasp the situation around the vehicle 1.

[0055] The first overhead image includes, for example, a captured image (camera image) based on the imaging data of camera 111, and a virtual image that is a virtual three-dimensional image generated based on the imaging data of camera 111. The captured image in the first overhead image is, for example, an image that shows vehicle 1 viewed from above and behind the center of the vehicle, obtained by performing image processing to reconstruct the above-mentioned composite image three-dimensionally, rotating vehicle 1 and the area around vehicle 1, and including the traveling direction in the angle of view.

[0056] The virtual image in the first overhead image is, for example, an image expressed by CG (Computer Graphics) as the vehicle 1 is viewed from above and behind the center of the vehicle, with the traveling direction included in the angle of view. For example, in the virtual image in the first overhead image, objects (e.g., obstacle 202) present around the vehicle 1 are represented by polygons or the like.

[0057] Similarly, the second overhead image includes, for example, a captured image (camera image) based on the imaging data of camera 111, and a virtual image that is a virtual three-dimensional image generated based on the imaging data of camera 111. The captured image in the second overhead image is, for example, an image that shows vehicle 1 viewed from above in the vertical direction by performing image processing to reconstruct the above-mentioned composite image three-dimensionally and rotating vehicle 1 and the area around vehicle 1.

[0058] The virtual image in the second overhead view image is, for example, an image that uses CG to represent the vehicle 1 as seen from above in the vertical direction. For example, in the virtual image in the second overhead view image, objects (e.g., obstacle 202) that exist around the vehicle 1 are represented by polygons or the like.

[0059] In the following description, for convenience of explanation, the captured image in the first overhead-view image will also be referred to as the "first captured image," the three-dimensional image in the first overhead-view image will also be referred to as the "first virtual image," the captured image in the second overhead-view image will also be referred to as the "second captured image," and the second overhead-view image will also be referred to as the "second virtual image." Furthermore, when there is no need to distinguish between "first" and "second," they will also be simply referred to as "overhead image," "captured image," and "virtual image."

[0060] Furthermore, image processing unit 32 sets a mask area in the generated images (first overhead image and second overhead image). The mask area refers to an area set to hide the body of vehicle 1 reflected in the image captured by camera 111. The mask area is set as an area having a shape that surrounds vehicle 1. Image processing unit 32 displays a vehicle image showing vehicle 1 superimposed on a portion of the mask area that corresponds to the space in which vehicle 1 is located. The vehicle images are the first overhead image, which is "an image of vehicle 1 viewed from above behind the center of vehicle 1, with the traveling direction of vehicle 1 included in the angle of view," and the second overhead image, which is "an image of vehicle 1 viewed from above in the vertical direction of vehicle 1," i.e., a vehicle image corresponding to the first overhead image and a vehicle image corresponding to the second overhead image.

[0061] The image generated by the image processing unit 32 may be only an image to be displayed on a display unit such as the touch panel 22, or may include not only the image to be displayed but also images other than the image to be displayed. That is, the image processing unit 32 may generate only an image to be displayed on the touch panel 22 (for example, a first virtual image) according to the surrounding situation of the moving vehicle 1, or may generate an image including other images (the first captured image, the second overhead image, and the second captured image).

[0062] The control unit 33 automatically moves the vehicle 1 to the parking position 211 based on the acquired surrounding conditions of the vehicle 1. Specifically, the control unit 33 performs drive control, braking control, and steering control of the vehicle 1 via the drive ECU 51, brake ECU 61, and EPS ECU 45, and performs automatic driving control with the parking position 211 specified by the user as the target parking position.

[0063] The display control unit 34 displays the image generated by the image processing unit 32 on a display unit such as the touch panel 22. Specifically, the display control unit 34 displays the first captured image and the second overhead image generated by the image processing unit 32 on the touch panel 22.

[0064] For example, when the distance from the current position of vehicle 1 to designated parking position 211 is equal to or greater than a predetermined value, display control unit 34 causes touch panel 22 to display the first overhead image. The image to be displayed on touch panel 22 while vehicle 1 is moving may be set, for example, by the user. However, in this embodiment, as described above, depending on the driving situation, it may be desirable to grasp the situation in the direction of travel of vehicle 1. For example, when the distance to the parking position is not close, in other words, when the distance is equal to or greater than a predetermined value, it may be desirable to grasp the situation in the direction of travel of vehicle 1 rather than the situation around vehicle 1. Therefore, display control unit 34 causes touch panel 22 to display the first overhead image generated by image processing unit 32. That is, display control unit 34 displays an image of an overhead view from above behind vehicle 1 to indicate the situation in the direction of travel.

[0065] Furthermore, it is preferable that the display control unit 34 displays the first virtual image on the touch panel 22 out of the first virtual image in the first overhead image and the first captured image. This is because the captured image may display more information than the virtual image, which may result in information overload for the user. The setting of which image to display, the first virtual image or the first captured image, may be appropriately set by the manufacturer of the vehicle 1, the user, etc. In this embodiment, the first virtual image is basically displayed unless a predetermined condition is met.

[0066] On the other hand, when the distance from the current position of vehicle 1 to parking position 211 is short, in other words, when the distance is less than a predetermined value, display control unit 34 causes touch panel 22 to display the second overhead image. That is, display control unit 34 causes an overhead image of vehicle 1 to be displayed, showing the situation around vehicle 1. This is because, since the parking position (i.e., the target position) is approaching, it is preferable that the image displayed on touch panel 22 be an image showing the situation around vehicle 1 rather than an image showing the situation in the direction of travel of vehicle 1.

[0067] In this way, the display control unit 34 switches the display between the first overhead image and the second overhead image depending on the distance between the current position of the vehicle 1 and the parking position 211 (depending on whether the distance is equal to or greater than a predetermined value or less than a predetermined value). Note that, because information about the inside of the private property 210, such as the driveway 201, is information that is not included in the map information database 24, the display control unit 34 determines whether the distance to the parking position 211 is less than a predetermined value by referring to the route data 35a. That is, when the route data 35a is recorded in the storage unit 35, the display control unit 34 refers to the route data 35a and switches the display between the first overhead image and the second overhead image depending on the distance to the parking position 211.

[0068] Conversely, if route data 35a is not recorded in storage unit 35 (for example, if it is a route that vehicle 1 is traveling for the first time, route data is not stored), the distance to parking position 211 cannot be determined even by referring to route data 35a, and therefore display control unit 34 switches the display between the first overhead image and the second overhead image according to the vehicle speed of vehicle 1. Specifically, display control unit 34 displays the first overhead image on touch panel 22 when the vehicle speed is equal to or greater than a predetermined speed, and displays the second overhead image on touch panel 22 when the vehicle speed is less than the predetermined speed. That is, display control unit 34 determines whether the distance to parking position 211 is less than the predetermined value based on the detection value of vehicle speed sensor 122, and switches the display between the first overhead image and the second overhead image. The vehicle speed may change depending on the sensing conditions of the external sensor 11, for example. For example, when the parking position 211 is within a predetermined distance during sensing, the vehicle speed is controlled to be less than the predetermined speed by the functions of the control unit 33, which performs drive control and braking control.

[0069] Furthermore, when the distance to parking position 211 is less than a predetermined value and the parking position 211 is not included in the imaging range of camera 111, display control unit 34 displays the second virtual image on touch panel 22. When the distance to parking position 211 is less than the predetermined value and the parking position 211 is included in the imaging range of camera 111, display control unit 34 displays the second captured image on touch panel 22. Specifically, when the distance from the current position of vehicle 1 to parking position 211 becomes less than the predetermined value, display control unit 34 switches the display of touch panel 22 from the first overhead image (first virtual image in this embodiment) to the second overhead image. At this time, the second virtual image is first displayed as the second overhead image. Then, when vehicle 1 further approaches parking position 211 while the second virtual image is being displayed and the parking position 211 is included in the imaging range of camera 111, display control unit 34 switches the display from the second virtual image to the second captured image.

[0070] Furthermore, when the distance to the parking position 211 is less than a predetermined value and the parking position 211 is included in the imaging range of the camera 111, the display control unit 34 causes the touch panel 22 to display a superimposed image in which the second captured image is superimposed on the second virtual image. The superimposed image is generated by the function of the image processing unit 32 described above. That is, the display control unit 34 causes the touch panel 22 to display the superimposed image generated by the image processing unit 32. Note that in this embodiment, the second virtual image and the second captured image are displayed on the touch panel 22 first, and therefore the superimposed image is configured to be superimposed on the displayed second virtual image. However, this relationship may be reversed. That is, the second virtual image may be superimposed on the second captured image.

[0071] Furthermore, it is preferable that the display control unit 34 display a captured image as the image to be displayed on the touch panel 22 when the distance to the parking position 211 is less than a predetermined value, or when the vehicle 1 passes through a location where an obstacle 202 is present or a location where the road is narrow during movement. This is because, when the vehicle 1 passes through a location where an obstacle 202 or the like is present or a location where the road is narrow, the user may want to grasp the sense of distance from the obstacle 202 or the road boundary. That is, when the vehicle 1 passes through a location where an obstacle 202 is present or a location where the road is narrow, the display control unit 34 displays, on the touch panel 22, the first captured image or the second captured image generated by the function of the image processing unit 32. Note that the image displayed on the touch panel 22 may be a captured image; however, as in the example of the parking position 211, the image to be displayed may be changed depending on, for example, the distance from the obstacle 202 or the imaging range of the camera 111. In other words, when the distance to the obstacle 202 is greater than or equal to a predetermined value or when the obstacle 202 is included in the imaging range, the first captured image may be displayed on the touch panel 22, and when the distance is less than the predetermined value or when the obstacle 202 is not included in the imaging range, the second captured image may be displayed on the touch panel 22.

[0072] Furthermore, when the vehicle 1 passes through a location where an obstacle 202 is present or a location where the road is narrow, the display control unit 34 causes the touch panel 22 to display a superimposed image in which the first captured image is superimposed on the first virtual image, or a superimposed image in which the second captured image is superimposed on the second virtual image. The superimposed image is generated by the function of the image processing unit 32 described above. That is, the display control unit 34 causes the superimposed image generated by the image processing unit 32 to be displayed on the touch panel 22. FIG. 5 is a diagram showing an example of a display on the touch panel 22 showing the superimposed image. When the vehicle 1 passes through a location where the obstacle 202 is present, an image in which the second captured image shown in a solid-line frame is superimposed on the second virtual image shown in a dashed-line frame is displayed. This allows a user viewing the display on the touch panel 22 to grasp an image of the vicinity of the obstacle 202, including the vehicle 1, from the captured image, and further to grasp the surrounding situation from the virtual image. In addition, the superimposed image generated by the image processing unit 32 may be generated by superimposing the first captured image (or the second captured image) on the first virtual image (or the second virtual image), or may be generated by superimposing the first virtual image (or the second virtual image) on the first captured image (or the second captured image).

[0073] When the first overhead image or the second overhead image is being displayed on the touch panel 22, the display control unit 34 updates the displayed image at predetermined intervals based on external environment information. That is, because the vehicle 1 is moving, the display control unit 34 dynamically updates the displayed image in accordance with the movement of the vehicle 1. When the display control unit 34 displays an overhead image on the touch panel 22 or switches the display of the overhead image, the display control unit 34 may, for example, notify the user through the speaker 23 of the overhead image to be displayed together. This is because the notification allows the user to easily understand which overhead image is currently being displayed.

[0074] [Processing performed by the control device] Next, an example of image display processing executed by the control device 30 will be described using a flowchart. FIG. 6 is a flowchart showing an example of the processing, and the processing is executed, for example, when a parking position, which is a target position, is specified and an operation switch for automatic driving is turned on. As described above, in this embodiment, as described with reference to FIG. 2, the vehicle 1 enters private property 210 from public road 200 and moves through driveway 201 to parking position 211. Furthermore, while the vehicle 1 is moving toward the parking position, external environment information is constantly acquired by external environment sensor 11, and the control device 30 generates overhead images based on the acquired external environment information at predetermined intervals. The generated overhead images are the first virtual image, the first captured image, the second virtual image, and the second captured image, as described above.

[0075] The control device 30 first determines whether the vehicle 1 has entered the driveway 201 (step S1), and if it is determined that the vehicle 1 has not entered the driveway 201 (No in step S1), it waits until the vehicle 1 enters the driveway 201. Conversely, if it is determined that the vehicle 1 has entered the driveway 201 (Yes in step S1), the control device 30 proceeds to step S2. Note that, when the information about the driveway 201 is recorded in the route data 35a, for example, it is possible to determine whether the vehicle 1 has entered the driveway 201 by referring to the route data 35a. On the other hand, for example, if it is the first time that vehicle 1 is entering driveway 201, information about driveway 201 will not be recorded in route data 35a. In this case, driveway 201 is detected by an external sensor 11 such as camera 111, and a determination is made as to whether or not driveway 201 has been entered based on the detected information.

[0076] In step S2, control device 30 starts displaying the first virtual image (step S2). That is, control device 30 causes the first virtual image generated by image processing unit 32 to be displayed on a display unit such as touch panel 22 using the function of display control unit 34. Note that the reason for displaying the first virtual image among the generated images is to display the situation in the traveling direction of vehicle 1 and because captured images may provide too much information to the user.

[0077] Next, the control device 30 determines whether or not the route data 35a is recorded in the storage unit 35 (step S3). This is because the means for determining whether the vehicle 1 has approached the parking position 211 may change depending on the recorded route data 35a. If it is determined in step S3 that the route data 35a is recorded in the storage unit 35 (Yes in step S3), the control device 30 proceeds to step S4. Note that the processing to be performed in the case where it is determined in step S3 that the route data 35a is not recorded in the storage unit 35 will be described later.

[0078] In step S4, the control device 30 determines whether the distance to the parking position 211 is less than a predetermined value. That is, the control device 30 refers to the route data 35a to recognize the current position of the vehicle 1, and determines whether the distance between the current position of the vehicle 1 and the parking position 211 is less than a predetermined value. If it is determined that the distance to the parking position 211 is equal to or greater than a threshold value (No in step S4), the control device 30 waits until the threshold value becomes less than the threshold value.

[0079] On the other hand, if it is determined that the distance to the parking position 211 is less than the threshold value (Yes in step S4), the control device 30 switches the image to be displayed to the second virtual image (step S5). That is, the control device 30 switches the image displayed on the touch panel 22 from the first virtual image to the second virtual image using the function of the display control unit 34. That is, because the parking position 211 is close, the control device 30 switches the image to be displayed from the image showing the situation in the traveling direction to an image showing the surroundings of the vehicle 1.

[0080] Next, the control device 30 determines whether the parking position 211 is within the imaging range (step S6). That is, the control device 30 determines whether the parking position 211 is included in the imaging range of the camera 111. If it is determined that the parking position 211 is not included in the imaging range (No in step S6), the control device 30 returns the process to step S4.

[0081] On the other hand, if it is determined that the parking position 211 is included in the imaging range (Yes in step S6), the control device 30 switches the image to be displayed to the second captured image (step S7). That is, the control device 30 switches the image displayed on the touch panel 22 from the second virtual image to the second captured image using the function of the display control unit 34. That is, the control device 30 switches the displayed image from the second virtual image to the second captured image in order to display a more detailed image of the surroundings of the vehicle 1 on the touch panel 22.

[0082] Next, the control device 30 determines whether parking at the parking position 211 is complete (step S8). That is, the control device 30 determines whether movement of the vehicle 1 to the target position designated by the user is complete. For example, the control device 30 identifies the position and attitude of the vehicle 1 based on detection by the camera 111 or the like using the function of the acquisition unit 31, and determines that parking at the parking position 211 is complete when the vehicle 1 fits into the parking space of the designated parking position 211 and the attitude of the vehicle 1 is parallel to the line indicating the parking space. Alternatively, the control device 30 determines that parking is complete when it determines that the shift position Ps of the shift device has become parking (P). Note that the shift position Ps is determined based on a signal input from the shift position sensor 53 to the control device 30, for example. In step S8, if it is determined that parking at the parking position 211 is not complete (No in step S8), the control device 30 waits until parking is complete.

[0083] On the other hand, if it is determined that parking at the parking position 211 is complete (Yes in step S8), the control device 30 ends the display of the second captured image that was being displayed on the touch panel 22 (step S9). That is, since the movement of the vehicle 1 is complete, the control device 30 ends the display of the image of the surroundings of the vehicle 1, etc.

[0084] Next, the processing to be performed when a negative determination is made in step S3 above will be described with reference to Fig. 7. Among the steps shown in Fig. 7, steps S11 to S14 are similar to the processing in steps S6 to S9 above. Therefore, detailed description will be omitted and the description will be simplified.

[0085] If it is determined in step S3 that the route data 35a is not recorded in the memory unit 35 (No in step S3), the control device 30 proceeds to step S10. In step S10, the control device 30 determines whether the vehicle speed of the vehicle 1 is less than a predetermined speed. That is, since the route data 35a is not recorded in the memory unit 35, the control device 30 determines whether the vehicle 1 is approaching the parking position 211 based on the vehicle speed. If it is determined that the vehicle speed is equal to or greater than the predetermined speed (No in step S10), the control device 30 waits until the vehicle speed becomes less than the predetermined speed.

[0086] On the other hand, if it is determined that the vehicle speed is less than the predetermined speed (Yes in step S10), the control device 30 determines whether the parking position 211 is within the imaging range of the camera 111 (step S11). If it is determined that the parking position 211 is not included in the imaging range (No in step S11), the control device 30 returns the process to step S10.

[0087] On the other hand, if it is determined that the parking position 211 is included in the imaging range (Yes in step S11), the image displayed on the touch panel 22 is switched from the first virtual image to the second captured image (step S12).

[0088] Next, the control device 30 determines whether parking at the parking position 211 is complete (step S13). If it is determined that parking at the parking position 211 is not complete (No in step S13), the control device 30 waits until parking is complete.

[0089] On the other hand, if it is determined that parking in the parking position 211 is complete (Yes in step S13), the control device 30 ends the display of the second captured image that has been displayed on the touch panel 22 (step S14).

[0090] Next, an example of an image display process for displaying a captured image when the presence of an obstacle 202 is recognized while the vehicle 1 is being moved to the parking position 211 will be described. Fig. 8 is a flowchart showing an example of this process. The process shown in this flowchart is executed in parallel with the processes described above in Figs. 6 and 7, for example, and is executed at predetermined intervals.

[0091] First, the control device 30 determines whether or not the vehicle 1 is traveling on the driveway 201 (step S20). For example, if information about the driveway 201 is recorded in the route data 35a, the control device 30 refers to the route data 35a to determine whether or not the vehicle 1 is traveling on the driveway 201. If information about the driveway 201 is not recorded in the route data 35a, the control device 30 determines whether or not the vehicle 1 is traveling on the driveway 201 based on the detection value from the external sensor 11. If it is determined in step S20 that the vehicle 1 is not traveling on the driveway 201 (No in step S20), the control device 30 temporarily terminates the processing of the flowchart shown in FIG. 8.

[0092] On the other hand, if it is determined that the vehicle 1 is traveling on the driveway 201 (Yes in step S20), the control device 30 determines whether the parking position 211 is within the imaging range of the camera 111 (step S21). If it is determined that the parking position 211 is included in the imaging range of the camera 111 (Yes in step S21), the control device 30 temporarily ends the processing of the flowchart shown in FIG. 8 because the second captured image is already displayed on the touch panel 22, as described above in step S7 of FIG. 6 and step S12 of FIG. 7.

[0093] On the other hand, if it is determined that the parking position 211 is not included in the imaging range of the camera 111 (No in step S21), the control device 30 determines whether or not there is an obstacle 202 around the vehicle 1 (step S22). For example, the control device 30 determines whether or not there is an obstacle 202 by referring to the route data 35a or based on the detection value of the external sensor 11. If it is determined that there is no obstacle 202 around the vehicle 1 (No in step S22), the control device 30 temporarily ends the processing of the flowchart shown in Fig. 8. On the other hand, if it is determined that there is an obstacle 202 around the vehicle 1 (Yes in step S22), the control device 30 proceeds to the processing of step S23.

[0094] In step S23, the control device 30 switches the display on the touch panel 22 to the first captured image or the second captured image. When the vehicle 1 is traveling in a situation where the presence of an obstacle 202 is not detected, it can be assumed that the first virtual image is displayed on the touch panel 22. Therefore, when it is determined that an obstacle 202 is present around the vehicle 1, the control device 30 switches the display on the touch panel 22 to the captured image using the function of the display control unit 34. The captured image displayed at this time may be either the first captured image or the second captured image. For example, when the obstacle 202 is not within the imaging range of the camera 111, the control device 30 displays the first captured image, and when the obstacle 202 is within the imaging range, the control device 30 displays the second captured image.

[0095] Next, the control device 30 determines whether the vehicle 1 has passed the periphery of the obstacle 202 (step S24). The control device 30 determines whether the vehicle 1 has passed the periphery of the obstacle 202, for example, by referring to the route data 35a or based on the detection value of the external sensor 11. If it is determined in step S24 that the vehicle 1 has not passed the periphery of the obstacle 202 (No in step S24), the control device 30 waits until the vehicle 1 has passed the periphery of the obstacle 202.

[0096] On the other hand, if it is determined that the vehicle 1 has passed through the periphery of the obstacle 202 (Yes in step S24), the control device 30 restores the image displayed before switching to the captured image (first captured image or second captured image) in step S23 (step S25). That is, if the first virtual image was displayed on the touch panel 22 before switching to the captured image in step S23, the display is switched to the first virtual image. Note that instead of this switching (i.e., restoring the display), the control device 30 may determine whether the distance to the parking position 211 when the vehicle 1 has passed the obstacle 202 is within a predetermined value, whether the parking position 211 is included in the captured image range, or whether the vehicle speed is less than a predetermined speed, and switch the image displayed on the touch panel 22 based on the determination result.

[0097] As described above, in this embodiment, the overhead image displayed on the touch panel 22 is switched depending on the distance to the parking position 211. That is, when the distance to the parking position 211 is equal to or greater than a predetermined value, the control device 30 displays a first overhead image that allows the user to grasp the situation in the traveling direction of the vehicle 1, and when the distance to the parking position 211 is less than the predetermined value, the control device 30 displays a second overhead image that allows the user to grasp the situation around the vehicle 1. Since the display image requested by the user may vary along the route, switching the display depending on the distance to the target position, such as a parking position, enables a display that is appropriate for the vehicle's traveling situation. By executing such processing, the control device 30 can ultimately improve traffic safety and contribute to the development of a sustainable transportation system.

[0098] Furthermore, in this embodiment, when switching the display between the first overhead image and the second overhead image, if route data 35a is recorded, the control device 30 switches the display between the first overhead image and the second overhead image based on the route data 35a. As described above, the driveway 201, etc., is a road on private property, and therefore no route exists in the map information database 24. Even in such a case, by recording route data 35a different from that in the map information database 24, the position of the vehicle 1 can be recognized and the display image can be switched based on the route data 35a.

[0099] Furthermore, when route data 35a is not recorded, control device 30 switches the display between the first overhead image and the second overhead image according to the vehicle speed. For example, control device 30 displays the first overhead image when the vehicle speed is equal to or greater than a predetermined speed, and displays the second overhead image when the vehicle speed is less than the predetermined speed. In this way, it is possible to determine whether the vehicle is approaching a target position such as a parking position according to changes in vehicle speed, and switch the overhead image to be displayed.

[0100] Furthermore, in this embodiment, control device 30 displays the second virtual image when the distance to parking position 211 is less than a predetermined speed and parking position 211 is not included in the imaging range of camera 111, and displays the second captured image when the distance to parking position 211 is less than a predetermined speed and parking position 211 is included in the imaging range. In this way, by switching the image to be displayed in the second overhead image depending on whether parking position 211 is included in the imaging range, it is possible to display a captured image that is more detailed than the virtual image when the distance to the parking position is closer.

[0101] Furthermore, when the vehicle 1 passes through an area where an obstacle or the like is present, the control device 30 also displays the first captured image or the second captured image on the touch panel 22. When passing through an area where an obstacle or the like is present, the user may want to see a detailed image. Therefore, by displaying such a captured image, it is possible to meet the user's request.

[0102] Furthermore, in this embodiment, the control device 30 superimposes the captured image on the virtual image when the parking position 211 is included in the imaging range or when the vehicle 1 passes through a location where an obstacle 202 is present. As described above, when the vehicle approaches a target position such as the parking position 211 or passes through a location where an obstacle 202 is present, it is preferable to display a captured image that can display a detailed image; however, the captured image alone may not provide enough information about the surrounding situation. Therefore, by superimposing the captured image on the virtual image that is displayed before the captured image is displayed, the surrounding image closer to the vehicle 1 is displayed by the captured image, and the area around the surrounding image is displayed by the virtual image. This reduces the possibility that the displayed image will not provide enough information about the surrounding situation of the vehicle 1.

[0103] In this embodiment, by displaying on the touch panel 22 an overhead image showing the situation in the direction of travel of the traveling vehicle 1 and an overhead image showing the situation around the vehicle 1, it becomes possible to execute an override that allows a user to intervene in an operation when, for example, an abnormality occurs in the behavior of the vehicle 1 during autonomous driving. In other words, by displaying not only an image of the situation around the vehicle 1 but also an image of the situation in the direction of travel, the user can execute an override with ample time to maneuver.

[0104] [Other embodiments] Next, other embodiments will be described. In the above-described embodiments, the direction in which the vehicle 1 moves forward has been described as the traveling direction. However, the vehicle 1 may also move in a reverse direction as the traveling direction. For example, using the example of the driveway 201 described above, a case can be envisioned in which the vehicle 1 moves backward from the parking position 211 and exits onto the public road 200 through the driveway 201. That is, there is a case in which the vehicle exits onto the public road 200 by backing out from the parking position 211. FIG. 9 shows an example of exiting from the driveway 201 onto the public road 200 during such backing out. In FIG. 9 , an overhead image of the vehicle 1 viewed from above behind the center of the vehicle (i.e., in front of the vehicle), with the traveling direction of the vehicle 1 (the direction of the arrow) included in the angle of view, is displayed on the touch panel 22 so that the user can grasp the situation in the direction in which the user wants to back out of the vehicle 1. This allows the user to grasp the situation in the direction in which the user wants to back out of the vehicle 1, thereby increasing the likelihood that backing out will be easier compared to, for example, when only an image of the vehicle 1's surroundings is displayed on the touch panel 22.

[0105] In the above embodiment, an example in which the image display process is executed during automatic driving has been described, but the process may be executed not only during automatic driving but also during manual driving. By executing the process during manual driving, the user can drive while viewing the overhead image displayed on the touch panel 22, thereby providing effective driving assistance to the user.

[0106] 8 is applied to an example where an obstacle 202 or the like is present, and may also be applied, for example, when passing through a location where the road is narrow. That is, when the vehicle 1 passes through a location where the road is narrower than the vehicle 1, the image displayed on the touch panel 22 may be switched to the first captured image or the second captured image. That is, this processing may be applied when passing through a location where caution or attention should be paid while traveling, in addition to a target location such as the parking position 211.

[0107] [others] Although one embodiment of the present invention has been described above with reference to the drawings, it goes without saying that the present invention is not limited to the above-described embodiment. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0108] For example, the above-mentioned image display processing was configured to be executed when an operation to turn on automatic driving is performed, but it may also be executed when, for example, vehicle 1 enters driveway 201 and an operation to turn on automatic parking is performed.

[0109] Furthermore, for example, the above-mentioned parking position 211 is set by the user's specification, but the parking position 211 may be directly specified by operating the touch panel 22 or the like, or, for example, the user may specify a home, and thereby a parking position 211 that is pre-linked to the home may be specified.

[0110] The control method described in the above-described embodiment can be realized by executing a prepared control program on a computer. The control program is recorded on a computer-readable storage medium and executed by being read from the storage medium. The control program may be provided in a form stored on a non-transitory storage medium such as a flash memory, or may be provided via a network such as the Internet. The computer that executes the control program may be included in a control device, or may be included in an electronic device such as a smartphone, tablet, or personal computer that can communicate with the control device, or may be included in a server device that can communicate with these control devices and electronic devices.

[0111] This specification describes at least the following: Note that the components in parentheses correspond to those in the above-described embodiments, but are not limited to these.

[0112] (1) A control device (control device 30) that moves a vehicle (vehicle 1) to a predetermined target position (parking position 211), an acquisition unit (acquisition unit 31) that acquires external environment information indicating the surrounding situation of the vehicle from an external environment sensor (external environment sensor 11) including an imaging device (camera 111) that captures an image of the surroundings of the vehicle; an image processing unit (image processing unit 32) that can generate, based on the acquired external environment information, a first overhead image of the vehicle viewed from above and behind the center of the vehicle, with the traveling direction of the vehicle included in the angle of view, and a second overhead image of the vehicle viewed from above in a vertical direction of the vehicle; a control unit (control unit 33) that controls the vehicle to move to the target position; a display control unit (display control unit 34) that displays the image generated by the image processing unit on a predetermined display unit (touch panel 22); The display control unit When the distance to the target position is equal to or greater than a predetermined value, the first overhead image is displayed on the display unit; When the distance to the target position is less than the predetermined value, the second overhead image is displayed on the display unit. Control device.

[0113] According to (1), when the distance to the target position is equal to or greater than a predetermined value, a first overhead image that allows the user to grasp the situation in the direction of travel of the vehicle is displayed, and when the distance to the target position is less than the predetermined value, a second overhead image that allows the user to grasp the situation around the vehicle is displayed. Since the display image requested by the user may vary along the route, switching the display in this way depending on the distance to the target position, such as a parking position, makes it possible to display an image that suits the vehicle's traveling situation. By performing such processing, the control device can ultimately improve traffic safety and contribute to the development of a sustainable transportation system.

[0114] (2) The control device according to (1), a storage unit (storage unit 35) for recording route data (route data 35a) which is a route history when the vehicle moves to the target position; The display control unit When the route data is recorded in the storage unit, the display of the first overhead image and the second overhead image is switched in accordance with the distance to the target position; When the route data is not recorded in the storage unit, the display of the first overhead image and the second overhead image is switched in accordance with the vehicle speed. Control device.

[0115] According to (2), it is possible to switch the display between the first overhead image and the second overhead image based on the route data.

[0116] (3) The control device according to (2), The display control unit When the route data is not recorded in the storage unit and the vehicle speed is equal to or greater than a predetermined speed, the first overhead image is displayed on the display unit; When the route data is not recorded in the storage unit and the vehicle speed is less than the predetermined speed, the second overhead image is displayed on the display unit. Control device.

[0117] According to (3), when no route data is recorded in the recording unit, the display can be switched between the first overhead image and the second overhead image in accordance with the vehicle speed.

[0118] (4) The control device according to (1), the first overhead image is displayed using at least one of a first captured image based on imaging data of the imaging device and a first virtual image which is a virtual three-dimensional image generated based on the imaging data; the second overhead image is displayed using at least one of a second captured image based on the imaging data and a second virtual image that is a virtual image generated based on the imaging data, The display control unit when the distance to the target position is less than the predetermined value and the target position is not included in an imaging range of the imaging device, displaying the second virtual image on the display unit; When the distance to the target position is less than the predetermined value and the target position is included in the imaging range, displaying the second captured image on the display unit; When an obstacle is present around the vehicle, the first captured image or the second captured image is displayed on the display unit. Control device.

[0119] According to (4), by switching the image displayed in the second overhead image between the second virtual image and the second captured image depending on whether the target position is included in the imaging range, it is possible to display a captured image that is more detailed than the virtual image when, for example, the distance to the parking position becomes closer. Also, when passing through a location where an obstacle or the like is present, it is possible to display a captured image that is more detailed than the virtual image.

[0120] (5) The control device according to (4), The image processing unit superimposing the second captured image on the second virtual image when the distance to the target position is less than the predetermined value and the target position is included in the imaging range; The display control unit displaying the superimposed image on the display unit; Control device.

[0121] According to (5), when the target position is included in the imaging range, by superimposing the captured image on the virtual image, it is possible to supplement information that may be lacking, for example, in the captured image alone, with the virtual image.

[0122] (6) The control device according to (4), The image processing unit When an obstacle is present around the vehicle, the first captured image is superimposed on the first virtual image, or the second captured image is superimposed on the second virtual image; The display control unit displaying the superimposed image on the display unit; Control device.

[0123] According to (6), when an obstacle is present, by superimposing the captured image on the virtual image, it is possible to supplement information that may be lacking from the captured image alone, for example, with the virtual image.

[0124] (7) A computer that moves a vehicle (vehicle 1) to a predetermined target position (parking position 211) Acquire external environment information indicating the surrounding situation of the vehicle from an external environment sensor (external environment sensor 11) including an imaging device (camera 111) that captures an image of the surroundings of the vehicle; generating a first overhead image of the vehicle viewed from above behind the center of the vehicle and a second overhead image of the vehicle viewed from above in a vertical direction of the vehicle, the first overhead image including the traveling direction of the vehicle in an angle of view based on the acquired external environment information; performing control to move the vehicle to the target position; The generated image is displayed on a predetermined display unit (touch panel 22), When the distance to the target position is equal to or greater than a predetermined value, the first overhead image is displayed on the display unit; When the distance to the target position is less than the predetermined value, a process is executed to display the second overhead image on the display unit. Control method.

[0125] According to (7), when the distance to the target position is equal to or greater than a predetermined value, a first overhead image that allows the user to grasp the situation in the direction of travel of the vehicle is displayed, and when the distance to the target position is less than the predetermined value, a second overhead image that allows the user to grasp the situation around the vehicle is displayed. Since the display image requested by the user may differ along the route, in this way, by switching the display depending on the distance to the target position such as a parking position, it is possible to display an image that suits the vehicle's traveling situation.

[0126] (8) A computer that moves a vehicle (vehicle 1) to a predetermined target position (parking position 211) Acquire external environment information indicating the surrounding situation of the vehicle from an external environment sensor (external environment sensor 11) including an imaging device (camera 111) that captures an image of the surroundings of the vehicle; generating a first overhead image of the vehicle viewed from above behind the center of the vehicle and a second overhead image of the vehicle viewed from above in a vertical direction of the vehicle, the first overhead image including the traveling direction of the vehicle in an angle of view based on the acquired external environment information; performing control to move the vehicle to the target position; The generated image is displayed on a predetermined display unit (touch panel 22), When the distance to the target position is equal to or greater than a predetermined value, the first overhead image is displayed on the display unit; When the distance to the target position is less than the predetermined value, a process is executed to display the second overhead image on the display unit. Control program.

[0127] According to (8), when the distance to the target position is equal to or greater than a predetermined value, a first overhead image that allows the user to grasp the situation in the direction of travel of the vehicle is displayed, and when the distance to the target position is less than the predetermined value, a second overhead image that allows the user to grasp the situation around the vehicle is displayed. Since the display image requested by the user may differ along the route, in this way, by switching the display depending on the distance to the target position such as a parking position, it is possible to display an image that suits the vehicle's traveling situation.

[0128] (9) A control device (control device 30) that moves a vehicle (vehicle 1) to a predetermined target position (parking position 211), an acquisition unit (acquisition unit 31) that acquires external environment information indicating the surrounding situation of the vehicle from an external environment sensor (external environment sensor 11) including an imaging device (camera 111) that captures an image of the surroundings of the vehicle; an image processing unit (image processing unit 32) capable of generating an overhead image of the vehicle from above and behind the center of the vehicle, with the traveling direction of the vehicle included in the angle of view, based on the acquired external environment information; a control unit (control unit 33) that controls the vehicle to move to the target position; a display control unit (display control unit 34) that displays the overhead image generated by the image processing unit on a predetermined display unit (touch panel 22), the display control unit causes the display unit to display the overhead image when the vehicle moves backward and exits into a predetermined location. Control device.

[0129] According to (9), an overhead image of the vehicle viewed from above in the direction behind the vehicle (i.e., in front of the vehicle) in the direction of travel is displayed on the display unit so that the user can grasp the situation in the direction in which the user wants to reverse the vehicle out of the parking lot. This allows the user to grasp the situation in the direction in which the user wants to reverse the vehicle, making it more likely that the user will be able to reverse out of the parking lot easier than, for example, when only an image of the vehicle's surroundings is displayed on the display unit. [Explanation of symbols]

[0130] 1 vehicle 11 External Sensors 22 Touch Panel 30 Control device 31 Acquisition Department 32 Image processing section 33 Control Unit 34 Display control unit 35 Storage section 35a Route Data 111 Camera 211 Parking location

Claims

1. A control device that moves a vehicle to a predetermined target position, an acquisition unit that acquires external environment information indicating a surrounding situation of the vehicle from an external environment sensor including an imaging device that captures an image of the surroundings of the vehicle; an image processing unit that generates a first bird's-eye view image and a second bird's-eye view image, each of which is a bird's-eye view of the vehicle at a different angle of view, based on the acquired external environment information, and that generates the first bird's-eye view image with a wider angle of view in the traveling direction of the vehicle than the second bird's-eye view image; a control unit that controls the vehicle to move to the target position; a display control unit that displays the image generated by the image processing unit on a predetermined display unit; a storage unit for recording route data that is a route history when the vehicle moves to the target position, The control unit performing control to move the vehicle to the target position based on the route data stored in the storage unit; The display control unit When the distance to the target position is equal to or greater than a predetermined value, the first overhead image is displayed on the display unit; When the distance to the target position is less than the predetermined value, the second overhead image is displayed on the display unit; When the route data is recorded in the storage unit, the route data is referenced and the display of the first overhead image and the second overhead image is switched in accordance with the distance to the target position. Control device.

2. The control device according to claim 1, The display control unit When the route data is not recorded in the storage unit, the display of the first overhead image and the second overhead image is switched in accordance with the vehicle speed. Control device.

3. The control device according to claim 2, The display control unit When the route data is not stored in the storage unit and the vehicle speed is equal to or higher than a predetermined speed, the first overhead image is displayed on the display unit; When the route data is not stored in the storage unit and the vehicle speed is less than the predetermined speed, the second overhead image is displayed on the display unit. Control device.

4. The control device according to claim 1, the first overhead image is displayed using at least one of a first captured image based on imaging data of the imaging device and a first virtual image which is a virtual three-dimensional image generated based on the imaging data, the second overhead image is displayed using at least one of a second captured image based on the imaging data and a second virtual image that is a virtual image generated based on the imaging data, The display control unit when the distance to the target position is less than the predetermined value and the target position is not included in an imaging range of the imaging device, displaying the second virtual image on the display unit; When the distance to the target position is less than the predetermined value and the target position is included in the imaging range, the second captured image is displayed on the display unit; When an obstacle is present around the vehicle, the first captured image or the second captured image is displayed on the display unit. Control device.

5. The control device according to claim 4, The image processing unit superimposing the second captured image on the second virtual image when the distance to the target position is less than the predetermined value and the target position is included in the imaging range; The display control unit displaying the superimposed image on the display unit; Control device.

6. The control device according to claim 4, The image processing unit When an obstacle is present around the vehicle, the first captured image is superimposed on the first virtual image, or the second captured image is superimposed on the second virtual image; The display control unit displaying the superimposed image on the display unit; Control device.

7. A computer that moves a vehicle to a predetermined target position acquiring external environment information indicating a surrounding situation of the vehicle from an external environment sensor including an imaging device that captures an image of the surroundings of the vehicle; a first overhead image and a second overhead image, each of which is a bird's-eye view of the vehicle at a different angle of view, can be generated based on the acquired external environment information, and the first overhead image has a wider angle of view in the traveling direction of the vehicle than the second overhead image; performing control to move the vehicle to the target position; Displaying the generated image on a predetermined display unit; execute a process to record route data that is a route history when the vehicle moves to the target position; In the process of moving the vehicle to the target position, the vehicle is moved to the target position based on the recorded route data; In the process of displaying the generated image on the display unit, when the distance to the target position is equal to or greater than a predetermined value, the first overhead image is displayed on the display unit; When the distance to the target position is less than the predetermined value, the second overhead image is displayed on the display unit; When the route data is recorded, the route data is referenced and the display of the first overhead image and the second overhead image is switched in accordance with the distance to the target position. Control method.

8. A computer that moves a vehicle to a predetermined target position acquiring external environment information indicating a surrounding situation of the vehicle from an external environment sensor including an imaging device that captures an image of the surroundings of the vehicle; a first overhead image and a second overhead image, each of which is a bird's-eye view of the vehicle at a different angle of view, can be generated based on the acquired external environment information, and the first overhead image has a wider angle of view in the traveling direction of the vehicle than the second overhead image; performing control to move the vehicle to the target position; Displaying the generated image on a predetermined display unit; Execute a process to record route data that is a route history when the vehicle moves to the target position; In the process of moving the vehicle to the target position, the vehicle is moved to the target position based on the recorded route data; In the process of displaying the generated image on the display unit, when the distance to the target position is equal to or greater than a predetermined value, the first overhead image is displayed on the display unit; When the distance to the target position is less than the predetermined value, the second overhead image is displayed on the display unit; When the route data is recorded, the route data is referenced and the display of the first overhead image and the second overhead image is switched in accordance with the distance to the target position. Control program.

Images