Image display system

Abstract

Image display system. An image display system includes: an external environment information acquisition device configured to acquire external environment information around a mobile body; a display device configured to display the external environment information; and a control device configured to control display of the external environment information on the display device, wherein the control device is configured to generate a virtual viewpoint image that observes the mobile body and an object around the mobile body from a virtual viewpoint, cause the display device to display the virtual viewpoint image, move the virtual viewpoint along a track disposed around the mobile body, dispose a restricted section in the track so that the restricted section is arranged in front of the mobile body in a plan view, and perform viewpoint restriction control to restrict display of the virtual viewpoint image from the virtual viewpoint in the restricted section.

Description

TECHNICAL FIELD

[0001] The present application relates to an image display system. BACKGROUND

[0002] Conventionally, an image display system can display an image (hereinafter referred to as "virtual viewpoint image") that views a moving body such as a vehicle and an object around the moving body from a virtual viewpoint.

[0003] For example, JP 2020-43418 A discloses a surrounding area monitoring device including an image generation unit and an output unit. The image generation unit generates a virtual viewpoint image so that a target parking area and a vehicle are both at least partially retained in a display area. The output unit outputs the virtual viewpoint image to a display device.

[0004] In the image display system as described above, the virtual viewpoint can move along a trajectory that is set around the moving body. In this case, if a virtual viewpoint image that views the moving body and the object from the front is displayed, the user can have difficulty in understanding whether the object is on the left side or the right side of the moving body in the actual space. SUMMARY

[0005] In view of the above-described background, an object of the present application is to enable a user to easily understand whether an object is on the left side or the right side of a moving body in the actual space.

[0006] To achieve such an object, one aspect of the present application provides an image display system 1 including an external environment information acquisition device 19 configured to acquire external environment information around a moving body V, a display device 32 configured to display external environment information acquired by the external environment information acquisition device, and a control device 15 configured to control display of the external environment information on the display device, wherein the control device is configured to generate a virtual viewpoint image 51 that views the moving body and an object Y around the moving body from a virtual viewpoint A, cause the display device to display the virtual viewpoint image, move the virtual viewpoint along a trajectory B that is set around the moving body, set a restriction section C in the trajectory so that the restriction section is arranged in front of the moving body in a plan view, and perform viewpoint restriction control to restrict display of the virtual viewpoint image from the virtual viewpoint in the restriction section.

[0007] According to this aspect, by restricting display of the virtual viewpoint image that views the moving body and the object from the front, the user can easily understand whether the object is on the left side or the right side of the moving body in the actual space.

[0008] In the above aspect, preferably, the control device is configured to generate a reversed rearward image 52 in which a rearward image from the moving body is reversed in the lateral direction, and cause the display device to display the reversed rearward image.

[0009] According to this aspect, when the image on the display device is switched from one of the virtual viewpoint image and the reversed rearward image to the other, the position of the object included in these images can move from one of the left side and the right side of the moving body to the other. If such movement of the object occurs, the user can have difficulty in understanding whether the object is on the left side or the right side of the moving body in the actual space. In this way, by limiting the display of the virtual viewpoint image in which the moving body and the object are viewed from the front, it is possible to suppress the movement of the object as described above. Therefore, the user can easily understand whether the object is on the left side or the right side of the moving body in the actual space.

[0010] In the above aspect, preferably, the shift range of the moving body is switchable between a plurality of ranges including a forward range in which the moving body is caused to move forward and a rearward range in which the moving body is caused to move rearward, and the control device performs the viewpoint restriction control when the shift range of the moving body is set to the forward range.

[0011] According to this aspect, when the shift range of the moving body is set to the forward range, the user can easily understand whether the object is on the left side or the right side of the moving body in the actual space.

[0012] In the above aspect, preferably, the control device does not perform the viewpoint restriction control when the shift range of the moving body is set to the rearward range.

[0013] According to this aspect, when the shift range of the moving body is set to the rearward range, the display of the virtual viewpoint image is not restricted by the viewpoint restriction control, so the user can improve the visibility of the surrounding environment of the moving body.

[0014] In the above aspect, preferably, the control device is configured to generate a reversed rearward image in which a rearward image from the moving body is reversed in the lateral direction, and cause the display device to display the reversed rearward image when the shift range of the moving body is set to the rearward range.

[0015] According to this aspect, by displaying the reversed rearward image when the shift range of the moving body is set to the rearward range, the user can further improve the visibility of the surrounding environment of the moving body.

[0016] In the above aspect, preferably, in a case where the shift range of the moving body is switched from the forward range to the rearward range in a state in which the display device displays the virtual viewpoint image, the control device switches the image on the display device from the virtual viewpoint image to the reversed rearward image.

[0017] According to this aspect, in response to the switching of the shift range, the image on the display device is also switched after the switching to an image suitable for the shift range, so that the user can further improve the visibility of the surroundings of the moving body. On the other hand, if the image on the display device is switched from the virtual viewpoint image to the reversed rearward image as described above, the position of the object included in these images moves from one of the left and right sides of the moving body to the other side. In this way, by limiting the display of the virtual viewpoint image that views the moving body and the object from the front, it is possible to suppress the movement of the object as described above. Therefore, the user can easily understand whether the object is on the left side or the right side of the moving body in the actual space.

[0018] In the above aspect, preferably, the control device is configured to execute a search process to search for the stop position of the moving body, and the viewpoint restriction control is executed during the search process.

[0019] According to this aspect, by executing the viewpoint restriction control while searching for the stop position of the moving body, the user can easily understand whether the stop position is on the left side or the right side of the moving body.

[0020] In the above aspect, preferably, during the viewpoint restriction control, the control device restricts the display of the virtual viewpoint image as long as the virtual viewpoint is within the restricted section while allowing the virtual viewpoint to enter the restricted section.

[0021] According to this aspect, even during the viewpoint restriction control, the virtual viewpoint can pass through the restricted section. Therefore, it is possible to quickly move the virtual viewpoint from a region on one side of the restricted section to another region on the other side. Therefore, it is possible to improve the convenience of the user.

[0022] In the above aspect, preferably, during the viewpoint restriction control, the control device restricts the virtual viewpoint from entering the restricted section.

[0023] According to this aspect, by restricting the entry of the virtual viewpoint into the restricted section itself, it is possible to reliably make the user recognize that the display of the virtual viewpoint image from the virtual viewpoint in the restricted section is restricted.

[0024] Therefore, according to the above aspect, the user can easily understand whether the object is on the left side or the right side of the moving body in the actual space. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a functional block diagram showing a vehicle provided with a parking assistance system according to an embodiment of the present application;

[0026] Figure 2 is a flowchart of an automatic parking process in the parking assistance system according to the embodiment;

[0027] Figure 3A is a view showing display of a screen of a touch panel during receiving processing at a target parking position in the parking assistance system according to the embodiment;

[0028] Figure 3B is a view showing display of a screen of a touch panel during driving processing in the parking assistance system according to the embodiment;

[0029] Figure 3C is a view showing display of a screen of a touch panel when automatic parking is completed in the parking assistance system according to the embodiment;

[0030] Figure 4 is a plan view showing a state in which a virtual viewpoint is located outside a limit section in the parking assistance system according to the embodiment;

[0031] Figure 5 is a plan view showing a state in which a virtual viewpoint is located inside a limit section in the parking assistance system according to the embodiment;

[0032] Figure 6A and Figure 6B are views each showing a screen of a touch panel on which a virtual viewpoint image and an overhead image are displayed in the parking assistance system according to the embodiment;

[0033] Figure 6C is a view showing a screen of a touch panel on which an inverted rearward image and an overhead image are displayed in the parking assistance system according to the embodiment; and

[0034] Figure 7 is a table showing images displayed on a touch panel during search processing in the parking assistance system according to the embodiment. DETAILED DESCRIPTION

[0035] Hereinafter, an embodiment of the present application will be described in detail with reference to the accompanying drawings.

[0036] A parking assistance system 1 (an example of an image display system) is installed on a vehicle V (an example of a moving body: an automobile in the present embodiment) provided with a vehicle control system 2 configured to cause the vehicle V to autonomously travel.

[0037] As shown in Figure 1 , the vehicle control system 2 includes a powertrain 4, a braking device 5, a steering device 6, an external environment sensor 7, a vehicle sensor 8, a navigation device 10, an operation input member 11, a driving operation sensor 12, a state detection sensor 13, a human-machine interface (HMI) 14, and a control device 15. The above components of the vehicle control system 2 are connected to each other so that signals can be transmitted therebetween via a communication mechanism such as a controller area network (CAN).

[0038] The powertrain 4 is a device configured to apply a driving force to the vehicle V. The powertrain 4 includes a power source and a transmission. The power source includes at least one of an internal combustion engine such as a gasoline engine and a diesel engine and an electric motor. In the present embodiment, the powertrain 4 includes an automatic transmission 16 (an example of a power transmission device) and a shift actuator 17 for changing a shift range (a gear of the vehicle V) of the automatic transmission 16. The brake device 5 is a device configured to apply a braking force to the vehicle V. For example, the brake device 5 includes a brake caliper configured to press a brake pad against a brake rotor and an electric cylinder configured to supply a brake pressure (an oil pressure) to the brake caliper. The brake device 5 can include an electric parking brake device configured to restrict the rotation of a wheel via a cable. The steering device 6 is a device for changing the steering angle of a wheel. For example, the steering device 6 includes a rack-and-pinion mechanism configured to steer (turn) a wheel and an electric motor configured to drive the rack-and-pinion mechanism. The powertrain 4, the brake device 5, and the steering device 6 are controlled by the control device 15.

[0039] The outside environment sensor 7 functions as an outside environment information acquisition device for detecting electromagnetic waves, sound waves, and the like from the surroundings of the vehicle V to detect an object outside the vehicle V and acquire the surroundings information of the vehicle V. The outside environment sensor 7 includes sonars 18 and an outside camera 19. The outside environment sensor 7 can also include a millimeter wave radar and / or a laser radar. The outside environment sensor 7 outputs the detection results to the control device 15.

[0040] Each sonar 18 is composed of a so-called ultrasonic sensor. Each sonar 18 emits an ultrasonic wave to the surroundings of the vehicle V and captures an ultrasonic wave reflected by an object, thereby detecting the position (distance and direction) of the object. A plurality of sonars 18 are provided at each of the rear and the front of the vehicle V. In the present embodiment, a pair of sonars 18 is provided on the rear bumper so as to be laterally spaced apart from each other, a pair of sonars 18 is provided on the front bumper so as to be laterally spaced apart from each other, and four sonars 18 are respectively provided at the front end and the rear end of each of the left and right side surfaces of the vehicle V. That is, a total of eight sonars 18 are provided in the vehicle V. The sonars 18 provided on the rear bumper mainly detect the position of an object behind the vehicle V. The sonars 18 provided on the front bumper mainly detect the position of an object in front of the vehicle V. The sonars 18 provided at the front end of each of the left and right side surfaces of the vehicle V respectively detect the position of an object on the outer side of the front end of the vehicle V. The sonars 18 provided at the rear end of each of the left and right side surfaces of the vehicle V respectively detect the position of an object on the outer side of the rear end of the vehicle V.

[0041] The exterior cameras 19 are devices configured to capture images of the surroundings of the vehicle V. For example, each of the exterior cameras 19 is composed of a digital camera using a solid imaging element such as a CCD or a CMOS. The exterior cameras 19 include a front camera for capturing an image of the front of the vehicle V and a rear camera for capturing an image of the rear of the vehicle V. The exterior cameras 19 can include a pair of left and right cameras disposed near the rearview mirrors of the vehicle V to capture images of the left and right sides of the vehicle V.

[0042] The vehicle sensors 8 include a vehicle speed sensor 8a configured to detect the vehicle speed of the vehicle V, an acceleration sensor configured to detect the acceleration of the vehicle V, a yaw rate sensor configured to detect the angular velocity around the vertical axis of the vehicle V, and a direction sensor configured to detect the direction of the vehicle V. For example, the vehicle speed sensor 8a is composed of a plurality of wheel speed sensors. Each wheel speed sensor is configured to detect the wheel speed (the rotational speed of each wheel). For example, the yaw rate sensor is composed of a gyro sensor.

[0043] The navigation device 10 is a device configured to obtain the current position of the vehicle V and provide route guidance to a destination or the like. The navigation device 10 includes a GPS receiving unit 20 and a map storage unit 21. The GPS receiving unit 20 identifies the position (latitude and longitude) of the vehicle V based on a signal received from an artificial satellite (positioning satellite). The map storage unit 21 is composed of a known storage device such as a flash memory or a hard disk, and stores map information.

[0044] The operation input member 11 is provided in the cabin to receive an input operation by an occupant (user) to control the vehicle V. The operation input member 11 includes a steering wheel 22, an accelerator pedal 23, a brake pedal 24 (brake input member), and a shift lever 25 (shift member). The shift lever 25 is configured to receive an operation for switching the gear of the vehicle V.

[0045] The drive operation sensor 12 detects the operation amount of the operation input member 11. The drive operation sensor 12 includes a steering angle sensor 26 configured to detect the steering angle of the steering wheel 22, a brake sensor 27 configured to detect the depression amount of the brake pedal 24, and an acceleration sensor 28 configured to detect the depression amount of the accelerator pedal 23. The drive operation sensor 12 outputs the detected operation amount to the control device 15.

[0046] The state detection sensor 13 is a sensor configured to detect a change in the state of the vehicle V according to an operation by the occupant. The operation by the occupant detected by the state detection sensor 13 includes an operation indicating the occupant's intention to get off the vehicle V and an operation indicating the occupant's intention not to check the environment around the vehicle V during the autonomous parking operation or the autonomous departure operation. The state detection sensor 13 includes a door opening / closing sensor 29 configured to detect the opening and / or closing of the door and a seat belt sensor 30 configured to detect the fastening state of the seat belt as sensors for detecting the operation indicating the intention to get off. The state detection sensor 13 includes a rearview mirror position sensor 31 configured to detect the position of the rearview mirror as a sensor for detecting the operation corresponding to the intention to get off. The state detection sensor 13 outputs a signal indicating the detected change in the state of the vehicle V to the control device 15.

[0047] The HMI 14 is an input / output device for receiving an input operation by the occupant and notifying the occupant of various information via display and / or voice. The HMI 14 includes, for example, a touch panel 32 including a display screen such as a liquid crystal display or an organic EL display and configured to receive an input operation by the occupant, a sound producing device 33 such as a buzzer or a speaker, a parking main switch 34, and a selection input member 35. The parking main switch 34 receives an input operation by the occupant to select one of the automatic parking process (the autonomous parking operation) and the automatic departure process (the autonomous departure operation). The parking main switch 34 is a so-called momentary switch that is turned on only when the occupant performs a press operation (a push-down operation). The selection input member 35 receives a selection operation by the occupant related to the selection of the automatic parking process and the automatic departure process. The selection input member 35 can be composed of a rotary selection switch that preferably requires a press as the selection operation.

[0048] The control device 15 is composed of an electronic control unit (ECU) including a CPU, a nonvolatile memory such as a ROM, a volatile memory such as a RAM, and the like. The CPU performs an operation process according to a program, whereby the control device 15 performs various types of vehicle control. The control device 15 can be composed of one hardware, or can be composed of units including a plurality of hardware. Furthermore, the functions of the control device 15 can be at least partially performed by hardware such as LSIs, ASICs, and FPGAs, or can be performed by a combination of software and hardware.

[0049] Further, the control device 15 performs arithmetic processing in accordance with a program, thereby performing conversion processing of an image (video) captured by the external camera 19 to generate an overhead image 47 corresponding to a plan view of the vehicle V and its surrounding area, and a bird's-eye image 48 corresponding to a three-dimensional image of the portion of the vehicle V and its surrounding area located in the direction of travel when viewed from above. The control device 15 can generate the overhead image 47 by combining the images of the front camera, the rear camera, and the left and right side cameras, and can generate the bird's-eye image 48 by combining the image captured by the front camera or the rear camera facing the direction of travel and the images captured by the left and right side cameras.

[0050] The parking assistance system 1 is a system for performing a so-called automatic parking process and a so-called automatic drive-off process in which the vehicle V autonomously moves to a prescribed target position (a target parking position or a target drive-off position) selected by an occupant to park the vehicle V or to drive off the vehicle V.

[0051] The parking assistance system 1 includes the external cameras 19 (examples of external environment information acquisition devices), the touch panel 32 (examples of display devices and input devices), and the control device 15. Each of the external cameras 19 captures a surrounding image (an example of external environment information around the vehicle V) of the vehicle V. The touch panel 32 displays the surrounding image captured by each of the external cameras 19. The control device 15 controls the display of the surrounding image on the control touch panel 32 in accordance with an input operation by an occupant.

[0052] The control device 15 controls the powertrain 4, the brake device 5, and the steering device 6 to perform an autonomous parking operation to autonomously move the vehicle V to a target parking position and park the vehicle V at the target parking position, and to perform an autonomous drive-off operation to autonomously move the vehicle V to a target drive-off position and drive off the vehicle V at the target drive-off position. To perform such operations, the control device 15 includes an external environment recognition unit 41, a vehicle position recognition unit 42, a behavior planning unit 43, a travel control unit 44, and a display control unit 45.

[0053] The external environment recognition unit 41 recognizes an obstacle (e.g., a parked vehicle or a wall) present around the vehicle V based on the detection results of the external environment sensors 7, thereby obtaining information about the obstacle. Further, the external environment recognition unit 41 analyzes the image captured by the external camera 19 based on a known image analysis method such as pattern matching, thereby determining whether a vehicle immobilizer (a wheel immobilizer) or an obstacle is present and obtaining the size of the vehicle immobilizer or the obstacle if present. Further, the external environment recognition unit 41 can calculate the distance to the obstacle based on the signal from the sonar 18 to obtain the position of the obstacle.

[0054] Further, by analyzing the detection results of the external environment sensor 7 (more specifically, by analyzing the image captured by the external camera 19 based on a known image analysis method such as pattern matching), the external environment recognition unit 41 is able to acquire, for example, a lane on a road defined by a road sign, and a parking space defined by a white line or the like provided on the surface of a road, a parking lot, or the like.

[0055] The vehicle position recognition unit 42 recognizes the position of the vehicle V (the host vehicle) based on a signal from the GPS reception unit 20 of the navigation device 10. Further, in addition to the signal from the GPS reception unit 20, the vehicle position recognition unit 42 can obtain the vehicle speed and the yaw rate from the vehicle sensor 8, and recognize the position and the attitude of the vehicle V by means of so-called inertial navigation.

[0056] The travel control unit 44 controls the powertrain 4, the brake device 5, and the steering device 6 based on the travel control instruction from the action planning unit 43, to cause the vehicle V to travel.

[0057] The display control unit 45 generates a surrounding image of the vehicle V, and causes the touch panel 32 to display the generated surrounding image. The display control unit 45 switches the surrounding image displayed on the touch panel 32 based on an input operation by the occupant and an instruction from the action planning unit 43.

[0058] When the vehicle V is in a prescribed state and the HMI 14 or the parking master switch 34 receives a prescribed input of a request from the user corresponding to an automatic parking process or an automatic departure process, the action planning unit 43 executes the automatic parking process (autonomous parking operation) or the automatic departure process (autonomous departure operation). More specifically, when the vehicle V is stopped or the vehicle V is traveling at a low speed equal to or less than a prescribed vehicle speed (a vehicle speed at which a candidate parking position can be searched), the action planning unit 43 executes the automatic parking process in a case where a prescribed input corresponding to the automatic parking process is made. When the vehicle V is stopped, the action planning unit 43 executes the automatic departure process (and the departure operation) in a case where a prescribed input corresponding to the automatic departure process is made. The process to be executed (the automatic parking process or the automatic departure process) can be selected by the action planning unit 43 based on the state of the vehicle V. Alternatively, the above selection can also be made by the occupant via the touch panel 32 or the selection input member 35. The action planning unit 43, when executing the automatic parking process, first causes the touch panel 32 to display a parking search screen for setting a target parking position. After the target parking position is set, the action planning unit 43 causes the touch panel 32 to display a parking screen. When the automatic departure process is executed, the action planning unit 43 first causes the touch panel 32 to display a departure search screen for setting a target departure position. After the target departure position is set, the action planning unit 43 causes the touch panel 32 to display a departure screen.

[0059] The following text will refer to Figure 2 The automatic parking process is described. Action planning unit 43 first performs an acquisition process (step ST1) to acquire one or more parking spaces (if any). More specifically, when vehicle V is stopped, action planning unit 43 first causes the touch panel 32 of HMI 14 to display a notification instructing the occupant to move vehicle V straight. When the occupant sitting in the driver's seat (hereinafter referred to as the "driver") moves vehicle V straight, external environment identification unit 41 acquires the position and size of each detected obstacle and the position of the white lines set on the road surface based on signals from external environment sensors 7. Based on the acquired position and size of the obstacles and the acquired position of the white lines, external environment identification unit 41 extracts one or more undefined parking spaces and one or more defined parking spaces (if any) (hereinafter, undefined parking spaces and defined parking spaces are collectively referred to as "parking spaces"). Each undefined parking space is a space not defined by white lines, etc., large enough to park vehicle V, and available (i.e., without obstacles). Each defined parking space is marked by white lines, etc., and is large enough to park a vehicle V, and is available (i.e., there is no space for another vehicle (other than this vehicle)).

[0060] Next, the action planning unit 43 performs trajectory calculation processing (step ST2) to calculate the trajectory of vehicle V from its current position to each extracted parking space. When calculating the vehicle's trajectory for a specific parking space, the action planning unit 43 sets that parking space as a candidate parking location where vehicle V can be parked, and displays that candidate parking location on the screen (parking search screen) using the touch panel 32. If the trajectory of vehicle V cannot be calculated due to the presence of obstacles, the action planning unit 43 does not set that parking space as a candidate parking location, and does not display that parking location on the screen using the touch panel 32. When the action planning unit 43 sets multiple candidate parking locations (i.e., multiple parking locations where the trajectory of vehicle V can be calculated), the action planning unit 43 displays these candidate parking locations on the touch panel 32.

[0061] Next, the action planning unit 43 performs a target parking location receiving process (step ST3) to receive a selection operation performed by the occupant to select a target parking location, which is the parking location where the occupant wants to park the vehicle V, and is selected from one or more candidate parking locations displayed on the touch panel 32. More specifically, the action planning unit 43 causes the touch panel 32 to... Figure 3A The parking search screen shown displays a top-down view 47 and a bird's-eye view 48 along the direction of travel. When the action planning unit 43 acquires at least one candidate parking location, the action planning unit 43 causes the touch panel 32 to display the top-down view 47 and the bird's-eye view 48. Figure 3Athe candidate parking position and an icon corresponding to the frame. The icon consists of a symbol indicating the candidate parking position (see Figure 3A In addition, the action planning unit 43 causes the touch panel 32 to display a parking search screen including a notification instructing the driver to stop the vehicle V and select a target parking position, so that the touch panel 32 receives a selection operation of the target parking position. The selection operation of the target parking position can be performed via the touch panel 32, or can be performed via the selection input member 35.

[0062] After the vehicle V is stopped and the driver selects the target parking position, the action planning unit 43 causes the touch panel 32 to switch the screen from the parking search screen to a parking screen. As shown in Figure 3B The parking screen is a screen in which an image in the advancing direction of the vehicle V (hereinafter referred to as "advancing direction image 49") is displayed on the left half of the touch panel 32, and the overhead image 47 including the vehicle V and the surrounding area thereof is displayed on the right half of the touch panel 32, as shown in

[0063] After the target parking position is selected and the screen of the touch panel 32 is switched to the parking screen, the action planning unit 43 performs a driving process (step ST4) to cause the vehicle V to advance along the calculated trajectory. At this time, the action planning unit 43 controls the vehicle V based on the position of the vehicle V acquired by the GPS receiving unit 20 and signals from the external camera 19, the vehicle sensor 8, and the like, so that the vehicle V advances along the calculated trajectory. At this time, the action planning unit 43 controls the powertrain 4, the brake device 5, and the steering device 6 to perform a switching operation (a reverse operation for reversing the advancing direction of the vehicle V) for switching the advancing direction of the vehicle V. The switching operation can be repeatedly performed, or can be performed only once.

[0064] In the driving process, the action planning unit 43 can acquire the advancing direction image 49 from the external camera 19, and cause the touch panel 32 to display the acquired advancing direction image 49 on the left half thereof. For example, as shown in Figure 3BAs shown in FIG. 6, when the vehicle V moves backward, the action planning unit 43 can cause the touch panel 32 to display an image of the rear of the vehicle V captured by the external camera 19 on the left half of the touch panel 32. When the action planning unit 43 is executing the driving process, the surrounding image of the vehicle V (the host vehicle) in the overhead image 47 displayed on the right half of the touch panel 32 changes as the vehicle V moves. When the vehicle V reaches the target parking position, the action planning unit 43 stops the vehicle V and ends the driving process.

[0065] When the driving process ends, the action planning unit 43 executes a parking process (step ST5). In the parking process, the action planning unit 43 first drives the shift actuator 17 to set the shift range (gear position) to the parking range (parking position). Thereafter, the action planning unit 43 drives the parking brake device, and causes the touch panel 32 to display a pop-up window indicating that automatic parking of the vehicle V has been completed (see FIG. 7). This pop-up window can be displayed on the screen of the touch panel 32 for a prescribed period of time. Thereafter, the action planning unit 43 can cause the touch panel 32 to switch the screen to the operation screen or the map screen of the navigation device 10. Figure 3C ). This pop-up window can be displayed on the screen of the touch panel 32 for a prescribed period of time. Thereafter, the action planning unit 43 can cause the touch panel 32 to switch the screen to the operation screen or the map screen of the navigation device 10.

[0066] In the parking process, there can be a case where the shift range cannot be changed to the parking range due to an abnormality of the shift actuator 17, or a case where the parking brake device 53 cannot be driven due to an abnormality of the parking brake device 53. In these cases, the action planning unit 43 can cause the touch panel 32 to display the reason for the abnormality on the screen thereof.

[0067] <Shift Range>

[0068] The action planning unit 43 activates the shift actuator 17 in accordance with an operation of the shift lever 25 by an occupant (an example of a user) to switch the shift range of the vehicle V between at least a forward range (travel range), a rearward range (reverse range), a neutral range, and a parking range. The forward range is a shift range that moves the vehicle V forward. The rearward range is a shift range that moves the vehicle V backward. The neutral range is a shift range in which the wheels are rotatable and the driving force of the powertrain 4 is not transmitted to the wheels. The parking range is a shift range in which the rotation of the wheels is restricted and the driving force of the powertrain 4 is not transmitted to the wheels.

[0069] <Virtual Viewpoint Image 51 and Reversed Rear Image 52>

[0070] Hereinafter, the virtual viewpoint image 51 and the reversed rear image 52 (both are examples of surrounding images) will be described with reference to FIGS. 8A and 8B. Figures 4 to 6C Figure 4 and Figure 5 The straight line LI in FIGS. 8A and 8B indicates a front-rear center line LI of the vehicle V (a straight line passing through the front-rear center of the vehicle V and extending in the lateral direction), and the virtual viewpoint image 51 is an image of the surroundings of the vehicle V as viewed from a virtual viewpoint VV located on the front-rear center line LI of the vehicle V.​Figure 4 and Figure 5 The straight line L2 in

[0071] With reference to Figure 4 and Figure 5 , the display control unit 45 sets a trajectory B (hereinafter simply referred to as "trajectory B") of the virtual viewpoint A around the vehicle V. The virtual viewpoint A is set diagonally above the vehicle V (set outside the vehicle V). In another embodiment, the virtual viewpoint A can be set at the same height as the vehicle V. The trajectory B has a true circular shape centered on the vehicle V in a plan view. In another embodiment, the trajectory B can have a shape other than the above-described shape (for example, an elliptical or polygonal shape centered on the vehicle V in a plan view, an arc shape centered on the vehicle V in a side view, or a dome shape centered on the vehicle V). Figure 4 and Figure 5 The angle θ in

[0072] The display control unit 45 sets an arc-shaped limit section C in the trajectory B so that the limit section C is arranged in front of the vehicle V in a plan view. In the present embodiment, the entire limit section C is arranged further forward than the front end Va of the vehicle V. In another embodiment, a part of the limit section C can be arranged further rearward than the front end Va of the vehicle V. The limit section C intersects the lateral center line L2 of the vehicle V in a plan view. The lateral center of the limit section C is arranged on the lateral center line L2 of the vehicle V. Incidentally, Figure 4 shows a state in which the virtual viewpoint A is outside the limit section C, and Figure 5 shows a state in which the virtual viewpoint A is located within the limit section C (more specifically, at the lateral center of the limit section C).

[0073] With reference to Figure 6A and Figure 6B , the display control unit 45 generates a virtual viewpoint image 51 by performing image processing on the images captured by each of the external cameras 19, and causes the touch panel 32 to display the generated virtual viewpoint image 51 together with the above-described overhead image 47. The virtual viewpoint image 51 is a three-dimensional image (normal image) in which the vehicle V and the objects Y around the vehicle V are viewed (observed) from the virtual viewpoint A. Figure 6AA virtual viewpoint image 51 from a virtual viewpoint A (see Figure 4 ) outside the restricted section C is shown. Figure 6B A virtual viewpoint image 51 from a virtual viewpoint A (see Figure 5 ) in the restricted section C is shown.

[0074] The display control unit 45 continuously moves the virtual viewpoint A in the horizontal direction along the locus B in accordance with a lateral sliding operation (an example of an input operation) of the occupant on the touch panel 32. As the virtual viewpoint A is continuously moved in this way, the angle of the vehicle V and the position of the object Y in the virtual viewpoint image 51 are continuously changed. In another embodiment, the display control unit 45 can continuously move the virtual viewpoint A in the up-down direction in accordance with an up-down sliding operation of the occupant on the touch panel 32. Further, in yet another embodiment, the display control unit 45 can move the virtual viewpoint A in accordance with an operation (for example, a tapping operation) other than the sliding operation of the occupant, or move the virtual viewpoint A in accordance with an operation of the occupant on a component other than the touch panel 32 (for example, a button provided around the touch panel 32).

[0075] With reference to Figure 6A and Figure 6B , the display control unit 45 causes the touch panel 32 to display an operation facilitation image 54 so that the operation facilitation image 54 is superimposed on the lower portion of the virtual viewpoint image 51. The operation facilitation image 54 is an image that facilitates the sliding operation of the occupant on the touch panel 32. The operation facilitation image 54 includes a hand image and a pair of arrow images arranged on the left and right sides of the hand image. In another embodiment, the operation facilitation image 54 can include images other than the hand image and the arrow images.

[0076] With reference to Figure 6C , the display control unit 45 generates a reversed rearward image 52 by performing image processing on the images captured by each of the external cameras 19, and causes the touch panel 32 to display the generated reversed rearward image 52 together with the overhead image 47 described above. The reversed rearward image 52 is an image (a mirror image) in which a rearward image from the vehicle V is reversed in the lateral direction (left-right reversed). The display control unit 45 displays a guide frame 55 so that the guide frame 55 is superimposed on the reversed rearward image 52. The guide frame 55 is a frame that shows the outline (an anticipated outline) of the vehicle V when the vehicle V is moved backward by a prescribed distance from the current position.

[0077] <Images displayed on the touch panel 32 during the search processing>

[0078] The action plan unit 43 is configured to execute a search process to search for a parking position X of the vehicle V. The search process is a process corresponding to the acquisition process, the trajectory calculation process, and the target parking position reception process in the automatic parking process described above. Hereinafter, the bird's-eye image 48 of the front of the vehicle V (refer to Figure 3A ) will be referred to as a "forward bird's-eye image 48". Hereinafter, only the image displayed on the left half of the touch panel 32 during the search process will be described, and the description of the image (i.e., the overhead image 47) displayed on the right half of the touch panel 32 during the search process will be omitted.

[0079] Referring to Figure 7 When the occupant has not selected a parking position X and the shift range of the vehicle V is set to the forward range, the display control unit 45 causes the touch panel 32 to display the forward bird's-eye image 48 or the virtual viewpoint image 51. On the other hand, when the occupant has not selected a parking position X and the shift range of the vehicle V is set to the reverse range, the display control unit 45 causes the touch panel 32 to display the reversed rearward image 52.

[0080] In a state where the occupant has not selected a parking position X and the touch panel 32 displays the forward bird's-eye image 48 or the virtual viewpoint image 51, the shift range of the vehicle V can be switched from the forward range to the reverse range. In this case, the display control unit 45 switches the image on the touch panel 32 from the forward bird's-eye image 48 or the virtual viewpoint image 51 to the reversed rearward image 52. Conversely, in a state where the occupant has not selected a parking position X and the touch panel 32 displays the reversed rearward image 52, the shift range of the vehicle V can be switched from the reverse range to the forward range. In this case, the display control unit 45 switches the image on the touch panel 32 from the reversed rearward image 52 to the forward bird's-eye image 48 or the virtual viewpoint image 51.

[0081] When the occupant has selected a parking position X and the shift range of the vehicle V is set to the forward range, the display control unit 45 causes the touch panel 32 to display the forward bird's-eye image 48. On the other hand, when the occupant has selected a parking position X and the shift range of the vehicle V is set to the reverse range, the display control unit 45 causes the touch panel 32 to display the reversed rearward image 52.

[0082] In a state where the occupant has selected the parking position X and the touch panel 32 displays the forward-looking image 48, the shift range of the vehicle V can be switched from the forward range to the rearward range. In this case, the display control unit 45 switches the image on the touch panel 32 from the forward bird's-eye image 48 to the reversed rearward image 52. Conversely, in a state where the occupant has selected the parking position X and the touch panel 32 displays the reversed rearward image 52, the shift range of the vehicle V can be switched from the rearward range to the forward range. In this case, the display control unit 45 switches the image on the touch panel 32 from the reversed rearward image 52 to the forward bird's-eye image 48.

[0083] As described above, in the search processing, the display control unit 45 causes the touch panel 32 to display the virtual viewpoint image 51 only in a case where the shift range of the vehicle V is set to the forward range. In another embodiment, during the search processing, the display control unit 45 can cause the touch panel 32 to display the virtual viewpoint image 51 even in a case where the shift range of the vehicle V is set to a range other than the forward range (for example, the rearward range).

[0084] Further, in a case where the shift range of the vehicle V is set to the rearward range during the search processing, the display control unit 45 causes the touch panel 32 (more specifically, the left half of the touch panel 32) to display only the reversed rearward image 52. Thus, the occupant can improve the visibility of the surrounding environment of the vehicle V. In another embodiment, in a case where the shift range of the vehicle V is set to the rearward range during the search processing, the display control unit 45 can cause the touch panel 32 to display an image other than the reversed rearward image 52 (for example, the virtual viewpoint image 51).

[0085] <VIEWPOINT RESTRICTION CONTROL>

[0086] As described above, when the shift range of the vehicle V is switched from the forward range to the rearward range in a state where the occupant has not selected the parking position X and the touch panel 32 displays the virtual viewpoint image 51, the display control unit 45 switches the image on the touch panel 32 from the virtual viewpoint image 51 to the reversed rearward image 52. Hereinafter, a problem caused by this switching from the virtual viewpoint image 51 to the reversed rearward image 52 will be described.

[0087] Figure 6B The virtual viewpoint image 51 (normal image) from the virtual viewpoint A in the restriction section C (the virtual viewpoint A in front of the vehicle V) is shown. In this virtual viewpoint image 51, the obstacle Y2 is displayed on the left side of the vehicle V. On the other hand, the reversed rearward image 52 (mirror image) from the rearward view of the vehicle V is shown. In this reversed rearward image 52, the obstacle Y2 is displayed on the right side of the vehicle V. Figure 6C The virtual viewpoint image 51 (normal image) from the virtual viewpoint A in the restriction section C (the virtual viewpoint A in front of the vehicle V) is shown. In this virtual viewpoint image 51, the obstacle Y2 is displayed on the left side of the vehicle V. On the other hand, the reversed rearward image 52 (mirror image) from the rearward view of the vehicle V is shown. In this reversed rearward image 52, the obstacle Y2 is displayed on the right side of the vehicle V. Figure 6BThe same obstacle Y2 as the obstacle Y2 shown is displayed on the right side of the vehicle V. In this way, as the image on the touch panel 32 is switched from the virtual viewpoint image 51 to the reversed backward image 52, the image on the touch panel 32 is switched from the normal image to the mirror image, and thus the position of each object Y included in these images can be moved from one of the left and right sides of the vehicle V to the other side thereof. If such a movement of the object Y occurs, the occupant can have difficulty in understanding whether the object Y is on the left or right side of the vehicle V in the actual space.

[0088] In this way, when the shift range of the vehicle V is set to the forward range during the search processing, the display control unit 45 performs viewpoint restriction control. In this viewpoint restriction control, the display control unit 45 restricts the display of the virtual viewpoint image 51 from the virtual viewpoint A in the restriction section C. Thus, the display of the virtual viewpoint image 51 from the front looking down (viewing) the vehicle V and the object Y is restricted, and thus the movement of each object Y as described above can be suppressed. Therefore, the occupant can easily understand whether the object Y and the parking position X are on the left or right side of the vehicle V in the actual space.

[0089] Further, during the viewpoint restriction control, while the virtual viewpoint A is allowed to enter the restriction section C, as long as the virtual viewpoint A is in the restriction section C, the display control unit 45 restricts the display of the virtual viewpoint image 51. By allowing the virtual viewpoint A to enter the restriction section C during the viewpoint restriction control, the virtual viewpoint A can pass through the restriction section C even during the viewpoint restriction control. Thus, it is possible to make the virtual viewpoint A move quickly from a region on one side of the restriction section C to another region on the other side thereof. Therefore, it is possible to improve the convenience of the occupant.

[0090] More specifically, during the viewpoint restriction control, as long as the virtual viewpoint A is in the restriction section C, the display control unit 45 causes the touch panel 32 to display a monochrome image (e.g., a black image) instead of the virtual viewpoint image 51. At this time, as long as the virtual viewpoint A is in the restriction section C, the display control unit 45 can cause the touch panel 32 to display a message indicating that the display of the virtual viewpoint image 51 is restricted, so that the message is superimposed on the monochrome image. In another embodiment, as long as the virtual viewpoint A is in the restriction section C, the display control unit 45 can reduce the resolution of the virtual viewpoint image 51 to a level at which the vehicle V and the object Y cannot be recognized. In this way, "restricting the display of the virtual viewpoint image 51" means both "not displaying the virtual viewpoint image 51" and "reducing the visibility of the virtual viewpoint image 51".

[0091] In another embodiment, during the viewpoint restriction control, the display control unit 45 restricts the entry of the virtual viewpoint A into the restriction section C itself. Thus, it is possible to reliably make the occupant recognize that the display of the virtual viewpoint image 51 from the virtual viewpoint A in the restriction section C is restricted.

[0092] Further, when the shift range of the vehicle V is set to the reverse range, the display control unit 45 does not perform the viewpoint restriction control. Thus, when the shift range of the vehicle V is set to the reverse range, the display of the virtual viewpoint image 51 is not restricted by the viewpoint restriction control, so that the occupant can improve the visibility of the surroundings of the vehicle V.

[0093] <Other Modified Embodiments>

[0094] In the above-described embodiment, the touch panel 32 is used as an example of the display device and the input device. On the other hand, in another embodiment, a communication device (for example, a smartphone, a tablet PC, a mobile phone, a PDA, or the like) configured to communicate with the vehicle V can be used as an example of the display device and the input device. That is, the display device and the input device are not necessarily mounted on the vehicle V. In addition, in the above-described embodiment, the touch panel 32 functions as both the display device and the input device. On the other hand, in another embodiment, the display device and the input device can be composed of different components.

[0095] In the above-described embodiment, the occupant is used as an example of the user. On the other hand, in another embodiment, a person other than the occupant (that is, a person outside the vehicle V) can be used as an example of the user.

[0096] In the above-described embodiment, the image captured by each external camera 19 is used as an example of the external environment information acquired by the external environment information acquisition device. On the other hand, in another embodiment, information other than the image captured by each external camera 19 (for example, an image captured by a vehicle-mounted camera, point cloud data acquired by a laser radar) can be used as an example of the external environment information acquired by the external environment information acquisition device.

[0097] In the above-described embodiment, the vehicle V (more specifically, an automobile) is used as an example of the moving body. On the other hand, in another embodiment, a vehicle V other than an automobile (for example, a two-wheeled vehicle) can be used as an example of the moving body, or a moving body other than the vehicle V (for example, a Segway, a ship, or an airplane) can be used as an example of the moving body.

[0098] In the above-described embodiment, the parking assistance system 1 is used as an example of the image display system. On the other hand, in another embodiment, a system other than the parking assistance system 1 (for example, a system that causes a moving body to travel) can be used as an example of the image display system.

[0099] The specific embodiments of the present application have been described above, but the present application should not be limited by the foregoing embodiments, and various modifications and changes can be made within the scope of the present application.

Claims

1. An image display system comprising: external environment information acquisition means configured to acquire external environment information around a mobile body; display means configured to display the external environment information acquired by the external environment information acquisition means; and control means configured to control display of the external environment information on the display means, wherein the control means is configured to: generate a virtual viewpoint image in which the mobile body and objects around the mobile body are viewed from a virtual viewpoint, cause the display means to display the virtual viewpoint image, move the virtual viewpoint along a trajectory provided around the mobile body, provide a restricted section in the trajectory such that the restricted section is arranged in front of the mobile body in a plan view, and perform viewpoint restriction control to restrict display of the virtual viewpoint image from a virtual viewpoint in the restricted section, wherein, during the viewpoint restriction control, the control means restricts display of the virtual viewpoint image as long as the virtual viewpoint is within the restricted section while allowing the virtual viewpoint to enter the restricted section. the control means is configured to:

2. The image display system of claim 1, wherein, generate a reversed rearward image in which a rearward image from the mobile body is reversed in a lateral direction, and cause the display means to display the reversed rearward image. the control means is configured to perform search processing to search for a stop position of the mobile body, and perform the viewpoint restriction control during the search processing.

3. The image display system according to claim 1 or 2, wherein during the viewpoint restriction control, the control means restricts the virtual viewpoint from entering the restricted section.

4. The image display system of claim 2, wherein, 5. An image display system comprising: external environment information acquisition means configured to acquire external environment information around a mobile body; display means configured to display the external environment information acquired by the external environment information acquisition means; and control means configured to control display of the external environment information on the display means, wherein the control means is configured to: generate a virtual viewpoint image in which the mobile body and objects around the mobile body are viewed from a virtual viewpoint, cause the display means to display the virtual viewpoint image, move the virtual viewpoint along a trajectory provided around the mobile body, provide a restricted section in the trajectory such that the restricted section is arranged in front of the mobile body in a plan view, and perform viewpoint restriction control to restrict display of the virtual viewpoint image from a virtual viewpoint in the restricted section, wherein a shift range of the mobile body is switchable between a plurality of ranges including a forward range in which the mobile body moves forward and a rearward range in which the mobile body moves rearward, and when the shift range of the mobile body is set to the forward range, the control means performs the viewpoint restriction control. ​ ​ 6. The image display system of claim 5, wherein, The control device does not execute the viewpoint restriction control when the shift range of the mobile body is set to the rearward range.

7. The image display system of claim 6, wherein, The control device is configured to generate a reversed rearward image in which a rearward image from the mobile body is reversed in a lateral direction, and The control device causes the display device to display the reversed rearward image when the shift range of the mobile body is set to the rearward range.

8. The image display system of claim 7, wherein, In a case where the shift range of the mobile body is switched from the forward range to the rearward range in a state where the display device displays the virtual viewpoint image, the control device switches the image on the display device from the virtual viewpoint image to the reversed rearward image.

9. The image display system according to any one of claims 5 to 8, wherein, The control device is configured to execute a search process to search for a stop position of the mobile body, and execute the viewpoint restriction control during the search process.

10. The image display system according to any one of claims 5 to 8, wherein, The control device restricts the virtual viewpoint from entering into the restricted section during the viewpoint restriction control.

Images