Control device, control method, and program product

Abstract

The present application provides a control device, a control method, and a control program that can display a surrounding image of a moving body and enable a specified object to be quickly recognized. The control device includes an image processing section (57) that generates an overhead image and a three-dimensional image representing a vehicle and a surrounding of the vehicle based on each photographing data acquired by a front camera (12Fr), a rear camera (12Rr), a left side camera (12L), and a right side camera (12R) of the vehicle (10); a display control section (55) that causes a touch panel (42) to display the overhead image and the three-dimensional image generated by the image processing section; and an object presence / absence determination section (56) that determines whether a specified object exists in a boundary region of each photographing data in the overhead image and the three-dimensional image. The image processing section preferentially changes the boundary region of the three-dimensional image in the displayed overhead image and the three-dimensional image when the object presence / absence determination section determines that the specified object exists in the boundary region.

Description

TECHNICAL FIELD

[0001] The present application relates to a control device, a control method, and a control program. BACKGROUND

[0002] In recent years, as a specific countermeasure against global climate change, initiatives to achieve a low-carbon society or a decarbonized society are very active. In the field of vehicles, there is a strong demand for reduction of CO2 emissions, and introduction of automatic driving and driving assistance of vehicles that contribute to improvement of fuel efficiency is rapidly advancing.

[0003] In the related art, an image generation method is known in which each of cameras mounted on a vehicle photographs a prescribed range and acquires an image, a surrounding image (for example, an overhead image) of the vehicle and the surroundings of the vehicle is generated based on a combined image of the photographed images, and a three-dimensional image is generated based on the overhead image. In Patent Literature 1, a vehicle surrounding monitoring device is described that changes a photographing range of an image photographed by each camera in accordance with opening and closing of a side mirror of the vehicle, and changes a boundary position between photographed images in a combined image of the photographed images to generate an overhead image. In addition, in Patent Literature 2, a vehicle surrounding monitoring device is described that, for a target object whose entirety is not displayed on a generated overhead image, changes a boundary line on the overhead image to display the entirety of the target object.

[0004]

Related Art

[0005]

Patent Literature

[0006] Patent Literature 1: Japanese Patent No. 5112998

[0007] Patent Literature 2: Japanese Patent Application Laid-Open No. 2013-093865

[0008] However, for example, at the time of parking assistance of a vehicle or the like, an overhead image and a three-dimensional image are sometimes displayed on a display device of the vehicle. At this time, a boundary line between photographed images in the overhead image and the three-dimensional image is sometimes displayed overlapping in a parking frame of a predetermined parking or in a parking frame of a parking. In this case, an image of the parking frame in which the boundary line is displayed overlapping becomes a distorted image, visibility is reduced, and it is undesirable as an image at the time of parking. However, in Patent Literature 1 and Patent Literature 2, a correspondence relationship between a parking frame at the time of parking assistance and a boundary line between photographed images is not described. Therefore, there is room for improvement regarding visibility of an overhead image and a three-dimensional image at the time of parking assistance. SUMMARY

[0009] The present application provides a control device, control method, and control program that enable a surrounding image of a mobile body to be displayed, the surrounding image of the mobile body enabling a prescribed object to be quickly recognized.

[0010] The present application provides a control device including:

[0011] an image processing section that generates an overhead image and a three-dimensional image representing the mobile body and a surrounding of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body;

[0012] a display control section that causes a display device to display the overhead image and the three-dimensional image generated by the image processing section; and

[0013] a determination section that determines whether a prescribed object exists in a boundary region of the each photographing data in the overhead image and the three-dimensional image,

[0014] the image processing section preferentially changing the boundary region of the three-dimensional image in the displayed overhead image and the three-dimensional image in a case where the determination section determines that the prescribed object exists in the boundary region.

[0015] The present application provides a control method executed by a processor that generates an overhead image and a three-dimensional image representing a mobile body and a surrounding of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body, and causes a display device to display the generated overhead image and the three-dimensional image, the control method including:

[0016] determining, by the processor, whether a prescribed object exists in a boundary region of the each photographing data in the overhead image and the three-dimensional image; and

[0017] preferentially changing, by the processor, the boundary region of the three-dimensional image in the displayed overhead image and the three-dimensional image in a case where it is determined that the prescribed object exists in the boundary region.

[0018] The present application provides a control program for causing a processor to execute a process that generates an overhead image and a three-dimensional image representing a mobile body and a surrounding of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body, and causes a display device to display the generated overhead image and the three-dimensional image, the process including:

[0019] determining whether a prescribed object exists in a boundary region of the each photographing data in the overhead image and the three-dimensional image; and

[0020] In a case where it is determined that the prescribed object exists in the boundary region, the boundary region of the displayed bird's-eye image and the three-dimensional image is preferentially changed.

[0021] The control device, the control method, and the control program according to the present application can display a surrounding image of a mobile body, which enables a prescribed object to be quickly recognized. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a side view showing one example of a vehicle on which the control device of the present embodiment is mounted.

[0023] Figure 2 is a plan view showing Figure 1 the vehicle.

[0024] Figure 3 is a block diagram showing an internal structure of the vehicle shown in Figure 1

[0025] Figure 4 is a diagram showing one example of a composite image generated from each captured data of a plurality of cameras.

[0026] Figure 5 is a diagram showing a composite image obtained by changing a boundary region of the composite image shown in Figure 4

[0027] Figure 6 is a flowchart showing display control of the control ECU in the first embodiment.

[0028] Figure 7 is a diagram showing one example of a bird's-eye image generated based on each captured data of a plurality of cameras.

[0029] Figure 8 is a diagram showing one example of a bird's-eye image and a three-dimensional image displayed on a touch panel of a vehicle.

[0030] Figure 9 is a flowchart showing display control of the control ECU in the second embodiment.

[0031] Figure 10 is a diagram showing one example of a bird's-eye image generated based on each captured data of a plurality of cameras in the third embodiment.

[0032] Figure 11 is a diagram showing one example of a three-dimensional image generated based on each captured data of a plurality of cameras.

[0033] Figure 12 is a diagram showing one example of a bird's-eye image and a three-dimensional image displayed on a touch panel of a vehicle.​​

[0034] --Explanation of Reference Numerals--

[0035] 10 vehicle (moving body), 12Fr front camera (imaging device), 12Rr rear camera (imaging device), 12L left side camera (imaging device), 12R right side camera (imaging device), 20 control ECU (control device), 42 touch panel (display device), 55 display control section, 56 object presence / absence determination section (determination section), 57 image processing section, 65, 75 shield area, 66a to 66d, 69c, 69d boundary line, 68 obstacle (predetermined object), 70, 80A bird's-eye view image, 76a to 76d, 78c boundary line, 80B three-dimensional image, 86a, 86b, 89c, 89d boundary line, 88 parking frame line (predetermined object), P1 to P5 parking frame (predetermined object). DETAILED DESCRIPTION

[0036] Hereinafter, one embodiment of the control device, the control method, and the control program of the present application will be described based on the drawings. Note that the drawings are viewed in the direction of the symbol. In addition, in this specification and the like, each of the directions of front, rear, left, right, up, and down is described in the order of Fr, Rr, L, R, U, and D, respectively, as viewed from the driver of the vehicle 10 shown in the drawings, and in the drawings, the front of the vehicle 10 is denoted as Fr, the rear is denoted as Rr, the left is denoted as L, the right is denoted as R, the upper is denoted as U, and the lower is denoted as D. Figure 1 and Figure 2

[0037] <Vehicle 10 equipped with the control device of the present application>

[0038] Figure 1 is a side view of the vehicle 10 equipped with the control device of the present application. Figure 2 is Figure 1 a plan view of the vehicle 10 shown in FIG. 1. The vehicle 10 is one example of the moving body of the present application.

[0039] ​Vehicle 10 is an automobile with a drive source (not shown) and wheels, the wheels including drive wheels driven by the power of the drive source and steering wheels capable of steering. In this embodiment, vehicle 10 is a four-wheeled automobile with a pair of front wheels on the left and right and a pair of rear wheels on the left and right. The drive source of vehicle 10 is, for example, an electric motor. The drive source of vehicle 10 can be an internal combustion engine such as a gasoline engine or a diesel engine, or a combination of an electric motor and an internal combustion engine. In addition, the drive source of vehicle 10 can drive the pair of front wheels on the left and right, the pair of rear wheels on the left and right, or all four wheels. The front wheels and rear wheels can be steering wheels capable of steering in both directions, or steering wheels capable of steering in either direction.

[0040] The vehicle 10 also includes side rearview mirrors 11L and 11R. The side rearview mirrors 11L and 11R are mirrors (rearview mirrors) located on the outer sides of the front doors of the vehicle 10, used by the driver to check the rear and rear sides. The side rearview mirrors 11L and 11R are fixed to the main body of the vehicle 10 via a rotating axis extending vertically, and can be opened and closed by rotating around this axis. The opening and closing of the side rearview mirrors 11L and 11R is electrically operated, for example, by the driver using an operating unit located near the driver's seat of the vehicle 10. The width of the vehicle 10 with the side rearview mirrors 11L and 11R closed is narrower than the width of the vehicle 10 with the side rearview mirrors 11L and 11R open. Therefore, for example, when entering a narrow parking space, the driver often closes the side rearview mirrors 11L and 11R to prevent the vehicle 10 from colliding with surrounding obstacles.

[0041] The vehicle 10 also includes a front camera 12Fr, a rear camera 12Rr, a left-side camera 12L, and a right-side camera 12R. The front camera 12Fr is a digital camera positioned at the front of the vehicle 10 and taking pictures of the front of the vehicle 10. The rear camera 12Rr is a digital camera positioned at the rear of the vehicle 10 and taking pictures of the rear of the vehicle 10. The left-side camera 12L is a digital camera positioned in the left-side rearview mirror 11L of the vehicle 10 and taking pictures of the left side of the vehicle 10. The right-side camera 12R is a digital camera positioned in the right-side rearview mirror 11R of the vehicle 10 and taking pictures of the right side of the vehicle 10. The front camera 12Fr, rear camera 12Rr, left-side camera 12L, and right-side camera 12R are an example of the shooting device of the present invention.

[0042] <Internal structure of vehicle 10>

[0043] Figure 3 It means Figure 1 A block diagram illustrating an example of the internal structure of vehicle 10. (See diagram for reference.) Figure 3As shown, the vehicle 10 has a sensor group 16, a navigation device 18, a control ECU (Electronic Control Unit) 20, an EPS (Electric Power Steering) system 22, and a communication section 24. The vehicle 10 also has a drive force control system 26 and a brake force control system 28. The control ECU 20 is an example of the control device of the present application.

[0044] The sensor group 16 acquires various detection values used in the control of the control ECU 20. The sensor group 16 includes a front camera 12Fr, a rear camera 12Rr, a left side camera 12L, and a right side camera 12R. In addition, the sensor group 16 includes a front sonar group 32a, a rear sonar group 32b, a left side sonar group 32c, and a right side sonar group 32d. In addition, the sensor group 16 includes wheel sensors 34a, 34b, a vehicle speed sensor 36, and an operation detection section 38.

[0045] The front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R output peripheral images obtained by capturing the periphery of the vehicle 10. The peripheral images captured by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R are respectively referred to as a front image, a rear image, a left side image, and a right side image. An image composed of the left side image and the right side image can also be referred to as a side image.

[0046] The front sonar group 32a, the rear sonar group 32b, the left side sonar group 32c, and the right side sonar group 32d emit sound waves toward the periphery of the vehicle 10, and receive reflected sounds from other objects. The front sonar group 32a, for example, includes four sonars. The sonars constituting the front sonar group 32a are respectively provided at the left oblique front, the front left side, the front right side, and the right oblique front of the vehicle 10. The rear sonar group 32b, for example, includes four sonars. The sonars constituting the rear sonar group 32b are respectively provided at the left oblique rear, the rear left side, the rear right side, and the right oblique rear of the vehicle 10. The left side sonar group 32c, for example, includes two sonars. The sonars constituting the left side sonar group 32c are respectively provided at the left side portion front and the left side portion rear of the vehicle 10. The right side sonar group 32d, for example, includes two sonars. The sonars constituting the right side sonar group 32d are respectively provided at the right side portion front and the right side portion rear of the vehicle 10.

[0047] The wheel sensors 34a, 34b detect a rotation angle of a wheel of the vehicle 10. The wheel sensors 34a, 34b can be constituted by an angle sensor or a displacement sensor. The wheel sensors 34a, 34b output a detection pulse every time the wheel rotates a predetermined angle. The detection pulse output from the wheel sensors 34a, 34b is used for calculation of a rotation angle of the wheel and a rotation speed of the wheel. A travel distance of the vehicle 10 is calculated on the basis of the rotation angle of the wheel. The wheel sensor 34a, for example, detects a rotation angle θa of the left rear wheel. The wheel sensor 34b, for example, detects a rotation angle θb of the right rear wheel.

[0048] The vehicle speed sensor 36 detects a speed of a vehicle body of the vehicle 10, that is, a vehicle speed V, and outputs the detected vehicle speed V to the control ECU 20. The vehicle speed sensor 36, for example, detects the vehicle speed V on the basis of a rotation of a countershaft of a transmission.

[0049] The operation detection section 38 detects a content of an operation performed by a user using the operation input section 14, and outputs the detected operation content to the control ECU 20. The operation input section 14, for example, includes various user interfaces such as a door mirror switch that switches an opening / closing state of the side mirrors 11L, 11R, a shift lever (gear lever, selector), and the like.

[0050] The navigation device 18, for example, detects a current position of the vehicle 10 using a GPS (Global Positioning System), and guides a route to a destination to a user. The navigation device 18 has a not-illustrated storage device that has a map information database.

[0051] The navigation device 18 has a touch panel 42 and a speaker 44. The touch panel 42 functions as an input device and a display device of the control ECU 20. The user inputs various instructions via the touch panel 42. In addition, various screens are displayed on the touch panel 42. The user can input, for example, an instruction related to the parking assistance via the touch panel 42. In addition, a screen related to the parking assistance can be displayed on the touch panel 42. For example, a parking assistance assist button that requests the parking assistance of the vehicle 10 is displayed on the touch panel 42. The parking assistance assist button includes an automatic parking assistance button that requests automatic parking based on automatic steering control of the control ECU 20, a parking assistance assist button that requests assistance at the time of parking by the operation of the driver. Note that a constituent element other than the touch panel 42, for example, a smartphone or the like can be used as the input device or the display device. The speaker 44 outputs various guide information to an occupant of the vehicle 10 by voice.

[0052] The control ECU 20 has an input / output section 50, an arithmetic section 52, and a storage section 54. The arithmetic section 52 is configured by, for example, a CPU (Central Processing Unit). The arithmetic section 52 controls each section based on a program stored in the storage section 54, thereby performing various controls.

[0053] The arithmetic section 52 includes a display control section 55, an object presence / absence determination section 56, and an image processing section 57. The image processing section 57 generates a surrounding image of the vehicle 10 based on captured data acquired by the cameras of the vehicle 10. Specifically, the image processing section 57 generates a composite image by synthesizing each captured data acquired by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R, and generates an overhead image that represents a situation in which the vehicle 10 and the surroundings of the vehicle 10 are observed from above.

[0054] In addition, the image processing section 57 performs image processing that three-dimensionally reconstructs the composite image of each captured data acquired by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R, and generates a three-dimensional image that virtually represents a situation in which the vehicle 10 and the surroundings of the vehicle 10 are rotated and, for example, observed from an oblique upper side.

[0055] In addition, the image processing section 57 sets a masking region in the generated surrounding image (overhead image and three-dimensional image). The masking region refers to a region that is set in order to hide the vehicle body of the vehicle 10 that is reflected in the captured image of the camera. The masking region is set as a region having a shape that surrounds the vehicle 10. The image processing section 57 superimposes and displays a vehicle image that represents the vehicle 10 in a portion of the masking region that corresponds to a space in which the vehicle 10 is present. The vehicle image is an image that represents a situation in which the vehicle 10 is observed from above, and is generated in advance and stored in the storage section 54 or the like. The image processing section 57 can also set a masking region in a side image (left side image and right side image) acquired by the left side camera 12L and the right side camera 12R.

[0056] In addition, the image processing section 57 performs re-synthesis processing on the composite image of the captured data acquired by each camera, to change a boundary region between adjacent captured images in the composite image. The image processing section 57, for example, changes the boundary region by performing re-synthesis processing in accordance with the position of a predetermined object in a case where the predetermined object present in the captured image overlaps with the boundary region. The predetermined object refers to a parking frame (parking space), a parking frame line, an obstacle, or the like, which is an object that the driver of the vehicle 10 should pay attention to.

[0057] The object presence / absence determination unit 56 determines whether a predetermined object exists in a boundary region of each of the captured data in the bird's-eye image and the three-dimensional image generated by the image processing unit 57.

[0058] The display control unit 55 causes the display device of the vehicle 10 to display the surrounding image generated by the image processing unit 57. Specifically, the display control unit 55 causes the touch panel 42 to display the bird's-eye image and the three-dimensional image of the vehicle 10 generated by synthesizing each of the captured data of the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R. In addition, the display control unit 55 causes the touch panel 42 to display the bird's-eye image and the three-dimensional image on which the re-synthesis processing of the captured data is performed, based on the determination result of the object presence / absence determination unit 56.

[0059] Furthermore, the parking assistance of the vehicle 10 is implemented by automatic steering control in which the operation of the steering wheel 110 is automatically implemented by the control of the control ECU 20. This parking assistance is, for example, control of automatic steering control at the time of parking so as to park the vehicle 10 in a parking frame selected by the driver of the vehicle 10. In the assistance of automatic steering control, the operation of the accelerator pedal (not shown), the brake pedal (not shown), and the operation input unit 14 is automatically performed. In addition, the control ECU 20 performs assistance in which the vehicle 10 is parked by the operation of the accelerator pedal, the brake pedal, and the operation input unit 14 performed by the user.

[0060] The EPS system 22 has a steering angle sensor 100, a torque sensor 102, an EPS motor 104, a resolver 106, and an EPS ECU 108. The steering angle sensor 100 detects a steering angle θst of the steering wheel 110. The torque sensor 102 detects a torque TQ applied to the steering wheel 110.

[0061] The EPS motor 104 can provide the driver with operation assistance of the steering wheel 110 and automatic steering control at the time of parking assistance by imparting a driving force or a reaction force to a steering column 112 coupled to the steering wheel 110. The resolver 106 detects a rotation angle θm of the EPS motor 104. The EPS ECU 108 is responsible for the overall control of the EPS system 22. The EPS ECU 108 has an input / output unit (not shown), a calculation unit (not shown), and a storage unit (not shown).

[0062] The communication unit 24 can perform wireless communication with other communication devices 120. The other communication devices 120 refer to a base station, a communication device of another vehicle, an information terminal such as a smartphone held by an occupant of the vehicle 10, and the like.

[0063] The drive force control system 26 includes a drive ECU 130. The drive force control system 26 performs drive force control of the vehicle 10. The drive ECU 130 controls the engine (not shown) and other components based on the user's operation of the accelerator pedal (not shown), thereby controlling the drive force of the vehicle 10.

[0064] The braking force control system 28 includes a braking ECU 132. The braking force control system 28 performs braking force control of the vehicle 10. The braking ECU 132 controls the braking mechanism (not shown) and other components based on the user's operation of the brake pedal (not shown), thereby controlling the braking force of the vehicle 10.

[0065] <Changes to the boundary region performed by the image processing unit 57>

[0066] Next, refer to Figure 4 as well as Figure 5 The process of altering boundary regions in the composite image from various captured data is explained.

[0067] Figure 4 This is a diagram illustrating an example of a composite image generated from data captured by the front camera 12Fr, the rear camera 12Rr, the left-side camera 12L, and the right-side camera 12R. Figure 5 It indicates a change Figure 4 An example of a synthetic image generated by analyzing the boundary regions in the synthetic image shown.

[0068] like Figure 4 As shown, when generating the composite image 60, the image processing unit 57 corrects image distortion and performs viewpoint transformation on the shooting data of each shooting area captured by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R to obtain an image that presents a top-down view from a predetermined viewpoint position vertically above the vehicle 10. Furthermore, from each converted image obtained through this conversion process, the image processing unit 57 extracts a front image 61, a left side image 62, a right side image 63, and a rear image 64, each with a predetermined field of view range set by matching the images on both sides of the boundary region. Then, the image processing unit 57 composites these images 61 to 64 to generate the composite image 60. A masking region 65 is provided in the center of the composite image 60 to surround the vehicle 10. Additionally, a vehicle image 67 representing the vehicle 10 can be displayed within the masking region 65.

[0069] Between the adjacent captured images in the front image 61, the left side image 62, the right side image 63, and the rear image 64, there are boundary lines 66a to 66d that are boundary regions of the captured images. The range of the field of view angle extracted from the front image 61, the left side image 62, the right side image 63, and the rear image 64 is not limited to a unique range as long as the boundary regions between the adjacent captured images match each other. Therefore, depending on the range of the field of view angle extracted, the positions of the boundary lines 66a to 66d between the adjacent captured images can also vary.

[0070] The range of the field of view angle of each image that can be captured by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R is set to a width at which the adjacent captured images somewhat overlap each other in a region. Therefore, the boundary regions between the adjacent captured images can be arbitrarily extracted from the overlapping region with the matching of the boundary regions as a condition.

[0071] The images of the boundary regions, particularly the images at the boundary lines 66a to 66d, are extracted in a manner that the images on both sides of the boundary regions match each other, but since they are parts of different images, distortion occurs and visibility is often reduced. Therefore, the image processing portion 57, in the case where a prescribed object for which good visibility is required is overlapped on the boundary lines 66a to 66d in the synthesized image 60, changes the positions of the boundary regions (the boundary lines 66a to 66d) so that the boundary lines 66a to 66d do not overlap the object.

[0072] For example, Figure 4 The synthesized image 60 shown is a synthesized image that represents a situation where a vehicle is intended to be parked in a certain parking space P. In the parking space P, there is an obstacle 68 behind the vehicle, and in this case, the obstacle 68 requires good visibility as a prescribed object. However, in the synthesized image 60, the obstacle 68 is present on the boundary line 66c between the left side image 62 and the rear image 64, and on the boundary line 66d between the right side image 63 and the rear image 64.

[0073] Therefore, the image processing portion 57, for example, as shown in Figure 5 changes the boundary regions so that the boundary line 69c between the left side image 62 and the rear image 64 and the boundary line 69d between the right side image 63 and the rear image 64 do not overlap the obstacle 68. Specifically, the changes are made so that the boundary line 69c between the left side image 62 and the rear image 64 and the boundary line 69d between the right side image 63 and the rear image 64 are respectively shifted to the side. Thereby, when the vehicle is parked in the parking space P, the visibility of the obstacle 68 is improved, and the obstacle 68 can be quickly and reliably recognized.

[0074] Display control by the control ECU 20

[0075] Next, the display control performed by the control ECU 20 will be described.

[0076] [First Embodiment]

[0077] Reference Figures 6 to 8 The first embodiment of the display control performed by the control ECU 20 will be described.

[0078] Figure 6 is a flowchart showing the display control performed by the control ECU 20 when a parking frame is selected to park the vehicle 10. Figure 7 is a diagram showing one example of the bird's-eye view image generated from the respective captured data acquired by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R of the vehicle 10. Figure 8 is a diagram showing one example of the bird's-eye view image and the three-dimensional image displayed on the touch panel 42 of the vehicle 10.

[0079] For example, assume that the driver of the vehicle 10 wants to park the vehicle 10 in a parking lot. The control ECU 20 determines whether or not there is a request for parking assistance from the driver of the vehicle 10 (step Sll). The request for parking assistance is output to the control ECU 20 as a parking assistance signal, for example, based on the driver's operation of the automatic parking assistance button, the parking assistance support button, or the like in the operation input section 14.

[0080] In step Sll, in the case where there is no request for parking assistance (step Sll: No), the control ECU 20 stands by until there is a request for parking assistance.

[0081] In step Sll, in the case where there is a request for parking assistance (step Sll: Yes), the control ECU 20 generates a bird's-eye view image and a three-dimensional image based on a composite image of the respective captured data acquired by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R through the image processing section 57 (step S12).

[0082] Next, the control ECU 20 determines whether or not there is a parking frame in which the vehicle 10 can be parked, i.e., an empty parking frame in which no other vehicle is parked, in the parking lot based on the generated image through the image processing section 57 (step S13). The determination of whether or not there is an empty parking frame can be made based on the composite image of the respective captured data acquired by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R, or can be made based on the image of the respective captured data before the composition. Alternatively, the determination can be made based on the bird's-eye view image and the three-dimensional image generated from the composite image.

[0083] For example, in Figure 7 An overhead image 70 for determining whether there is a parking frame in which parking is possible is shown in FIG. 7. In the overhead image 70, a case is shown in which 3 of the parking frames P1 to P5, P1, P2, P4 at 5 are parked with other vehicles V, and 2 of the parking frames P3, P5 are not parked with other vehicles V. Boundary lines 76a to 76d are shown in the boundary portions of the front image 71, the left side image 72, the right side image 73, and the rear image 74. A vehicle image 77 representing the vehicle 10 as the host vehicle is shown in the masking region 75.

[0084] The control ECU 20 determines that the parking frames P3, P5 are parking frames in which parking is possible on the basis of the overhead image 70 in step S13.

[0085] Next, the control ECU 20 determines whether the boundary lines 76a to 76d overlap the parking frames P3, P5 determined to be possible to park by the image processing portion 57 (step S14).

[0086] In the case where the boundary lines 76a to 76d do not overlap the parking frames P3, P5 possible to park in step S14 (step S14: No), the control ECU 20 displays the overhead image and the three-dimensional image generated in step S12 as they are as a selection screen of the parking frames in which the vehicle 10 is parked on the touch panel 42 of the vehicle 10 by the display control portion 55 (step S16).

[0087] In the case where the boundary lines 76a to 76d overlap the parking frames P3, P5 possible to park in step S14 (step S14: Yes), the control ECU 20 changes the boundary lines by the image processing portion 57 again to perform the synthesis processing of each of the captured data so that the boundary lines of the boundary lines 76a to 76d overlapping the parking frames P3, P5 in the three-dimensional image generated in step S12 do not overlap the parking frames P3, P5 (step S15).

[0088] Next, the control ECU 20 displays the overhead image as it is and the three-dimensional image in which the boundary lines 76a to 76d do not overlap the parking frames P3, P5 generated in step S15 as a selection screen of the parking frames on the touch panel 42 by the display control portion 55 (step S16).

[0089] For example, in Figure 7In the overhead view 70 shown, boundary line 76c of boundary lines 76a to 76d overlaps with the parking frame P5 where parking is possible. Therefore, the control ECU 20 reprocesses the captured data by the image processing unit 57 to make the boundary line 76c not overlap with the parking frame P5, thereby changing it to a boundary line 78c that does not overlap with the parking frame P5.

[0090] Then, the control ECU20 communicates with the display control unit 55, such as... Figure 8 As shown, in the first display area 42a and the second display area 42b of the touch panel 42, the first display area 42a displays a three-dimensional image in which the boundary lines no longer overlap on the parking frame P5, and the second display area 42b displays the overhead view image generated in the above step S12 (overhead view image 70 without changing the boundary line 76c).

[0091] The driver of vehicle 10 selects a parking frame by touching the parking frames (e.g., any one of parking frames P3 and P5) displayed on the touch panel 42. The control ECU 20 provides parking support by automatically steering the vehicle 10 to park in the selected parking frame.

[0092] [Second Embodiment]

[0093] Reference Figure 9 The flowchart shown illustrates a second embodiment of display control executed by the control ECU 20. In the first embodiment described above, the display control was described in which the boundary region (boundary line) in the top-view image and the three-dimensional image displayed on the touch panel 42 was changed when a specified object (parking frame) was present on the boundary line. In the second embodiment, the display control was described in which the boundary region (boundary line) in both the three-dimensional image and the top-view image was changed.

[0094] The control ECU 20 determines whether there is a parking support request from the driver of the vehicle 10 (step S21). Similar to the first embodiment, the parking support request is output to the control ECU 20 based on the operation of the automatic parking support button and the parking assist support button.

[0095] In step S21, if there is no request for parking support (step S21: No), the ECU20 is kept on standby until a request for parking support is received.

[0096] In step S21, if there is a request for parking support (step S21: Yes), the control ECU20 generates an overhead image and a three-dimensional image by means of the image processing unit 57 based on the composite image of the shooting data obtained by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R (step S22).

[0097] Next, the control ECU 20 determines, based on the generated images, whether or not there is a parking frame in which the vehicle 10 can be parked, i.e., an empty parking frame in which no other vehicle is parked, in the parking lot (step S23) by the image processing section 57. The determination of whether or not there is an empty parking frame can be made based on the composite image of each captured data, based on the images of each captured data before the composition, or based on the overhead image and the three-dimensional image generated from the composite image, as in the first embodiment.

[0098] Next, the control ECU 20 determines, as in the first embodiment, whether or not the boundary lines 76a to 76d are superimposed on the parking frames P3, P5 determined to be parkable (refer to FIG. 12) (step S24). Figure 7

[0099] In the case where the boundary lines 76a to 76d are not superimposed on the parking frames P3, P5 determined to be parkable (step S24: No), the control ECU 20 displays, by the display control section 55, the overhead image and the three-dimensional image generated in step S22 as they are as the selection screen of the parking frames on the touch panel 42 of the vehicle 10 (step S25).

[0100] In the case where the boundary lines 76a to 76d are superimposed on the parking frames P3, P5 determined to be parkable (step S24: Yes), the control ECU 20 changes the boundary lines by again performing the composition processing of each captured data by the image processing section 57 so that the boundary lines 76a to 76d in the overhead image and the three-dimensional image generated in step S22, first in the three-dimensional image, which are superimposed on the parking frames P3, P5, are not superimposed on the parking frames P3, P5 (step S26).

[0101] The control ECU 20 displays, by the display control section 55, the three-dimensional image re-generated in step S26 so that the boundary lines 76a to 76d are not superimposed on the parking frames P3, P5 as the selection screen of the parking frames on the touch panel 42 (step S27).

[0102] Next, the control ECU 20 changes the boundary lines by again performing the composition processing of each captured data so that the boundary lines 76a to 76d in the overhead image and the three-dimensional image generated in step S22, first in the overhead image, which are superimposed on the parking frames P3, P5, are not superimposed on the parking frames P3, P5 (step S28).

[0103] The control ECU 20 displays, by the display control section 55, the overhead image re-generated in step S28 so that the boundary lines 76a to 76d are not superimposed on the parking frames P3, P5 as the selection screen of the parking frames on the touch panel 42 (step S29).​

[0104] In this case, the bird's-eye view image displayed in the second display area 42b of the touch panel 42 is also displayed as a boundary line without overlapping the image on the parking frame P5, as with the three-dimensional image displayed in the first display area 42a. Figure 8

[0105] [Third Embodiment]

[0106] Referring to Figures 10 to 12 A third embodiment of the display control performed by the control ECU 20 will be described.

[0107] Figure 10 is a view showing one example of a bird's-eye view image generated from each of the captured data acquired by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R of the vehicle 10. Figure 11 is a view showing one example of a three-dimensional image generated from each of the captured data acquired by the same cameras 12Fr, 12Rr, 12L, and 12R. Figure 12 is a view showing one example of a bird's-eye view image and a three-dimensional image displayed in the touch panel 42 of the vehicle 10.

[0108] The third embodiment describes the display control performed by the control ECU 20 in a case where the vehicle 10 is backing up to park in a prescribed parking space P. As shown in Figure 10 and Figure 11 The parking frame line 88 that becomes a parking target position of the vehicle 10 is provided in the parking space P. Therefore, in this backing up to park, in order to be able to accurately recognize the positional relationship between the vehicle 10 (vehicle image 87) and the parking frame line 88, an image of the parking frame line 88 with good visibility is required.

[0109] Therefore, the control ECU 20, for example, as shown in the bird's-eye view image 80A of Figure 10 determines that the parking frame line 88 is overlapped and displayed on the boundary line 86c between the left side image 82 and the rear image 84 and on the boundary line 86d between the right side image 83 and the rear image 84 when the vehicle 10 is backing up, the control ECU 20 causes the image processing portion 57 to perform the synthesis processing of each of the captured data again so that the parking frame line 88 is not overlapped with the boundary lines 86c and 86d.

[0110] Then, as shown in the bird's-eye view image 80A of Figure 10 and the three-dimensional image 80B of Figure 11 ​to the boundary lines, the boundary lines are changed to boundary lines that are offset to the side that does not overlap with the parking frame line 88, like the boundary lines 89c, 89d shown in the three-dimensional image 80B. Note that, regarding the change of the boundary lines, the boundary lines in the three-dimensional image 80B can be changed as explained in the first embodiment described above, or the boundary lines in the overhead view image 80A can be further changed after the boundary lines in the three-dimensional image 80B are changed as explained in the second embodiment.

[0111] Thus, in the image displayed on the touch panel 42 of the vehicle 10, at least the boundary lines of the three-dimensional image do not overlap with the parking frame line 88, that is, the three-dimensional image 80B in which the visibility of the parking frame line 88 is good is displayed on the touch panel 42. Note that, in the case where the boundary lines of the overhead view image 80A and the three-dimensional image 80B are changed, the boundary lines of the three-dimensional image 80B are changed first, and then the boundary lines of the overhead view image 80A are changed. Figure 12 In the first display area 42a and the second display area 42b of the touch panel 42 shown in FIG. 6, the three-dimensional image 80B and the overhead view image 80A in which the boundary lines do not overlap with the parking frame line 88 are displayed, respectively.

[0112] As explained above, in the case where it is determined by the object presence / absence determination portion 56 that the prescribed object exists on the boundary line, the control ECU 20 changes the boundary lines of the displayed overhead view image and the three-dimensional image of the three-dimensional image by the image processing portion 57 with priority.

[0113] Note that, the priority change of the boundary lines of the overhead view image and the three-dimensional image of the three-dimensional image means that the boundary lines of the three-dimensional image are changed with priority as long as there is no other factor that should change the boundary lines of the overhead view image with priority, such as the user specifying the change of the boundary lines of the overhead view image, or the user referring to the overhead view image more frequently than the three-dimensional image.

[0114] Thus, in the case where the prescribed object exists on the boundary line, the boundary lines of the three-dimensional image, which is easier for the driver of the vehicle 10 to recognize the situation of the periphery than the overhead view image, are changed with priority, and thus the prescribed object can be quickly recognized by the driver. Therefore, for example, in the process of entering a narrow parking space or in the process of exiting from a narrow parking space, it is possible to accurately confirm whether the vehicle 10 collides with an obstacle in the periphery or the like. In addition, in the process of entering a narrow parking space, it is easy to confirm whether there is a space in which the occupant of the vehicle 10 can easily get off the vehicle 10 after the vehicle 10 is parked, or the like. In addition, in the process of parking the vehicle 10, it is easy to confirm whether there is an obstacle or the like that the occupant of the vehicle 10 contacts when getting off the vehicle 10.

[0115] In addition, in the case where it is determined by the object presence / absence determination portion 56 that the prescribed object exists on the boundary line, the control ECU 20 changes the boundary lines of only the three-dimensional image of the overhead view image and the three-dimensional image by the image processing portion 57. Thus, the boundary lines of at least the three-dimensional image, which is easier to recognize the situation of the periphery, are changed, and thus quick recognition of the prescribed object can be performed.

[0116] In addition, in a case where it is determined by the object presence / absence determination portion 56 that the prescribed object is present on the boundary line, the control ECU 20 changes the boundary line of the overhead view image and displays it on the touch panel 42 after changing the boundary line of the three-dimensional image and displaying it on the touch panel 42 by the image processing portion 57. Thereby, quick recognition of the prescribed object is possible by the three-dimensional image, and the object is also able to be confirmed in an image with good visibility by the overhead view image, and thus convenience is improved.

[0117] Note that, in the above-described embodiment, a case where the boundary line of the three-dimensional image is always prioritized to be changed in a case where it is determined that the prescribed object is displayed in superposition on the boundary line has been described, but the present application is not limited thereto. For example, the control ECU 20 can prioritize the boundary line of either one of the overhead view image and the three-dimensional image based on information related to the user of the vehicle 10 (for example, the driver of the vehicle 10).

[0118] The information related to the user is, for example, a setting by the user. That is, the control ECU 20 can also prioritize the boundary line of the overhead view image to be changed in a case where the boundary line of the overhead view image is set to be prioritized to be changed by the user of the vehicle 10 (for example, the driver of the vehicle 10) from the overhead view image and the three-dimensional image. Thereby, the usability of the present function in the vehicle 10 can be improved.

[0119] Alternatively, the information related to the user can also be history information of the past user's reference to the overhead view image and the three-dimensional image. For example, the control ECU 20 can also determine which one of the overhead view image and the three-dimensional image the driver of the vehicle 10 frequently refers to based on the history information of the past user's reference to the overhead view image and the three-dimensional image, and prioritize the boundary line of the image that is more frequently referred to to be changed. Thereby, the usability of the present function in the vehicle 10 can be improved.

[0120] The history information of the past user's reference to the overhead view image and the three-dimensional image is acquired, for example, based on a detection result of a line-of-sight sensor that detects the line of sight of the driver of the vehicle 10 provided to the vehicle 10. In addition, in a case where either one of the overhead view image and the three-dimensional image is able to be displayed by the touch panel 42 by the operation of the driver, the history information of the past user's reference to the overhead view image and the three-dimensional image can also be acquired based on a switching history of the operation of the driver to display the overhead view image and the three-dimensional image.

[0121] The three-dimensional image and the overhead view image displayed at the time of selecting the parking frame and at the time of parking the vehicle 10 have been described, but the present application is not limited thereto, and can also be applied to the three-dimensional image and the overhead view image displayed at the time of starting the vehicle 10.

[0122] The above describes embodiments of the present application, but the present application is not limited to the above-described embodiments and can be appropriately modified, improved, and the like.

[0123] For example, in the above-described embodiments, the case where the control ECU 20 displays the bird's-eye image and the three-dimensional image on the touch panel 42 of the vehicle 10 is described, but is not limited thereto. For example, the control ECU 20 can also display the bird's-eye image and the three-dimensional image on a display screen of an information terminal (for example, a smartphone or the like) held by an occupant of the vehicle 10 via the communication section 24.

[0124] Further, in the above-described embodiments, the case where the mobile body is a vehicle is described, but is not limited thereto. The idea of the present application is not limited to a vehicle and can also be applied to a robot, a ship, an airplane, or the like that has a drive source and is capable of moving by the power of the drive source.

[0125] Note that the control method described in the above-described embodiments can be realized by a computer executing a control program prepared in advance. The control program is stored in a storage medium that is readable by a computer and is executed by being read from the storage medium. Further, the control program can be provided in the form of being stored in a non-transitory storage medium such as a flash memory or can be provided via a network such as the Internet. The computer that executes the control program can be included in the control device, can be included in an electronic device such as a smartphone, a tablet terminal, or a personal computer that is capable of communicating with the control device, or can be included in a server device that is capable of communicating with the control device and the electronic device.

[0126] Further, at least the following matters are described in the present specification. Note that although the corresponding constituent elements and the like in the above-described embodiments are shown in parentheses, the present application is not limited thereto.

[0127] (1) A control device including:

[0128] an image processing section (image processing section 57) that generates a bird's-eye image and a three-dimensional image that represent a mobile body and a periphery of the mobile body on the basis of each photographing data acquired by a plurality of photographing devices (front camera 12Fr, rear camera 12Rr, left side camera 12L, right side camera 12R) of the mobile body (vehicle 10);

[0129] a display control section (display control section 55) that causes a display device (touch panel 42) to display the bird's-eye image and the three-dimensional image generated by the image processing section; and

[0130] a determination unit (object presence determination unit 56) that determines whether a prescribed object (obstacle 68, parking frame line 88) is present in a boundary region of the respective pieces of captured data in the overhead image and the three-dimensional image,

[0131] The image processing unit preferentially changes the boundary region of the three-dimensional image in the overhead image and the three-dimensional image that is displayed in the case where the determination unit determines that the prescribed object is present in the boundary region.

[0132] According to (1), in the case where the prescribed object is present in the boundary region, by preferentially changing the boundary region of the three-dimensional image that is easier for the user to recognize the situation of the surroundings, the prescribed object can be quickly recognized by the driver.

[0133] (2) The control device according to (1), wherein

[0134] The image processing unit changes the boundary region of only the three-dimensional image in the overhead image and the three-dimensional image in the case where the determination unit determines that the prescribed object is present in the boundary region.

[0135] According to (2), at least the boundary region of the three-dimensional image that is easier to recognize the situation of the surroundings is changed, and thus quick recognition of the prescribed object can be performed.

[0136] (3) The control device according to (1), wherein

[0137] The image processing unit, in the case where the determination unit determines that the prescribed object is present in the boundary region, changes the boundary region of the three-dimensional image and outputs to the display control unit, and then changes the boundary region of the overhead image and outputs to the display control unit.

[0138] According to (3), quick recognition of the prescribed object can be performed by the three-dimensional image, and the prescribed object can also be confirmed in an image with good visibility by the overhead image, and thus convenience is improved.

[0139] (4) The control device according to any one of (1) to (3), wherein

[0140] The image processing unit, in the case where the determination unit determines that the prescribed object is present in the boundary region, preferentially changes the boundary region of either one of the overhead image and the three-dimensional image based on information related to a user of the moving body.

[0141] According to (4), by preferentially changing the boundary region of the image corresponding to the information related to the user in the overhead image and the three-dimensional image, usability can be improved.

[0142] (5) The control device according to (4), in which

[0143] The information related to the user of the moving body includes history information of the user having referred to the overhead image and the three-dimensional image in the past, respectively.

[0144] According to (5), by preferentially changing the boundary region of the image that the user more frequently refers to between the overhead image and the three-dimensional image, it is possible to improve the usability.

[0145] (6) A control method executed by a processor that generates an overhead image and a three-dimensional image representing a moving body and a periphery of the moving body based on each photographing data acquired by a plurality of photographing devices of the moving body, and causes a display device to display the generated overhead image and three-dimensional image, the control method comprising:

[0146] determining, by the processor, whether a prescribed object exists in a boundary region of the each photographing data in the overhead image and the three-dimensional image; and

[0147] in a case where it is determined that the prescribed object exists in the boundary region, preferentially changing, by the processor, the boundary region of the three-dimensional image in the displayed overhead image and three-dimensional image.

[0148] According to (6), in a case where a prescribed object exists in a boundary region, by preferentially changing the boundary region of the three-dimensional image that is easier for the user to recognize the situation of the periphery, it is possible for the driver to quickly recognize the prescribed object.

[0149] (7) A control program for causing a processor to execute a process that generates an overhead image and a three-dimensional image representing a moving body and a periphery of the moving body based on each photographing data acquired by a plurality of photographing devices of the moving body, and causes a display device to display the generated overhead image and three-dimensional image, the process comprising:

[0150] determining whether a prescribed object exists in a boundary region of the each photographing data in the overhead image and the three-dimensional image; and

[0151] in a case where it is determined that the prescribed object exists in the boundary region, preferentially changing the boundary region of the three-dimensional image in the displayed overhead image and three-dimensional image.

[0152] According to (7), in a case where a prescribed object exists in a boundary region, by preferentially changing the boundary region of the three-dimensional image that is easier for the user to recognize the situation of the periphery, it is possible for the driver to quickly recognize the prescribed object.

Claims

1. A control device comprising: an image processing section that generates an overhead image and a three-dimensional image representing a mobile body and a periphery of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body; a display control section that causes a display device to display the overhead image and the three-dimensional image generated by the image processing section; and a determination section that determines whether a prescribed object exists in a boundary region of the each photographing data in the overhead image and the three-dimensional image, the image processing section preferentially changing the boundary region of the three-dimensional image in the displayed overhead image and the three-dimensional image in a case where the determination section determines that the prescribed object exists in the boundary region.

2. The control device according to claim 1, wherein the image processing section changes only the boundary region of the three-dimensional image in the overhead image and the three-dimensional image in a case where the determination section determines that the prescribed object exists in the boundary region.

3. The control device according to claim 1, wherein the image processing section changes the boundary region of the overhead image and outputs to the display control section after changing the boundary region of the three-dimensional image and outputting to the display control section in a case where the determination section determines that the prescribed object exists in the boundary region.

4. The control device according to any one of claims 1 to 3, wherein the image processing section preferentially changes the boundary region of the three-dimensional image in the three-dimensional image based on information related to a user of the mobile body in a case where the determination section determines that the prescribed object exists in the boundary region.

5. The control device according to claim 4, wherein the information related to the user of the mobile body includes history information of the user respectively referring to the overhead image and the three-dimensional image in the past.

6. A control method executed by a processor that generates an overhead image and a three-dimensional image representing a mobile body and a periphery of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body, and causes a display device to display the generated overhead image and the three-dimensional image, the control method comprising: determining, by the processor, whether a prescribed object exists in a boundary region of the each photographing data in the overhead image and the three-dimensional image; and preferentially changing, by the processor, the boundary region of the three-dimensional image in the displayed overhead image and the three-dimensional image in a case where it is determined that the prescribed object exists in the boundary region.

7. A program product including a control program for causing a processor to execute a process that generates an overhead image and a three-dimensional image representing a mobile body and a periphery of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body, and causes a display device to display the generated overhead image and the three-dimensional image, the process comprising: determining whether a prescribed object exists in a boundary region of the each photographing data in the overhead image and the three-dimensional image; and preferentially changing the boundary region of the three-dimensional image in the displayed overhead image and the three-dimensional image in a case where it is determined that the prescribed object exists in the boundary region. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ In a case where it is determined that the prescribed object exists in the boundary region, the boundary region of the displayed bird's-eye view image and the three-dimensional image is preferentially changed.

Images