Control device, control method, and control program

Abstract

The present application provides a control device, control method, and control program capable of displaying a good surrounding image regardless of the opening and closing of a side mirror. The control device includes an image processing section (57) that generates an overhead image (61, 71) of a vehicle (10) based on captured data obtained by a left side camera (12L) and a right side camera (12R) provided to a side mirror (11L, 11R) of the vehicle (10) that can be opened and closed; a display control section (55) that causes a touch panel (42) to display the overhead image (61, 71) generated by the image processing section (57); and an opening and closing determination section (56) that determines the opening and closing state of the side mirror (11L, 11R). The image processing section (57) sets a masking region (62, 72) in the overhead image (61, 71) based on the range of the determination result of the opening and closing state of the opening and closing determination section (56).

Description

TECHNICAL FIELD

[0001] The present application relates to a control device, a control method, and a control program. BACKGROUND

[0002] In recent years, as a specific countermeasure against global climate change, initiatives to achieve a low-carbon society or a decarbonized society are very active. In the field of vehicles as well, there is a strong demand for reduction in CO2 emissions, and introduction of automatic driving and driving assistance of vehicles that contribute to improvement in fuel efficiency is rapidly advancing.

[0003] For example, an image generation method is known in which each of cameras mounted on a vehicle photographs a prescribed range and acquires an image, and a surrounding image (for example, an overhead image) of the vehicle and the surroundings of the vehicle is generated based on a combined image of the photographed respective images. In Patent Literature 1, a vehicle surrounding monitoring device is described that changes a photographing range of an image photographed by each camera in accordance with opening and closing of a side mirror of the vehicle, and changes a boundary position between photographed images in a combined image of the photographed images to generate an overhead image.

[0004] PRIOR ART DOCUMENTS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: Japanese Patent No. 5112998

[0007] However, in a case where the surroundings of the vehicle are photographed using cameras mounted on the vehicle, there is a case where a part of the host vehicle is reflected in the photographed image. Therefore, in order to prevent reflection of the host vehicle, a masking region is set in the vicinity of the host vehicle in the overhead image, for example. However, when the side mirror of the vehicle is opened and closed, the photographing range of the camera changes, and therefore the reflection range of the host vehicle also changes in accordance with opening and closing of the side mirror. However, in Patent Literature 1, generation of an overhead image corresponding to the change in the reflection range of the host vehicle accompanying opening and closing of the side mirror is not described. Therefore, there is room for improvement in this regard in the related art. SUMMARY

[0008] An object of the present application is to provide a control device, a control method, and a control program that can display a good surrounding image without being affected by opening and closing of a side mirror.

[0009] The present application provides a control device including:

[0010] an image processing section that generates a surrounding image of a moving body based on photographing data acquired by a photographing device provided to a side mirror of the moving body that is openable and closable;

[0011] a display control section that causes a display device to display the surrounding image generated by the image processing section; and

[0012] an opening / closing determination section that determines an opening / closing state of the side mirror, wherein

[0013] the image processing section sets a masking region in the surrounding image based on a range of a determination result of the opening / closing state by the opening / closing determination section.

[0014] A control method is provided, which is executed by a control device that has a processor that generates a surrounding image of a moving body based on captured data acquired by a capturing device provided to a side mirror of the moving body and causes a display device to display the generated surrounding image, the control method including:

[0015] determining, by the processor, an opening / closing state of the side mirror; and

[0016] setting, by the processor, a masking region in the surrounding image based on a range of a determination result of the opening / closing state.

[0017] A control program is provided, which causes a processor of a control device that generates a surrounding image of a moving body based on captured data acquired by a capturing device provided to a side mirror of the moving body and causes a display device to display the generated surrounding image to execute a process, the process including:

[0018] determining an opening / closing state of the side mirror; and

[0019] setting a masking region in the surrounding image based on a range of a determination result of the opening / closing state.

[0020] According to the control device, the control method, and the control program of the present application, it is possible to display a surrounding image of a moving body that enables a prescribed object to be quickly recognized. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a side view showing one example of a vehicle on which the control device of the present embodiment is mounted.

[0022] Figure 2 is a plan view showing a state in which the side mirror is opened in the vehicle shown in Figure 1 .

[0023] Figure 3 is a plan view showing a state in which the side mirror is closed in the vehicle shown in Figure 1 .

[0024] Figure 4 is a plan view showingFigure 1 a block diagram showing an internal structure of the vehicle.

[0025] Figure 5 is a flowchart showing one example of display control by the control ECU.

[0026] Figure 6 is a view showing one example of a masking region set when the side mirror is in an open state.

[0027] Figure 7 is a view showing one example of a masking region set when the side mirror is in a closed state.

[0028] Figure 8 is a view showing one example of an overhead view image displayed on the touch panel.

[0029] --BRIEF DESCRIPTION OF THE DRAWINGS--

[0030] 10 vehicle (moving body), 11L, 11R side mirror, 12Fr front camera (imaging device), 12Rr rear camera (imaging device), 12L left side camera (imaging device), 12R right side camera (imaging device), 20 control ECU (control device), 42 touch panel (display device), 55 display control portion, 56 opening / closing determination portion, 57 image processing portion, 61, 71 overhead view image, 62, 72 masking region, 63, 73 vehicle image. DETAILED DESCRIPTION

[0031] Hereinafter, one embodiment of the control device, the control method, and the control program of the present application will be described based on the drawings. Note that the drawings are viewed in the direction of the orientation of symbols. In addition, in this specification and the like, each of the directions of front, rear, left, right, upper, and lower is described in the order of Fr, Rr, L, R, U, and D, respectively, as viewed from the driver of the vehicle 10 shown in the drawings, and in the drawings, the front of the vehicle 10 is denoted as Fr, the rear is denoted as Rr, the left is denoted as L, the right is denoted as R, the upper is denoted as U, and the lower is denoted as D. Figures 1-3

[0032] <vehicle 10 to which the control device of the present application is mounted>

[0033] Figure 1 is a side view of the vehicle 10 to which the control device of the present application is mounted. Figure 2 , Figure 3 is a plan view of the vehicle 10 shown in Figure 1

[0034] ​​Vehicle 10 is an automobile with a drive source (not shown) and wheels, the wheels including drive wheels driven by the power of the drive source and steering wheels capable of steering. In this embodiment, vehicle 10 is a four-wheeled automobile with a pair of front wheels on the left and right and a pair of rear wheels on the left and right. The drive source of vehicle 10 is, for example, an electric motor. The drive source of vehicle 10 can be an internal combustion engine such as a gasoline engine or a diesel engine, or a combination of an electric motor and an internal combustion engine. In addition, the drive source of vehicle 10 can drive the pair of front wheels on the left and right, the pair of rear wheels on the left and right, or all four wheels. The front wheels and rear wheels can be steering wheels capable of steering in both directions, or steering wheels capable of steering in either direction.

[0035] The vehicle 10 also has side rearview mirrors 11L and 11R. The side rearview mirrors 11L and 11R are mirrors (rearview mirrors) located on the outside of the front doors of the vehicle 10 and used to allow the driver to see behind and to the sides.

[0036] The side rearview mirrors 11L and 11R are fixed to the main body of the vehicle 10 via a rotating shaft extending in the vertical direction, and can be opened and closed by rotating around the rotating shaft. The opening and closing of the side rearview mirrors 11L and 11R are electrically operated, for example, by the driver operating an operating unit located near the driver's seat of the vehicle 10.

[0037] like Figure 2 As shown, the side mirrors 11L and 11R are set to the open state when in use (e.g., when the vehicle 10 is in motion). The width (length in the left-right direction) of the vehicle 10 including the side mirrors 11L and 11R at this time is set to width D1.

[0038] like Figure 3 As shown, the side mirrors 11L and 11R are set to the closed state when they are not in use (e.g., when the vehicle 10 is parked). The width (length in the left-right direction) of the vehicle 10 including the side mirrors 11L and 11R at this time is set to width D2.

[0039] like Figure 2 , Figure 3 As shown, the width D2 of the vehicle 10 when the side mirrors 11L and 11R are closed is narrower than the width D1 of the vehicle 10 when the side mirrors 11L and 11R are open. Therefore, for example, when entering a narrow parking space, the driver often closes the side mirrors 11L and 11R to prevent the vehicle 10 from colliding with surrounding obstacles.

[0040] The vehicle 10 also includes a front camera 12Fr, a rear camera 12Rr, a left-side camera 12L, and a right-side camera 12R. The front camera 12Fr is a digital camera positioned at the front of the vehicle 10 and taking pictures of the front of the vehicle 10. The rear camera 12Rr is a digital camera positioned at the rear of the vehicle 10 and taking pictures of the rear of the vehicle 10. The left-side camera 12L is a digital camera positioned in the left-side rearview mirror 11L of the vehicle 10 and taking pictures of the left side of the vehicle 10. The right-side camera 12R is a digital camera positioned in the right-side rearview mirror 11R of the vehicle 10 and taking pictures of the right side of the vehicle 10. The front camera 12Fr, rear camera 12Rr, left-side camera 12L, and right-side camera 12R are an example of the shooting device of the present invention.

[0041] <Internal structure of vehicle 10>

[0042] Figure 4 It means Figure 1 A block diagram illustrating an example of the internal structure of vehicle 10. (See diagram for reference.) Figure 4 As shown, vehicle 10 includes a sensor group 16, a navigation device 18, a control ECU (Electronic Control Unit) 20, an EPS (Electric Power Steering) system 22, and a communication unit 24. Vehicle 10 also includes a drive force control system 26 and a braking force control system 28. The control ECU 20 is an example of the control device of the present invention.

[0043] Sensor group 16 acquires various detection values ​​used in the control of ECU 20. Sensor group 16 includes a front camera 12Fr, a rear camera 12Rr, a left-side camera 12L, and a right-side camera 12R. Additionally, sensor group 16 includes a front sonar group 32a, a rear sonar group 32b, a left-side sonar group 32c, and a right-side sonar group 32d. Furthermore, sensor group 16 includes wheel sensors 34a and 34b, a vehicle speed sensor 36, and an operation detection unit 38.

[0044] The front camera 12Fr, rear camera 12Rr, left-side camera 12L, and right-side camera 12R output peripheral images obtained by capturing the area around the vehicle 10. The peripheral images captured by the front camera 12Fr, rear camera 12Rr, left-side camera 12L, and right-side camera 12R are respectively referred to as the front image, rear image, left-side image, and right-side image. The left-side image and the right-side image constitute the side image.

[0045] The front sonar group 32a, the rear sonar group 32b, the left side sonar group 32c, and the right side sonar group 32d emit sound waves toward the periphery of the vehicle 10, and receive reflected sounds from other objects. The front sonar group 32a includes, for example, four sonars. The sonars that constitute the front sonar group 32a are respectively provided in the left oblique front, the front left side, the front right side, and the right oblique front of the vehicle 10. The rear sonar group 32b includes, for example, four sonars. The sonars that constitute the rear sonar group 32b are respectively provided in the left oblique rear, the rear left side, the rear right side, and the right oblique rear of the vehicle 10. The left side sonar group 32c includes, for example, two sonars. The sonars that constitute the left side sonar group 32c are respectively provided in the left side portion front and the left side portion rear of the vehicle 10. The right side sonar group 32d includes, for example, two sonars. The sonars that constitute the right side sonar group 32d are respectively provided in the right side portion front and the right side portion rear of the vehicle 10.

[0046] The wheel sensors 34a, 34b detect the rotation angle of the wheels of the vehicle 10. The wheel sensors 34a, 34b can be constituted by angle sensors, or can be constituted by displacement sensors. The wheel sensors 34a, 34b output a detection pulse each time the wheel rotates a prescribed angle. The detection pulse output from the wheel sensors 34a, 34b is used for the calculation of the rotation angle of the wheel and the rotation speed of the wheel. The distance of movement of the vehicle 10 is calculated on the basis of the rotation angle of the wheel. The wheel sensor 34a, for example, detects the rotation angle θa of the left rear wheel. The wheel sensor 34b, for example, detects the rotation angle θb of the right rear wheel.

[0047] The vehicle speed sensor 36 detects the speed of the vehicle body of the vehicle 10, that is, the vehicle speed V, and outputs the detected vehicle speed V to the control ECU 20. The vehicle speed sensor 36, for example, detects the vehicle speed V on the basis of the rotation of the countershaft of the transmission.

[0048] The operation detection portion 38 detects the content of the operation performed by the user using the operation input portion 14, and outputs the detected operation content to the control ECU 20. The operation input portion 14 includes, for example, a side mirror switch that switches the open state and the closed state of the side mirrors 11L, 11R. In addition, the operation input portion 14 can include various user interfaces such as a shift lever (gear lever, selector), and the like.

[0049] The navigation device 18 detects the current position of the vehicle 10 using, for example, a GPS (Global Positioning System), and guides the user of the path to the destination. The navigation device 18 has a storage device (not shown) that has a map information database.

[0050] The navigation device 18 is provided with a touch panel 42 and a speaker 44. The touch panel 42 functions as an input device and a display device that control the ECU 20. The user inputs various instructions via the touch panel 42. In addition, various screens are displayed on the touch panel 42. Note that a constituent element other than the touch panel 42, such as a smartphone, can be used as an input device or a display device. The speaker 44 outputs various guidance information to the occupant of the vehicle 10 through sound.

[0051] The control ECU 20 has an input-output section 50, an arithmetic section 52, and a storage section 54. The arithmetic section 52 is constituted by, for example, a CPU (Central Processing Unit). The arithmetic section 52 controls each section based on a program stored in the storage section 54, thereby performing various controls.

[0052] The arithmetic section 52 includes a display control section 55, an opening-closing determination section 56, and an image processing section 57.

[0053] The opening-closing determination section 56 determines whether the side mirror 11L, 11R is in an open state or a closed state. The determination of the opening-closing determination section 56 is performed based on, for example, a detection result of the operation of the side mirror switch included in the operation input section 14 by the operation detection section 38. The determination of the opening-closing determination section 56 can also be performed based on a detection result of the state of the side mirror 11L, 11R by a sensor (not shown) provided to the side mirror 11L, 11R.

[0054] The image processing section 57 generates a surrounding image of the vehicle 10 based on captured data acquired by the cameras of the vehicle 10. Specifically, the image processing section 57 generates an overhead image that represents a situation of observing the surroundings of the vehicle 10 from above by synthesizing captured data acquired by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R.

[0055] In addition, the image processing section 57 sets a masking region in the generated surrounding image (overhead image). The masking region refers to a region set to hide the vehicle body of the vehicle 10 reflected in the captured image. The image processing section 57 can also set a masking region in a side image (left side image and right side image) acquired by the left side camera 12L and the right side camera 12R.

[0056] The range in which the vehicle body of the vehicle 10 is reflected in the captured image differs depending on the captured data acquired by the cameras. Therefore, the masking region set in the surrounding image also differs depending on the captured data acquired by the cameras. For example, the side image acquired by the left side camera 12L and the right side camera 12R differs depending on whether the side mirrors 11L and 11R in which the left side camera 12L and the right side camera 12R are installed are opened or closed. Therefore, the range in which the vehicle body of the vehicle 10 is reflected, that is, the masking region set in the surrounding image of the vehicle 10 differs depending on whether the side mirrors 11L and 11R are opened or closed. The image processing section 57 sets the masking region having a different size in the surrounding image of the vehicle 10 on the basis of the determination result of the opening / closing state of the side mirrors 11L and 11R by the opening / closing determination section 56.

[0057] The display control section 55 causes the display device of the vehicle 10 to display the surrounding image generated by the image processing section 57. Specifically, the display control section 55 causes the touch panel 42 to display the bird's-eye image of the vehicle 10 generated by synthesizing the captured data of the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R. In addition, the display control section 55 causes the touch panel 42 to display the bird's-eye image on the basis of the determination result of the opening / closing determination section 56. Specifically, the display control section 55 causes the touch panel 42 to display the bird's-eye image including the masking region having a size switched depending on the opening / closing of the side mirrors 11L and 11R.

[0058] The control ECU 20 can also assist in parking of the vehicle 10 by performing automatic steering control. That is, the operation of the steering wheel 110 can also be automatically performed by the control of the control ECU 20. Specifically, the operation of the accelerator pedal (not shown), the brake pedal (not shown), and the operation input section 14 can also be automatically performed.

[0059] The EPS system 22 has a steering angle sensor 100, a torque sensor 102, an EPS motor 104, a resolver 106, and an EPS ECU 108. The steering angle sensor 100 detects a steering angle θst of the steering wheel 110. The torque sensor 102 detects a torque TQ applied to the steering wheel 110.

[0060] The EPS motor 104 can provide the driver with operation assistance of the steering wheel 110 and automatic steering control at the time of parking assistance by imparting a driving force or a reaction force to a steering column 112 coupled to the steering wheel 110. The resolver 106 detects a rotation angle θm of the EPS motor 104. The EPS ECU 108 is responsible for the overall control of the EPS system 22. The EPS ECU 108 has an input / output section (not shown), a calculation section (not shown), and a storage section (not shown).

[0061] The communication section 24 is capable of wireless communication with other communication devices 120. The other communication devices 120 refer to base stations, communication devices of other vehicles, information terminals such as smartphones held by occupants of the vehicle 10, and the like.

[0062] The drive force control system 26 is provided with a drive ECU 130. The drive force control system 26 performs drive force control of the vehicle 10. The drive ECU 130 controls an engine and the like, which are not shown, on the basis of an operation of an accelerator pedal, which is not shown, by a user, thereby controlling the drive force of the vehicle 10.

[0063] The brake force control system 28 is provided with a brake ECU 132. The brake force control system 28 performs brake force control of the vehicle 10. The brake ECU 132 controls a brake mechanism and the like, which are not shown, on the basis of an operation of a brake pedal, which is not shown, by a user, thereby controlling the brake force of the vehicle 10.

[0064] <Display Control by Control ECU 20>

[0065] Next, the display control by the control ECU 20 will be described with reference to Figures 5 to 8 The display control by the control ECU 20 will be described. Figure 5 is a flowchart showing one example of the display control by the control ECU 20. Figure 6 is a view showing one example of the masking region set when the side mirrors 11L, 11R are in the open state. Figure 7 is a view showing one example of the masking region set when the side mirrors 11L, 11R are in the closed state. Figure 8 is a view showing one example of the bird's-eye image displayed on the touch panel 42.

[0066] The control ECU 20 starts the processing shown in Figure 5 for example at the time of turning on of an ignition switch.

[0067] First, the control ECU 20 causes the opening / closing determination section 56 to determine the opening / closing state of the side mirrors 11L, 11R (step S11). The control ECU 20 determines whether or not the side mirrors 11L, 11R are in the open state on the basis of the determination result of the opening / closing determination section 56 in step S11 (step S12).

[0068] In step S12, in the case where the side mirrors 11L, 11R are in the open state (step S12: YES), the control ECU 20 generates a bird's-eye image in which a relatively narrow masking region is set, by the image processing section 57 (step S13). The relatively narrow masking region refers to a masking region that is narrower than the masking region generated when the side mirrors 11L, 11R are in the closed state.

[0069] Figure 6The illustrated overhead image 61 is an image indicating a situation of observing the periphery of the vehicle 10 from above, which is generated by synthesizing the captured data of the front image, the rear image, the left side image, and the right side image obtained by the front camera 12Fr, the rear camera 12Rr, and the left side camera 12L and the right side camera 12R in the opened state. The overhead image 61 is generated, for example, as an image having a rectangular shape. The masking region 62 is generated in the central portion of the overhead image 61. The masking region 62 is generated, for example, as a region having a rectangular shape, similarly to the overhead image 61. In the masking region 62, the vehicle image 63 indicating the vehicle 10 is displayed in superposition in a portion (for example, the central portion) corresponding to the space in which the vehicle 10 is present. The vehicle image 63 is an image indicating a situation of observing the vehicle 10 from above, and is an image generated (captured) in advance and stored in the storage section 54 or the like. With the display of the vehicle image 63, the driver can easily grasp the positional relationship between the periphery of the vehicle 10 and the vehicle 10.

[0070] For example, if the lateral width (width in the left-right direction) of the masking region 62 is set to a lateral width Ml, the lateral width Ml of the masking region 62 generated when the side mirrors 11L, 11R are in the opened state is set to be smaller than the lateral width of the masking region (described later in Figure 7

[0071] On the other hand, in the case where the side mirrors 11L, 11R are not in the opened state in step S12 (step S12: No), the control ECU 20 generates an overhead image in which a wider masking region is set, using the image processing section 57 (step S14). The wider masking region refers to a masking region that is wider than the masking region set when the side mirrors 11L, 11R are in the opened state, that is, the masking region 62 described above. The control ECU 20 generates an overhead image in which a masking region wider than the masking region 62 is set. Figure 6

[0072] Figure 7 The illustrated overhead image 71 is an image indicating a situation of observing the periphery of the vehicle 10 from above, which is generated by synthesizing the captured data of the front image, the rear image, the left side image, and the right side image obtained by the front camera 12Fr, the rear camera 12Rr, and the left side camera 12L and the right side camera 12R in the stowed state. The masking region 72 is generated in the central portion of the overhead image 71. The vehicle image 73 is displayed in superposition in the masking region 72.

[0073] ​​For example, if the lateral width of the shield region 72 is set to be lateral width M2, the lateral width M2 of the shield region 72 generated when the side mirrors 11L, 11R are in the closed state is set to be larger than the lateral width Ml of the shield region 62 generated when the side mirrors 11L, 11R are in the open state.

[0074] The bird's-eye image 71 is an image generated by synthesizing the captured data acquired by the front camera 12Fr, the rear camera 12Rr, and the left side camera 12L and the right side camera 12R in the closed state as described above. The control ECU 20, for example, performs conversion processing on the captured data acquired by the left side camera 12L and the right side camera 12R in the closed state, and generates the bird's-eye image 71 by synthesizing the captured data of the left side camera 12L and the right side camera 12R subjected to the conversion processing with the captured data of the front camera 12Fr and the rear camera 12Rr.

[0075] The conversion processing on the captured data is processing to offset the change in the capturing conditions of the left side camera 12L and the right side camera 12R accompanying the displacement of the side mirrors 11L, 11R. For example, if the side mirror 11L is displaced from the open state to the closed state, the left side camera 12L provided to the side mirror 11L is also displaced, and thus the capturing conditions such as the capturing position, the capturing direction, and the like of the left side camera 12L change. Therefore, the control ECU 20 performs conversion processing for bringing the captured data obtained by the left side camera 12L when the side mirror 11L is in the closed state close to the captured data obtained by the left side camera 12L when the side mirror 11L is in the open state.

[0076] As an example, when the side mirror 11L changes from an open state to a closed state, the shooting direction of the left-side camera 12L changes to face rearward and towards the vehicle body 10. In this case, the shooting data acquired by the left-side camera 12L becomes shooting data that includes an image of the rear of the vehicle 10, and also becomes shooting data that includes an image of the vehicle body 10. The control ECU 20 performs a conversion process that significantly stretches the shooting data acquired by the left-side camera 12L to the forward side when the side mirror 11L is closed. Furthermore, the conversion process includes a process to correct for distortions such as those accompanying the significant forward stretching of the shooting data. As a result, it is possible to make the shooting data acquired by the left-side camera 12L when the side mirror 11L is closed similar to the shooting data obtained when the side mirror 11L is open, but the shooting data includes an image of the vehicle body 10. Therefore, when the side mirror 11L is closed, it is necessary to ensure a wide obscuring area in order to hide the image of the vehicle body 10. It should be noted that the conversion processing for the shooting data obtained by the left-side camera 12L has been explained, but the conversion processing for the shooting data obtained by the right-side camera 12R is the same.

[0077] Next, in step S13, if an overhead view 61 with a relatively narrow obscuring area 62 is generated, the control ECU 20 moves to step S15, and the generated overhead view 61 is displayed on the touch panel 42 of the vehicle 10's display device via the display control unit 55. Similarly, in step S14, if an overhead view 71 with a relatively wide obscuring area 72 is generated, the control ECU 20 moves to step S15, and the generated overhead view 71 is displayed on the touch panel 42 of the vehicle 10 via the display control unit 55.

[0078] like Figure 8 As shown, the touch panel 42 is provided with, for example, a first display area 42a and a second display area 42b. The control ECU 20 causes the second display area 42b to display the overhead view 61 generated in step S13 or the overhead view 71 generated in step S14. Figure 8 In the example shown, a top-down image 61 is displayed, showing a relatively narrow obscuring area 62 generated when the side mirrors 11L and 11R are open. In the top-down image 61, adjacent vehicles 100A and 100B are shown parked on either side of the vehicle image 63, which is an image of vehicle 10. Thus, the driver can confirm the surroundings of vehicle 10 based on the top-down image 61, which has a good and sufficient field of vision, by obscuring, for example, the body of vehicle 10, through the relatively narrow obscuring area 62.

[0079] It should be noted that, although inFigure 8 Although not shown, any image such as a front image based on captured data acquired by the front camera 12Fr, a rear image based on captured data acquired by the rear camera 12Rr, a button image for various instructions to the occupant of the vehicle 10, and the like can also be displayed in the first display area 42a.

[0080] In addition, in a case where the overhead image 71 generated when the side mirrors 11L, 11R are in the closed state is displayed in the second display area 42b, the reflection of the vehicle body of the vehicle 10 included in the conversion processing or the like that stretches the captured data greatly can be masked with the relatively wide masking area 72. Thereby, even when the side mirrors 11L, 11R are in the closed state, the driver can confirm the surroundings of the vehicle 10 based on the good overhead image 71.

[0081] As explained above, the ECU 20 controls, with the use of the image processing section 57, in a manner that sets the prescribed range of the masking area 62, 72 within the overhead image 61, 71 based on the determination result of the opening / closing determination section 56 that determines the opening / closing state of the side mirrors 11L, 11R. Thereby, the masking area 62, 72 of the overhead image 61, 71 is set based on the opening / closing state of the side mirrors 11L, 11R, and thus even when the photographing ranges of the left side camera 12L and the right side camera 12R change due to the opening / closing of the side mirrors 11L, 11R, it is possible to display a wide range of good overhead images 61, 71 without the reflection of the vehicle body. Therefore, for example, during entry into a narrow parking space or during exit from a narrow parking space, it is possible to accurately confirm whether or not the vehicle 10 collides with an obstacle in the surroundings or the like. In addition, during entry into a narrow parking space, it is easy to confirm whether or not there is a space where the occupant of the vehicle 10 can easily get off the vehicle 10 after the vehicle 10 is parked or the like. In addition, during parking of the vehicle 10, it is easy to confirm whether or not there is an obstacle or the like that the occupant of the vehicle 10 contacts when getting off the vehicle 10.

[0082] In addition, the control ECU 20 sets, with the use of the image processing section 57, the masking area 72 wider than in a case where the side mirrors 11L, 11R are in the open state in a case where the side mirrors 11L, 11R are in the closed state. Thereby, in the state where the side mirrors 11L, 11R are closed, by setting the wider masking area 72, it is possible to mask the reflection of the vehicle 10 in the overhead image 71, and in the state where the side mirrors 11L, 11R are open, by setting the narrower masking area 62, it is possible to secure a wider visible range in the overhead image 61.

[0083] The above describes an embodiment of the present application, but the present application is not limited to the above-described embodiment, and can be appropriately modified, improved, or the like.

[0084] For example, in the above-described embodiment, the case where the control ECU 20 displays the bird's-eye images 61, 71 on the touch panel 42 of the vehicle 10 is described, but is not limited thereto. For example, the control ECU 20 can also display the bird's-eye images 61, 71 on a display screen of an information terminal (for example, a smartphone or the like) held by an occupant of the vehicle 10 via the communication section 24.

[0085] In addition, in the above-described embodiment, the case where the shield regions 62, 72 are generated in a rectangular shape is described, but is not limited thereto, and for example, the shield regions 62, 72 can also be generated in an elliptical shape or a circular shape. The size of a region (a front region) of the shield regions 62, 72 that is located in the advancing direction of the vehicle 10 (the moving body) can also be changed from the size indicated by the arrow in FIG. 6. Figure 6 Figure 7 The size or shape of the shield regions 62, 72 can be set on a per-vehicle 10 (per-moving body) basis, or can be set by user input. In addition, the size or shape of the shield regions 62, 72 can also be set on the basis of a time period, the surrounding environment of the vehicle 10 (the moving body) (for example, brightness of illuminance or the like, weather, presence or absence of surrounding moving bodies, state), or the like.

[0086] In addition, in the above-described embodiment, the case where the moving body is a vehicle is described, but is not limited thereto. The idea of the present application is not limited to a vehicle, and can also be applied to a robot, a ship, an airplane, or the like that has a drive source and is capable of moving by the power of the drive source.

[0087] Note that the control method described in the above-described embodiment can be realized by a computer executing a control program prepared in advance. The control program is stored in a storage medium that is readable by a computer, and is executed by being read from the storage medium. In addition, the control program can be provided in the form of being stored in a non-transitory storage medium such as a flash memory, or can be provided via a network such as the Internet. The computer that executes the control program can be included in the control device, or can be included in an electronic device such as a smartphone, a tablet terminal, or a personal computer that is capable of communicating with the control device, or can be included in a server device that is capable of communicating with the control device and the electronic device.

[0088] In addition, at least the following matters are described in the present specification. Note that although the corresponding constituent elements and the like in the foregoing embodiments are indicated in parentheses, the present application is not limited thereto.

[0089] (1) A control device including:

[0090] ​An image processing section (image processing section 57) generates a surrounding image (overhead image 61, 71) of a mobile body (vehicle 10) based on captured data acquired by a camera (left side camera 12L, right side camera 12R) provided to a side mirror (side mirror 11L, 11R) of the mobile body that is openable and closable;

[0091] A display control section (display control section 55) causes a display device (touch panel 42) to display the surrounding image generated by the image processing section; and

[0092] An open / close determination section (open / close determination section 56) determines an open / close state of the side mirror,

[0093] The image processing section sets a masking region (masking region 62, 72) of a range based on a determination result of the open / close state of the side mirror in the surrounding image.

[0094] According to (1), since the masking region of the surrounding image is set based on the open / close state of the side mirror, even if the captured range of the camera changes due to the opening / closing of the side mirror, a good surrounding image can be displayed.

[0095] (2) The control device according to (1), wherein

[0096] The image processing section sets a wider masking region in a case where the side mirror is in a closed state than in a case where the side mirror is in an open state.

[0097] According to (2), the masking region can be expanded to suppress the reflection of the mobile body in the surrounding image in a state where the side mirror is closed, and the masking region can be reduced to expand the visible range in the surrounding image in a state where the side mirror is open.

[0098] (3) The control device according to (1) or (2), wherein

[0099] The camera includes a plurality of cameras,

[0100] The surrounding image is an overhead image that represents a situation of the surroundings of the mobile body viewed from above, which is generated by synthesizing each captured data acquired by the plurality of cameras.

[0101] According to (3), the driver can intuitively grasp the situation of the surroundings of the vehicle.

[0102] (4) The control device according to any one of (1) to (3), wherein

[0103] The surrounding image includes an image (vehicle image 63, 73) of the mobile body.

[0104] According to (4), the driver can easily grasp the positional relationship between the periphery of the vehicle and the vehicle.

[0105] (5) A control method executed by a control device that has a processor that generates a periphery image of a moving body based on captured data acquired by a camera and causes a display device to display the generated periphery image, the camera being provided to a side mirror of the moving body, the control method including:

[0106] determining, by the processor, an opening / closing state of the side mirror; and

[0107] setting, by the processor, a masking region of a range based on a determination result of the opening / closing state in the periphery image.

[0108] According to (5), since the masking region of the periphery image is set based on the opening / closing state of the side mirror, even if the capturing range of the camera changes due to the opening / closing of the side mirror, a good periphery image can be displayed.

[0109] (6) A control program that causes a processor of a control device to execute a process that generates a periphery image of a moving body based on captured data acquired by a camera and causes a display device to display the generated periphery image, the camera being provided to a side mirror of the moving body, the process including:

[0110] determining an opening / closing state of the side mirror; and

[0111] setting a masking region of a range based on a determination result of the opening / closing state in the periphery image.

[0112] According to (6), since the masking region of the periphery image is set based on the opening / closing state of the side mirror, even if the capturing range of the camera changes due to the opening / closing of the side mirror, a good periphery image can be displayed.

Claims

1. A control device comprising: an image processing section that generates a surrounding image of a moving body based on captured data acquired by a capturing device provided to a side mirror of the moving body that is openable and closable; a display control section that causes a display device to display the surrounding image generated by the image processing section; and an open / close determination section that determines an open / close state of the side mirror, the image processing section setting a wider masking region in the surrounding image in a case where the side mirror is in a closed state than in a case where the side mirror is in an open state, based on a determination result of the open / close state by the open / close determination section, the masking region being set as a rectangular region in a portion corresponding to a space in which the moving body is located, the masking region in the case where the side mirror is in the closed state being larger than the masking region in the case where the side mirror is in the open state in a width direction of the moving body.

2. The control device according to claim 1, wherein the capturing device includes a plurality of capturing devices, the surrounding image is a bird's-eye view image that represents a situation of the surroundings of the moving body as viewed from above, which is generated by synthesizing each of the captured data acquired by the plurality of capturing devices.

3. The control device according to claim 1 or 2, wherein the surrounding image includes an image of the moving body.

4. A control method executed by a control device that includes a processor that generates a surrounding image of a moving body based on captured data acquired by a capturing device provided to a side mirror of the moving body and causes a display device to display the generated surrounding image, the control method comprising: determining, by the processor, an open / close state of the side mirror; and setting, by the processor, a wider masking region in the surrounding image in a case where the side mirror is in a closed state than in a case where the side mirror is in an open state, based on a determination result of the open / close state, the masking region being set as a rectangular region in a portion corresponding to a space in which the moving body is located, the masking region in the case where the side mirror is in the closed state being larger than the masking region in the case where the side mirror is in the open state in a width direction of the moving body.

5. A computer-readable recording medium storing a control program that causes a processor of a control device to execute a process that generates a surrounding image of a moving body based on captured data acquired by a capturing device provided to a side mirror of the moving body and causes a display device to display the generated surrounding image, the process comprising: determining an open / close state of the side mirror; and setting a wider masking region in the surrounding image in a case where the side mirror is in a closed state than in a case where the side mirror is in an open state, based on a determination result of the open / close state, the masking region being set as a rectangular region in a portion corresponding to a space in which the moving body is located, the masking region in the case where the side mirror is in the closed state being larger than the masking region in the case where the side mirror is in the open state in a width direction of the moving body. ​ ​ ​

Images