Control device, control method, and storage medium

Abstract

The present application provides a kind of control device, control method and storage medium capable of improving the visual recognition of rotatable three-dimensional image displayed on display device.The control device has: image processing unit (57), which is based on each shooting data obtained by front camera (12Fr) of vehicle (10), rear camera (12Rr), left side camera (12L), right side camera (12R), to generate three-dimensional image indicating the space including vehicle (10) and the periphery of vehicle (10), and the space in three-dimensional image can rotate;And display control unit (55), it makes touch panel (42) display three-dimensional image generated by image processing unit (57).In the case where it is predicted that the boundary region (66a to 66d) of each shooting data in three-dimensional image exists in specific region (73) when automatic rotation stops, image processing unit changes the rotation amount of automatic rotation.

Description

TECHNICAL FIELD

[0001] The present application relates to a control device, a control method, and a storage medium. BACKGROUND

[0002] In recent years, as a specific countermeasure against global climate change, initiatives to achieve a low-carbon society or a decarbonized society are very active. In the field of vehicles, there is a strong demand for reduction in the amount of CO2 emissions, and the introduction of automated driving and driving assistance, which contribute to the improvement of fuel efficiency, is rapidly advancing.

[0003] In addition, an image generation method is known in which each camera mounted on a vehicle around the front, rear, left, and right of the vehicle photographs a prescribed range and acquires an image, and a surrounding image (for example, an overhead image and a three-dimensional image) of the vehicle and the surroundings of the vehicle is generated based on a combined image of the photographed images. In Patent Literature 1, a vehicle surrounding monitoring device is described that changes the photographing range of the images photographed by each camera in accordance with the opening and closing of the side mirrors of the vehicle, and changes the boundary positions between the photographed images in the combined image of the photographed images to generate an overhead image. In addition, in Patent Literature 2, a vehicle surrounding monitoring device is described that, with respect to a target object that is not entirely displayed on the generated overhead image, changes the boundary line on the overhead image to display the entire target object.

[0004] PRIOR ART DOCUMENTS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: Japanese Patent No. 5112998

[0007] Patent Literature 2: Japanese Patent Application Laid-Open No. 2013-093865 SUMMARY

[0008] PROBLEMS TO BE SOLVED BY THE INVENTION

[0009] However, a three-dimensional image representing a vehicle and the surroundings of the vehicle, as an image for confirming the vehicle and the surroundings of the vehicle, is sometimes displayed as a rotatable three-dimensional image on a display device such as a display of the vehicle. In the case where the three-dimensional image is displayed as a rotatable image, for example, there is sometimes a demand for the visual recognition of the image at the time when the rotating three-dimensional image is stopped. In particular, in the case where the three-dimensional image is generated by combining a plurality of photographed images, there is sometimes a demand for the visual recognition of the image at the boundary region between the photographed images of the three-dimensional image at the time of stopping rotation. However, in Patent Literature 1 and Patent Literature 2, the visual recognition of the image in the display device at the time when the rotating image is stopped is not described. Therefore, there is room for improvement in the visual recognition of the rotatable three-dimensional image displayed on the display device.

[0010] An object of the present application is to provide a control device, a control method, and a storage medium that can improve visual recognition of a rotatable three-dimensional image displayed on a display device.

[0011] Means for solving the problem

[0012] The present application provides a control device in which,

[0013] The control device includes:

[0014] an image processing section that generates a three-dimensional image representing a space including a mobile body and a periphery of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body, and that the space in the three-dimensional image is rotatable; and

[0015] a display control section that causes a display device to display the three-dimensional image generated by the image processing section,

[0016] in a case where it is predicted that a boundary region of the each photographing data in the three-dimensional image exists in a specific region at the time when the rotation is stopped, the image processing section changes a rotation amount of the rotation.

[0017] The present application provides a control method in which,

[0018] The control method is executed by a control device that generates a three-dimensional image representing a space including a mobile body and a periphery of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body, and that the space in the three-dimensional image is rotatable, the control device causes a display device to display the three-dimensional image,

[0019] in the control method, in a case where it is predicted that a boundary region of the each photographing data in the three-dimensional image exists in a specific region at the time when the rotation is stopped, a processor of the control device changes a rotation amount of the rotation.

[0020] The present application provides a storage medium that stores a control program in which,

[0021] The control program is used to cause a processor of a control device to execute a process, the control device generates a three-dimensional image representing a space including a mobile body and a periphery of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body, and that the space in the three-dimensional image is rotatable, the control device causes a display device to display the three-dimensional image,

[0022] in the process, in a case where it is predicted that a boundary region of the each photographing data in the three-dimensional image exists in a specific region at the time when the rotation is stopped, a rotation amount of the rotation is changed.

[0023] Inventive Effects

[0024] The control device, the control method, and the storage medium according to the present application can improve the visual recognition of a rotatable three-dimensional image displayed on a display device. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a side view showing one example of a vehicle on which the control device of the present embodiment is mounted.

[0026] Figure 2 is a plan view showing the vehicle shown in Figure 1

[0027] Figure 3 is a block diagram showing the internal structure of the vehicle shown in Figure 1

[0028] Figure 4 is a view showing one example of a composite image generated from each of the captured data of a plurality of cameras.

[0029] Figure 5 is a flowchart showing one example of display control by the control ECU.

[0030] Figure 6 is a view showing one example of an outside viewpoint three-dimensional image in which a boundary region exists outside a specific region.

[0031] Figure 7 is a view showing an outside viewpoint three-dimensional image in which the rotation amount of the image has been changed so that the boundary region shown in Figure 6 does not exist in the specific region.

[0032] Figure 8 is a flowchart showing another example of display control by the control ECU.

[0033] Figure 9 is a view showing one example of an inside viewpoint three-dimensional image in which a boundary region exists inside a specific region.

[0034] Figure 10 is a view showing an inside viewpoint three-dimensional image in which the rotation amount of the image has been changed so that the boundary region shown in Figure 9 does not exist in the specific region.

[0035] Explanation of Reference Signs:

[0036] 10 vehicle (moving body)

[0037] 11L, 11R sideview mirrors

[0038] ​​12Fr front camera (imaging device)

[0039] 12Rr rear camera (imaging device)

[0040] 12L left side camera (imaging device)

[0041] 12R right side camera (imaging device)

[0042] 20 control ECU (control device)

[0043] 42 touch panel (display device)

[0044] 55 display control section

[0045] 57 image processing section

[0046] 73 specific region

[0047] 60 synthesized image

[0048] 66 boundary region

[0049] 66a left front boundary region (boundary region)

[0050] 66b right front boundary region (boundary region)

[0051] 66c left rear boundary region (boundary region)

[0052] 66d right rear boundary region (boundary region)

[0053] 70A, 70B outside viewpoint three-dimensional image (three-dimensional image)

[0054] 80A, 80B inside viewpoint three-dimensional image (three-dimensional image) DETAILED DESCRIPTION

[0055] One embodiment of the control device, the control method, and the storage medium of the present application will be described below based on the accompanying drawings. Note that the drawings are viewed in the direction of the arrow of the reference numerals. In this specification and the like, each of the directions of front, back, left, right, up, and down is described in the order as viewed from the driver of the vehicle 10 shown in the drawings, and in the drawings, the front of the vehicle 10 is denoted as Fr, the rear is denoted as Rr, the left is denoted as L, the right is denoted as R, the up is denoted as U, and the down is denoted as D. Figure 1 and Figure 2 In the drawings, the front of the vehicle 10 is denoted as Fr, the rear is denoted as Rr, the left is denoted as L, the right is denoted as R, the up is denoted as U, and the down is denoted as D.

[0056] Vehicle 10 equipped with the control device of the present application

[0057] Figure 1 is a side view of the vehicle 10 equipped with the control device of the present application.Figure 2 is Figure 1 a plan view of the vehicle 10. The vehicle 10 is one example of a mobile body of the present application.

[0058] The vehicle 10 is an automobile having a drive source (omitted from the drawing) and wheels including drive wheels driven by power of the drive source and steerable wheels. In the present embodiment, the vehicle 10 is an automobile having four wheels of a pair of front wheels on the left and right and a pair of rear wheels on the left and right. The drive source of the vehicle 10 is, for example, an electric motor. In addition, the drive source of the vehicle 10 can be an internal combustion engine such as a gasoline engine or a diesel engine, or a combination of an electric motor and an internal combustion engine. In addition, the drive source of the vehicle 10 can drive the pair of front wheels on the left and right, or the pair of rear wheels on the left and right, or all of the four wheels of the pair of front wheels on the left and right and the pair of rear wheels on the left and right. The front wheels and the rear wheels can be steerable wheels that are both steerable, or steerable wheels that are either steerable.

[0059] The vehicle 10 further has side rearview mirrors 11L, 11R. The side rearview mirrors 11L, 11R are mirrors (rearview mirrors) provided on the outside of the front seat door of the vehicle 10 for the driver to confirm the rear and the rear side. The side rearview mirrors 11L, 11R are each fixed to the main body of the vehicle 10 by a rotation shaft extending in the vertical direction, and are openable and closable by rotating about the rotation shaft. The opening and closing of the side rearview mirrors 11L, 11R is performed, for example, electrically by operation of an operation portion provided in the vicinity of the driver's seat. The width of the vehicle 10 in the closed state of the side rearview mirrors 11L, 11R is narrower than the width of the vehicle 10 in the open state of the side rearview mirrors 11L, 11R. Therefore, the side rearview mirrors 11L, 11R are mostly set to the closed state so as not to collide with obstacles around when entering a narrow parking space, and the like.

[0060] The vehicle 10 further has a front camera 12Fr, a rear camera 12Rr, a left side camera 12L, and a right side camera 12R. The front camera 12Fr is a digital camera provided in the front of the vehicle 10 and photographing the front of the vehicle 10. The rear camera 12Rr is a digital camera provided in the rear of the vehicle 10 and photographing the rear of the vehicle 10. The left side camera 12L is a digital camera provided in the left side rearview mirror 11L of the vehicle 10 and photographing the left side of the vehicle 10. The right side camera 12R is a digital camera provided in the right side rearview mirror 11R of the vehicle 10 and photographing the right side of the vehicle 10. The front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R are one example of a photographing device of the present application.

[0061] <Internal structure of the vehicle 10>

[0062] Figure 3 is a plan viewFigure 1 a block diagram of one example of an internal structure of the vehicle 10. As shown in Figure 3 As shown, the vehicle 10 has a sensor group 16, a navigation device 18, a control ECU (Electronic Control Unit) 20, an EPS (Electric Power Steering) system 22, and a communication section 24. The vehicle 10 also has a drive force control system 26 and a brake force control system 28. The control ECU 20 is one example of a control device of the present application.

[0063] The sensor group 16 acquires various detection values used in the control of the control ECU 20. The sensor group 16 includes a front camera 12Fr, a rear camera 12Rr, a left side camera 12L, and a right side camera 12R. In addition, the sensor group 16 includes a front sonar group 32a, a rear sonar group 32b, a left side sonar group 32c, and a right side sonar group 32d. In addition, the sensor group 16 includes wheel sensors 34a, 34b, a vehicle speed sensor 36, and an operation detection section 38.

[0064] The front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R output peripheral images obtained by capturing the periphery of the vehicle 10. The peripheral images captured by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R are respectively referred to as a front image, a rear image, a left side image, and a right side image. An image composed of the left side image and the right side image is also referred to as a side image.

[0065] The front sonar group 32a, the rear sonar group 32b, the left side sonar group 32c, and the right side sonar group 32d emit sound waves toward the periphery of the vehicle 10, and receive reflected sounds from other objects. The front sonar group 32a, for example, includes four sonars. The sonars constituting the front sonar group 32a are respectively provided at a left oblique front of the vehicle 10, a front left side, a front right side, and a right oblique front. The rear sonar group 32b, for example, includes four sonars. The sonars constituting the rear sonar group 32b are respectively provided at a left oblique rear of the vehicle 10, a rear left side, a rear right side, and a right oblique rear. The left side sonar group 32c, for example, includes two sonars. The sonars constituting the left side sonar group 32c are respectively provided at a left side portion front of the vehicle 10 and a left side portion rear. The right side sonar group 32d, for example, includes two sonars. The sonars constituting the right side sonar group 32d are respectively provided at a right side portion front of the vehicle 10 and a right side portion rear.

[0066] The wheel sensors 34a, 34b detect the rotation angle of the wheels of the vehicle 10. The wheel sensors 34a, 34b can be constituted by angle sensors, or can be constituted by displacement sensors. The wheel sensors 34a, 34b output a detection pulse each time the wheel rotates a prescribed angle. The detection pulse output from the wheel sensors 34a, 34b is used for the calculation of the rotation angle of the wheel and the rotation speed of the wheel. The distance of movement of the vehicle 10 is calculated on the basis of the rotation angle of the wheel. The wheel sensor 34a, for example, detects the rotation angle θa of the left rear wheel. The wheel sensor 34b, for example, detects the rotation angle θb of the right rear wheel.

[0067] The vehicle speed sensor 36 detects the speed of the vehicle body of the vehicle 10, that is, the vehicle speed V, and outputs the detected vehicle speed V to the control ECU 20. The vehicle speed sensor 36, for example, detects the vehicle speed V on the basis of the rotation of the countershaft of the transmission.

[0068] The operation detection portion 38 detects the content of the operation performed by the user using the operation input portion 14, and outputs the detected operation content to the control ECU 20. The operation input portion 14, for example, includes a side mirror switch that switches the open / close state of the side mirrors 11L, 11R, a shift lever (gear lever, selector), and various user interfaces.

[0069] The navigation device 18, for example, detects the current position of the vehicle 10 using a GPS (Global Positioning System), and guides the user of the path to the destination. The navigation device 18 has a storage device not shown that has a map information database.

[0070] The navigation device 18 is provided with a touch panel 42 and a speaker 44. The touch panel 42 functions as an input device and a display device of the control ECU 20. The user can input various instructions via the touch panel 42. In addition, various screens are displayed on the touch panel 42. Note that a constituent element other than the touch panel 42, such as a smartphone, can be used as an input device or a display device. The speaker 44 outputs various guide information to the occupant of the vehicle 10 by sound.

[0071] The control ECU 20 is provided with an input / output portion 50, a calculation portion 52, and a storage portion 54. The calculation portion 52 is constituted by a CPU (Central Processing Unit), for example. The calculation portion 52 controls each portion on the basis of a program stored in the storage portion 54, whereby various controls are performed.

[0072] The calculation portion 52 includes a display control portion 55, a prediction portion 56, and an image processing portion 57.

[0073] The image processing unit 57 generates a peripheral image of the vehicle 10 based on the shooting data acquired by the cameras of the vehicle 10. Specifically, the image processing unit 57 synthesizes the shooting data acquired by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R to generate a composite image, performs image processing to reconstruct the composite image in three dimensions, and generates a three-dimensional image that virtually represents the space encompassing the vehicle 10 and its periphery. Within the space of the three-dimensional image, an outer viewpoint three-dimensional image (e.g., representing the space of the three-dimensional image observed from outside the vehicle 10) is included. Figure 6 , 7 The image shown), and an interior viewpoint three-dimensional image representing a three-dimensional image of space viewed from the inside of vehicle 10 (e.g., an image shown from the inside of vehicle 10). Figure 9 , 10 (The images shown). The outer viewpoint 3D image is an example of the first image of the present invention, and the inner viewpoint 3D image is an example of the second image of the present invention.

[0074] In addition, the image processing unit 57 synthesizes the shooting data obtained by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R to generate a composite image, and generates an overhead view of the vehicle 10 and its surroundings, showing the view of the composite image from above.

[0075] Furthermore, the image processing unit 57 sets masking regions in the generated peripheral images (three-dimensional images and overhead images). A masking region is an area set to hide the body of the vehicle 10 in the captured image from the camera. The masking region is set to have a shape that surrounds the vehicle 10. The image processing unit 57 overlays a vehicle image representing the vehicle 10 within the masking region, corresponding to the space where the vehicle 10 is located. The vehicle image is a two-dimensional or three-dimensional image representing the view of the vehicle 10 from above; this vehicle image is pre-generated (captured) and stored in the storage unit 54, etc. Additionally, the image processing unit 57 can also set masking regions in the side images (left side image and right side image) acquired by the left side camera 12L and the right side camera 12R.

[0076] Furthermore, the image processing unit 57 is capable of spatially rotating the generated three-dimensional image. For example, the image processing unit 57 can perform spatial rotation of the three-dimensional image by manually rotating it manually or automatically rotating it automatically. In this embodiment, manual rotation refers to rotation that begins based on a user-defined operation and continues for the duration of that operation. Automatic rotation refers to rotation that begins based on a user-defined operation and continues regardless of whether the defined operation continues.

[0077] For example, right and left rotation buttons for rotating the three-dimensional image are provided on the touch panel 42, and the manual rotation includes rotation in which, in a case where the right rotation button is pressed down, the three-dimensional image is spatially rotated to the right during the period in which the right rotation button is in the pressed-down state, and rotation in which, in a case where the left rotation button is pressed down, the three-dimensional image is spatially rotated to the left during the period in which the left rotation button is in the pressed-down state. In addition, in a case where the configuration is such that the three-dimensional image is spatially rotated by sliding on the touch panel 42, the rotation of the three-dimensional image based on the sliding is included in the manual rotation. Furthermore, the inertia rotation in which the three-dimensional image is slightly rotated and then stopped due to inertia at the time of the sliding is also included in the manual rotation.

[0078] In contrast, for example, in a case where the configuration is such that a rotation button for rotating the three-dimensional image is provided on the touch panel 42 and the three-dimensional image is spatially rotated for a predetermined time (for example, 5 seconds) set in advance based on one press of the rotation button, the rotation based on the press is included in the automatic rotation. In addition, in a case where the configuration is such that a three-dimensional image for demonstration is displayed on the touch panel 42 for a predetermined time at the time of turning on the ignition switch, at the time of idling, or the like, the rotation is included in the automatic rotation.

[0079] In addition, in a case where it is predicted that the boundary region of each of the captured data in the three-dimensional image exists in a predetermined specific region at the time when the rotation of the three-dimensional image that has started by the manual rotation or the automatic rotation is stopped, the image processing section 57 changes the amount of rotation of the three-dimensional image so that the boundary region at the time of the stop does not exist in the specific region. The specific region refers to a substantially central region of the three-dimensional image displayed on the touch panel 42, and is a region that is more conspicuous to the user.

[0080] The prediction section 56 predicts whether or not the boundary region in the three-dimensional image exists in the specific region at the time when the rotation of the three-dimensional image is stopped, before the rotation of the three-dimensional image is stopped.

[0081] The display control section 55 causes the display device of the vehicle 10 to display the surrounding image generated by the image processing section 57. Specifically, the display control section 55 causes the touch panel 42 to display the three-dimensional image of the vehicle 10 and the bird's-eye image generated by synthesizing each of the captured data of the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R. In addition, the display control section 55 causes the touch panel 42 to display a rotation operation button for causing the image processing section 57 to perform the rotation processing of the three-dimensional image, such as an automatic rotation button for the automatic rotation and a manual rotation button for the manual rotation, and the like.

[0082] Also, the control ECU 20 can perform parking assistance of the vehicle 10 based on automatic steering operation in which the operation of the steering wheel 110 is automatically performed by the control of the control ECU 20. In the assistance of the automatic steering operation, the operation of an accelerator pedal (not shown), a brake pedal (not shown), and the operation input portion 14 is automatically performed. In addition, the control ECU 20 can perform assistance assistance when the vehicle 10 is parked by the user's operation of the accelerator pedal, the brake pedal, and the operation input portion 14.

[0083] The EPS system 22 has a steering angle sensor 100, a torque sensor 102, an EPS motor 104, a resolver 106, and an EPS ECU 108. The steering angle sensor 100 detects a steering angle θst of the steering wheel 110. The torque sensor 102 detects a torque TQ applied to the steering wheel 110.

[0084] The EPS motor 104 can provide the occupant with operation assistance of the steering wheel 110, automatic steering operation at the time of parking assistance, by imparting a driving force or a reaction force to a steering column 112 coupled to the steering wheel 110. The resolver 106 detects a rotation angle θm of the EPS motor 104. The EPS ECU 108 is responsible for the overall control of the EPS system 22. The EPS ECU 108 has an input / output portion (not shown), a calculation portion (not shown), and a storage portion (not shown).

[0085] The communication portion 24 can perform wireless communication with other communication devices 120. The other communication devices 120 refer to a base station, a communication device of another vehicle, an information terminal such as a smartphone held by the user of the vehicle 10, and the like.

[0086] The drive force control system 26 has a drive ECU 130. The drive force control system 26 performs drive force control of the vehicle 10. The drive ECU 130 controls an engine and the like not shown based on the operation of an accelerator pedal not shown by the user, thereby controlling the drive force of the vehicle 10.

[0087] The brake force control system 28 has a brake ECU 132. The brake force control system 28 performs brake force control of the vehicle 10. The brake ECU 132 controls a brake mechanism and the like not shown based on the operation of a brake pedal not shown by the user, thereby controlling the brake force of the vehicle 10.

[0088] <Generation of a composite image by the image processing portion 57>

[0089] Next, the generation of a composite image by the image processing portion 57 based on each of the captured data will be described with reference to Figure 4 The generation of a composite image by the image processing portion 57 based on each of the captured data will be described with reference to Figure 4is a view showing one example of a composite image generated from the respective captured data captured by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R.

[0090] As shown in Figure 4 When generating the composite image 60, the image processing portion 57 performs view point conversion on the captured data of the respective captured regions captured by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R while correcting distortion and the like of the images, to become images obtained by looking down from a prescribed view point position on the vertical above the vehicle 10. Further, from the respective converted images obtained by this conversion processing, the image processing portion 57 extracts the front image 61, the left side image 62, the right side image 63, and the rear image 64 of the prescribed field angle range set for each converted image in a manner that the images in the boundary regions of the adjacent converted images match each other. Then, the image processing portion 57 generates the composite image 60 by synthesizing these images 61 to 64. In addition, in the central portion of the composite image 60, there is provided a masking region 65 in a manner that surrounds the vehicle 10.

[0091] In the front image 61, the left side image 62, the right side image 63, and the rear image 64, there are provided boundary regions 66 (66a to 66d) indicating the boundaries of the captured images between the adjacent captured images. For example, there is provided a left front boundary region 66a between the captured images of the front image 61 and the left side image 62. There is provided a right front boundary region 66b between the captured images of the front image 61 and the right side image 63. There is provided a left rear boundary region 66c between the captured images of the left side image 62 and the rear image 64. There is provided a right rear boundary region 66d between the captured images of the right side image 63 and the rear image 64.

[0092] As described above, the front image 61, the left side image 62, the right side image 63, and the rear image 64 constituting the composite image 60 are extracted as images of the prescribed field angle range set for each converted image in a manner that the adjacent captured images match each other. However, since it is an image on which conversion processing has been performed on captured data captured by different cameras, there are many cases where a certain degree of distortion occurs in the images in the boundary regions 66a to 66d of the composite image and the visual recognition is reduced.

[0093] <Display Control by Control ECU 20>

[0094] Next, the display control of the three-dimensional image performed by the control ECU 20 will be described.

[0095] [First Display Control Example]

[0096] Referring to Figures 5 to 7 A first display control example in which the control ECU 20 performs display control of the three-dimensional image will be described. Figure 5 is a flowchart showing the first display control example in which the control ECU 20 performs display control of the three-dimensional image. Figure 6 is a diagram showing one example of the outside-viewpoint three-dimensional image in which a boundary region exists in a specific region. Figure 7 is a diagram showing one example of the outside-viewpoint three-dimensional image in which the boundary region shown in Figure 6 is changed.

[0097] When an on operation is performed on a rotation operation button (for example, an automatic rotation button 74, a manual right rotation button 75a, or a manual left rotation button 75b) for causing the space of the three-dimensional image to rotate by an occupant of the vehicle 10, the control ECU 20 starts Figure 6 the processing shown in Figure 5 In the present display control example, a case in which the automatic rotation button 74 for causing the space of the three-dimensional image to automatically rotate is subjected to the on operation will be described.

[0098] First, the control ECU 20 causes the automatic rotation of the outside-viewpoint three-dimensional image (for example, the three-dimensional image shown in Figure 6 , Figure 7 ) representing the space of the three-dimensional image observed from the outside of the vehicle 10 to start by the image processing section 57 (step S11). The automatic rotation in the present display control example is configured such that, when the automatic rotation button 74 is pressed, the space of the three-dimensional image is rotated clockwise for 5 seconds.

[0099] Next, the control ECU 20 predicts the position of the boundary region in the outside-viewpoint three-dimensional image at the time of the stop of the automatic rotation of the outside-viewpoint three-dimensional image started by the image processing section 57 in step S11 by the prediction section 56 (step S12). As described above, the automatic rotation is performed clockwise for 5 seconds, and thus the position of the boundary region after 5 seconds from the pressing of the automatic rotation button 74 is predicted.

[0100] Next, the control ECU 20 determines whether or not the position of the boundary region at the time of the stop predicted in step S12 exists in the specific region (step S13). The specific region is, for example, as shown in Figure 6 , Figure 7 , a region corresponding to the front (front face) of the vehicle 10 in the outside-viewpoint three-dimensional images 70A and 70B, and becomes the specific region 73 divided into a rectangular shape in the substantially central portion.

[0101] The three-dimensional image is actually a two-dimensional image that represents a three-dimensional space in two dimensions. The specific region 73 is, for example, a two-dimensional region in the two-dimensional image that is located in the center in the left-right direction and is located slightly below the center in the up-down direction. In this case, the control ECU 20 performs the determination of step S13 by two-dimensional coordinate calculation. Alternatively, the specific region 73 can also be a three-dimensional region that is located on the front side in the three-dimensional space represented by the three-dimensional image. In this case, the control ECU 20 performs the determination of step S13 by three-dimensional coordinate calculation.

[0102] In the case where the position of the boundary region at the time of stop exists within the specific region (step S13: YES), the control ECU 20 changes the rotation amount of the automatic rotation of the outside-viewpoint three-dimensional image by the image processing section 57 so that the boundary region at the time of stop does not exist in the specific region (step S14).

[0103] For example, it is assumed that the outside-viewpoint three-dimensional image is stopped at the time of automatic rotation stop, like the outside-viewpoint three-dimensional image 70A shown in FIG. 7A. As shown in FIG. 7A, the outside-viewpoint three-dimensional image 70A includes a three-dimensional surrounding image 71 that is a synthetic image of the surroundings of the vehicle 10 on which image processing is performed to have three-dimensional visual features, and a three-dimensional vehicle image 72 that is superimposed and displayed within a masking region set in the synthetic image of the surroundings, which represents the vehicle 10. In this case, it is predicted by the prediction section 56 that the left-front boundary region 66a between the photographed images of the front image 61 and the left-side image 62 included in the three-dimensional surrounding image 71 exists within the specific region 73. Figure 6 Figure 6 Therefore, the control ECU 20 changes the rotation amount of the automatic rotation in the clockwise direction by the image processing section 57, and further performs clockwise rotation so that the left-front boundary region 66a at the time of stop does not exist within the specific region 73, for example, to obtain the effect presented by the outside-viewpoint three-dimensional image 70B shown in FIG. 7B. At this time, other boundary regions included in the three-dimensional surrounding image 71, that is, the right-front boundary region 66b, the left-rear boundary region 66c, and the right-rear boundary region 66d also do not exist within the specific region 73.

[0104] Therefore, the control ECU 20 changes the rotation amount of the automatic rotation in the clockwise direction by the image processing section 57, and further performs clockwise rotation so that the left-front boundary region 66a at the time of stop does not exist within the specific region 73, for example, to obtain the effect presented by the outside-viewpoint three-dimensional image 70B shown in FIG. 7B. At this time, other boundary regions included in the three-dimensional surrounding image 71, that is, the right-front boundary region 66b, the left-rear boundary region 66c, and the right-rear boundary region 66d also do not exist within the specific region 73. Figure 7

[0105] Next, the control ECU 20 determines whether the automatic rotation corresponding to the changed rotation amount in step S14 has ended (step S15) by the image processing section 57.

[0106] In the case where the automatic rotation has not ended in step S15 (step S15: NO), the control ECU 20 repeatedly performs the process of step S15 by the image processing section 57 until the automatic rotation ends.

[0107] ​​If the automatic rotation ends in step S15 (step S15: Yes), the control ECU 20 displays a three-dimensional image of the outer viewpoint (e.g., where the boundary regions 66a to 66d do not exist within the specific region 73) via the display control unit 55. Figure 7 The three-dimensional image 70B from the outer viewpoint shown is displayed statically on the touch panel 42, and then this display control ends.

[0108] Additionally, if the location of the boundary region at the time of stopping in step S13 does not exist within the specific region (step S13: No), the control ECU20 proceeds to step S15 to determine whether the automatic rotation in step S11 has ended, and then ends this display control.

[0109] As explained above, in the first display control example of the control ECU20, if it is predicted that the boundary region 66a of each captured data in the three-dimensional image 70A of the outer viewpoint exists in a specific region 73 when the automatic rotation stops, the rotation amount of the automatic rotation is changed by the image processing unit 57, for example, by obtaining... Figure 7 The effect presented by the 3D image 70B from the outer viewpoint. Therefore, when automatic rotation stops, the boundary region 66a of the 3D image from the outer viewpoint is not present in a conspicuous area. Thus, the visual recognizability of the 3D image from the outer viewpoint displayed on the touch panel 42 can be improved, and the sense of disharmony for the user viewing the 3D image from the outer viewpoint can be reduced.

[0110] Furthermore, in the first display control example controlling ECU20, the specific region 73 is the region in the outer viewpoint three-dimensional image that corresponds to the foreground of the three-dimensional vehicle image 72 (vehicle 10). In the case of the outer viewpoint three-dimensional image, preferably, the boundary region does not exist in the region corresponding to the foreground of the three-dimensional vehicle image 72 (vehicle 10).

[0111] [Second Display Control Example]

[0112] Reference Figures 8 to 10 This describes a second display control example where the ECU20 controls the display of a 3D image. Figure 8 This is a flowchart illustrating a second display control example where the ECU20 controls the display of a three-dimensional image. Figure 9 This is an example of a three-dimensional image of an inner viewpoint representing a boundary region within a specific area. Figure 10 It means in order to make Figure 9 The diagram shows an example of an inner-view 3D image where the boundary region does not exist in a specific region, and the amount of rotation of the image has been changed.

[0113] Similar to the first display control example described above, when the automatic rotation button 74 for automatically rotating the three-dimensional image in space is turned on, the control ECU 20 starts...Figure 8 The processing shown.

[0114] First, the control ECU 20 uses the image processing unit 57 to generate an outer viewpoint three-dimensional image (e.g., representing a three-dimensional image of space observed from the outside of the vehicle 10) representing a three-dimensional image of space. Figure 6 , Figure 7 Three-dimensional images as shown), and three-dimensional images of the interior viewpoint representing the space observed from the interior of vehicle 10 (e.g., Figure 9 , Figure 10 The 3D image shown begins to rotate automatically (step S21). In this display control example, the 3D image is also configured to rotate clockwise for 5 seconds when the automatic rotation button 74 is pressed.

[0115] Next, the control ECU20 predicts, via the prediction unit 56, the position of the boundary region in the outer and inner viewpoint 3D images when the automatic rotation of the outer viewpoint 3D image, which started in step S21, stops, and when the automatic rotation of the inner viewpoint 3D image stops (step S22). Since the automatic rotation proceeds clockwise for 5 seconds, the position of the boundary region is predicted 5 seconds after the automatic rotation button 74 is pressed.

[0116] Next, the control ECU 20 determines whether the position of the boundary region predicted in step S22 when the 3D image of the outer viewpoint stops exists within a specific region (step S23). Similar to the first display control example, the specific region becomes... Figure 6 as well as Figure 7 Like the specific area 73.

[0117] If the location of the boundary region when the 3D image of the outer viewpoint stops exists within a specific region in step S23 (step S23: Yes), the control ECU20 changes the rotation amount of the automatic rotation of the 3D image of the outer viewpoint through the image processing unit 57 so that the boundary region at the time of stopping does not exist within the specific region (step S24).

[0118] The specific processing content is the same as that in the first display control example, for example, to Figure 6 The amount of rotation of the automatically rotated 3D image 70A shown from the outer viewpoint is changed to Figure 7 The amount of rotation is automatically rotated, as shown in the 3D image 70B from the outer viewpoint.

[0119] Next, the control ECU20 determines whether the position of the boundary region predicted in step S22 when the inner viewpoint 3D image stops exists within a specific region (step S25). For example, such as Figure 9 , Figure 10As shown, the specific region is a region in the inner side viewpoint three-dimensional image 80A, 80B that corresponds to the inner side of the vehicle 10 and becomes a roughly central portion divided into a rectangular shape, and is the specific region 83.

[0120] As described above, the three-dimensional image is actually a two-dimensional image that represents a three-dimensional space in two dimensions. The specific region 83 is, for example, a two-dimensional region in the two-dimensional image that is located in the center in the left-right direction and also in the center in the up-down direction. In this case, the control ECU 20 performs the determination of step S25 by two-dimensional coordinate calculation. Alternatively, the specific region 83 can also be a three-dimensional region located in the inner side in the three-dimensional space represented by the three-dimensional image. In this case, the control ECU 20 performs the determination of step S25 by three-dimensional coordinate calculation.

[0121] In addition, in a case where the position of the boundary region at the time of the inner side viewpoint three-dimensional image stop is not present in the specific region in step S23 (step S23: No), the control ECU 20 directly proceeds to step S25 to perform the determination of whether the position of the boundary region at the time of the inner side viewpoint three-dimensional image stop is present in the specific region.

[0122] In a case where the position of the boundary region at the time of the inner side viewpoint three-dimensional image stop is present in the specific region in step S25 (step S25: Yes), the control ECU 20 changes the rotation amount of the automatic rotation of the inner side viewpoint three-dimensional image by the image processing section 57 so that the boundary region at the time of the stop is not present in the specific region (step S26).

[0123] For example, it is assumed that the inner side viewpoint three-dimensional image stops as shown in the inner side viewpoint three-dimensional image 80A as shown in FIG. 8A. As shown in FIG. 8B, the inner side viewpoint three-dimensional image 80A includes a three-dimensional surrounding image 81 that is a synthetic image of the surroundings of the vehicle 10 on which image processing is performed to have three-dimensional visual features, and a three-dimensional vehicle image 82 that is displayed so as to overlap in a masking region set in the synthetic image of the surroundings, which represents the vehicle 10. In this case, it is predicted by the prediction section 56 that the right front boundary region 66b between the photographed images of the front image 61 and the right side image 63 included in the three-dimensional surrounding image 81 is present in the specific region 83. Figure 9 Figure 9 Therefore, the control ECU 20 changes the rotation amount of the automatic rotation by the image processing section 57 so that the right front boundary region 66b at the time of the stop is not present in the specific region 83, for example, to achieve the effect as shown in the inner side viewpoint three-dimensional image 80B as shown in FIG. 8B. At this time, other boundary regions included in the three-dimensional surrounding image 81, that is, the left front boundary region 66a, the left rear boundary region 66c, and the right rear boundary region 66d are also not present in the specific region 83.

[0124] Therefore, the control ECU 20 changes the rotation amount of the automatic rotation by the image processing section 57 so that the right front boundary region 66b at the time of the stop is not present in the specific region 83, for example, to achieve the effect as shown in the inner side viewpoint three-dimensional image 80B as shown in FIG. 8B. At this time, other boundary regions included in the three-dimensional surrounding image 81, that is, the left front boundary region 66a, the left rear boundary region 66c, and the right rear boundary region 66d are also not present in the specific region 83. Figure 10 Therefore, the control ECU 20 changes the rotation amount of the automatic rotation by the image processing section 57 so that the right front boundary region 66b at the time of the stop is not present in the specific region 83, for example, to achieve the effect as shown in the inner side viewpoint three-dimensional image 80B as shown in FIG. 8B. At this time, other boundary regions included in the three-dimensional surrounding image 81, that is, the left front boundary region 66a, the left rear boundary region 66c, and the right rear boundary region 66d are also not present in the specific region 83.​

[0125] Next, the control ECU 20 determines whether the automatic rotation of the outside-viewpoint three-dimensional image of step S21 or the automatic rotation of the outside-viewpoint three-dimensional image corresponding to the rotation amount changed in step S24, and the automatic rotation of the inside-viewpoint three-dimensional image corresponding to the rotation amount changed in step S26 is ended (step S27) by the image processing section 57.

[0126] In the case where the automatic rotation is not ended in step S27 (step S27: No), the control ECU 20 repeatedly performs the process of step S27 by the image processing section 57 until the automatic rotation is ended.

[0127] In the case where the automatic rotation is ended in step S27 (step S27: Yes), the control ECU 20 still displays the outside-viewpoint three-dimensional image (for example, the outside-viewpoint three-dimensional image 70B shown in FIG. 6) in which the boundary regions 66a to 66d are not present in the specific region 73 or the inside-viewpoint three-dimensional image (for example, the inside-viewpoint three-dimensional image 80B shown in FIG. 8) in which the boundary regions 66a to 66d are not present in the specific region 83 by the display control section 55, and then ends the present display control. Note that, as to whether the touch panel 42 displays the outside-viewpoint three-dimensional image or the inside-viewpoint three-dimensional image, the selection can be made at the time of the on operation of the automatic rotation button 74, or the switching can be made in the middle of the automatic rotation. In addition, the outside-viewpoint three-dimensional image and the inside-viewpoint three-dimensional image can be displayed in parallel on the touch panel 42. Figure 7 Figure 10 In the case where the automatic rotation is not ended in step S27 (step S27: No), the control ECU 20 repeatedly performs the process of step S27 by the image processing section 57 until the automatic rotation is ended.

[0128] In the case where the position of the boundary region at the time when the inside-viewpoint three-dimensional image is stopped in step S25 is not present in the specific region (step S25: No), the control ECU 20 proceeds to step S27, performs the determination of whether the automatic rotation of the outside-viewpoint three-dimensional image of step S21 or the automatic rotation of the outside-viewpoint three-dimensional image corresponding to the rotation amount changed in step S24, and the automatic rotation of the inside-viewpoint three-dimensional image of step S21 is ended, and then ends the present display control.

[0129] ​As explained above, in the second display control example of the control ECU 20, the outside viewpoint three-dimensional image representing the space of the three-dimensional image viewed from the outside of the vehicle 10 and the inside viewpoint three-dimensional image representing the space of the three-dimensional image viewed from the inside of the vehicle 10 are each rotated by the image processing portion 57, and in a case where it is predicted that the boundary regions 66a to 66d exist in the specific regions 73, 83 at the time of rotation stop, the image processing portion 57 changes the rotation amount of the respective rotations. Thereby, for the outside viewpoint three-dimensional image and the inside viewpoint three-dimensional image of the vehicle 10, it is possible to make the boundary regions 66a to 66d not exist in the conspicuous regions at the time of rotation stop, respectively. Therefore, it is possible to improve the visual recognizability of the outside viewpoint three-dimensional image and the inside viewpoint three-dimensional image displayed on the touch panel 42, and it is possible to reduce the sense of incongruity given to the user. For example, there are cases where the boundary region that is hidden in the inside of the vehicle 10 and is not conspicuous when the outside viewpoint three-dimensional image is displayed exists in the conspicuous region when the inside viewpoint three-dimensional image is displayed. Therefore, in such a case, it is possible to improve the visual recognizability of the inside viewpoint three-dimensional image by individually changing the rotation amount of the inside viewpoint three-dimensional image, and it is possible to reduce the sense of incongruity given to the user.

[0130] The embodiments of the present application have been described above, but the present application is not limited to the above-described embodiments, and can be appropriately modified, improved, and the like.

[0131] For example, in the above-described embodiments, a case where the display control of the automatic rotation of the space of the three-dimensional image based on the operation of the automatic rotation button 74 is explained, but the present application is not limited thereto. For example, it can be a case where the display control of the manual rotation is performed based on the operation of the manual right rotation button 75a and the manual left rotation button 75b that manually rotate the space of the three-dimensional image. Specifically, it can be possible for the prediction portion 56 to predict the positions of the boundary regions 66a to 66d at the time of stop of the space of the three-dimensional image based on the rotation amount of the manual rotation corresponding to the pressing time of the manual right rotation button 75a or the manual left rotation button 75b. Also, in a case where it is predicted that the boundary regions 66a to 66d at the time of rotation stop exist within the specific regions 73, 83, it can be possible to change the rotation amount of the manual rotation of the space of the three-dimensional image so that the boundary regions 66a to 66d at the time of stop do not exist within the specific regions 73, 83. In a case where the display control is performed in this way, it is also possible to obtain the same effects as the first and second display control examples in the above-described embodiments.

[0132] In addition, in the above-described embodiment, a case where the control ECU 20 displays the three-dimensional image on the touch panel 42 of the vehicle 10 is described, but the present application is not limited thereto. For example, the control ECU 20 can also display the three-dimensional image on a display screen of an information terminal (for example, a smartphone or the like) held by an occupant of the vehicle 10 via the communication section 24.

[0133] In addition, in the above-described embodiment, a case where the rotation button displayed on the touch panel 42 is touched in order to automatically rotate or manually rotate the three-dimensional image is described, but the present application is not limited thereto. For example, the automatic rotation or manual rotation can be performed by operation of a mechanical button, operation based on a sound instruction, or operation based on detection of the driver's line of sight.

[0134] In addition, in the above-described embodiment, a case where the photographing data is acquired by the plurality of photographing devices (the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R) is described, but for example, the photographing data can be acquired by a single 360-degree camera.

[0135] In addition, in the above-described embodiment, an example in which the mobile body is a vehicle is described, but the present application is not limited thereto. The idea of the present application is not limited to a vehicle, and can be applied to a robot, a ship, an aircraft, or the like that has a drive source and is capable of moving by power of the drive source.

[0136] Note that the control method described in the above-described embodiment can be realized by a computer executing a control program prepared in advance. The control program is stored in a storage medium that is readable by a computer, and is executed by being read from the storage medium. In addition, the control program can be provided in the form of being stored in a non-transitory storage medium such as a flash memory, or can be provided via a network such as the Internet. The computer that executes the control program can be included in the control device, or can be included in an electronic device such as a smartphone, a tablet terminal, or a personal computer that is capable of communicating with the control device, or can be included in a server device that is capable of communicating with the control device and the electronic device.

[0137] In addition, at least the following matters are described in the present specification. Note that although the corresponding constituent elements and the like in the above-described embodiment are shown in parentheses, the present application is not limited thereto.

[0138] (1) A control device, wherein

[0139] The control device has:

[0140] an image processing section (image processing section 57) that generates a three-dimensional image representing a space including the mobile body and the periphery of the mobile body based on each photographing data acquired by a plurality of photographing devices (front camera 12Fr, rear camera 12Rr, left side camera 12L, right side camera 12R) of the mobile body (vehicle 10), and that the space in the three-dimensional image is rotatable; and

[0141] a display control section (display control section 55) that causes a display device to display the three-dimensional image generated by the image processing section,

[0142] In a case where it is predicted that a boundary region (boundary regions 66a to 66d) of the each photographing data in the three-dimensional image exists in a specific region (specific region 73) at the time of rotation stop, the image processing section changes the rotation amount of the rotation.

[0143] According to (1), it is possible to make a boundary region of a three-dimensional image not exist in a conspicuous region at the time of rotation stop, and it is possible to reduce a sense of incongruity given to a user.

[0144] (2) The control device according to (1), wherein

[0145] The specific region is a region corresponding to the vicinity of the mobile body in the space.

[0146] According to (2), preferably, a boundary region of a three-dimensional image does not exist in a region corresponding to the vicinity of the mobile body.

[0147] (3) The control device according to (1) or (2), wherein

[0148] The three-dimensional image includes a first image (outside viewpoint three-dimensional image 70A, 70B) representing the space viewed from the outside of the mobile body and a second image (inside viewpoint three-dimensional image 80A, 80B) representing the space viewed from the inside of the mobile body,

[0149] With respect to the first image and the second image, the rotation is respectively possible, and in a case where it is predicted that the boundary region exists in a specific region (specific region 73, 83) at the time of rotation stop, the image processing section changes the rotation amount of the rotation.

[0150] According to (3), regarding the first image representing the space viewed from the outside of the mobile body and the second image representing the space viewed from the inside of the mobile body, it is possible to make the boundary region not exist in a conspicuous region at the time of rotation stop, and it is possible to reduce the sense of incongruity given to the user. For example, a boundary region that is hidden in the inside of the mobile body and is not conspicuous in the first image is sometimes conspicuous in the second image, and in such a case, the rotation amount of the second image is changed alone.

[0151] (4) A control method in which,

[0152] The control method is executed by a control device that generates a three-dimensional image representing a space including a mobile body and a periphery of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body, and the space in the three-dimensional image is rotatable, and the control device causes a display device to display the three-dimensional image,

[0153] In the control method, in a case where it is predicted that a boundary region of the each photographing data in the three-dimensional image exists in a specific region at the time of rotation stop, a processor of the control device changes a rotation amount of the rotation.

[0154] According to (4), it is possible to make a boundary region of a three-dimensional image not exist in a conspicuous region at the time of rotation stop, and it is possible to reduce the sense of incongruity given to the user.

[0155] (5) A storage medium that stores a control program,

[0156] The control program is for causing a processor of a control device to execute a process, the control device generating a three-dimensional image representing a space including a mobile body and a periphery of the mobile body based on each photographing data acquired by a plurality of photographing devices of the mobile body, and the space in the three-dimensional image is rotatable, and the control device causing a display device to display the three-dimensional image,

[0157] In the process, in a case where it is predicted that a boundary region of the each photographing data in the three-dimensional image exists in a specific region at the time of rotation stop, a rotation amount of the rotation is changed.

[0158] According to (5), it is possible to make a boundary region of a three-dimensional image not exist in a conspicuous region at the time of rotation stop, and it is possible to reduce the sense of incongruity given to the user.

Claims

1. A control device, wherein, The control device includes: An image processing unit generates a three-dimensional image representing a space encompassing the moving body and its periphery, based on image data acquired by multiple imaging devices of the moving body, wherein the space in the three-dimensional image is rotatable; and The display control unit causes the display device to display the three-dimensional image generated by the image processing unit. If it is predicted that the boundary regions of the captured data in the 3D image exist in a specific region when the rotation stops, the image processing unit changes the amount of rotation so that the boundary regions do not exist in the specific region when the rotation stops. The boundary region is the area between the captured images. The specific region is the area in the space that corresponds to the area in front of the moving body.

2. The control device according to claim 1, wherein, The three-dimensional image includes a first image representing the space as seen from the outside of the moving body and a second image representing the space as seen from the inside of the moving body. Regarding the first image and the second image, the rotation can be performed respectively. If it is predicted that the boundary region exists in the specific region when the rotation stops, the image processing unit changes the amount of rotation.

3. A control method, wherein, The control method is executed by a control device that generates a three-dimensional image representing the space encompassing the moving body and its surroundings, based on image data acquired by multiple imaging devices of the moving body. This three-dimensional image contains rotatable space. The control device then causes a display device to display the three-dimensional image. In the control method, if it is predicted that the boundary regions of each captured data in the 3D image exist in a specific region when the rotation stops, the processor of the control device changes the amount of rotation so that the boundary regions do not exist in the specific region when the rotation stops. The boundary region is the area between the captured images. The specific region is the area in the space that corresponds to the area in front of the moving body.

4. A storage medium storing a control program, wherein, The control program is used to cause the processor of the control device to perform processing. The control device generates a three-dimensional image representing a space encompassing the moving body and its surroundings, based on image data acquired by multiple imaging devices of the moving body. The space in the three-dimensional image is rotatable. The control device causes a display device to display the three-dimensional image. In the process, if it is predicted that the boundary regions of the captured data in the 3D image exist in a specific region when the rotation stops, the amount of rotation is changed so that the boundary regions do not exist in the specific region when the rotation stops. The boundary region is the area between the captured images. The specific region is the area in the space that corresponds to the area in front of the moving body.

Images