Control Device, Control Method, and Storage Medium

Abstract

A control device, a control method, and a storage medium that can improve the visual recognition of a three-dimensional image when the three-dimensional image is displayed on a display device, and the three-dimensional image can be switched between automatic rotation display and manual rotation display and then displayed. The control device includes: an image processing unit that generates a three-dimensional image representing a space including a vehicle and its surroundings based on shooting data obtained by a front camera, a rear camera, a left-side camera, and a right-side camera of the vehicle, and can perform manual rotation for manually rotating the space in the three-dimensional image and automatic rotation for automatically rotating the space in the three-dimensional image; and a display control unit that causes a touch panel to display the three-dimensional image generated by the image processing unit. When switching from manual rotation to automatic rotation by an operation of a user of the vehicle, the image processing unit starts automatic rotation from a position based on the stop position of the space of the three-dimensional image after manual rotation.

Description

Technical Field

[0001] The present invention relates to a control device, a control method, and a storage medium. Background Art

[0002] In recent years, as a specific measure to address global climate change, initiatives to achieve a low-carbon society or a decarbonized society have been very active. In the case of vehicles, there is also a strong demand to reduce CO2 emissions, and the introduction of autonomous driving and driving support for vehicles that contribute to improved fuel efficiency is rapidly advancing. Conventionally, an image generation system is known that causes cameras mounted on the front, rear, left, and right of a vehicle to photograph a specified range and acquire images, and generates a surrounding image (such as an overhead image or a three-dimensional image) of the vehicle and its surroundings based on a combined image of the images acquired by the photographing. In Patent Document 1, a control system that can automatically rotate or manually rotate the generated three-dimensional image is described.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2013-236374 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] For example, when a rotatable three-dimensional image is displayed on a display screen or the like, the ease of viewing of the image based on the display method is required. In particular, when switching between automatic rotation and manual rotation to display a three-dimensional image, the ease of viewing of the displayed image at the time of the switch is required. However, in Patent Document 1, the visual recognition of the image displayed on the display screen when switching between automatic rotation and manual rotation is not described. Therefore, there is room for improvement in the visual recognition of the image when switching between automatic rotation display and manual rotation display to display a three-dimensional image.

[0008] An object of the present invention is to provide a control device, a control method, and a storage medium that can improve the visual recognition of a three-dimensional image when the three-dimensional image is displayed on a display device, and the three-dimensional image can be switched between automatic rotation display and manual rotation display.

[0009] Means for Solving the Problems

[0010] The present invention provides a control device, wherein,

[0011] the control device includes:

[0012] An image processing unit that generates a three-dimensional image representing a space including the moving body and the periphery of the moving body based on captured data obtained by a capturing device of the moving body, and is capable of performing manual rotation for manually rotating the space in the three-dimensional image and automatic rotation for automatically rotating the space in the three-dimensional image; and

[0013] A display control unit that causes a display device to display the three-dimensional image generated by the image processing unit,

[0014] When switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the image processing unit starts the automatic rotation from a position based on the stop position of the space after the manual rotation.

[0015] Advantageous Effects of the Invention

[0016] According to the control device, control method, and storage medium of the present invention, it is possible to improve the visual recognition of the three-dimensional image when the three-dimensional image is displayed on the display device, and the three-dimensional image can be displayed while switching between automatic rotation display and manual rotation display. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 FIG. is a side view showing an example of a vehicle equipped with the control device of the present embodiment.

[0018] Figure 2 FIG. is a top view of the vehicle shown in Figure 1 FIG.

[0019] Figure 3 FIG. is a block diagram showing the internal structure of the vehicle shown in Figure 1 FIG.

[0020] Figure 4 FIG. is a diagram showing an example of a three-dimensional image generated from the captured data of multiple cameras.

[0021] Figure 5 FIG. is a diagram showing the three-dimensional image when the three-dimensional image shown in Figure 4 FIG. is rotated by a predetermined angle.

[0022] Figure 6 FIG. is a flowchart showing an example of the display control of the control ECU.

[0023] Figure 7 FIG. schematically shows Figure 6 FIG. is a diagram showing the display viewing point of the three-dimensional image at the time of rotation switching in the display control of

[0024] Figure 8 FIG. is a flowchart showing another example of the display control of the control ECU.

[0025] Figure 9 It schematically shows Figure 8 a diagram of the display viewpoint of a three-dimensional image during rotation switching in display control.

[0026] Explanation of reference numerals:

[0027] 10 Vehicle (moving body)

[0028] 12Fr Front camera (imaging device)

[0029] 12Rr Rear camera (imaging device)

[0030] 12L Left-side camera (imaging device)

[0031] 12R Right-side camera (imaging device)

[0032] 20 Control ECU (control device)

[0033] 42 Touch panel (display device)

[0034] 60, 60a, 60b Three-dimensional images. Detailed implementation mode

[0035] Hereinafter, an embodiment of the control device, control method, and storage medium of the present invention will be described based on the drawings. It should be noted that the drawings are viewed according to the orientation of the reference numerals. In addition, in this specification, etc., in order to make the description simple and clear, the front, rear, left, and right, and up and down directions are described according to the Figure 1 and Figure 2 direction observed by the driver of the vehicle 10 shown. And in the drawings, the front of the vehicle 10 is represented as Fr, the rear is represented as Rr, the left side is represented as L, the right side is represented as R, the upper side is represented as U, and the lower side is represented as D.

[0036] <Vehicle 10 equipped with the control device of the present invention>

[0037] Figure 1 It is a side view showing the vehicle 10 equipped with the control device of the present invention. Figure 2 It is Figure 1 a top view of the vehicle 10 shown. The vehicle 10 is an example of the moving body of the present invention.

[0038] The vehicle 10 is an automobile having a drive source (not shown) and wheels, and the wheels include drive wheels driven by the power of the drive source and steerable steerable wheels. In the present embodiment, the vehicle 10 is a four-wheeled automobile having a pair of left and right front wheels and a pair of left and right rear wheels. The drive source of the vehicle 10 is, for example, an electric motor. In addition, the drive source of the vehicle 10 may be an internal combustion engine such as a gasoline engine or a diesel engine, or a combination of an electric motor and an internal combustion engine. In addition, the drive source of the vehicle 10 may drive a pair of left and right front wheels, may drive a pair of left and right rear wheels, or may drive the four wheels of a pair of left and right front wheels and a pair of left and right rear wheels. The front wheels and the rear wheels may be steerable wheels that can both be steered, or may be steerable wheels that can be steered on either side.

[0039] The vehicle 10 further includes side mirrors 11L and 11R. The side mirrors 11L and 11R are mirrors (rearview mirrors) provided on the outside of the front seat doors of the vehicle 10 for the driver to confirm the rear and the rear sides. The side mirrors 11L and 11R are respectively fixed to the main body of the vehicle 10 by a rotation axis extending in the vertical direction, and can be opened and closed by rotating about the rotation axis. The opening and closing of the side mirrors 11L and 11R are electrically performed, for example, by operating an operation unit provided near the driver's seat. The width of the vehicle 10 in the closed state of the side mirrors 11L and 11R is narrower than the width of the vehicle 10 in the open state of the side mirrors 11L and 11R. Therefore, when entering a narrow parking space or the like, the side mirrors 11L and 11R are mostly set in the closed state to avoid collision with surrounding obstacles.

[0040] The vehicle 10 further includes a front camera 12Fr, a rear camera 12Rr, a left side camera 12L, and a right side camera 12R. The front camera 12Fr is a digital camera provided in front of the vehicle 10 and photographing the front of the vehicle 10. The rear camera 12Rr is a digital camera provided behind the vehicle 10 and photographing the rear of the vehicle 10. The left side camera 12L is a digital camera provided in the left side mirror 11L of the vehicle 10 and photographing the left side of the vehicle 10. The right side camera 12R is a digital camera provided in the right side mirror 11R of the vehicle 10 and photographing the right side of the vehicle 10. The front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R are an example of the photographing device of the present invention.

[0041] <Internal structure of the vehicle 10>

[0042] Figure 3 is a block diagram showing Figure 1 an example of the internal structure of the vehicle 10 shown. As Figure 3As shown, the vehicle 10 includes a sensor group 16, a navigation device 18, a control ECU (Electronic Control Unit) 20, an EPS (Electric Power Steering) system 22, and a communication unit 24. The vehicle 10 also includes a driving force control system 26 and a braking force control system 28. The control ECU 20 is an example of the control device of the present invention.

[0043] The sensor group 16 obtains various detection values used in the control of the control ECU 20. The sensor group 16 includes a front camera 12Fr, a rear camera 12Rr, a left-side camera 12L, and a right-side camera 12R. In addition, the sensor group 16 includes a front sonar group 32a, a rear sonar group 32b, a left-side sonar group 32c, and a right-side sonar group 32d. In addition, the sensor group 16 includes wheel sensors 34a, 34b, a vehicle speed sensor 36, and an operation detection unit 38.

[0044] The front camera 12Fr, the rear camera 12Rr, the left-side camera 12L, and the right-side camera 12R output surrounding images obtained by photographing the periphery of the vehicle 10. The surrounding images photographed by the front camera 12Fr, the rear camera 12Rr, the left-side camera 12L, and the right-side camera 12R are respectively referred to as a front image, a rear image, a left-side image, and a right-side image. The image composed of the left-side image and the right-side image is also referred to as a side image.

[0045] The front sonar group 32a, the rear sonar group 32b, the left-side sonar group 32c, and the right-side sonar group 32d emit sound waves to the periphery of the vehicle 10 and receive reflected sounds from other objects. The front sonar group 32a includes, for example, four sonars. The sonars constituting the front sonar group 32a are respectively provided at the left front diagonal, the front left, the front right, and the right front diagonal of the vehicle 10. The rear sonar group 32b includes, for example, four sonars. The sonars constituting the rear sonar group 32b are respectively provided at the left rear diagonal, the rear left, the rear right, and the right rear diagonal of the vehicle 10. The left-side sonar group 32c includes, for example, two sonars. The sonars constituting the left-side sonar group 32c are respectively provided in front of and behind the left side portion of the vehicle 10. The right-side sonar group 32d includes, for example, two sonars. The sonars constituting the right-side sonar group 32d are respectively provided in front of and behind the right side portion of the vehicle 10.

[0046] Wheel sensors 34a and 34b detect the rotational angles of the wheels of vehicle 10. Wheel sensors 34a and 34b may be constituted by angle sensors or displacement sensors. Wheel sensors 34a and 34b output detection pulses each time the wheel rotates by a predetermined angle. The detection pulses output from wheel sensors 34a and 34b are used for calculating the rotational angle and rotational speed of the wheel. The moving distance of vehicle 10 is calculated based on the rotational angle of the wheel. For example, wheel sensor 34a detects the rotational angle θa of the left rear wheel. For example, wheel sensor 34b detects the rotational angle θb of the right rear wheel.

[0047] Vehicle speed sensor 36 detects the speed of the vehicle body of vehicle 10, that is, vehicle speed V, and outputs the detected vehicle speed V to control ECU 20. Vehicle speed sensor 36 detects vehicle speed V based on the rotation of the countershaft of the transmission, for example.

[0048] Operation detection unit 38 detects the content of the operation performed by the user using operation input unit 14, and outputs the detected operation content to control ECU 20. Operation input unit 14 includes various user interfaces such as side mirror switches for switching the open / closed states of side mirrors 11L and 11R, a shift lever (gearshift, selector), etc., for example.

[0049] Navigation device 18 detects the current position of vehicle 10 using GPS (Global Positioning System), for example, and guides the path to the destination to the user. Navigation device 18 has a storage device (not shown) having a map information database.

[0050] Navigation device 18 includes a touch panel 42 and a speaker 44. Touch panel 42 functions as an input device and a display device for control ECU 20. The user inputs various instructions via touch panel 42. In addition, various screens are displayed on touch panel 42. It should be noted that components other than touch panel 42, such as a smartphone, etc., may be used as the input device or display device. Speaker 44 outputs various guidance information to the passengers of vehicle 10 by sound.

[0051] Control ECU 20 includes an input / output unit 50, an arithmetic unit 52, and a storage unit 54. Arithmetic unit 52 is constituted by a CPU (Central Processing Unit), for example. Arithmetic unit 52 controls each unit based on the program stored in storage unit 54, thereby performing various controls.

[0052] The operation unit 52 includes a display control unit 55, a stop position storage unit 56, and an image processing unit 57. The image processing unit 57 generates a surrounding image of the vehicle 10 based on the captured data obtained by the camera of the vehicle 10. Specifically, the image processing unit 57 synthesizes the respective captured data obtained by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R to generate a synthesized image, and performs image processing for three-dimensionally reconstructing the synthesized image to generate a three-dimensional image that virtually represents the space including the vehicle 10 and the surroundings of the vehicle 10.

[0053] In addition, the image processing unit 57 synthesizes the respective captured data obtained by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R to generate a synthesized image, and generates an aerial view image of the vehicle 10 and the surroundings of the vehicle 10 showing the situation of observing the synthesized image from above.

[0054] In addition, the image processing unit 57 sets a shielding area in the generated surrounding images (three-dimensional image and aerial view image). The shielding area is an area set to hide the body of the vehicle 10 in the captured image captured by the camera. The shielding area is set as an area having a shape surrounding the vehicle 10. The image processing unit 57 overlaps and displays a vehicle image representing the vehicle 10 in a part corresponding to the space where the vehicle 10 is located within the shielding area. The vehicle image is a two-dimensional or three-dimensional image showing the situation of observing the vehicle 10 from above, and the vehicle image is pre-generated (captured) and stored in the storage unit 54 or the like. In addition, the image processing unit 57 may also set a shielding area in the side images (left side image and right side image) obtained by the left side camera 12L and the right side camera 12R.

[0055] In addition, the image processing unit 57 can rotate the space of the generated three-dimensional image. Rotating the space of the three-dimensional image means generating a three-dimensional image that is continuous in time, so as to present the effect of the rotation of the space represented by the three-dimensional image (recognized by the user). For example, the image processing unit 57 enables manual rotation for manually rotating the space of the three-dimensional image and automatic rotation for automatically rotating the space of the three-dimensional image. In the present embodiment, the manual rotation means starting based on a specified operation of the user (such as an operation of rotating a certain amount) and rotating during the period when the specified operation continues. The automatic rotation means starting based on a specified operation of the user (such as an operation for starting the automatic rotation) and continuing regardless of whether the specified operation continues.

[0056] For example, a right rotation button and a left rotation button are provided on the touch panel 42. Manual rotations include: when the right rotation button is pressed, a spatial right rotation of the three-dimensional image during the period when the right rotation button is in the pressed state; when the left rotation button is pressed, a spatial left rotation of the three-dimensional image during the period when the left rotation button is in the pressed state. Additionally, in a configuration where the spatial rotation of the three-dimensional image can be achieved by swiping on the touch panel 42, the spatial rotation of the three-dimensional image based on the swipe is included in the manual rotation. Moreover, the inertial rotation in which the spatial rotation of the three-dimensional image stops slightly after rotating due to inertia during the swipe is also included in the manual rotation.

[0057] Additionally, for example, in a configuration where the spatial rotation of the three-dimensional image rotates 360 degrees based on a single press of the rotation button, the rotation based on the press is included in the automatic rotation. Additionally, in a configuration where a three-dimensional image for demonstration rotates and is displayed on the touch panel 42 when the ignition switch is turned on or during idling, etc., the rotation is included in the automatic rotation.

[0058] Additionally, when the rotation of the three-dimensional image is switched from manual rotation to automatic rotation by the operation of a user riding in the vehicle 10, the image processing unit 57 starts automatic rotation from the position based on the stop position of the space in the three-dimensional image after manual rotation. For example, when the rotation of the three-dimensional image is switched from manual rotation to automatic rotation by the operation of the user, the image processing unit 57 may also start automatic rotation from the stop position of the space in the three-dimensional image after manual rotation. Additionally, when the rotation of the three-dimensional image is switched from manual rotation to automatic rotation by the operation of the user, the image processing unit 57 may also start automatic rotation from the position reached by rotating back a specified amount in the direction opposite to the manual rotation from the stop position of the space in the three-dimensional image after manual rotation.

[0059] When the rotation of the three-dimensional image is switched from manual rotation to automatic rotation by the operation of the user, the stop position storage unit 56 stores the stop position of the space in the three-dimensional image after manual rotation in the storage unit 54. The stop position of the space in the three-dimensional image after manual rotation is, for example, the rotation position of the space in the three-dimensional image at the time point when switching from manual rotation to automatic rotation.

[0060] The display control unit 55 causes the display device of the vehicle 10 to display the surrounding image generated by the image processing unit 57. Specifically, the display control unit 55 causes the touch panel 42 to display a three-dimensional image and an overhead image of the vehicle 10 generated by synthesizing the shooting data of the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R. In addition, the display control unit 55 causes the touch panel 42 to display operation buttons for causing the image processing unit 57 to perform rotation processing of the three-dimensional image, such as an auto-rotation button for automatic rotation and a manual rotation button for manual rotation, etc.

[0061] Moreover, the control ECU 20 can also perform parking support of the vehicle 10 based on automatic steering operation. In the automatic steering operation, the operation of the steering wheel 110 is automatically performed by the control of the control ECU 20. In the support of the automatic steering operation, the operations of the accelerator pedal (not shown), the brake pedal (not shown), and the operation input unit 14 are automatically performed. In addition, the control ECU 20 can also perform auxiliary support when the user operates the accelerator pedal, the brake pedal, and the operation input unit 14 to stop the vehicle 10.

[0062] The EPS system 22 includes a steering angle sensor 100, a torque sensor 102, an EPS motor 104, a resolver 106, and an EPS ECU 108. The steering angle sensor 100 detects the steering angle θst of the steering wheel 110. The torque sensor 102 detects the torque TQ applied to the steering wheel 110.

[0063] The EPS motor 104 can provide operation support for the occupant of the steering wheel 110 and automatic steering operation during parking support by applying a driving force or a reaction force to the steering column 112 connected to the steering wheel 110. The resolver 106 detects the rotation angle θm of the EPS motor 104. The EPS ECU 108 is responsible for the overall control of the EPS system 22. The EPS ECU 108 includes an input / output unit (not shown), an arithmetic unit (not shown), and a storage unit (not shown).

[0064] The communication unit 24 can perform wireless communication with other communication devices 120. The other communication devices 120 refer to information terminals such as base stations, communication devices of other vehicles, and smart phones held by users of the vehicle 10.

[0065] The driving force control system 26 includes a drive ECU 130. The driving force control system 26 performs driving force control of the vehicle 10. The drive ECU 130 controls an engine (not shown), etc. based on the operation of the accelerator pedal (not shown) by the user, thereby controlling the driving force of the vehicle 10.

[0066] The braking force control system 28 includes a brake ECU 132. The braking force control system 28 performs braking force control of the vehicle 10. The brake ECU 132 controls a braking mechanism (not shown) and the like based on the operation of a brake pedal (not shown) by the user, thereby controlling the braking force of the vehicle 10.

[0067] <Rotation processing of three-dimensional images performed by the image processing unit 57>

[0068] Next, with reference to Figure 4 and Figure 5 the rotation processing of the three-dimensional images displayed on the touch panel 42 will be described. Figure 4 is a diagram showing an example of a three-dimensional image of the vehicle 10 and the surroundings thereof generated based on a composite image of the respective captured data obtained by the front camera 12Fr, the rear camera 12Rr, the left side camera 12L, and the right side camera 12R. Figure 5 is a diagram showing Figure 4 the three-dimensional image of the vehicle 10 and the surroundings thereof obtained by rotating the shown three-dimensional image by a predetermined angle.

[0069] As shown in Figure 4 and Figure 5 the three-dimensional images 60 (three-dimensional images 60a, 60b) displayed on the touch panel 42 include a three-dimensional surrounding image 61 obtained by performing image processing on the composite image of the surroundings of the vehicle 10 to have three-dimensional visual characteristics, and a three-dimensional vehicle image 62 representing the vehicle 10 overlappingly displayed within a shielding area set in the surrounding composite image.

[0070] Figure 4 The three-dimensional image 60a shown in Figure 5 is an image obtained by rotating the vehicle 10 (three-dimensional vehicle image 62) so as to be viewable from the upper left front obliquely. Figure 4 The three-dimensional image 60b shown in

[0071] is an image obtained by, for example, rotating the three-dimensional image 60a shown in

[0072] to the right so as to be viewable from the upper left rear obliquely. In addition, on the touch panel 42, there are displayed an automatic rotation button 63 which is an operation button for automatically rotating the three-dimensional images 60a, 60b, a right rotation button 64a which is an operation button for manual rotation, and a left rotation button 64b. When the automatic rotation button 63 is pressed, the three-dimensional image displayed on the touch panel 42 rotates, for example, 360 degrees to the right or left. When the automatic rotation button 63 is pressed again during the automatic rotation, the automatic rotation stops. Then, when the automatic rotation button 63 is pressed again, the automatic rotation starts again. The rotation speed of the three-dimensional image during the automatic rotation is preset, but it can also be set by the user.

[0073] On the other hand, when the right rotation button 64a is pressed, the three-dimensional image displayed on the touch panel 42 rotates to the right corresponding to the period during which the pressing operation is performed. Further, when the left rotation button 64b is pressed, the three-dimensional image displayed on the touch panel 42 rotates to the left corresponding to the period during which the pressing operation is performed.

[0074] <Display control executed by the control ECU 20>

[0075] Next, the display control of the three-dimensional image executed by the control ECU 20 will be described.

[0076] [First display control example]

[0077] Refer to Figure 6 and Figure 7 to describe the first display control example in which the control ECU 20 performs display control on the three-dimensional image. Figure 6 is a flowchart showing the first display control example in which the control ECU 20 performs display control on the three-dimensional image. Figure 7 schematically shows Figure 6 the display viewpoint of the three-dimensional image at the time of rotation switching in the display control of. For example, when the occupant of the vehicle 10 turns on a three-dimensional image display button (not shown) for causing the touch panel 42 to display a three-dimensional image, the control ECU 20 starts Figure 6 the process shown.

[0078] First, the control ECU 20 causes the touch panel 42 to display a three-dimensional image (for example, Figure 4 the three-dimensional image 60a) representing a space including the vehicle 10 and the periphery of the vehicle 10 through the display control unit 55. Then, the control ECU 20 causes the space of the displayed three-dimensional image to automatically rotate from a preset initial position through the image processing unit 57 (step S11).

[0079] The initial position at which the automatic rotation starts is set, for example, as shown in the state 701 of Figure 7 the position of the viewing point 71 for observing the vehicle 10 obliquely from above the front. When the automatic rotation of the space of the three-dimensional image starts, the position of the viewing point for displaying the three-dimensional image changes clockwise (to the right) from the position of the viewing point 71 as shown by the arrow A with respect to the vehicle 10. The control ECU 20 generates three-dimensional images of the vehicle 10 observed from the changed viewing points through the image processing unit 57, and causes the touch panel 42 to display the generated three-dimensional images through the display control unit 55.

[0080] Next, the control ECU 20 determines whether an operation for manually rotating the space of the three-dimensional image is received (step S12). Specifically, the control ECU 20 determines, for example, Figure 4Whether the right rotation button 64a or the left rotation button 64b for manual rotation displayed on the touch panel 42 is operated.

[0081] In step S12, when an operation for manually rotating the space of the three-dimensional image is received (step S12: YES), the control ECU20 switches the rotation of the space of the three-dimensional image from automatic rotation to manual rotation, and starts rotation processing corresponding to the operation of the manual rotation through the image processing unit 57 (step S13). That is, the control ECU20 causes the space of the three-dimensional image to rotate to the right through the image processing unit 57 when the right rotation button 64a is operated, and causes the space of the three-dimensional image to rotate to the left when the left rotation button 64b is operated.

[0082] For example, it is assumed that: as Figure 7 shown in the state 701, an automatic rotation is performed from the position of the viewing point 71 to the position of the viewing point 72, and the left rotation button 64b is pressed when reaching the position of the viewing point 72. The control ECU20 stops the automatic rotation of the three-dimensional image at the position of the viewing point 72, and as shown in the state 702, starts a manual rotation of the three-dimensional image in the left direction from the position of the viewing point 72 as indicated by the arrow B according to the operation of the left rotation button 64b through the image processing unit 57.

[0083] In step S12, when an operation for manually rotating the space of the three-dimensional image is not received (step S12: NO), the control ECU20 determines whether an operation for automatically rotating the space of the three-dimensional image is received (step S14). Specifically, the control ECU20 determines whether, for example Figure 4 the automatic rotation button 63 for performing automatic rotation displayed on the touch panel 42 is operated.

[0084] In step S14, when an operation for automatically rotating the space of the three-dimensional image is not received (step S14: NO), the control ECU20 returns to step S12. When an operation for automatically rotating the space of the three-dimensional image is received (step S14: YES), the control ECU20 determines whether it is in the manual rotation started through step S13 (step S15).

[0085] In step S15, when it is in the manual rotation (step S15: YES), the control ECU20 stores the current rotation position of the space of the three-dimensional image as the stop position of the manual rotation in the storage unit 54 through the stop position storage unit 56 (step S16).

[0086] Next, the control ECU 20 switches the rotation of the space of the three-dimensional image from manual rotation to automatic rotation, reads the "stop position of manual rotation" stored in step S16 from the storage unit 54, and starts automatic rotation from this stop position by the image processing unit 57 (step S17), and then returns to step S12.

[0087] For example, it is assumed that: through the rotation process corresponding to the operation of manual rotation in step S13, as Figure 7 shown in state 702, the position of the viewpoint for displaying the three-dimensional image reaches the position of viewpoint 73 due to the change in the position relative to the vehicle 10, and the manual rotation stops at the position of viewpoint 73. The control ECU 20 stores the position of viewpoint 73 as the stop position of manual rotation in the storage unit 54. And, when the control ECU 20 receives an operation to perform automatic rotation during manual rotation, it reads the position of viewpoint 73 as the stop position of the most recent manual rotation stored in the storage unit 54, and as shown in state 703, starts automatic rotation of the three-dimensional image in the clockwise direction from the position of viewpoint 73 as indicated by arrow C by the image processing unit 57.

[0088] In step S15, when not in manual rotation (step S15: No), the control ECU 20 switches the rotation of the space of the three-dimensional image from manual rotation to automatic rotation, for example, and starts automatic rotation from a preset initial position (step S18), and then returns to step S12. Additionally, in step S18, it may also be that when it is in the case after manual rotation (i.e., when an operation to perform automatic rotation is received after manual rotation stops), the control ECU 20 starts automatic rotation from the stop position of this manual rotation by the image processing unit 57.

[0089] [Second display control example]

[0090] Refer to Figure 8 and Figure 9 to describe the second display control example in which the control ECU 20 performs display control on the three-dimensional image. Figure 8 is a flowchart showing the second display control example in which the control ECU 20 performs display control on the three-dimensional image. Figure 9 is schematically showing Figure 8 the display viewpoint of the three-dimensional image at the time of rotation switching in the display control of. Similar to the above-described first display control example, for example, when an on operation is performed on a three-dimensional image display button (not shown), the control ECU 20 starts Figure 8 the processing shown.

[0091] First, similar to step S11 of the first display control example, the control ECU 20 displays the space of the three-dimensional image (for example, refer to Figure 4a three-dimensional image 60a) therein, and an automatic rotation of the space of the three-dimensional image displayed starting from the initial position (step S21).

[0092] The initial position at which the automatic rotation starts is set, for example, as Figure 9 shown in the state 801, and in the same manner as in the first display control example, it is set to the position of the viewpoint 81 that observes the vehicle 10 obliquely from above the front. When the automatic rotation of the space of the three-dimensional image starts, the position of the viewpoint for displaying the three-dimensional image with respect to the vehicle 10 changes clockwise (to the right) from the position of the viewpoint 81 as shown by the arrow D. The control ECU 20 generates three-dimensional images of the vehicle 10 observed from the changed viewpoints respectively, and causes the touch panel 42 to display the generated three-dimensional images.

[0093] Next, in the same manner as in step S12 of the first display control example, the control ECU 20 determines whether an operation for manually rotating the space of the three-dimensional image, such as an operation of the right rotation button 64a or the left rotation button 64b, is received (step S22).

[0094] In step S22, when an operation for manually rotating the space of the three-dimensional image is received (step S22: YES), in the same manner as in step S13 of the first display control example, the control ECU 20 switches the rotation of the space of the three-dimensional image from automatic rotation to manual rotation, and starts the rotation process corresponding to the manual rotation operation (step S23).

[0095] For example, it is assumed that: as Figure 9 shown in the state 801, an automatic rotation is performed from the position of the viewpoint 81 to the position of the viewpoint 82, and when the position of the viewpoint 82 is reached, the left rotation button 64b is pressed. The control ECU 20 stops the automatic rotation of the three-dimensional image at the position of the viewpoint 82, and as shown in the state 802, according to the operation of the left rotation button 64b, starts a manual rotation of the three-dimensional image in the left direction as shown by the arrow E through the image processing unit 57.

[0096] In step S22, when an operation for manually rotating the space of the three-dimensional image is not received (step S22: NO), in the same manner as in step S14 of the first display control example, the control ECU 20 determines whether an operation for automatically rotating the space of the three-dimensional image, such as an operation of the automatic rotation button 63, is received (step S24).

[0097] In step S24, when an operation for automatically rotating the space of the three-dimensional image is not received (step S24: NO), the control ECU 20 returns to step S22. When an operation for automatically rotating the space of the three-dimensional image is received (step S24: YES), the control ECU 20 determines whether it is in the manual rotation started through step S23 (step S25).

[0098] In step S25, when in the manual rotation state (step S25: Yes), the control ECU20 stores the stop position of the manual rotation as the stop position of the manual rotation in the storage unit 54 through the stop position storage unit 56 (step S26).

[0099] Next, the control ECU20 switches the rotation of the three-dimensional image space from manual rotation to automatic rotation, reads the "stop position of manual rotation" stored in step S26 from the storage unit 54, and starts automatic rotation from a position, for example, 90 degrees in front of this stop position (the position reached by returning 90 degrees in the direction of the above initial position) (step S27), and then returns to step S22. It should be noted that the angle of the front position relative to the stop position is not limited to 90 degrees, and can be arbitrarily set by the user of the vehicle 10.

[0100] For example, it is assumed that: through the rotation process corresponding to the operation of manual rotation in step S23, as Figure 9 shown in state 802, the position of the viewpoint for displaying the three-dimensional image reaches the position of viewpoint 83 due to the change in the position relative to the vehicle 10, and the manual rotation stops at the position of viewpoint 83. The control ECU20 stores the position of viewpoint 83 as the stop position of the manual rotation in the storage unit 54 through the stop position storage unit 56. And when the control ECU20 receives an operation to perform automatic rotation during the manual rotation, it reads the position of viewpoint 83 as the stop position of the nearest manual rotation stored in the storage unit 54. As shown in state 803, the position of the viewpoint returns from the position of viewpoint 83 to the position of viewpoint 84 which is 90 degrees in front, and the control starts the automatic rotation of the three-dimensional image in the clockwise direction from the position of viewpoint 84 as indicated by the arrow F through the image processing unit 57.

[0101] In step S25, when not in the manual rotation state (step S25: No), the control ECU20, for example, switches the rotation of the three-dimensional image space from manual rotation to automatic rotation, and starts automatic rotation from a preset initial position (step S28), and then returns to step S22. Additionally, in step S28, it can also be that when it is in the state after manual rotation (i.e., when an operation to perform automatic rotation is received after the manual rotation stops), the control ECU20 starts automatic rotation from a position, for example, 90 degrees in front of the stop position of this manual rotation through the image processing unit 57.

[0102] As described above, when the control ECU 20 switches from the manual rotation in which the user of the vehicle 10 manually rotates the space of the three-dimensional image to the automatic rotation in which the space of the three-dimensional image is automatically rotated, the automatic rotation starts from the position based on the stop position of the manual rotation. Thereby, when switching from manual rotation to automatic rotation by the operation of the user, the automatic rotation of the three-dimensional image can start from a position close to the position of the three-dimensional image seen by the user. Therefore, the visual recognition when switching from manual rotation to automatic rotation can be improved. Therefore, for example, when starting the vehicle 10 from the place where it was previously parked, it is possible to accurately and quickly confirm whether there are obstacles or the like in the vicinity. In addition, during the process of driving into a narrow parking space or driving out of a narrow parking space, it is possible to accurately and quickly confirm whether the vehicle 10 will collide with surrounding obstacles or the like. In addition, during the process of entering a narrow parking space, it is easy to confirm whether there is a space where the passengers of the vehicle 10 can get off after the vehicle 10 stops. In addition, during the process of the vehicle 10 stopping, it is easy to confirm whether there are obstacles that the passengers of the vehicle 10 will contact when getting off.

[0103] In addition, the control ECU 20 may also start the automatic rotation from the stop position of the manual rotation when switching from the manual rotation to the automatic rotation by the operation of the user of the vehicle 10 through the image processing unit 57. Thereby, when switching from manual rotation to automatic rotation by the operation of the user, the automatic rotation can start from the position of the three-dimensional image seen by the user. Therefore, the visual recognition when switching from manual rotation to automatic rotation can be improved.

[0104] In addition, the control ECU 20 may also start the automatic rotation from a position at a predetermined angle (for example, 90 degrees) in front of the stop position of the manual rotation when switching from the manual rotation to the automatic rotation by the operation of the user of the vehicle 10 through the image processing unit 57. Thereby, when switching from manual rotation to automatic rotation by the operation of the user, the automatic rotation can start from a position slightly retracted to the front from the position of the three-dimensional image seen by the user. Therefore, the visual recognition when switching from manual rotation to automatic rotation can be improved.

[0105] In addition, in the control ECU 20, the user of the vehicle 10 may also set the angle of returning to the front from the stop position after the manual rotation. Thereby, a position that is easy for the user to observe can be set, and the visual recognition can be further improved.

[0106] As described above, the embodiments of the present invention have been described, but the present invention is not limited to the above embodiments, and can be appropriately modified, improved, etc.

[0107] For example, in the above-described embodiment, it has been described that when the rotation of the three-dimensional image space is switched from manual rotation to automatic rotation, the automatic rotation starts from the position reached by returning a predetermined angle set by the user from the stop position of the three-dimensional image space after manual rotation. However, the present invention is not limited thereto. The predetermined angle returned from the stop position may be, for example, an angle corresponding to the rotation speed of the automatic rotation of the three-dimensional image space. Specifically, the return angle may be decreased when the rotation speed of the automatic rotation is slow, and increased when the rotation speed is fast. Thereby, when the speed of the automatic rotation is slow, when switching from manual rotation to automatic rotation, it is possible to start the automatic rotation from a position close to the position of the three-dimensional image that the user saw during manual rotation. Therefore, the visual recognition when switching from manual rotation to automatic rotation is improved.

[0108] In addition, in the above-described embodiment, the case where the rotation of the three-dimensional image is switched from manual rotation to automatic rotation by the operation of the user has been described. However, the present invention is not limited thereto. For example, the rotation start position of the automatic rotation may also be set when the rotation of the three-dimensional image is switched from manual rotation to automatic rotation according to a predetermined condition that does not depend on the operation of the user. Specifically, for example, during the stable driving of the vehicle 10, since there is no need to observe surrounding information, the operation of the user on the display of the three-dimensional image on the touch panel 42 is often reduced. In this case, when the state of no operation continues for a predetermined time, the rotation of the three-dimensional image is switched to automatic rotation, and a demonstration three-dimensional image of the vehicle 10, for example, is displayed. Therefore, when switching from manual rotation to automatic rotation under such conditions, for example, as the display of the demonstration three-dimensional image, the automatic rotation may start from a predetermined angle set in advance. Furthermore, it may be that when the ignition switch of the vehicle 10 is turned on, idling, etc., the demonstration three-dimensional image is also displayed on the touch panel 42, and in this case, the automatic rotation of the demonstration three-dimensional image may also start from a predetermined angle.

[0109] In addition, in the above-described embodiment, the case where the control ECU 20 displays the three-dimensional image on the touch panel 42 of the vehicle 10 has been described. However, the present invention is not limited thereto. For example, the control ECU 20 may also display the three-dimensional image on the display screen of an information terminal (such as a smartphone, etc.) held by a passenger of the vehicle 10 via the communication unit 24.

[0110] In addition, in the above-described embodiment, the case where a touch operation is performed on buttons (automatic rotation button 63, right rotation button 64a, left rotation button 64b) displayed on the touch panel 42 to automatically or manually rotate the three-dimensional image has been described. However, the present invention is not limited thereto. For example, automatic rotation or manual rotation may also be performed by operating a mechanical button, an operation based on a voice instruction, or an operation that detects the driver's line of sight and is based on the line of sight.

[0111] In addition, in the above-described embodiment, the case where shooting data is acquired using a plurality of shooting devices (front camera 12Fr, rear camera 12Rr, left side camera 12L, right side camera 12R) has been described. However, for example, shooting data may also be acquired using a single 360-degree camera.

[0112] In addition, in the above-described embodiment, an example in which the moving body is a vehicle has been described. However, the present invention is not limited thereto. The idea of the present invention is not limited to vehicles and can also be applied to robots, ships, aircraft, etc. that have a drive source and can move by the power of the drive source.

[0113] It should be noted that the control method described in the above-described embodiment can be implemented by a computer executing a pre-prepared control program. This control program is stored in a computer-readable storage medium and is executed by being read out from the storage medium. In addition, this control program can be provided in a form stored in a non-transitory storage medium such as a flash memory, or can be provided via a network such as the Internet. The computer that executes this control program may be included in the control device, may be included in an electronic device such as a smartphone, a tablet terminal, or a personal computer that can communicate with the control device, or may be included in a server device that can communicate with these control devices and electronic devices.

[0114] In addition, at least the following matters are described in this specification. It should be noted that although the corresponding components, etc. in the above-described embodiment are shown in parentheses, the present invention is not limited thereto.

[0115] (1) A control device, wherein,

[0116] The control device includes:

[0117] An image processing unit (image processing unit 57) that generates a three-dimensional image representing a space including the moving body and the periphery of the moving body based on shooting data acquired by a shooting device (front camera 12Fr, rear camera 12Rr, left side camera 12L, right side camera 12R) of the moving body (vehicle 10), and is capable of performing manual rotation for manually rotating the space in the three-dimensional image and automatic rotation for automatically rotating the space in the three-dimensional image; and

[0118] A display control unit (display control unit 55) that causes the display device to display the three-dimensional image generated by the image processing unit.

[0119] When switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the image processing unit starts the automatic rotation from a position based on the stop position of the space after the manual rotation.

[0120] According to (1), when switching from the manual rotation to the automatic rotation by an operation of a user, it is possible to start the automatic rotation from a position close to the position observed by the user, and thus the visual recognition when switching from the manual rotation to the automatic rotation can be improved.

[0121] (2) The control device according to (1), wherein

[0122] When switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the image processing unit starts the automatic rotation from the stop position of the space after the manual rotation.

[0123] According to (2), when switching from the manual rotation to the automatic rotation by an operation of a user, it is possible to start the automatic rotation from the position seen by the user, and thus the visual recognition when switching from the manual rotation to the automatic rotation can be improved.

[0124] (3) The control device according to (1), wherein

[0125] When switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the image processing unit starts the automatic rotation from a position reached by rotating a predetermined amount in a direction opposite to the manual rotation from the stop position of the space after the manual rotation.

[0126] According to (3), when switching from the manual rotation to the automatic rotation by an operation of a user, it is possible to start the automatic rotation from a position reached by slightly returning from the position observed by the user, and thus the visual recognition when switching from the manual rotation to the automatic rotation can be improved.

[0127] (4) The control device according to (3), wherein

[0128] The predetermined amount is an amount set by a user of the moving body.

[0129] According to (4), it is possible to set a position that is easy for the user to observe, and the visual recognition can be improved.

[0130] (5) The control device according to (3) or (4), wherein

[0131] The specified amount is an amount corresponding to the speed of the automatic rotation.

[0132] According to (5), when switching from manual rotation to automatic rotation by the operation of the user, the automatic rotation can start from a position easily observable by the user according to the speed of the automatic rotation, so that the visual recognition when switching from manual rotation to automatic rotation can be improved.

[0133] (6) The control device according to any one of (1) to (5), wherein

[0134] When switching from the manual rotation to the automatic rotation according to a specified condition independent of the operation of the user of the moving body, the automatic rotation starts from a preset initial position.

[0135] According to (6), even when switching from manual rotation to automatic rotation independent of the operation of the user, the visual recognition can be improved.

[0136] (7) The control device according to any one of (1) to (6), wherein

[0137] The photographing device includes a plurality of photographing devices,

[0138] The three-dimensional image is an image generated by synthesizing each piece of photographing data obtained by the plurality of photographing devices.

[0139] According to (7), the driver can intuitively grasp the situation around the vehicle.

[0140] (8) A control method, wherein

[0141] The control method is executed by a processor, and the processor generates a three-dimensional image representing a space including the moving body and the periphery of the moving body based on photographing data obtained by a photographing device of the moving body, and can perform a manual rotation of manually rotating the space in the three-dimensional image and an automatic rotation of automatically rotating the space in the three-dimensional image. The processor causes a display device to display the generated three-dimensional image,

[0142] In the control method, when switching from the manual rotation to the automatic rotation by the operation of the user of the moving body, the automatic rotation starts from a position based on the stop position of the space after the manual rotation.

[0143] According to (8), when switching from manual rotation to automatic rotation by the operation of the user, the automatic rotation can start from a position close to the position observed by the user, so that the visual recognition when switching from manual rotation to automatic rotation can be improved.

[0144] (9) A storage medium stores a control program, wherein,

[0145] the control program is used to cause a processor to perform processing, and the processor generates a three-dimensional image of a space including the moving body and the periphery of the moving body based on captured data obtained by a capturing device of the moving body, and is capable of performing a manual rotation for manually rotating the space in the three-dimensional image and an automatic rotation for automatically rotating the space in the three-dimensional image. The processor causes a display device to display the generated three-dimensional image.

[0146] In the processing, when switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the automatic rotation starts from a position based on the stop position of the space after the manual rotation.

[0147] According to (9), when switching from the manual rotation to the automatic rotation by an operation of a user, it is possible to start the automatic rotation from a position close to the position observed by the user, so that the visual recognition when switching from the manual rotation to the automatic rotation can be improved.

Claims

1. A control device, wherein, the control device includes: an image processing unit that generates a three-dimensional image representing a space including the moving body and the periphery of the moving body based on captured data obtained by a capturing device of the moving body, and is capable of performing a manual rotation for manually rotating the space in the three-dimensional image and an automatic rotation for automatically rotating the space in the three-dimensional image; and a display control unit that causes a display device to display the three-dimensional image generated by the image processing unit, when switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the image processing unit starts the automatic rotation from a position based on the stop position of the space after the manual rotation, when switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the image processing unit starts the automatic rotation from a position reached by rotating a predetermined amount in a direction opposite to the manual rotation with respect to the stop position of the space after the manual rotation.

2. The control device according to claim 1, wherein, the predetermined amount is an amount set by a user of the moving body.

3. The control device according to claim 1 or 2, wherein, the predetermined amount is an amount corresponding to the speed of the automatic rotation.

4. The control device according to claim 1 or 2, wherein, when switching from the manual rotation to the automatic rotation according to a predetermined condition that does not depend on an operation of a user of the moving body, the automatic rotation starts from a preset initial position.

5. The control device according to claim 1 or 2, wherein, the capturing device includes a plurality of capturing devices, the three-dimensional image is an image generated by synthesizing the respective captured data obtained by the plurality of capturing devices.

6. A control method, wherein, the control method is executed by a processor that generates a three-dimensional image representing a space including the moving body and the periphery of the moving body based on captured data obtained by a capturing device of the moving body, and is capable of performing a manual rotation for manually rotating the space in the three-dimensional image and an automatic rotation for automatically rotating the space in the three-dimensional image, and the processor causes a display device to display the generated three-dimensional image, in the control method, when switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the automatic rotation starts from a position based on the stop position of the space after the manual rotation, when switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the automatic rotation starts from a position reached by rotating a predetermined amount in a direction opposite to the manual rotation with respect to the stop position of the space after the manual rotation.

7. A storage medium that stores a control program, wherein, The control program is used to cause a processor to execute processing. The processor generates a three-dimensional image representing a space including the moving body and the periphery of the moving body based on captured data acquired by a capturing device of the moving body, and is capable of performing a manual rotation for manually rotating the space in the three-dimensional image and an automatic rotation for automatically rotating the space in the three-dimensional image. The processor causes a display device to display the generated three-dimensional image. In the processing, when switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the automatic rotation starts from a position based on the stop position of the space after the manual rotation. when switching from the manual rotation to the automatic rotation by an operation of a user of the moving body, the automatic rotation starts from a position reached by rotating a predetermined amount in a direction opposite to the manual rotation with respect to the stop position of the space after the manual rotation.

Images