Control device, method of operation of control device, and storage medium

Abstract

This invention relates to a control device, a method of operating the control device, and a storage medium. The control device controls the imaging of fisheye cameras positioned at the front, rear, and left and right sides of a vehicle. The control device includes: a detection unit that detects the vehicle's attitude; and a control unit that, based on the vehicle's attitude, controls the position of the transformation center for transforming the fisheye images from each fisheye camera into planar images. The control device accurately acquires peripheral information about a desired area regardless of changes in the vehicle's attitude.

Description

Technical Field

[0001] This invention relates to a control device, a method of operating the control device, and a storage medium. Background Technology

[0002] Objects are detected from images captured around a vehicle for driver assistance and other control purposes. To expand the detection range, a fisheye camera with a large field of view is considered. However, because fisheye cameras produce distorted images, detection accuracy may decrease when using object detection techniques that rely on undistorted images from conventional cameras.

[0003] Patent document 1 discloses the following technology: performing distortion correction processing on a distorted image and using the corrected image for object detection.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2008-48443 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] However, the technology described in Patent Document 1 completely fails to consider changes in the vehicle's posture when equipped with a fisheye camera (e.g., tilting to the right or left while driving, or tilting forward or backward due to acceleration and deceleration). Therefore, there is a problem that images of the desired area may not be acquired.

[0009] The present invention was made in view of the above-mentioned problems, and provides a technique for acquiring peripheral information about a desired area with good accuracy, independent of changes in the attitude of the vehicle.

[0010] Solution for solving the problem

[0011] The control device of the present invention, which achieves the above-mentioned objective, controls the capturing images of fisheye cameras arranged at the front, rear, and left and right sides of a vehicle. The control device includes:

[0012] A detection unit that detects the attitude of the vehicle; and

[0013] The control unit controls the position of the transformation center for transforming fisheye images from each fisheye camera into planar images, based on the vehicle's attitude.

[0014] Furthermore, the operating method of the control device involved in the present invention, which achieves the above-mentioned objective, controls the shooting of fisheye cameras arranged at the front, rear, and left and right sides of a vehicle. The operating method of the control device includes:

[0015] The inspection process includes detecting the vehicle's attitude; and

[0016] The control process controls the position of the transformation center used to transform the fisheye images from each fisheye camera into planar images, based on the vehicle's posture.

[0017] The effects of the invention

[0018] According to the present invention, it is possible to acquire peripheral information about a desired area with high accuracy, regardless of changes in the vehicle's attitude. Therefore, various processes such as driver assistance and autonomous driving that utilize the acquired peripheral information can be performed with high precision. Attached Figure Description

[0019] Figure 1 This is a block diagram of a vehicle according to one embodiment of the present invention.

[0020] Figure 2 (a) shows the horizontal shooting range of each fisheye camera. Figure 2 (b) shows the vertical shooting range of the fisheye camera on the right side. Figure 2 (c) shows the vertical shooting range of the rear fisheye camera.

[0021] Figure 3 This is an illustration of distortion correction processing for fisheye images.

[0022] Figure 4 Figures (a) to (c) are illustrations showing an example of the attitude change of a vehicle in the longitudinal direction according to an embodiment of the present invention.

[0023] Figure 5 Figures (a) to (c) are illustrations showing an example of the left-right attitude change of a vehicle according to an embodiment of the present invention.

[0024] Figure 6 This is a flowchart illustrating the steps of a process implemented by a control device according to an embodiment of the present invention.

[0025] Figure 7 This is a diagram illustrating an example of rotation correction according to an embodiment of the present invention.

[0026] Explanation of reference numerals in the attached figures

[0027] 1: Vehicle; 2: Control device; 22, 23: ECU; 41-44: Fisheye camera; 5: Gyroscope sensor; 7c: Vehicle speed sensor. Detailed Implementation

[0028] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the following embodiments are not intended to limit the scope of the invention as claimed, and the present invention does not require a combination of all the features described in the embodiments. Alternatively, two or more features from the plurality of features described in the embodiments may be arbitrarily combined. Additionally, the same or identical structures will be given the same reference numerals, and repeated descriptions will be omitted.

[0029] (Implementation Method)

[0030] <Structure>

[0031] Figure 1 This is a block diagram of a vehicle 1 according to one embodiment of the present invention. Figure 1 The diagram shows a general outline of vehicle 1 using top and side views. Vehicle 1 is an example of a four-wheeled passenger car. Vehicle 1 can be such a four-wheeled vehicle, or it can be a two-wheeled vehicle or other types of vehicles.

[0032] Vehicle 1 includes a vehicle control device 2 (hereinafter referred to as control device 2) that controls vehicle 1. Control device 2 includes multiple ECUs (Electronic Control Units) 20-29 that are communicatively connected via an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit), a memory such as a semiconductor memory, and an interface to external devices. The memory stores the program executed by the processor, the data used by the processor in processing, etc. Each ECU may also have multiple processors, memories, and interfaces. For example, ECU 20 has one or more processors 20a and one or more memories 20b. The processor 20a executes commands, including the program stored in the memory 20b, thereby ECU 20 performs processing. Alternatively, ECU 20 may have a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for executing the processing of ECU 20. The same applies to other ECUs.

[0033] The functions of each ECU 20 to 29 are explained below. Furthermore, the number of ECUs and their functions can be appropriately designed, allowing for a more detailed or integrated approach than this embodiment.

[0034] ECU 20 performs controls related to the autonomous driving of vehicle 1. In autonomous driving, it automatically controls at least one of the following: wheel turning, acceleration, and deceleration of vehicle 1. Alternatively, the autonomous driving performed by ECU 20 may include: autonomous driving that does not require driver intervention (also known as autonomous driving) and autonomous driving that assists the driver in performing driving operations (also known as driving assistance).

[0035] ECU 21 controls the electric power steering system 3. The electric power steering system 3 includes a mechanism that turns the front wheels according to the driver's driving operation (wheel operation) on the steering wheel 31. Additionally, the electric power steering system 3 includes: a motor that generates driving force to assist in wheel operation or to automatically turn the front wheels; and sensors that detect the wheel angle, etc. When the vehicle 1 is in automatic driving mode, ECU 21 automatically controls the electric power steering system 3 according to instructions from ECU 20 to control the direction of travel of the vehicle 1.

[0036] ECU 22 and ECU 23 control the detection unit that detects the surrounding environment of the vehicle and process the detection results. Vehicle 1 includes a standard camera 40 and four fisheye cameras 41-44 as detection units for detecting the surrounding environment of the vehicle. Standard camera 40, fisheye cameras 42 and 44 are connected to ECU 22. Fisheye cameras 41 and 43 are connected to ECU 23. ECU 22 and ECU 23 analyze the images captured by the standard camera 40 and the fisheye cameras 41-44, thereby extracting the outlines of body markers, lane markings (white lines, etc.) on the road.

[0037] Fisheye cameras 41-44 are cameras equipped with fisheye lenses. The structure of fisheye camera 41 will be described below. Other fisheye cameras 42-44 may also have the same structure. The field of view of fisheye camera 41 is larger than that of standard camera 40. Therefore, compared to standard camera 40, fisheye camera 41 can capture a larger area. Compared to images captured by standard camera 40, images captured by fisheye camera 41 have greater distortion. Therefore, it is possible that ECU 23 performs distortion reduction transformation processing (hereinafter referred to as "distortion correction processing") on the image captured by fisheye camera 41 before analyzing it. Alternatively, it is possible that ECU 22 does not perform distortion correction processing on the image captured by standard camera 40 before analyzing it. Thus, standard camera 40 is an imaging device that captures images that are not subject to distortion correction processing, while fisheye camera 41 is an imaging device that captures images that are subject to distortion correction processing. Alternatively, instead of the standard camera 40, other shooting devices that capture images that are not subject to distortion correction processing can be used, such as a camera equipped with a wide-angle lens or a telephoto lens.

[0038] A standard camera 40 is mounted at the center of the front of vehicle 1 to capture the surrounding environment in front of vehicle 1. A fisheye camera 41 is mounted at the center of the front of vehicle 1 to capture the surrounding environment in front of vehicle 1. Figure 1 The diagram shows a standard camera 40 and a fisheye camera 41 arranged horizontally. However, the configuration of the standard camera 40 and the fisheye camera 41 is not limited to this; for example, these cameras can also be arranged vertically. Alternatively, at least one of the standard camera 40 and the fisheye camera 41 can be mounted on the front of the roof of the vehicle 1 (e.g., inside the front window). The fisheye camera 42 is mounted on the center of the right side of the vehicle 1 to capture the surrounding environment on the right side of the vehicle 1. The fisheye camera 43 is mounted on the center of the rear of the vehicle 1 to capture the surrounding environment at the rear of the vehicle 1. The fisheye camera 44 is mounted on the center of the left side of the vehicle 1 to capture the surrounding environment on the left side of the vehicle 1.

[0039] The types, number, and installation locations of cameras in vehicle 1 are not limited to the examples described above. Alternatively, vehicle 1 may include lidar (Light Detection and Ranging) or millimeter-wave radar as detection units used to detect objects around vehicle 1 or to measure distances to objects.

[0040] ECU 22 controls the standard camera 40, fisheye camera 42, and fisheye camera 44, and processes the detection results. ECU 23 controls the fisheye camera 41 and fisheye camera 43, and processes the detection results. By dividing the detection unit that detects the vehicle's surroundings into two systems, the reliability of the detection results can be improved.

[0041] ECU 24 controls gyroscope sensor 5, GPS sensor 24b, and communication device 24c, and processes the detection or communication results. Gyroscope sensor 5 detects the rotational motion of vehicle 1. Based on the detection results of gyroscope sensor 5, wheel speed, etc., the direction of travel of vehicle 1 can be determined. GPS sensor 24b detects the current position of vehicle 1. Communication device 24c wirelessly communicates with a server providing map information and traffic information to obtain this information. ECU 24 can access a map information database 24a built in memory, and ECU 24 performs path exploration from its current location to its destination. ECU 24, map database 24a, and GPS sensor 24b constitute a navigation device.

[0042] The ECU 25 is equipped with a communication device 25a for vehicle-to-vehicle communication. The communication device 25a wirelessly communicates with other vehicles in the vicinity and exchanges information between the vehicles.

[0043] ECU 26 controls powertrain 6. Powertrain 6 is a mechanism that outputs driving force to rotate the drive wheels of vehicle 1, and includes, for example, an engine and a transmission. ECU 26 controls the engine output in response to driver operations (accelerator operation or acceleration operation) detected by operation detection sensor 7a located on accelerator pedal 7A, and switches transmission gears based on information such as vehicle speed detected by vehicle speed sensor 7c. When vehicle 1 is in autonomous driving mode, ECU 26 controls powertrain 6 in response to instructions from ECU 20 to control the acceleration and deceleration of vehicle 1.

[0044] ECU 27 controls lighting devices (headlights, taillights, etc.), including the turn indicator 8. Figure 1 In the case of this example, the direction indicator 8 is located at the front of the vehicle 1, the door rearview mirror, and the rear.

[0045] ECU 28 controls input / output device 9. Input / output device 9 outputs information to the driver and accepts information input from the driver. Voice output device 91 notifies the driver of information by sound. Display device 92 notifies the driver of information by displaying images. Display device 92 is, for example, disposed in front of the driver's seat, forming an instrument panel, etc. Moreover, although sound and display are exemplified here, information can also be notified by vibration or light. In addition, multiple combinations of sound, display, vibration, or light can be used to notify information. Furthermore, the combination or notification method can be different depending on the level of information to be notified (e.g., urgency). Input device 93 is disposed in a position that the driver can operate, and may be a group of switches for instructing the vehicle 1, or may include a voice input device.

[0046] ECU 29 controls the braking device 10 and the parking brake (not shown). The braking device 10 is, for example, a disc brake, installed on each wheel of the vehicle 1, applying resistance to the rotation of the wheels, thereby decelerating or stopping the vehicle 1. ECU 29 controls the operation of the braking device 10 in accordance with the driver's driving operation (brake operation) detected by the operation detection sensor 7b installed on the brake pedal 7B. When the vehicle 1 is in automatic driving mode, ECU 29 automatically controls the braking device 10 in accordance with the instructions from ECU 20 to control the deceleration and stopping of the vehicle 1. The braking device 10 and the parking brake can also be activated to maintain the vehicle 1 in a stopped state. In addition, if the transmission of the powertrain 6 is equipped with a parking lock mechanism, it can also be activated to maintain the vehicle 1 in a stopped state.

[0047] <Shooting Area>

[0048] Then refer to Figure 2 This describes the shooting range of the standard camera 40 and the fisheye cameras 41-44. Figure 2 (a) shows the horizontal shooting range of each camera. Figure 2 (b) shows the vertical shooting range of the fisheye camera 42 mounted on the right side of vehicle 1. Figure 2 (c) shows the vertical shooting range of the fisheye camera 43 mounted at the rear of vehicle 1.

[0049] First, refer to Figure 2 (a) illustrates the shooting range of the top view of vehicle 1 (i.e., the horizontal direction of vehicle 1). Standard camera 40 captures the scenery included in the shooting range 200. The shooting center 200C of standard camera 40 faces directly forward of vehicle 1. The horizontal field of view of standard camera 40 can be less than 90°, for example, it can be 45° or 30°.

[0050] Fisheye camera 41 captures the scenery within the shooting range 201. The shooting center 201C of fisheye camera 41 faces directly in front of vehicle 1. Fisheye camera 42 captures the scenery within the shooting range 202. The shooting center 202C of fisheye camera 42 faces the right side of vehicle 1. Fisheye camera 43 captures the scenery within the shooting range 203. The shooting center 203C of fisheye camera 43 faces directly behind vehicle 1. Fisheye camera 44 captures the scenery within the shooting range 204. The shooting center 204C of fisheye camera 44 faces the left side of vehicle 1. The horizontal field of view of fisheye cameras 41-44 can be greater than 0°, greater than 150°, or greater than 180°, for example, 180°. Figure 2 (a) shows an example of a fisheye camera 41-44 with a horizontal field of view of 180°.

[0051] The shooting range 201 can be divided into an area 201L located diagonally forward to the left of vehicle 1, an area 201F located directly in front of vehicle 1, and an area 201R located diagonally forward to the right of vehicle 1. The shooting range 202 can be divided into an area 202L located diagonally forward to the right of vehicle 1, an area 202F located to the right of vehicle 1, and an area 202R located diagonally behind to the right of vehicle 1. The shooting range 203 can be divided into an area 203L located to the right of vehicle 1, an area 203F located directly behind vehicle 1, and an area 203R located diagonally behind to the left of vehicle 1. The shooting range 204 can be divided into an area 204L located diagonally behind to the left of vehicle 1, an area 204F located to the left of vehicle 1, and an area 204R located diagonally forward to the left of vehicle 1. The shooting range 201 can also be equally divided into three areas 201L, 201F, and 201R (i.e., in a manner that makes the field of view of each area equal). The other shooting ranges 202 to 204 can also be equally divided into three.

[0052] The standard camera 40 and the fisheye cameras 41-44 have shooting ranges 200-204 as described above, thus the shooting ranges of the two cameras respectively cover the front of the vehicle 1 and four diagonal directions. Specifically, the shooting range 200 of the standard camera 40 and the area 201F of the shooting range 201 of the fisheye camera 41 cover the front of the vehicle 1 on both sides. The area 201R of the shooting range 201 of the fisheye camera 41 and the area 202L of the shooting range 202 of the fisheye camera 42 cover the right diagonal front of the vehicle 1 on both sides. The same applies to the other three diagonal directions of the vehicle 1.

[0053] Then, refer to Figure 2 (b) and Figure 2(c) indicates the vertical shooting range of vehicle 1. Figure 2 In (b), the vertical shooting range of the fisheye camera 42 is described. Figure 2 In section (c), the vertical shooting range of fisheye camera 43 is described. The vertical shooting range of other fisheye cameras 41 and 44 can also be the same.

[0054] The vertical field of view of the fisheye cameras 41 to 44 can be greater than 90°, greater than 150°, or greater than 180°, for example, it can be 180°. Figure 2 (b) and Figure 2 (c) shows an example where the vertical field of view of fisheye cameras 41-44 is 180°. In the illustrated example, the shooting center 203C of fisheye camera 43 is oriented downwards (towards the ground) relative to a direction parallel to the ground. Alternatively, the shooting center 203C of fisheye camera 43 may be oriented parallel to the ground, or it may be oriented upwards (towards the opposite side of the ground) relative to a direction parallel to the ground. Furthermore, the shooting centers 201C-204C of fisheye cameras 41-44 may also be oriented in different directions in the vertical direction.

[0055] Reference Figure 3 This describes the distortion correction processing applied to the images captured by fisheye cameras 41-44. Image 300 is an image of the scenery to the right of vehicle 1 captured by fisheye camera 42. As shown, image 300 exhibits significant distortion, particularly in the peripheral areas.

[0056] The ECU 22, connected to the fisheye camera 42, performs distortion correction processing (transformation from a fisheye image to a planar image) on the image 300. Specifically, the ECU 22 sets a point within the image 300 as a correction center point 301. The ECU 22 cuts out a rectangular region 302 centered on the correction center point 301 from the image 300. The ECU 22 performs distortion correction processing on this region 302, thereby generating a distortion-reduced image 303. The distortion correction process reduces distortion as the image approaches the correction center point 301, while distortion does not decrease or even increases at locations far from the correction center point 301. Therefore, in one embodiment, the ECU 22 sets the correction center point 301 in a region of interest within the environment surrounding the vehicle 1 and generates a distortion-reduced image for that region.

[0057] Moreover, in Figure 3In the example, the correction center point 301 and the rectangular area 302 are set to the right of the shooting center 351 of the fisheye image (the line 352 passing through the vertical direction of the shooting center 351). However, the correction center point 301 and the rectangular area 302 can also be set to the left. Alternatively, the shooting center 351 and the correction center point 301 of the fisheye image can be set together on the line 352.

[0058] <Vehicle attitude changes>

[0059] The posture of vehicle 1 changes with acceleration and deceleration during driving. Figure 4 (a)~ Figure 4 (c) is a diagram showing an example of the attitude of vehicle 1 in relation to acceleration and deceleration in the direction of travel. Figure 4 (a) shows the attitude of vehicle 1 during acceleration. Figure 4 (b) shows the posture of vehicle 1 when it is traveling at a constant speed or when it is parked. Figure 4 (c) shows the posture of vehicle 1 during deceleration.

[0060] First, when driving at a constant speed or when parked, such as Figure 4 As shown in (b), the fisheye camera 41 and fisheye camera 43, which are arranged in the front-rear direction of the vehicle 1, are oriented toward a predetermined shooting direction (shooting center).

[0061] When accelerating, Figure 4 As shown in (a), the front of vehicle 1 rises vertically upwards and the rear descends vertically downwards due to acceleration. Simultaneously, the shooting direction (shooting center) of the fisheye camera 41 positioned at the front of vehicle 1 changes upwards compared to when vehicle 1 is horizontal. Furthermore, the shooting direction (shooting center) of the fisheye camera 43 positioned at the rear of vehicle 1 changes downwards compared to when vehicle 1 is horizontal.

[0062] Conversely, when decelerating, Figure 4 As shown in (c), the front of vehicle 1 descends vertically downwards due to deceleration, while the rear rises vertically upwards. Simultaneously, the shooting direction (shooting center) of the fisheye camera 41 positioned at the front of vehicle 1 changes downwards compared to when vehicle 1 is horizontal. Furthermore, the shooting direction (shooting center) of the fisheye camera 43 positioned at the rear of vehicle 1 changes upwards compared to when vehicle 1 is horizontal.

[0063] Furthermore, the attitude of vehicle 1 also changes due to the rotation of the steering wheel (lateral acceleration) during travel. Specifically, acceleration is generated in the left and right directions (lateral direction) based on the speed of vehicle 1 and the amount of rotation of the steering wheel 31. Figure 5 (a)~ Figure 5 Figure (c) shows an example of the posture of vehicle 1 when the wheel is turning. Figure 5(a) shows the posture of vehicle 1 when it is traveling at a constant speed or when it is parked without the wheel being turned. Figure 5 (b) shows the posture of vehicle 1 when performing a wheel rotation operation to the left. Figure 5 (c) shows the posture of vehicle 1 when performing a wheel rotation operation in the right direction.

[0064] When turning left at a left turn lane (when operating steering wheel 31 to turn left), a right-direction acceleration is generated, such as... Figure 5 As shown in (b), the left side of vehicle 1 is raised vertically upwards due to the rotating wheel, while the right side is lowered vertically downwards. Correspondingly, the shooting direction (shooting center) of the fisheye camera 44 positioned on the left side of vehicle 1 changes upwards compared to when vehicle 1 is horizontal. Furthermore, the shooting direction (shooting center) of the fisheye camera 42 positioned on the right side of vehicle 1 changes downwards compared to when vehicle 1 is horizontal.

[0065] Conversely, when turning the wheel to the right in a right-turn lane (when operating steering wheel 31 to turn right), a left-direction acceleration is generated, such as... Figure 5 As shown in (c), the right side of vehicle 1 is raised vertically upwards due to the rotating wheel, while the left side is lowered vertically downwards. Correspondingly, the shooting direction (shooting center) of the fisheye camera 42 positioned on the right side of vehicle 1 changes upwards compared to when vehicle 1 is horizontal. Furthermore, the shooting direction (shooting center) of the fisheye camera 44 positioned on the left side of vehicle 1 changes downwards compared to when vehicle 1 is horizontal.

[0066] Thus, the range of information that can be acquired from the surrounding environment changes due to the acceleration, deceleration, and wheel rotation of vehicle 1. In this embodiment, the positions of the transformation centers used to transform the fisheye images captured by each fisheye camera into planar images are corrected according to the attitude of vehicle 1.

[0067] For example, the changes in the attitude of vehicle 1 corresponding to acceleration and deceleration during acceleration and deceleration (the degree to which the shooting direction (shooting center) changes) can be pre-stored as table data or functions, and the current acceleration or deceleration can be obtained, thereby estimating what kind of attitude change vehicle 1 is undergoing.

[0068] Similarly, the changes in the attitude of vehicle 1 corresponding to the lateral acceleration during wheel rotation (the degree to which the shooting direction (shooting center) changes) are pre-stored as table data or functions. The current lateral acceleration is then obtained, allowing estimation of the attitude changes occurring in vehicle 1. For example, the lateral acceleration can be calculated based on the vehicle 1's speed and wheel rotation.

[0069] In this way, the position of the transformation center used to transform the fisheye image into a planar image is corrected according to the attitude of vehicle 1, thereby correcting the effects caused by the attitude change of vehicle 1 and obtaining the desired surrounding information with good accuracy.

[0070] <Processing>

[0071] Next refer to Figure 6 The flowchart illustrates the processing steps implemented by the control device 2 in this embodiment.

[0072] In S601, ECU 22 and ECU 23 detect the attitude of vehicle 1. Various methods can be used for attitude detection. For example, the acceleration and deceleration of vehicle 1 can be calculated based on the speed change of vehicle 1 detected by vehicle speed sensor 7c, and the calculated acceleration or deceleration and the current attitude of vehicle 1 can be pre-stored as a table or function. Furthermore, the corresponding attitude of vehicle 1 can be calculated based on the current acceleration or deceleration of vehicle 1, thereby enabling attitude detection. Similarly, the lateral acceleration of vehicle 1 can be calculated based on the speed of vehicle 1 and the amount of steering wheel rotation 31, and the calculated lateral acceleration and the current attitude of vehicle 1 can be pre-stored as a table or function. Furthermore, the corresponding attitude of vehicle 1 can be calculated based on the current lateral acceleration of vehicle 1, thereby enabling attitude detection. Alternatively, the attitude of vehicle 1 can also be detected based on the detection results of gyroscope sensor 5.

[0073] In S602, ECU 22 and ECU 23 control the transformation center positions for transforming the fisheye images from each fisheye camera (fisheye cameras 41-44) into planar images based on the attitude of vehicle 1. For example, the relationship between the attitude of vehicle 1 and the transformation center positions is stored in advance, thereby enabling the transformation center positions corresponding to the attitude to be derived. Various examples of the processing in S602 are described below.

[0074] [Control example during braking (deceleration)]

[0075] <Fisheye Camera 41>

[0076] First, let's explain the control example during acceleration. For example, in S602, when the attitude of vehicle 1 changes (during braking (deceleration)) as the front of vehicle 1 moves downward and the rear of vehicle 1 moves upward, ECU 23, based on this changed attitude, sets the transformation center position of the fisheye camera 41, which is positioned at the front of vehicle 1 and captures images of the front of vehicle 1, to be higher than the predetermined position (the transformation center position when vehicle 1 is horizontal). Compared to the horizontal position, the shooting center of fisheye camera 41 drops downward by an amount equivalent to the drop in the front of vehicle 1, but the direction for acquiring peripheral information is upward. Therefore, the transformation center position is corrected upward by an amount equivalent to the drop in the front of vehicle 1. In the case of large deceleration, the drop in the front of vehicle 1 also increases, therefore, the transformation center position is corrected upward by an amount equivalent to that.

[0077] <Fisheye Camera 43>

[0078] Furthermore, the ECU 23 sets the transformation center position of the fisheye camera 43, which is positioned at the rear of vehicle 1 and captures images of the rear of vehicle 1, to be lower than the predetermined position (the transformation center position when vehicle 1 is horizontal). Compared to the horizontal position, the shooting center of the fisheye camera 43 is raised upward by an amount equivalent to the rear of vehicle 1 being raised, but the direction for acquiring peripheral information is downward. Therefore, the transformation center position is corrected downward by an amount equivalent to the rear of vehicle 1 being raised. Under large deceleration, the rear of vehicle 1 is also raised more, therefore, the transformation center position is corrected downward by an amount equivalent to that.

[0079] <Fisheye Camera 42, 44>

[0080] Furthermore, when the front of vehicle 1 moves downward and the rear of vehicle 1 moves upward (during braking (deceleration), the ECU 22, based on the changed posture, rotates the center position of the fisheye camera 42, which is positioned on the right side of vehicle 1 and captures images of the right side of vehicle 1, clockwise around the center of the fisheye camera 42's capture, thereby performing correction. Conversely, when the front of vehicle 1 moves downward and the rear of vehicle 1 moves upward (during braking (deceleration), the ECU 22, based on the changed posture, rotates the center position of the fisheye camera 44, which is positioned on the left side of vehicle 1 and captures images of the left side of vehicle 1, counterclockwise around the center of the fisheye camera 44's capture, thereby performing correction.

[0081] here, Figure 7(a) is an example of a fisheye image (shooting center 703) taken by a fisheye camera 42 in a horizontal state without acceleration, deceleration, or lateral acceleration, taken from the right side. A rectangular region 702 is defined centered on the correction center point 701. During a change in attitude (braking (deceleration)) when the front of the vehicle 1 moves downward and the rear of the vehicle 1 moves upward, the correction center point 751 and the rectangular region 752 rotate and change as follows: Figure 7 As in (b), in the illustrated example, peripheral information is acquired from an area higher than the original desired location. To correct this, the transformation center position is rotated clockwise around the shooting center 703, as indicated by arrow symbol 753. The transformation center position of the fisheye camera 44, positioned on the left side of vehicle 1 and capturing images of the left side of vehicle 1, is rotated in the opposite direction, thus performing the reverse rotation correction.

[0082] Therefore, it is possible to obtain the surrounding information of the desired location with good accuracy.

[0083] [Example of control during acceleration]

[0084] Next, the control example during acceleration will be explained. During acceleration, the opposite of during deceleration, the front of vehicle 1 rises and the rear of vehicle 1 falls, thus correcting the change center position in the opposite direction to deceleration. The correction method can be the same as that used during deceleration.

[0085] For example, in S602, during the attitude change (acceleration) when the front of vehicle 1 moves upward and the rear of vehicle 1 moves downward, ECU 23 sets the transformation center position of the fisheye camera 41, which is positioned at the front of vehicle 1 and captures images of the front of vehicle 1, to be lower than the predetermined position (the transformation center position when vehicle 1 is horizontal), based on the changed attitude. Additionally, ECU 23 sets the transformation center position of the fisheye camera 43, which is positioned at the rear of vehicle 1 and captures images of the rear of vehicle 1, to be higher than the predetermined position (the transformation center position when vehicle 1 is horizontal).

[0086] Furthermore, during a posture change where the front of vehicle 1 moves upward and the rear of vehicle 1 moves downward (e.g., a backward lean during acceleration), the ECU 22, based on this changed posture, rotates the center of rotation of the fisheye camera 42, which is positioned on the right side of vehicle 1 and captures images of the right side of the vehicle, to the left, centered on the center of capture of the fisheye camera 42, thereby performing correction. Additionally, during a posture change where the front of vehicle 1 moves upward and the rear of vehicle 1 moves downward (during acceleration), the ECU 22, based on this changed posture, rotates the center of rotation of the fisheye camera 44, which is positioned on the left side of vehicle 1 and captures images of the left side of vehicle 1, to the right, centered on the center of capture of the fisheye camera 44, thereby performing correction.

[0087] Therefore, it is possible to obtain the surrounding information of the desired location with good accuracy.

[0088] [Control example in cases of lateral acceleration]

[0089] Not only during acceleration and deceleration, but also when driving on curves, the lateral acceleration generated by wheel rotation during movement causes the right side of vehicle 1 to lift and the left side to sink (tilt to the left), or vice versa. For example, when driving in a left turn lane, wheel rotation generates right-direction acceleration, causing vehicle 1 to tilt to the right. Conversely, when driving in a right turn lane, wheel rotation generates left-direction acceleration, causing vehicle 1 to tilt to the left.

[0090] First, let's explain a control example when vehicle 1 is tilted to the right.

[0091] <Fisheye Camera 42, 44>

[0092] For example, in S602, in a posture change state (rightward tilt state) where the right side of vehicle 1 moves downward and the left side of vehicle 1 moves upward, ECU 22, based on the changed posture, adjusts the transformation center position of the fisheye camera 42, which is positioned on the right side of vehicle 1 and captures images of the right side of vehicle 1, to be higher than its predetermined position (the transformation center position when vehicle 1 is horizontal). When vehicle 1 tilts to the right, the transformation center position of the fisheye camera 42 capturing images of the right side will be oriented lower than the area where surrounding information is originally intended to be acquired, thus requiring an upward correction by an amount equivalent to the change.

[0093] On the other hand, in a tilted-right state (rightward tilt), where the right side of the vehicle moves downward and the left side moves upward, the ECU 22, based on the changed posture, adjusts the transformation center position of the fisheye camera 44, which is positioned on the left side of the vehicle and captures images of the left side, to be lower than its predetermined position (the transformation center position when the vehicle 1 is level). When the vehicle 1 tilts to the right, the transformation center position of the fisheye camera 42 capturing images of the left side will be oriented higher than the area where surrounding information is originally intended to be acquired, thus requiring a downward correction by an amount equivalent to the change.

[0094] <Fisheye Camera 41, 43>

[0095] In a tilted-right state (rightward tilt), where the right side of vehicle 1 moves downward and the left side moves upward, ECU 23, based on the changed posture, rotates the center of rotation of the fisheye camera 41, which is positioned at the front of vehicle 1 and captures images of the front of vehicle 1, to the left, centered on the center of capture of fisheye camera 41, thereby performing correction. Similarly, in a tilted-right state (rightward tilt), where the right side of vehicle 1 moves downward and the left side moves upward, ECU 23, based on the changed posture, rotates the center of rotation of the fisheye camera 43, which is positioned at the rear of vehicle 1 and captures images of the rear of vehicle 1, to the right, centered on the center of capture of fisheye camera 43, thereby performing correction.

[0096] When vehicle 1 tilts to the right, the fisheye camera 41, which is shooting forward, rotates clockwise by an amount equivalent to the clockwise rotation, and then performs a counterclockwise rotation correction. Similarly, when vehicle 1 tilts to the right, the fisheye camera 43, which is shooting backward, rotates counterclockwise by an amount equivalent to the counterclockwise rotation, and then performs a counterclockwise rotation correction.

[0097] Next, a control example will be described when the vehicle 1 is tilted to the left. This control example is the opposite of the control example when the vehicle 1 is tilted to the right.

[0098] <Fisheye Camera 42, 44>

[0099] For example, in S602, in a posture change state (leftward tilt state) where the right side of vehicle 1 moves upward and the left side of vehicle 1 moves downward, ECU 22, based on the changed posture, alters the transformation center position of the fisheye camera 42, which is positioned on the right side of vehicle 1 and captures images of the right side of vehicle 1, to a position lower than its predetermined position (the transformation center position when vehicle 1 is horizontal). When vehicle 1 tilts to the left, the transformation center position of the fisheye camera 42 capturing images of the right side will be oriented higher than the area where surrounding information is originally intended to be acquired, thus requiring a downward correction by an amount equivalent to the change.

[0100] On the other hand, in a tilted-left state (right side moving upwards and left side moving downwards), the ECU 22, based on the changed posture, adjusts the transformation center position of the fisheye camera 44, which is positioned on the left side of the vehicle and captures images of the left side, to be higher than its predetermined position (the transformation center position when the vehicle 1 is level). When the vehicle 1 tilts to the left, the transformation center position of the fisheye camera 42 capturing images of the left side will be oriented lower than the area where surrounding information is originally intended to be acquired, thus requiring an upward correction equivalent to the change.

[0101] <Fisheye Camera 41, 43>

[0102] In a tilted-left state (right side moving upwards and left side moving downwards), the ECU 23, based on the changed posture, rotates the center of rotation of the fisheye camera 41, which is positioned at the front of the vehicle 1 and captures images of the front of the vehicle 1, clockwise around the center of its capture position, thereby performing correction. Similarly, in a tilted-left state (right side moving upwards and left side moving downwards), the ECU 23, based on the changed posture, rotates the center of rotation of the fisheye camera 43, which is positioned at the rear of the vehicle 1 and captures images of the rear of the vehicle 1, counterclockwise around the center of its capture position, thereby performing correction.

[0103] When vehicle 1 tilts to the left, the camera rotates counterclockwise based on the transformation center position of the fisheye camera 41 that is shooting forward, and then performs a clockwise rotation correction by an amount equivalent to that rotation. Similarly, when vehicle 1 tilts to the left, the camera rotates clockwise based on the transformation center position of the fisheye camera 43 that is shooting backward, and then performs a counterclockwise rotation correction by an amount equivalent to that rotation.

[0104] The above examples illustrate various processing methods based on the attitude changes of vehicle 1, such as acceleration, deceleration, and lateral acceleration. However, combinations of these processing methods can also be executed. For instance, in the case of deceleration and driving in a left-turn lane, the height changes of the front and rear of vehicle 1, as well as the height changes of the left and right sides of vehicle 1, can both be processed together according to their respective corresponding methods.

[0105] Based on the transformation center position set (corrected) in S602, an image transformation from a fisheye image to a planar image is performed. The transformed planar image is then used to perform various actions such as acquiring surrounding information and providing driving assistance. Figure 6 The series of processes have concluded.

[0106] As explained above, in this embodiment, the position of the transformation center for transforming fisheye images from each fisheye camera positioned at the front, rear, and left and right sides of the vehicle into planar images is controlled based on the vehicle's posture.

[0107] Therefore, it is possible to acquire surrounding information about the desired area with high accuracy, regardless of changes in vehicle attitude. Consequently, it is possible to perform various processing operations, such as driver assistance and autonomous driving, using the acquired surrounding information with high precision.

[0108] [Variation Example]

[0109] Alternatively, control can be implemented such that the frequency of image transformations by the fisheye cameras 42 and 44 positioned on the left and right sides of vehicle 1 is lower than the frequency of image transformations by the fisheye cameras 41 and 43 positioned at the front and rear of vehicle 1. For example, when driving on a one-way road without changing lanes, changes in the environment (surrounding information) in the front and rear directions have a greater impact on driving than changes in the environment (surrounding information) in the left and right directions. Therefore, by making the frequency of image transformations (image acquisition frequency) in the front and rear directions higher than in the left and right directions, more accurate information can be acquired in a timely manner, and the processing load can be reduced by suppressing the frequency of transformations in the left and right directions.

[0110] Furthermore, in the above embodiments, examples of ECU 22 and ECU 23 performing processing separately have been described. However, it is also possible to configure these ECUs into a single ECU, and have one ECU perform the processing of the above embodiments or variations.

[0111] Furthermore, the shape, position, and size of the rectangular region for cutting shown in the above embodiments are just examples and are not limited to the illustrated example. A rectangular region larger than the illustrated rectangle may also be used, as may a smaller rectangular region.

[0112] (Other implementation methods)

[0113] Furthermore, a program that implements one or more functions described in each embodiment is provided to the system or device via a network or storage medium, and one or more processors in the computer of the system or device can read and execute the program. The present invention can also be implemented in this manner.

[0114] The invention is not limited to the above-described embodiments, and various modifications and alterations can be made within the scope of the invention's intent.

[0115] <Summary of Implementation Methods>

[0116] The control device (e.g., 2) of the first type controls the shooting of fisheye cameras 41-44 arranged at the front, rear, left and right sides of the vehicle (e.g., 1), and the control device includes:

[0117] Detection units (e.g., 22, 23, 5, 7c) detect the attitude of the vehicle; and

[0118] A control unit (e.g., 22) controls the position (e.g., 301, 701, 751) of the transformation center for transforming the fisheye images from each fisheye camera into planar images, based on the attitude of the vehicle.

[0119] Therefore, it is possible to acquire surrounding information about the desired area with high accuracy, regardless of changes in vehicle attitude. Consequently, it is possible to perform various processing functions, such as driver assistance and autonomous driving, using the acquired surrounding information with high precision.

[0120] In the control device of the second type (e.g. 2), the control unit sets the transformation center position of the first fisheye camera (e.g. 41) that is positioned at the front of the vehicle and captures images of the front of the vehicle to a position higher than a predetermined position (e.g., a horizontal position) based on the attitude change state (e.g., a forward tilt state during deceleration) of the vehicle's front moving downward and the vehicle's rear moving upward.

[0121] Therefore, the influence of changes in posture in the forward and backward directions can be reduced, and the fisheye camera that shoots in front can obtain the surrounding information of the desired area with good accuracy.

[0122] In the third type of control device (e.g., 2), the control unit sets the transformation center position of the second fisheye camera (e.g., 43) that is positioned at the rear of the vehicle and captures images of the rear of the vehicle to a lower position than a predetermined position (e.g., a horizontal position) based on the attitude change state (e.g., a forward tilt state during deceleration) of the vehicle's front moving downward and the vehicle's rear moving upward.

[0123] Therefore, the influence of changes in posture in the front and rear directions can be reduced, and fisheye cameras that shoot from the rear can obtain the surrounding information of the desired area with good accuracy.

[0124] In the fourth type of control device (e.g., 2), the control unit is based on the attitude change state of the vehicle, where the front of the vehicle moves downward and the rear of the vehicle moves upward (e.g., a forward tilt during deceleration).

[0125] Correction is performed by rotating the transformation center position of the third fisheye camera (e.g., 42), which is positioned on the right side of the vehicle and captures images of the right side of the vehicle, clockwise around the shooting center of the third fisheye camera.

[0126] Correction is performed by rotating the transformation center position of the fourth fisheye camera (e.g., 44), which is positioned on the left side of the vehicle and captures images of the left side of the vehicle, counterclockwise around the shooting center of the fourth fisheye camera.

[0127] This reduces the impact of changes in posture in the front-to-back direction, allowing each fisheye camera shooting left and right to accurately acquire surrounding information of the desired area.

[0128] In the control device of the fifth method (e.g. 2), the control unit sets the transformation center position of the first fisheye camera (e.g. 41) that is positioned at the front of the vehicle and captures images of the front of the vehicle to a lower position than a predetermined position (e.g., a horizontal position) based on the attitude change state (e.g., the rearward tilt state during acceleration) of the vehicle as the front of the vehicle moves upward and the rear of the vehicle moves downward.

[0129] Therefore, the influence of changes in posture in the forward and backward directions can be reduced, and the fisheye camera that shoots in front can obtain the surrounding information of the desired area with good accuracy.

[0130] In the control device of the sixth method (e.g. 2), the control unit sets the transformation center position of the second fisheye camera (e.g. 43) that is positioned at the rear of the vehicle and captures images of the rear of the vehicle above the front and below the rear of the vehicle (e.g., a backward tilt during acceleration) above a predetermined position (e.g., a horizontal position) based on the attitude change state of the vehicle, in which the front of the vehicle moves upward and the rear of the vehicle moves downward.

[0131] Therefore, the influence of changes in posture in the front and rear directions can be reduced, and fisheye cameras that shoot from the rear can obtain the surrounding information of the desired area with good accuracy.

[0132] In the seventh type of control device (e.g., 2), the control unit is based on the attitude change state of the vehicle, where the front of the vehicle moves upward and the rear of the vehicle moves downward (e.g., the rearward tilt state during acceleration).

[0133] Correction is performed by rotating the transformation center position of the third fisheye camera (e.g., 42), which is positioned on the right side of the vehicle and captures images of the right side of the vehicle, counterclockwise around the shooting center of the third fisheye camera.

[0134] Correction is performed by rotating the transformation center position of the fourth fisheye camera, which is positioned on the left side of the vehicle and captures images of the left side of the vehicle, clockwise around the shooting center of the fourth fisheye camera (e.g., 44).

[0135] Therefore, the transformation center position of the fisheye cameras on the left and right sides, which rotate and move due to changes in posture in the front-to-back direction, can be corrected to an appropriate position. Consequently, each fisheye camera shooting from the left and right can acquire peripheral information of the desired area with good accuracy.

[0136] In the control device of the eighth method (e.g. 2), the control unit changes the position of the transformation center of the third fisheye camera (e.g. 42) that is positioned on the right side of the vehicle and captures images of the right side of the vehicle to a position higher than the predetermined position (e.g., the position when it is horizontal) based on the attitude change state of the right side of the vehicle moving downward and the left side of the vehicle moving upward (when a lateral acceleration in the right direction is generated).

[0137] Therefore, it is possible to correct for the effects of vehicle attitude changes (right tilt) when performing wheel rotation operations, for example, while driving in a left-turn lane, and obtain an appropriate right-side image.

[0138] In the control device of the ninth method (e.g. 2), the control unit changes the position of the transformation center of the fourth fisheye camera (e.g. 42) that is positioned on the left side of the vehicle and captures images of the left side of the vehicle to a position lower than the predetermined position (e.g., the position when it is horizontal) based on the attitude change state of the right side of the vehicle moving downward and the left side of the vehicle moving upward (when a lateral acceleration is generated in the right direction, i.e., the right tilt state).

[0139] Therefore, it is possible to correct for the effects of vehicle attitude changes (right tilt) when performing wheel operations, for example, while driving in a left turn lane, and obtain an appropriate left-side image.

[0140] In the control device of the tenth method (e.g., 2), the control unit is based on the attitude change state of the vehicle, where the right side of the vehicle moves downward and the left side of the vehicle moves upward (when a lateral acceleration in the right direction is generated, i.e., a right tilt state).

[0141] Correction is achieved by rotating the transformation center position of the first fisheye camera (e.g., 41), which is positioned at the front of the vehicle and captures images of the front of the vehicle, counterclockwise around the image center of the first fisheye camera.

[0142] Correction is performed by rotating the transformation center position of the second fisheye camera (e.g., 43), which is positioned at the rear of the vehicle and captures images of the rear of the vehicle, clockwise around the shooting center of the second fisheye camera.

[0143] Therefore, it is possible to correct for the effects of vehicle attitude changes (right tilt) when performing wheel operation, for example, while driving in a left turn lane, and obtain appropriate front-to-back images.

[0144] In the control device of the eleventh method (e.g. 2), the control unit changes the position of the transformation center of the third fisheye camera (e.g. 2) that is positioned on the right side of the vehicle and captures images of the right side of the vehicle to a lower position than the predetermined position, based on the attitude change state of the vehicle's right side moving upward and the vehicle's left side moving downward (when a lateral acceleration in the left direction is generated, i.e., a left tilt state).

[0145] Therefore, it is possible to correct for the effects of vehicle attitude changes (left tilt) when performing wheel operations, for example, while driving in a right-turn lane, and obtain an appropriate right-side image.

[0146] In the control device of the twelfth method (e.g. 2), the control unit changes the position of the transformation center of the fourth fisheye camera (e.g. 44) that is positioned on the left side of the vehicle and captures images of the left side of the vehicle to a position higher than the predetermined position, based on the attitude change state of the vehicle's right side moving upward and the vehicle's left side moving downward (when a lateral acceleration in the left direction is generated, i.e., a left tilt state).

[0147] Therefore, it is possible to correct for the effects of vehicle attitude changes (left tilt) when performing wheel rotation operations, for example, while driving in a right-turn lane, and obtain an appropriate left-side image.

[0148] In the control device of the thirteenth method (e.g., 2), the control unit is based on the attitude change state of the vehicle, where the right side of the vehicle moves upward and the left side of the vehicle moves downward (when a lateral acceleration in the left direction is generated, i.e., a left tilt state).

[0149] Correction is performed by rotating the transformation center position of the first fisheye camera (e.g., 41), which is positioned at the front of the vehicle and captures images of the front of the vehicle, clockwise around the image center of the first fisheye camera.

[0150] Correction is performed by rotating the transformation center position of the second fisheye camera (e.g., 43), which is positioned at the rear of the vehicle and captures images of the rear of the vehicle, counterclockwise around the shooting center of the second fisheye camera.

[0151] Therefore, it is possible to correct for the effects of vehicle attitude changes (left tilt) when performing wheel operation, for example, while driving in a right turn lane, and obtain appropriate front-to-back images.

[0152] In the control device of the fourteenth method (e.g., 2), the detection unit detects the attitude of the vehicle based on at least one of the vehicle's acceleration or deceleration and the lateral acceleration perpendicular to the vehicle's direction of travel.

[0153] Therefore, it is possible to easily detect changes in the vehicle's posture in the forward and backward directions and / or left and right directions.

[0154] In the control device of the fifteenth method (e.g., 2), the detection unit calculates the lateral acceleration based on the vehicle's speed and the amount of wheel rotation.

[0155] Therefore, it is easy to obtain the acceleration in the lateral direction (left and right).

[0156] In the control device of the sixteenth method (e.g., 2), the detection unit uses a gyroscope sensor (e.g., 5) to detect the attitude of the vehicle.

[0157] Therefore, the vehicle's attitude can be easily detected.

[0158] The seventeenth type of control device (e.g., 2) operates by controlling the shooting of fisheye cameras (e.g., 41-44) arranged at the front, rear, and left and right sides of a vehicle (e.g., 1), and the operation method of the control device includes:

[0159] The detection process (e.g., S601) detects the attitude of the vehicle; and

[0160] The control process (e.g., S602) controls the transformation center positions (e.g., 301, 701, 751) for transforming the fisheye images from each fisheye camera into planar images based on the vehicle's posture.

[0161] Therefore, it is possible to acquire surrounding information about the desired area with high accuracy, regardless of changes in vehicle attitude. Consequently, it is possible to perform various processing functions, such as driver assistance and autonomous driving, using the acquired surrounding information with high precision.

[0162] The storage medium stores the program of the eighteenth method, in which...

[0163] Stores a program for enabling the computer to function as a control device in any of the first to sixteenth modes.

[0164] Therefore, the actions of the control device can be realized through a computer.

Claims

1. A control device that controls imaging by fisheye cameras arranged on the front and rear and the left and right sides of a vehicle, the control device comprising: a detection unit that detects a posture of the vehicle; and a control unit that controls a transformation center position for transforming a fisheye image of each fisheye camera into a planar image, based on the posture of the vehicle, wherein the control unit controls in such a manner that a transformation frequency of each of the fisheye cameras arranged on the left and right sides of the vehicle is lower than a transformation frequency of each of the fisheye cameras arranged on the front and rear of the vehicle.

2. The control device according to claim 1, wherein the control unit sets the transformation center position of a first fisheye camera arranged on a front portion of the vehicle and imaging a front of the vehicle to be higher than a predetermined position, based on a posture change state in which the front portion of the vehicle moves downward and a rear portion of the vehicle moves upward.

3. The control device according to claim 1, wherein the control unit sets the transformation center position of a second fisheye camera arranged on the rear portion of the vehicle and imaging a rear of the vehicle to be lower than a predetermined position, based on a posture change state in which the front portion of the vehicle moves downward and the rear portion of the vehicle moves upward.

4. The control device according to claim 1, wherein the control unit, based on a posture change state in which the front portion of the vehicle moves downward and the rear portion of the vehicle moves upward, rotates the transformation center position of a third fisheye camera arranged on a right side portion of the vehicle and imaging a right side of the vehicle in a right direction around a center of imaging of the third fisheye camera, thereby correcting, and rotates the transformation center position of a fourth fisheye camera arranged on a left side portion of the vehicle and imaging a left side of the vehicle in a left direction around a center of imaging of the fourth fisheye camera, thereby correcting.

5. The control device according to claim 1, wherein the control unit sets the transformation center position of a first fisheye camera arranged on a front portion of the vehicle and imaging a front of the vehicle to be lower than a predetermined position, based on a posture change state in which the front portion of the vehicle moves upward and a rear portion of the vehicle moves downward.

6. The control device according to claim 1, wherein the control unit sets the transformation center position of a second fisheye camera arranged on a rear portion of the vehicle and imaging a rear of the vehicle to be higher than a predetermined position, based on a posture change state in which the front portion of the vehicle moves upward and the rear portion of the vehicle moves downward.

7. The control device according to claim 1, wherein the control unit, based on a posture change state in which the front portion of the vehicle moves upward and the rear portion of the vehicle moves downward, ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ correcting the transformation center position of a third fisheye camera configured on the right side of the vehicle and capturing the right side of the vehicle by rotating the transformation center position of the third fisheye camera counterclockwise around the center of the third fisheye camera, correcting the transformation center position of a fourth fisheye camera configured on the left side of the vehicle and capturing the left side of the vehicle by rotating the transformation center position of the fourth fisheye camera clockwise around the center of the fourth fisheye camera.

8. The control device according to claim 1, wherein the control unit changes the transformation center position of a third fisheye camera configured on the right side of the vehicle and capturing the right side of the vehicle to be higher than a predetermined position based on a posture change state in which the right portion of the vehicle moves downward and the left portion of the vehicle moves upward.

9. The control device according to claim 1, wherein the control unit changes the transformation center position of a fourth fisheye camera configured on the left side of the vehicle and capturing the left side of the vehicle to be lower than a predetermined position based on a posture change state in which the right portion of the vehicle moves downward and the left portion of the vehicle moves upward.

10. The control device according to claim 1, wherein the control unit changes the transformation center position of a third fisheye camera configured on the right side of the vehicle and capturing the right side of the vehicle to be higher than a predetermined position based on a posture change state in which the right portion of the vehicle moves downward and the left portion of the vehicle moves upward, correcting the transformation center position of a first fisheye camera configured on the front portion of the vehicle and capturing the front of the vehicle by rotating the transformation center position of the first fisheye camera counterclockwise around the center of the first fisheye camera, correcting the transformation center position of a second fisheye camera configured on the rear portion of the vehicle and capturing the rear of the vehicle by rotating the transformation center position of the second fisheye camera clockwise around the center of the second fisheye camera.

11. The control device according to claim 1, wherein the control unit changes the transformation center position of a third fisheye camera configured on the right side of the vehicle and capturing the right side of the vehicle to be lower than a predetermined position based on a posture change state in which the right portion of the vehicle moves upward and the left portion of the vehicle moves downward.

12. The control device according to claim 1, wherein the control unit changes the transformation center position of a fourth fisheye camera configured on the left side of the vehicle and capturing the left side of the vehicle to be higher than a predetermined position based on a posture change state in which the right portion of the vehicle moves upward and the left portion of the vehicle moves downward.

13. The control device according to claim 1, wherein the control unit changes the transformation center position of a third fisheye camera configured on the right side of the vehicle and capturing the right side of the vehicle to be lower than a predetermined position based on a posture change state in which the right portion of the vehicle moves upward and the left portion of the vehicle moves downward, correcting the transformation center position of a first fisheye camera configured on the front portion of the vehicle and capturing the front of the vehicle by rotating the transformation center position of the first fisheye camera clockwise around the center of the first fisheye camera, The center of transformation position of a second fisheye camera configured at the rear of the vehicle and capturing the rear of the vehicle is rotated counterclockwise around the center of capturing of the second fisheye camera, thereby being corrected.

14. The control device according to claim 1, wherein The detection unit detects the attitude of the vehicle based on at least one of an acceleration or deceleration of the vehicle, or an acceleration in a lateral direction perpendicular to a traveling direction of the vehicle.

15. The control device according to claim 14, wherein The detection unit calculates the acceleration in the lateral direction based on a speed of the vehicle and a steering amount of the vehicle.

16. The control device according to claim 1, wherein The detection unit detects the attitude of the vehicle using a gyro sensor.

17. A storage medium storing a program for causing a computer to function as the control device according to any one of claims 1 to 16.

18. A method of operation of a control device that controls capturing by fisheye cameras arranged at front and rear and left and right sides of a vehicle, the method of operation of the control device comprising: a detection process that detects an attitude of the vehicle; and a control process that controls a center of transformation position for transforming a fisheye image of each fisheye camera into a planar image based on the attitude of the vehicle, in the control process, the control is performed in such a manner that a frequency of transformation of each of the fisheye cameras arranged at the left and right sides of the vehicle is lower than a frequency of transformation of each of the fisheye cameras arranged at the front and rear of the vehicle. ​

Images