Control device, control device operation method and program

Abstract

To acquire peripheral information on a desired area with high precision regardless of a change in the posture of a vehicle.SOLUTION: A control device that controls the shooting by a fish-eye camera placed on the front, rear, and sides of a vehicle includes detection means for detecting the posture of the vehicle, and control means for controlling a transformation center position for transforming a fish-eye image of each fish-eye camera into a plane image on the basis of the posture of the vehicle.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a control device, an operation method of the control device, and a program. [Background technology]

[0002] It is common to detect objects from images of the vehicle's surroundings and use the results to control driving assistance and other applications. In this case, it is expected that a fisheye camera with a wide angle of view will be used to expand the detection range. However, since the images captured by a fisheye camera are distorted, the detection accuracy may decrease if object detection technology that assumes distortion-free images obtained from a normal camera is applied.

[0003] Patent Document 1 discloses a technique for performing distortion correction processing on a distorted image and performing object detection using the corrected image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2008-48443 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 does not take into consideration any changes in the posture of the vehicle on which the fisheye camera is placed (for example, leaning to the right or left while driving, leaning forward or backward due to acceleration or deceleration, etc.), which means that there is a problem that it is not always possible to obtain an image of the desired area.

[0006] The present invention has been made in consideration of the above problems, and has an object to provide a technique for accurately acquiring surrounding information about a desired area regardless of changes in the vehicle's posture. [Means for solving the problem]

[0007] The control device according to the present invention, which achieves the above object, comprises: A control device that controls photography using fisheye cameras installed at the front, rear, left and right sides of a vehicle, A detection means for detecting the attitude of the vehicle; a control means for controlling a conversion center position for converting a fisheye image of each fisheye camera into a planar image based on the attitude of the vehicle; Equipped with 、 The control means controls the conversion frequency of each of the fish-eye cameras arranged on the left and right sides of the vehicle to be lower than the conversion frequency of each of the fish-eye cameras arranged at the front and rear of the vehicle. It is characterized by: Effect of the Invention

[0008] According to the present invention, it is possible to obtain peripheral information about a desired area with high accuracy regardless of changes in the vehicle's attitude. Therefore, it is possible to perform various processes such as driving assistance and automatic driving using the obtained peripheral information with high accuracy. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram of a vehicle according to an embodiment of the present invention. [Diagram 2] (a) shows the horizontal shooting range of each fisheye camera, (b) shows the vertical shooting range of the right-side fisheye camera, and (c) shows the vertical shooting range of the rear fisheye camera. [Diagram 3] 11A and 11B are diagrams illustrating a process of correcting distortion of a fisheye image. [Figure 4] 5(a) to 5(c) are diagrams illustrating an example of a change in posture in the front-rear direction of a vehicle according to one embodiment of the present invention. [Diagram 5] 5(a) to 5(c) are diagrams illustrating an example of a change in posture in the left-right direction of a vehicle according to an embodiment of the present invention. [Figure 6] 4 is a flowchart showing a procedure of a process executed by a control device according to an embodiment of the present invention. [Figure 7] 11A and 11B are diagrams illustrating an example of rotation correction according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (Embodiment) <Configuration> Fig. 1 is a block diagram of a vehicle 1 according to an embodiment of the present invention. In Fig. 1, the vehicle 1 is generally shown in a plan view and a side view. As an example, the vehicle 1 is a sedan-type four-wheeled passenger car. The vehicle 1 may be such a four-wheeled vehicle, or may be a two-wheeled vehicle or another type of vehicle.

[0012] The vehicle 1 includes a vehicle control device 2 (hereinafter, simply referred to as the control device 2) that controls the vehicle 1. The control device 2 includes a plurality of ECUs (Electronic Control Units) 20 to 29 that are communicatively connected via an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit), a memory such as a semiconductor memory, an interface with an external device, and the like. The memory stores a program executed by the processor and data used by the processor for processing, and the like. Each ECU may include a plurality of processors, memories, interfaces, and the like. For example, the ECU 20 includes one or more processors 20a and one or more memories 20b. The processor 20a executes instructions including a program stored in the memory 20b, whereby the ECU 20 executes processing. Alternatively, the ECU 20 may include a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for executing processing by the ECU 20. The same applies to the other ECUs.

[0013] The following describes the functions and the like handled by each of the ECUs 20 to 29. Note that the number of ECUs and the functions handled by them can be designed appropriately, and they can be divided more finely or integrated than in this embodiment.

[0014] The ECU 20 executes control related to the automatic driving of the vehicle 1. In the automatic driving, at least one of the steering and the acceleration / deceleration of the vehicle 1 is automatically controlled. The automatic driving by the ECU 20 may include automatic driving that does not require driving operation by the driver (which may also be called automatic driving) and automatic driving that assists the driving operation by the driver (which may also be called driving assistance).

[0015] The ECU 21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism for steering the front wheels in response to a driver's driving operation (steering operation) on a steering wheel 31. The electric power steering device 3 also includes a motor that generates a driving force for assisting the steering operation and automatically steering the front wheels, a sensor that detects the steering angle, and the like. When the driving state of the vehicle 1 is autonomous driving, the ECU 21 automatically controls the electric power steering device 3 in response to an instruction from the ECU 20, and controls the traveling direction of the vehicle 1.

[0016] The ECUs 22 and 23 control the detection units that detect the surrounding conditions of the vehicle and process information on the detection results. The vehicle 1 includes one standard camera 40 and four fisheye cameras 41 to 44 as detection units that detect the surrounding conditions of the vehicle. The standard camera 40 and the fisheye cameras 42 and 44 are connected to the ECU 22. The fisheye cameras 41 and 43 are connected to the ECU 23. The ECUs 22 and 23 can extract the contours of targets and lane markings (white lines, etc.) on the road by analyzing images captured by the standard camera 40 and the fisheye cameras 41 to 44.

[0017] The fisheye cameras 41 to 44 are cameras to which a fisheye lens is attached. The configuration of the fisheye camera 41 will be described below. The other fisheye cameras 42 to 44 may have the same configuration. The angle of view of the fisheye camera 41 is wider than that of the standard camera 40. Therefore, the fisheye camera 41 can capture a wider range than the standard camera 40. The image captured by the fisheye camera 41 has a large distortion compared to the image captured by the standard camera 40. Therefore, the ECU 23 may perform a conversion process (hereinafter referred to as a "distortion correction process") to reduce the distortion on the image captured by the fisheye camera 41 before analyzing the image. On the other hand, the ECU 22 may not perform the distortion correction process on the image captured by the standard camera 40 before analyzing the image. In this way, the standard camera 40 is an imaging device that captures an image that is not subject to the distortion correction process, and the fisheye camera 41 is an imaging device that captures an image that is subject to the distortion correction process. Instead of the standard camera 40, other imaging devices that capture images that are not subject to distortion correction processing, such as a camera equipped with a wide-angle lens or a telephoto lens, may be used.

[0018] The standard camera 40 is attached to the center of the front of the vehicle 1 and photographs the surroundings in front of the vehicle 1. The fisheye camera 41 is attached to the center of the front of the vehicle 1 and photographs the surroundings in front of the vehicle 1. In FIG. 1, the standard camera 40 and the fisheye camera 41 are shown aligned horizontally. However, the arrangement of the standard camera 40 and the fisheye camera 41 is not limited to this, and for example, they may be aligned vertically. In addition, at least one of the standard camera 40 and the fisheye camera 41 may be attached to the front of the roof of the vehicle 1 (for example, on the inside of the passenger compartment of the front window). The fisheye camera 42 is attached to the center of the right side of the vehicle 1 and photographs the surroundings to the right of the vehicle 1. The fisheye camera 43 is attached to the center of the rear of the vehicle 1 and photographs the surroundings behind the vehicle 1. The fisheye camera 44 is attached to the center of the left side of the vehicle 1 and photographs the surroundings to the left of the vehicle 1.

[0019] The type, number and mounting positions of the cameras of the vehicle 1 are not limited to the above examples. The vehicle 1 may also include a lidar (Light Detection and Ranging) or a millimeter wave radar as a detection unit for detecting targets around the vehicle 1 and measuring the distance to the targets.

[0020] The ECU 22 controls the standard camera 40 and the fisheye cameras 42 and 44 and processes information on the detection results. The ECU 23 controls the fisheye cameras 41 and 43 and processes information on the detection results. By dividing the detection unit that detects the surrounding conditions of the vehicle into two systems, the reliability of the detection results can be improved.

[0021] The ECU 24 controls the gyro sensor 5, the GPS sensor 24b, and the communication device 24c, and processes information on the detection results or communication results. The gyro sensor 5 detects the rotational motion of the vehicle 1. The path of the vehicle 1 can be determined based on the detection results of the gyro sensor 5, the wheel speed, etc. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c wirelessly communicates with a server that provides map information and traffic information to obtain this information. The ECU 24 can access a database 24a of map information constructed in a memory, and the ECU 24 performs route searches from the current location to the destination, etc. The ECU 24, the map database 24a, and the GPS sensor 24b constitute a so-called navigation device.

[0022] The ECU 25 includes a communication device 25a for vehicle-to-vehicle communication. The communication device 25a performs wireless communication with other surrounding vehicles and exchanges information between the vehicles.

[0023] The ECU 26 controls the power plant 6. The power plant 6 is a mechanism that outputs a driving force to rotate the drive wheels of the vehicle 1, and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine in response to a driving operation (accelerator operation or acceleration operation) of the driver detected by an operation detection sensor 7a provided on the accelerator pedal 7A, for example, and switches the gear stage of the transmission based on information such as the vehicle speed detected by a vehicle speed sensor 7c. When the driving state of the vehicle 1 is autonomous, the ECU 26 automatically controls the power plant 6 in response to an instruction from the ECU 20, and controls the acceleration and deceleration of the vehicle 1.

[0024] The ECU 27 controls lighting devices (headlights, taillights, etc.) including turn signals 8 (blinkers). In the example of Fig. 1, the turn signals 8 are provided at the front, door mirrors, and rear of the vehicle 1.

[0025] The ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and accepts information input from the driver. The audio output device 91 notifies the driver of information by audio. The display device 92 notifies the driver of information by displaying an image. The display device 92 is arranged on the surface of the driver's seat, for example, and constitutes an instrument panel or the like. Note that, although audio and display are exemplified here, information may be notified by vibration or light. Information may be notified by a combination of audio, display, vibration, or light. Furthermore, the combination or the notification mode may be changed depending on the level of the information to be notified (for example, urgency). The input device 93 is a group of switches arranged in a position operable by the driver and used to give instructions to the vehicle 1, but may also include an audio input device.

[0026] The ECU 29 controls the brake device 10 and the parking brake (not shown). The brake device 10 is, for example, a disk brake device provided on each wheel of the vehicle 1, and applies resistance to the rotation of the wheel to decelerate or stop the vehicle 1. The ECU 29 controls the operation of the brake device 10 in response to the driver's driving operation (brake operation) detected by, for example, an operation detection sensor 7b provided on the brake pedal 7B. When the driving state of the vehicle 1 is automatic driving, the ECU 29 automatically controls the brake device 10 in response to an instruction from the ECU 20, and controls the deceleration and stop of the vehicle 1. The brake device 10 and the parking brake can also be operated to maintain the stopped state of the vehicle 1. In addition, when the transmission of the power plant 6 is equipped with a parking lock mechanism, this can also be operated to maintain the stopped state of the vehicle 1.

[0027] <Shooting range> Next, the shooting ranges of standard camera 40 and fisheye cameras 41 to 44 will be described with reference to Fig. 2. Fig. 2(a) shows the horizontal shooting range of each camera, Fig. 2(b) shows the vertical shooting range of fisheye camera 42 attached to the right side of vehicle 1, and Fig. 2(c) shows the vertical shooting range of fisheye camera 43 attached to the rear of vehicle 1.

[0028] First, the imaging range in a plan view of the vehicle 1 (i.e., the horizontal direction of the vehicle 1) will be described with reference to Fig. 2(a). The standard camera 40 captures the scenery included in the imaging range 200. The imaging center 200C of the standard camera 40 faces directly ahead of the vehicle 1. The horizontal angle of view of the standard camera 40 may be less than 90°, and may be, for example, about 45° or about 30°.

[0029] Fisheye camera 41 captures the scenery included in shooting range 201. The shooting center 201C of fisheye camera 41 faces directly in front of vehicle 1. Fisheye camera 42 captures the scenery included in shooting range 202. The shooting center 202C of fisheye camera 42 faces directly to the right of vehicle 1. Fisheye camera 43 captures the scenery included in shooting range 203. The shooting center 203C of fisheye camera 43 faces directly behind vehicle 1. Fisheye camera 44 captures the scenery included in shooting range 204. The shooting center 204C of fisheye camera 44 faces directly to the left of vehicle 1. The horizontal angle of view of fisheye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, or greater than 180°, and may be, for example, approximately 180°. FIG. 2(a) shows an example in which the horizontal angle of view of fish-eye cameras 41 to 44 is 180°.

[0030] The shooting range 201 can be divided into an area 201L diagonally forward to the left of the vehicle 1, an area 201F directly in front of the vehicle 1, and an area 201R diagonally forward to the right of the vehicle 1. The shooting range 202 can be divided into an area 202L diagonally forward to the right of the vehicle 1, an area 202F directly to the right of the vehicle 1, and an area 202R diagonally rearward to the right of the vehicle 1. The shooting range 203 can be divided into an area 203L diagonally rearward to the right of the vehicle 1, an area 203F directly behind the vehicle 1, and an area 203R diagonally rearward to the left of the vehicle 1. The shooting range 204 can be divided into an area 204L diagonally rearward to the right of the vehicle 1, an area 204F directly to the left of the vehicle 1, and an area 204R diagonally forward to the left of the vehicle 1. The imaging range 201 may be divided equally into three areas 201L, 201F, and 201R (that is, so that the angle of view of each area is equal). The other imaging ranges 202 to 204 may also be divided equally into three.

[0031] Since the standard camera 40 and the fisheye cameras 41-44 have the above-mentioned shooting ranges 200-204, the direct front and the four diagonal directions of the vehicle 1 are included in the shooting ranges of the two separate cameras. Specifically, the direct front of the vehicle 1 is included in both the shooting range 200 of the standard camera 40 and the area 201F of the shooting range 201 of the fisheye camera 41. The right diagonal front of the vehicle 1 is included in both the area 201R of the shooting range 201 of the fisheye camera 41 and the area 202L of the shooting range 202 of the fisheye camera 42. The same is true for the other three diagonal directions of the vehicle 1.

[0032] Next, the vertical imaging range of the vehicle 1 will be described with reference to Fig. 2(b) and Fig. 2(c). Fig. 2(b) describes the vertical imaging range of the fisheye camera 42, and Fig. 2(c) describes the vertical imaging range of the fisheye camera 43. The vertical imaging ranges of the other fisheye cameras 41 and 44 may be similar.

[0033] The vertical angle of view of the fish-eye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, or greater than 180°, for example, about 180°. FIGS. 2(b) and (c) show an example in which the vertical angle of view of the fish-eye cameras 41 to 44 is 180°. In the example shown, the imaging center 203C of the fish-eye camera 43 faces downward (toward the ground) from the direction parallel to the ground. Alternatively, the imaging center 203C of the fish-eye camera 43 may face in a direction parallel to the ground, or may face upward (opposite the ground) from the direction parallel to the ground. Furthermore, the imaging centers 201C to 204C of the fish-eye cameras 41 to 44 may face in different directions from each other in the vertical direction.

[0034] The distortion correction process for images captured by fisheye cameras 41 to 44 will be described with reference to Fig. 3. Image 300 is an image of the scenery to the right of vehicle 1 captured by fisheye camera 42. As shown, image 300 has large distortion, especially in the peripheral area.

[0035] The ECU 22 connected to the fisheye camera 42 performs distortion correction processing (processing for converting a fisheye image into a planar image) on the image 300. Specifically, the ECU 22 sets one point in the image 300 as a correction center point 301. The ECU 22 cuts out a rectangular area 302 centered on the correction center point 301 from the image 300. The ECU 22 performs distortion correction processing on this area 302 to generate an image 303 in which distortion is reduced. In the distortion correction processing, the distortion is reduced in a position closer to the correction center point 301, and the distortion is not reduced or is increased in a position farther from the correction center point 301. Therefore, in some embodiments, the ECU 22 sets the correction center point 301 in an area to be focused on in the environment around the vehicle 1, and generates an image in which distortion is reduced for this area.

[0036] In the example of Figure 3, the correction center point 301 and the rectangular area 302 are set to the right of the shooting center 351 of the fisheye image (the vertical line 352 passing through it), but the correction center point 301 and the rectangular area 302 may be set to the left, or the shooting center 351 of the fisheye image and the correction center point 301 may both be set on the line 352.

[0037] <Changes in vehicle posture> The attitude of the vehicle 1 changes depending on acceleration and deceleration while traveling. Figures 4(a) to 4(c) are diagrams showing an example of the attitude of the vehicle 1 with respect to acceleration and deceleration in the traveling direction. Figure 4(a) shows the attitude of the vehicle 1 during acceleration, Figure 4(b) shows the attitude of the vehicle 1 during constant speed traveling or when stopped, and Figure 4(c) shows the attitude of the vehicle 1 during deceleration.

[0038] First, when the vehicle is traveling at a constant speed or stopped, the fish-eye cameras 41 and 43 disposed in the front-rear direction of the vehicle 1 face a predetermined shooting direction (shooting center) as shown in FIG. 4(b).

[0039] 4(a), when accelerating, the front position of vehicle 1 rises vertically upward and the rear position drops vertically downward due to acceleration. Accordingly, the shooting direction (shooting center) of fish-eye camera 41 arranged at the front of vehicle 1 changes upward compared to when vehicle 1 is horizontal. Also, the shooting direction (shooting center) of fish-eye camera 43 arranged at the rear of vehicle 1 changes downward compared to when vehicle 1 is horizontal.

[0040] Conversely, when decelerating, as shown in Fig. 4(c), the front position of the vehicle 1 drops vertically downward and the rear position rises vertically upward due to deceleration. Accordingly, the shooting direction (shooting center) of the fish-eye camera 41 arranged at the front of the vehicle 1 changes downward compared to when the vehicle 1 is horizontal. Also, the shooting direction (shooting center) of the fish-eye camera 43 arranged at the rear of the vehicle 1 changes upward compared to when the vehicle 1 is horizontal.

[0041] The attitude of the vehicle 1 also changes depending on steering (lateral acceleration) while traveling. Specifically, acceleration occurs in the left-right direction (lateral direction) according to the speed of the vehicle 1 and the steering amount of the steering wheel 31. Fig. 5(a) to Fig. 5(c) are diagrams showing an example of the attitude of the vehicle 1 while steering. Fig. 5(a) shows the attitude of the vehicle 1 while traveling at a constant speed without steering operation or while stopped, Fig. 5(b) shows the attitude of the vehicle 1 while steering to the left, and Fig. 5(c) shows the attitude of the vehicle 1 while steering to the right.

[0042] When steering left (when steering wheel 31 is operated counterclockwise) for a left curve, for example, rightward acceleration occurs, and as a result of steering, the left side of vehicle 1 rises vertically upward and the right side falls vertically downward, as shown in Fig. 5(b). Accordingly, the shooting direction (shooting center) of fish-eye camera 44 arranged on the left side of vehicle 1 changes upward compared to when vehicle 1 is horizontal. Also, the shooting direction (shooting center) of fish-eye camera 42 arranged on the right side of vehicle 1 changes downward compared to when vehicle 1 is horizontal.

[0043] Conversely, when steering to the right, such as when making a right curve (when the steering wheel 31 is operated clockwise), leftward acceleration occurs, and as a result of steering, the right side of the vehicle 1 rises vertically upward and the left side falls vertically downward, as shown in Fig. 5(c). Accordingly, the shooting direction (shooting center) of the fish-eye camera 42 arranged on the right side of the vehicle 1 changes upward compared to when the vehicle 1 is horizontal. Also, the shooting direction (shooting center) of the fish-eye camera 44 arranged on the left side of the vehicle 1 changes downward compared to when the vehicle 1 is horizontal.

[0044] In this way, the attitude of the vehicle 1 changes due to acceleration / deceleration and steering of the vehicle 1, and therefore the range in which peripheral information is acquired changes. In this embodiment, the positions of the conversion centers for converting the fisheye images captured by the fisheye cameras into planar images are corrected according to the attitude of the vehicle 1.

[0045] For example, the change in the posture of vehicle 1 in response to acceleration and deceleration during acceleration and deceleration (how much the shooting direction (center of shooting) changes) can be stored in advance as table data or a function, and by acquiring the current acceleration or deceleration, it is possible to estimate how the posture of vehicle 1 is changing.

[0046] Similarly, the change in the attitude of the vehicle 1 according to the lateral acceleration during steering (how much the shooting direction (shooting center) changes) is stored in advance as table data or a function, and by acquiring the current lateral acceleration, it is possible to estimate how the attitude of the vehicle 1 is changing. The lateral acceleration can be calculated based on, for example, the speed of the vehicle 1 and the steering amount.

[0047] In this way, by correcting the conversion center position for converting a fisheye image into a planar image in accordance with the attitude of vehicle 1, it is possible to correct the effects of changes in the attitude of vehicle 1, and it becomes possible to accurately obtain the desired surrounding information.

[0048] <Processing> Next, the procedure of the process executed by the control device 2 according to this embodiment will be described with reference to the flowchart of FIG.

[0049] In S601, the ECU 22 and the ECU 23 detect the attitude of the vehicle 1. Various methods can be applied to detect the attitude. For example, the acceleration and deceleration of the vehicle 1 are calculated based on the change in the speed of the vehicle 1 detected by the vehicle speed sensor 7c, and the calculated acceleration or deceleration and the attitude of the vehicle 1 at that time are stored in advance as a table or a function. Then, the attitude can be detected by calculating the corresponding attitude of the vehicle 1 according to the current acceleration or deceleration of the vehicle 1. Similarly, the lateral acceleration of the vehicle 1 is calculated based on the speed of the vehicle 1 and the steering amount of the steering wheel 31, and the calculated lateral acceleration and the attitude of the vehicle 1 at that time are stored in advance as a table or a function. Then, the attitude can be detected by calculating the corresponding attitude of the vehicle 1 according to the current lateral acceleration of the vehicle 1. Alternatively, the attitude of the vehicle 1 may be detected from the detection result by the gyro sensor 5.

[0050] In S602, ECU 22 and ECU 23 each control the transformation center position for transforming the fisheye image of each fisheye camera (fisheye cameras 41-44) into a planar image based on the attitude of vehicle 1. For example, by previously storing the relationship between the attitude of vehicle 1 and the transformation center position in association with each other, it is possible to derive the transformation center position according to the attitude. Various examples of the process of S602 will be described below.

[0051] [Example of control during braking (deceleration)] <Fisheye Camera 41> First, an example of control during acceleration will be described. For example, in S602, when the vehicle 1 changes its attitude (when braking (deceleration)) such that the front of the vehicle 1 moves downward and the rear of the vehicle 1 moves upward, the ECU 23 sets the conversion center position of the fish-eye camera 41, which is disposed at the front of the vehicle 1 and captures the front of the vehicle 1, to a position higher than a predetermined position (the conversion center position when the vehicle 1 is horizontal) based on the attitude after the change. Since the front of the vehicle 1 is lowered, the capture center of the fish-eye camera 41 faces downward compared to when the vehicle is horizontal, but the direction in which peripheral information is desired to be acquired is further upward. Therefore, the conversion center position is corrected upward by the amount of the lowering of the front of the vehicle 1. In the case of a large deceleration, the lowering of the front of the vehicle 1 also becomes large, so the conversion center position is corrected upward by that amount.

[0052] <Fisheye Camera 43> Furthermore, ECU 23 sets the conversion center position of fisheye camera 43, which is disposed at the rear of vehicle 1 and captures the rear of vehicle 1, below a predetermined position (the conversion center position when vehicle 1 is horizontal). Since the rear of vehicle 1 is raised, the center of image capture by fisheye camera 43 faces upward compared to when vehicle 1 is horizontal, but the direction in which peripheral information is desired to be acquired is further downward. Therefore, the conversion center position is corrected upward by the amount of the rear of vehicle 1 being raised. In the case of large deceleration, the rear of vehicle 1 is also raised more significantly, so the conversion center position is corrected downward accordingly.

[0053] <Fisheye Camera 42, 44> Furthermore, when the vehicle 1 changes its posture (when braking (deceleration)) such that the front of the vehicle 1 moves downward and the rear of the vehicle 1 moves upward, the ECU 22 corrects the transformation center position of the fisheye camera 42, which is disposed on the right side of the vehicle 1 and photographs the right side of the vehicle 1, by rotating it clockwise around the center of photography of the fisheye camera 42, based on the posture after the change. Conversely, when the vehicle 1 changes its posture (when braking (deceleration)) such that the front of the vehicle 1 moves downward and the rear of the vehicle 1 moves upward, the ECU 22 corrects the transformation center position of the fisheye camera 44, which is disposed on the left side of the vehicle 1 and photographs the left side of the vehicle 1, by rotating it counterclockwise around the center of photography of the fisheye camera 44, based on the posture after the change.

[0054] Here, FIG. 7(a) is an example of a fisheye image (photography center 703) of the fisheye camera 42 photographing the right side in a horizontal state where no acceleration / deceleration or lateral acceleration occurs. A rectangular area 702 is set with a correction center point 701 as the center. When the vehicle 1 changes its posture (when braking (deceleration)) such that the front part of the vehicle 1 moves downward and the rear part of the vehicle 1 moves upward, the correction center point 751 and rectangular area 752 rotate as shown in FIG. 7(b), and in the illustrated example, peripheral information is acquired from an area above the position where peripheral information is originally to be acquired. To correct this, the conversion center position is rotated clockwise around the photography center 703 as shown by the arrow 753. The conversion center position of the fisheye camera 44, which is disposed on the left side of the vehicle 1 and photographs the left side of the vehicle 1, rotates in the opposite direction, so an opposite rotation correction is performed.

[0055] This makes it possible to obtain information about the surrounding area of ​​the desired location with high accuracy.

[0056] [Example of control during acceleration] Next, an example of control during acceleration will be described. During acceleration, the front of the vehicle 1 rises and the rear of the vehicle 1 falls, which is the opposite of during deceleration, so the conversion center position is corrected in the opposite direction to that during deceleration. The correction method can be the same as during deceleration.

[0057] For example, in S602, when the vehicle 1 changes its posture (accelerating) such that the front of the vehicle 1 moves upward and the rear of the vehicle 1 moves downward, the ECU 23 sets the transformation center position of the fish-eye camera 41, which is disposed at the front of the vehicle 1 and captures the view in front of the vehicle 1, below a predetermined position (the transformation center position when the vehicle 1 is horizontal) based on the changed posture. Also, the ECU 23 sets the transformation center position of the fish-eye camera 43, which is disposed at the rear of the vehicle 1 and captures the view behind the vehicle 1, above a predetermined position (the transformation center position when the vehicle 1 is horizontal).

[0058] In a posture change state in which the front of the vehicle 1 moves upward and the rear of the vehicle 1 moves downward (for example, a backward tilt state during acceleration), the ECU 22 corrects the transformation center position of the fisheye camera 42, which is disposed on the right side of the vehicle 1 and captures the right side of the vehicle, by rotating it counterclockwise around the capture center of the fisheye camera 42, based on the posture after the change. In addition, in a posture change in which the front of the vehicle 1 moves upward and the rear of the vehicle 1 moves downward (during acceleration), the ECU 22 corrects the transformation center position of the fisheye camera 44, which is disposed on the left side of the vehicle 1 and captures the left side of the vehicle 1, by rotating it clockwise around the capture center of the fisheye camera 44, based on the posture after the change.

[0059] This makes it possible to obtain information about the surrounding area of ​​the desired location with high accuracy.

[0060] [Example of control when lateral acceleration occurs] In addition to acceleration and deceleration, steering while traveling, such as when traveling around a curve, causes lateral acceleration, causing the right side of the vehicle 1 to lift up while the left side sinks (tilt to the left), or the left side of the vehicle 1 to lift up while the right side sinks (tilt to the right). For example, steering when traveling around a left curve causes rightward acceleration, causing the vehicle 1 to tilt to the right. Conversely, steering when traveling around a right curve causes leftward acceleration, causing the vehicle 1 to tilt to the left.

[0061] First, a control example when the attitude of the vehicle 1 tilts to the right will be described.

[0062] <Fisheye Camera 42, 44> For example, in S602, in a posture change state (right tilt state) in which the right side of vehicle 1 moves downward and the left side of vehicle 1 moves upward, ECU 22 changes the transformation center position of fisheye camera 42, which is disposed on the right side of vehicle 1 and captures the right side of vehicle 1, to a position above a predetermined position (the transformation center position when vehicle 1 is horizontal) based on the posture after the change. When vehicle 1 is tilted to the right, the transformation center position of fisheye camera 42 capturing the right side is facing downward from the area from which peripheral information is originally desired to be acquired, so the change is corrected upward.

[0063] On the other hand, in a posture change state (right tilt state) in which the right side of the vehicle 1 moves downward and the left side of the vehicle 1 moves upward, the ECU 22 changes the transformation center position of the fisheye camera 44, which is disposed on the left side of the vehicle and captures the left side of the vehicle, to a position lower than a predetermined position (the transformation center position when the vehicle 1 is horizontal) based on the posture after the change. When the vehicle 1 is tilted to the right, the transformation center position of the fisheye camera 42 capturing the left side is facing upward from the area from which peripheral information is originally desired to be acquired, so the change is corrected downward.

[0064] <Fisheye Camera 41, 43> In an attitude change state (right lean state) in which the right part of vehicle 1 moves downward and the left part of vehicle 1 moves upward, ECU 23 corrects the transformation center position of fisheye camera 41, which is arranged at the front of vehicle 1 and captures the front of vehicle 1, by rotating it counterclockwise around the capture center of fisheye camera 41, based on the attitude after the change. Also, in an attitude change state (right lean state) in which the right part of vehicle 1 moves downward and the left part of vehicle 1 moves upward, ECU 23 corrects the transformation center position of fisheye camera 43, which is arranged at the rear of vehicle 1 and captures the rear of vehicle 1, by rotating it clockwise around the capture center of fisheye camera 43, based on the attitude after the change.

[0065] When vehicle 1 tilts to the right, the transformation center position of fish-eye camera 41 capturing the front view rotates clockwise, so rotation correction is made to the left by the amount of rotation change. Similarly, when vehicle 1 tilts to the right, the transformation center position of fish-eye camera 43 capturing the rear view rotates counterclockwise, so rotation correction is made to the right by the amount of rotation change.

[0066] Next, a control example when the attitude of the vehicle 1 tilts to the left will be described. This control example is the reverse process of the control example when the attitude of the vehicle 1 tilts to the right.

[0067] <Fisheye Camera 42, 44> For example, in S602, in a posture change state (left tilt state) in which the right side of the vehicle 1 moves upward and the left side of the vehicle 1 moves downward, the ECU 22 changes the transformation center position of the fisheye camera 42, which is disposed on the right side of the vehicle 1 and captures the right side of the vehicle 1, to a position lower than a predetermined position (the transformation center position when the vehicle 1 is horizontal) based on the posture after the change. When the vehicle 1 is tilted to the left, the transformation center position of the fisheye camera 42 capturing the right side of the vehicle 1 faces upward from the area from which peripheral information is originally desired to be acquired, so the change is corrected downward.

[0068] On the other hand, in a posture change state (left tilt state) in which the right side of the vehicle 1 moves upward and the left side of the vehicle 1 moves downward, the ECU 22 changes the transformation center position of the fisheye camera 44, which is disposed on the left side of the vehicle and captures the left side of the vehicle, to a position above a predetermined position (the transformation center position when the vehicle 1 is horizontal) based on the posture after the change. When the vehicle 1 is tilted to the left, the transformation center position of the fisheye camera 42 capturing the left side is facing downward from the area from which peripheral information is originally desired to be acquired, so the change is corrected upward.

[0069] <Fisheye Camera 41, 43> In an attitude change state (left tilt state) in which the right part of vehicle 1 moves upward and the left part of vehicle 1 moves downward, ECU 23 corrects the transformation center position of fisheye camera 41, which is arranged at the front of vehicle 1 and captures the front of vehicle 1, by rotating it clockwise around the capture center of fisheye camera 41, based on the attitude after the change. Also, in an attitude change state (left tilt state) in which the right part of vehicle 1 moves upward and the left part of vehicle 1 moves downward, ECU 23 corrects the transformation center position of fisheye camera 43, which is arranged at the rear of vehicle 1 and captures the rear of vehicle 1, by rotating it counterclockwise around the capture center of fisheye camera 43, based on the attitude after the change.

[0070] When vehicle 1 tilts to the left, the transformation center position of fish-eye camera 41 capturing the front view rotates counterclockwise, so rotation correction is performed clockwise by the amount of rotation change. Similarly, when vehicle 1 tilts to the left, the transformation center position of fish-eye camera 43 capturing the rear view rotates clockwise, so rotation correction is performed counterclockwise by the amount of rotation change.

[0071] Although various processes have been described above in response to changes in the attitude of the vehicle 1, taking as examples cases where the vehicle 1 is accelerating, decelerating, or undergoing lateral acceleration, some of these processes may be executed in combination. For example, when the vehicle 1 is traveling around a left curve while decelerating, changes in height of both the front and rear parts of the vehicle 1 and the left and right parts of the vehicle 1 occur, and therefore processes corresponding to each of these changes may be executed together.

[0072] Based on the conversion center position set (corrected) in S602, image conversion from a fisheye image to a planar image is performed, and various operations such as obtaining peripheral information and driving support are performed using the converted planar image. This completes the series of processes in FIG.

[0073] As described above, in this embodiment, the conversion center position for converting fisheye images from each fisheye camera disposed at the front, rear, left and right sides of the vehicle into planar images is controlled based on the vehicle posture.

[0074] This makes it possible to obtain peripheral information about a desired area with high accuracy regardless of changes in the vehicle's attitude. Therefore, various processes such as driving assistance and automatic driving using the obtained peripheral information can be performed with high accuracy.

[0075] [Variations] The image conversion frequency of the fisheye cameras 42, 44 arranged on the left and right sides of the vehicle 1 may be controlled to be lower than the image conversion frequency of the fisheye cameras 41, 43 arranged at the front and rear of the vehicle 1. For example, when driving on a one-lane road or when lane changes are not performed, changes in the environment (surrounding information) in the front-rear direction often have a greater impact on driving than changes in the environment (surrounding information) in the left-right direction. Therefore, by increasing the frequency of image conversion in the front-rear direction (frequency of image acquisition) compared to the left-right direction, it is possible to acquire more accurate information in a timely manner, and by suppressing the frequency of conversion in the left-right direction, it is possible to reduce the processing load.

[0076] In the above embodiment, the ECU 22 and the ECU 23 perform processes separately. However, they may be configured as a single ECU, and the single ECU may perform the processes of the above embodiment or the modified examples.

[0077] The shape, position, and size of the rectangular region for extraction shown in the above embodiment are merely examples, and are not limited to the illustrated examples. A rectangular region larger or smaller than the illustrated rectangle may be used.

[0078] (Other embodiments) Furthermore, a program for realizing one or more functions described in each embodiment can be supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device can read and execute the program. The present invention can also be realized in such an embodiment.

[0079] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0080] <Summary of the embodiment> The control device (e.g. 2) according to the first aspect comprises: A control device that controls photography using fisheye cameras (41-44) arranged on the front, rear, left and right sides of a vehicle (e.g., 1), A detection means (e.g., 22, 23, 5, 7c) for detecting the attitude of the vehicle; A control means (e.g., 22) for controlling a conversion center position (e.g., 301, 701, 751) for converting the fisheye image of each fisheye camera into a planar image based on the posture of the vehicle; Equipped with.

[0081] This makes it possible to obtain peripheral information about a desired area with high accuracy regardless of changes in the vehicle's attitude. Therefore, various processes such as driving assistance and automatic driving using the obtained peripheral information can be performed with high accuracy.

[0082] In the control device according to the second aspect (e.g. 2), The control means sets the transformation center position of a first fisheye camera (e.g., 41) located at the front of the vehicle and photographing the area in front of the vehicle to a position higher than a predetermined position (e.g., a horizontal position) based on a posture change state in which the front of the vehicle moves downward and the rear of the vehicle moves upward (e.g., a forward leaning state during deceleration).

[0083] This reduces the effect of changes in posture in the forward and backward directions, making it possible to accurately obtain peripheral information of a desired area using a fish-eye camera capturing images in front of the vehicle.

[0084] In the control device according to the third aspect (e.g. 2), The control means sets the transformation center position of a second fisheye camera (e.g., 43) located at the rear of the vehicle and photographing the rear of the vehicle to a position lower than a predetermined position (e.g., a horizontal position) based on a posture change state in which the front of the vehicle moves downward and the rear of the vehicle moves upward (e.g., a forward leaning state during deceleration).

[0085] This reduces the effect of changes in posture in the forward / backward direction, making it possible to accurately obtain peripheral information of a desired area using a fish-eye camera capturing rearward images.

[0086] In the control device according to the fourth aspect (e.g. 2), The control means, based on a posture change state in which the front part of the vehicle moves downward and the rear part of the vehicle moves upward (for example, a forward leaning state during deceleration), The conversion center position of a third fish-eye camera (e.g., 42) that is disposed on the right side of the vehicle and captures the right side of the vehicle is corrected by rotating the conversion center position in a clockwise direction around the capture center of the third fish-eye camera; The transformation center position of a fourth fish-eye camera (eg, 44) that is disposed on the left side of the vehicle and captures the left side of the vehicle is corrected by rotating the fourth fish-eye camera counterclockwise around its capture center.

[0087] This reduces the effect of changes in posture in the forward and backward directions, making it possible to accurately obtain peripheral information of the desired area for each fisheye camera capturing images on the left and right.

[0088] In the control device (e.g. 2) according to the fifth aspect, The control means sets the transformation center position of a first fisheye camera (e.g., 41) that is positioned at the front of the vehicle and captures images in front of the vehicle to a position lower than a predetermined position (e.g., a horizontal position) based on a posture change state in which the front of the vehicle moves upward and the rear of the vehicle moves downward (e.g., a backward leaning state during acceleration).

[0089] This reduces the effect of changes in posture in the forward and backward directions, making it possible to accurately obtain peripheral information of a desired area using a fish-eye camera capturing images in front of the vehicle.

[0090] In the control device (e.g. 2) according to the sixth aspect, The control means sets the transformation center position of a second fisheye camera (e.g., 43) located at the rear of the vehicle and photographing the rear of the vehicle to a position higher than a predetermined position (e.g., a horizontal position) based on a posture change state in which the front of the vehicle moves upward and the rear of the vehicle moves downward (e.g., a backward leaning state during acceleration).

[0091] This reduces the effect of changes in posture in the forward / backward direction, making it possible to accurately obtain peripheral information of a desired area using a fish-eye camera capturing rearward images.

[0092] In the control device (e.g. 2) according to the seventh aspect, The control means, based on a posture change state in which the front part of the vehicle moves upward and the rear part of the vehicle moves downward (for example, a backward tilt state during acceleration), The conversion center position of a third fish-eye camera (e.g., 42) that is disposed on the right side of the vehicle and captures the right side of the vehicle is corrected by rotating the conversion center position counterclockwise around the capture center of the third fish-eye camera; The conversion center position of a fourth fish-eye camera that is disposed on the left side of the vehicle and captures the left side of the vehicle is corrected by rotating the fourth fish-eye camera (eg, 44) clockwise around its capture center.

[0093] This makes it possible to correct the positions of the transformation centers of the left and right side fisheye cameras, which rotate due to changes in posture in the forward and backward directions, to appropriate positions. Therefore, it becomes possible to obtain peripheral information of the desired area with high accuracy for each fisheye camera capturing images on the left and right.

[0094] In the control device (e.g. 2) according to the eighth aspect, The control means changes the conversion center position of a third fisheye camera (e.g., 42) located on the right side of the vehicle and photographing the right side of the vehicle to a position above a predetermined position (e.g., a horizontal position) based on the posture change state in which the right side of the vehicle is moving downward and the left side of the vehicle is moving upward (when lateral acceleration to the right is occurring, i.e., a right tilt state).

[0095] This makes it possible to correct the effect of a change in vehicle attitude (rightward tilt) that occurs when a steering operation is performed, for example, when traveling around a left curve, and to obtain an appropriate image of the right side.

[0096] In the control device (e.g., 2) according to the ninth aspect, The control means changes the conversion center position of a fourth fisheye camera (e.g., 42) that is disposed on the left side of the vehicle and photographs the left side of the vehicle to a position lower than a predetermined position (e.g., a horizontal position) based on the posture change state in which the right side of the vehicle is moving downward and the left side of the vehicle is moving upward (when lateral acceleration to the right is occurring, i.e., a right tilt state).

[0097] This makes it possible to correct the effect of changes in vehicle attitude (rightward tilt) that occur when a steering operation is performed, for example, when traveling around a left curve, and to obtain an appropriate image of the left side.

[0098] In a control device (e.g., 2) according to a tenth aspect, The control means, based on a posture change state in which a right part of the vehicle moves downward and a left part of the vehicle moves upward (when a lateral acceleration in the right direction is occurring, i.e., a right tilt state), The conversion center position of a first fish-eye camera (e.g., 41) that is disposed at the front of the vehicle and captures the front of the vehicle is corrected by rotating the conversion center position counterclockwise around the capture center of the first fish-eye camera; The transformation center position of a second fish-eye camera (eg, 43) that is disposed at the rear of the vehicle and captures an image of the rear of the vehicle is corrected by rotating the second fish-eye camera clockwise around its capture center.

[0099] This makes it possible to correct the effect of changes in vehicle posture (rightward tilt) that occur when steering a vehicle around a left curve, for example, and to acquire appropriate images in the forward and backward directions.

[0100] In a control device (e.g., 2) according to an eleventh aspect, The control means changes the transformation center position of a third fisheye camera (e.g., 2) disposed on the right side of the vehicle and photographing the right side of the vehicle to a position lower than a predetermined position based on the posture change state in which the right side of the vehicle is moving upward and the left side of the vehicle is moving downward (when lateral acceleration to the left is occurring, i.e., a left tilt state).

[0101] This makes it possible to correct the effect of a change in the vehicle's attitude (left tilt) when a steering operation is performed, for example, when traveling around a right curve, and to obtain an appropriate image of the right side.

[0102] In a control device (e.g., 2) according to a twelfth aspect, The control means changes the conversion center position of a fourth fisheye camera (e.g., 44) that is positioned on the left side of the vehicle and captures the left side of the vehicle to a position above a predetermined position based on the posture change state in which the right side of the vehicle is moving upward and the left side of the vehicle is moving downward (when lateral acceleration to the left is occurring, i.e., a left tilt state).

[0103] This makes it possible to correct the effect of a change in the vehicle's attitude (left lean) when a steering operation is performed, for example, when traveling around a right curve, and to obtain an appropriate image of the left side.

[0104] In a control device (e.g., 2) according to a thirteenth aspect, The control means, based on a posture change state in which a right part of the vehicle moves upward and a left part of the vehicle moves downward (when a lateral acceleration in the left direction is occurring, i.e., a left tilt state), The conversion center position of a first fish-eye camera (e.g., 41) that is disposed at the front of the vehicle and captures the front of the vehicle is corrected by rotating the conversion center position in a clockwise direction around the capture center of the first fish-eye camera; The transformation center position of a second fish-eye camera (eg, 43) that is disposed at the rear of the vehicle and captures an image of the rear of the vehicle is corrected by rotating the second fish-eye camera counterclockwise around its capture center.

[0105] This makes it possible to correct the effect of changes in vehicle posture (left tilt) that occur when steering a vehicle around a right curve, for example, and to acquire appropriate images in the forward and backward directions.

[0106] In a control device (e.g., 2) according to a fourteenth aspect, The detection means detects the attitude of the vehicle based on at least one of the acceleration or deceleration of the vehicle and the lateral acceleration perpendicular to the traveling direction of the vehicle.

[0107] This makes it possible to easily detect changes in the vehicle's posture in the longitudinal and / or lateral directions.

[0108] In a control device (e.g., 2) according to a fifteenth aspect, The detection means calculates the lateral acceleration based on the speed of the vehicle and the steering amount of the vehicle.

[0109] This makes it possible to easily obtain the lateral (left-right) acceleration.

[0110] In a control device (e.g., 2) according to a sixteenth aspect, The detection means detects the attitude of the vehicle using a gyro sensor (eg, 5).

[0111] This makes it possible to easily detect the vehicle attitude.

[0112] A method of operating a control device (e.g., 2) according to a seventeenth aspect includes the steps of: A method for controlling the photographing of a vehicle (e.g., 1) using fisheye cameras (e.g., 41 to 44) arranged on the front, rear, left and right sides of the vehicle, A detection step (e.g., S601) of detecting the attitude of the vehicle; A control step (e.g., S602) of controlling a transformation center position (e.g., 301, 701, 751) for transforming a fisheye image of each fisheye camera into a planar image based on the posture of the vehicle; has.

[0113] This makes it possible to obtain peripheral information about a desired area with high accuracy regardless of changes in the vehicle's attitude. Therefore, it becomes possible to perform various processes such as driving assistance and automatic driving using the obtained peripheral information with high accuracy.

[0114] A program according to an eighteenth aspect, A program for causing a computer to function as a control device according to any one of the first to sixteenth aspects.

[0115] This allows the operation of the control device to be realized by a computer. [Explanation of symbols]

[0116] 1: vehicle, 2: control device, 22, 23: ECU, 41-44: fisheye camera, 5: gyro sensor, 7c: vehicle speed sensor

Claims

1. A control device that controls photography using fisheye cameras installed at the front, rear, left and right sides of a vehicle, A detection means for detecting the attitude of the vehicle; a control means for controlling a conversion center position for converting a fisheye image of each fisheye camera into a planar image based on the attitude of the vehicle; Equipped with A control device characterized in that the control means controls the conversion frequency of each of the fisheye cameras arranged on the left and right sides of the vehicle to be lower than the conversion frequency of each of the fisheye cameras arranged at the front and rear of the vehicle.

2. The control device according to claim 1, characterized in that the control means sets the transformation center position of a first fisheye camera disposed at the front of the vehicle for photographing the area in front of the vehicle to a position higher than a predetermined position based on an attitude change state in which the front of the vehicle is moving downward and the rear of the vehicle is moving upward.

3. The control device according to claim 1 or 2, characterized in that the control means sets the conversion center position of a second fisheye camera disposed at the rear of the vehicle and photographing the rear of the vehicle to a position lower than a predetermined position based on an attitude change state in which the front of the vehicle is moving downward and the rear of the vehicle is moving upward.

4. The control means, based on a posture change state in which the front part of the vehicle is moving downward and the rear part of the vehicle is moving upward, correcting the conversion center position of a third fish-eye camera that is disposed on the right side of the vehicle and captures the right side of the vehicle by rotating the third fish-eye camera clockwise around the capture center of the third fish-eye camera; The control device according to any one of claims 1 to 3, characterized in that the conversion center position of a fourth fish-eye camera arranged on the left side of the vehicle and photographing the left side of the vehicle is corrected by rotating the fourth fish-eye camera counterclockwise around the photographing center of the fourth fish-eye camera.

5. The control device according to any one of claims 1 to 4, characterized in that the control means sets the transformation center position of a first fisheye camera arranged at the front of the vehicle and photographing the area in front of the vehicle to a position lower than a predetermined position based on an attitude change state in which the front of the vehicle is moving upward and the rear of the vehicle is moving downward.

6. The control device according to any one of claims 1 to 5, characterized in that the control means sets the conversion center position of a second fisheye camera disposed at the rear of the vehicle for photographing the rear of the vehicle to a position higher than a predetermined position based on an attitude change state in which the front of the vehicle moves upward and the rear of the vehicle moves downward.

7. The control means, based on a posture change state in which the front part of the vehicle moves upward and the rear part of the vehicle moves downward, correcting the conversion center position of a third fish-eye camera that is disposed on the right side of the vehicle and captures the right side of the vehicle by rotating the third fish-eye camera counterclockwise around the capture center of the third fish-eye camera; The control device according to any one of claims 1 to 6, characterized in that the conversion center position of a fourth fish-eye camera arranged on the left side of the vehicle and photographing the left side of the vehicle is corrected by rotating the fourth fish-eye camera clockwise around the photographing center of the fourth fish-eye camera.

8. The control device according to any one of claims 1 to 7, characterized in that the control means changes the conversion center position of a third fisheye camera arranged on the right side of the vehicle and photographing the right side of the vehicle to a position above a predetermined position based on an attitude change state in which the right side of the vehicle is moving downward and the left side of the vehicle is moving upward.

9. The control device according to any one of claims 1 to 8, characterized in that the control means changes the conversion center position of a fourth fisheye camera arranged on the left side of the vehicle and photographing the left side of the vehicle to a position lower than a predetermined position based on an attitude change state in which the right side of the vehicle is moving downward and the left side of the vehicle is moving upward.

10. The control means, based on a posture change state in which a right part of the vehicle is moving downward and a left part of the vehicle is moving upward, correcting the conversion center position of a first fish-eye camera that is disposed at the front of the vehicle and captures an image of the front of the vehicle by rotating the first fish-eye camera counterclockwise around a capture center of the first fish-eye camera; The control device according to any one of claims 1 to 9, characterized in that the transformation center position of a second fish-eye camera disposed at the rear of the vehicle and photographing the rear of the vehicle is corrected by rotating the transformation center position in a clockwise direction around the photographing center of the second fish-eye camera.

11. The control device according to any one of claims 1 to 10, characterized in that the control means changes the conversion center position of a third fisheye camera arranged on the right side of the vehicle and photographing the right side of the vehicle to a position lower than a predetermined position based on an attitude change state in which the right side of the vehicle moves upward and the left side of the vehicle moves downward.

12. The control device according to any one of claims 1 to 11, characterized in that the control means changes the conversion center position of a fourth fisheye camera arranged on the left side of the vehicle and photographing the left side of the vehicle to a position above a predetermined position based on an attitude change state in which the right side of the vehicle moves upward and the left side of the vehicle moves downward.

13. The control means, based on a posture change state in which a right part of the vehicle moves upward and a left part of the vehicle moves downward, correcting the conversion center position of a first fish-eye camera that is disposed at the front of the vehicle and captures an image of the front of the vehicle by rotating the conversion center position in a clockwise direction around a capture center of the first fish-eye camera; The control device according to any one of claims 1 to 12, characterized in that the transformation center position of a second fish-eye camera arranged at the rear of the vehicle and photographing the rear of the vehicle is corrected by rotating the transformation center position counterclockwise around the photographing center of the second fish-eye camera.

14. 14. The control device according to claim 1, wherein the detection means detects the attitude of the vehicle based on at least one of an acceleration or deceleration of the vehicle and a lateral acceleration perpendicular to a traveling direction of the vehicle.

15. 15. The control device according to claim 14, wherein the detection means calculates the lateral acceleration based on a speed of the vehicle and an amount of steering of the vehicle.

16. 14. The control device according to claim 1, wherein the detection means detects the attitude of the vehicle using a gyro sensor.

17. A method for controlling an image capturing apparatus using fisheye cameras disposed on the front, rear, left and right sides of a vehicle, comprising: a detection step of detecting an attitude of the vehicle; a control step of controlling a conversion center position for converting the fisheye images of each fisheye camera into a planar image based on the attitude of the vehicle; having A method for operating a control device, characterized in that in the control process, the conversion frequency of each of the fisheye cameras arranged on the left and right sides of the vehicle is controlled to be lower than the conversion frequency of each of the fisheye cameras arranged at the front and rear of the vehicle.

18. A program for causing a computer to function as the control device according to any one of claims 1 to 16.

Images