Control device, control device operation method and program

Abstract

To provide a technique for accurately acquiring appropriate peripheral information according to the vehicle speed.SOLUTION: A control device that controls the shooting by a fish-eye camera installed on the front, rear, and sides of a vehicle includes acquisition means for acquiring information on the speed of the vehicle, and control means for controlling a center position for transforming a fish-eye image of each fish-eye camera into a plane image on the basis of the speed.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a control device, an operation method of the control device, and a program. [Background technology]

[0002] It is common to detect objects from images of the vehicle's surroundings and use the results to control driving assistance and other applications. In this case, it is expected that a fisheye camera with a wide angle of view will be used to expand the detection range. However, since the images captured by a fisheye camera are distorted, the detection accuracy may decrease if object detection technology that assumes distortion-free images obtained from a normal camera is applied.

[0003] Patent Document 1 discloses a technique for performing distortion correction processing on a distorted image and performing object detection using the corrected image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2008-48443 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 does not take into consideration which area to use for corrective conversion from a fisheye image to a planar image depending on vehicle speed and other conditions, etc. Therefore, there is a problem in that it is difficult to accurately obtain appropriate surrounding information depending on the vehicle speed.

[0006] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a technique for accurately acquiring appropriate surrounding information according to vehicle speed. [Means for solving the problem]

[0007] The control device according to the present invention, which achieves the above object, comprises: A control device that controls photography using fisheye cameras arranged at the front, rear, and sides of a vehicle, An acquisition means for acquiring information on the speed of the vehicle; a control means for controlling a center position for converting a fisheye image of each fisheye camera into a planar image based on the speed; Equipped with 、 the control means sets the center position of each fish-eye camera to a predetermined position when the speed is less than a threshold value, and sets the center position of each fish-eye camera higher relative to the predetermined position as the speed is higher when the speed is equal to or greater than the threshold value; When the speed is equal to or higher than the threshold value, the higher the speed, the higher the center position of each fish-eye camera is set relative to the predetermined position, and the center positions of each fish-eye camera disposed at the front and rear of the vehicle are set higher than the center positions of each fish-eye camera disposed at the sides of the vehicle. It is characterized by the above. Effect of the Invention

[0008] According to the present invention, it is possible to accurately acquire appropriate surrounding information according to the vehicle speed. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram of a vehicle according to an embodiment of the present invention. [Diagram 2] (a) shows the horizontal shooting range of each fisheye camera, (b) shows the vertical shooting range of the right-side fisheye camera, and (c) shows the vertical shooting range of the rear fisheye camera. [Diagram 3] 11A and 11B are diagrams illustrating a process of correcting distortion of a fisheye image. [Figure 4] 4 is a flowchart showing a procedure of a process executed by a control device according to an embodiment of the present invention. [Diagram 5] 4 is a flowchart showing a procedure of a process executed by a control device according to an embodiment of the present invention. [Figure 6] FIG. 13 is a diagram showing an example of a transformation center position (correction center point) at a predetermined position regarding the fisheye image on the right side. [Figure 7] FIG. 1A is a diagram showing an example of a transformation center position (correction center point) at a predetermined position for a front fisheye image, and FIG. 1B is a diagram showing an example of a transformation center position (correction center point) set above the predetermined position for a front fisheye image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (Embodiment) <Configuration> Fig. 1 is a block diagram of a vehicle 1 according to an embodiment of the present invention. In Fig. 1, the vehicle 1 is generally shown in a plan view and a side view. As an example, the vehicle 1 is a sedan-type four-wheeled passenger car. The vehicle 1 may be such a four-wheeled vehicle, or may be a two-wheeled vehicle or another type of vehicle.

[0012] The vehicle 1 includes a vehicle control device 2 (hereinafter, simply referred to as the control device 2) that controls the vehicle 1. The control device 2 includes a plurality of ECUs (Electronic Control Units) 20 to 29 that are communicatively connected via an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit), a memory such as a semiconductor memory, an interface with an external device, and the like. The memory stores a program executed by the processor and data used by the processor for processing, and the like. Each ECU may include a plurality of processors, memories, interfaces, and the like. For example, the ECU 20 includes one or more processors 20a and one or more memories 20b. The processor 20a executes instructions including a program stored in the memory 20b, whereby the ECU 20 executes processing. Alternatively, the ECU 20 may include a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for executing processing by the ECU 20. The same applies to the other ECUs.

[0013] The following describes the functions and the like handled by each of the ECUs 20 to 29. Note that the number of ECUs and the functions handled by them can be designed appropriately, and they can be divided more finely or integrated than in this embodiment.

[0014] The ECU 20 executes control related to the automatic driving of the vehicle 1. In the automatic driving, at least one of the steering and the acceleration / deceleration of the vehicle 1 is automatically controlled. The automatic driving by the ECU 20 may include automatic driving that does not require driving operation by the driver (which may also be called automatic driving) and automatic driving that assists the driving operation by the driver (which may also be called driving assistance).

[0015] The ECU 21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism for steering the front wheels in response to a driver's driving operation (steering operation) on a steering wheel 31. The electric power steering device 3 also includes a motor that generates a driving force for assisting the steering operation and automatically steering the front wheels, a sensor that detects the steering angle, and the like. When the driving state of the vehicle 1 is autonomous driving, the ECU 21 automatically controls the electric power steering device 3 in response to an instruction from the ECU 20, and controls the traveling direction of the vehicle 1.

[0016] The ECUs 22 and 23 control the detection units that detect the surrounding conditions of the vehicle and process information on the detection results. The vehicle 1 includes one standard camera 40 and four fisheye cameras 41 to 44 as detection units that detect the surrounding conditions of the vehicle. The standard camera 40 and the fisheye cameras 42 and 44 are connected to the ECU 22. The fisheye cameras 41 and 43 are connected to the ECU 23. The ECUs 22 and 23 can extract the contours of targets and lane markings (white lines, etc.) on the road by analyzing images captured by the standard camera 40 and the fisheye cameras 41 to 44.

[0017] The fisheye cameras 41 to 44 are cameras to which a fisheye lens is attached. The configuration of the fisheye camera 41 will be described below. The other fisheye cameras 42 to 44 may have the same configuration. The angle of view of the fisheye camera 41 is wider than that of the standard camera 40. Therefore, the fisheye camera 41 can capture a wider range than the standard camera 40. The image captured by the fisheye camera 41 has a large distortion compared to the image captured by the standard camera 40. Therefore, the ECU 23 may perform a conversion process (hereinafter referred to as a "distortion correction process") to reduce the distortion on the image captured by the fisheye camera 41 before analyzing the image. On the other hand, the ECU 22 may not perform the distortion correction process on the image captured by the standard camera 40 before analyzing the image. In this way, the standard camera 40 is an imaging device that captures an image that is not subject to the distortion correction process, and the fisheye camera 41 is an imaging device that captures an image that is subject to the distortion correction process. Instead of the standard camera 40, other imaging devices that capture images that are not subject to distortion correction processing, such as a camera equipped with a wide-angle lens or a telephoto lens, may be used.

[0018] The standard camera 40 is attached to the center of the front of the vehicle 1 and photographs the surroundings in front of the vehicle 1. The fisheye camera 41 is attached to the center of the front of the vehicle 1 and photographs the surroundings in front of the vehicle 1. In FIG. 1, the standard camera 40 and the fisheye camera 41 are shown aligned horizontally. However, the arrangement of the standard camera 40 and the fisheye camera 41 is not limited to this, and for example, they may be aligned vertically. In addition, at least one of the standard camera 40 and the fisheye camera 41 may be attached to the front of the roof of the vehicle 1 (for example, on the inside of the passenger compartment of the front window). The fisheye camera 42 is attached to the center of the right side of the vehicle 1 and photographs the surroundings to the right of the vehicle 1. The fisheye camera 43 is attached to the center of the rear of the vehicle 1 and photographs the surroundings behind the vehicle 1. The fisheye camera 44 is attached to the center of the left side of the vehicle 1 and photographs the surroundings to the left of the vehicle 1.

[0019] The type, number and mounting positions of the cameras of the vehicle 1 are not limited to the above examples. The vehicle 1 may also include a lidar (Light Detection and Ranging) or a millimeter wave radar as a detection unit for detecting targets around the vehicle 1 and measuring the distance to the targets.

[0020] The ECU 22 controls the standard camera 40 and the fisheye cameras 42 and 44 and processes information on the detection results. The ECU 23 controls the fisheye cameras 41 and 43 and processes information on the detection results. By dividing the detection unit that detects the surrounding conditions of the vehicle into two systems, the reliability of the detection results can be improved.

[0021] The ECU 24 controls the gyro sensor 5, the GPS sensor 24b, and the communication device 24c, and processes information on the detection results or communication results. The gyro sensor 5 detects the rotational motion of the vehicle 1. The path of the vehicle 1 can be determined based on the detection results of the gyro sensor 5, the wheel speed, etc. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c wirelessly communicates with a server that provides map information and traffic information to obtain this information. The ECU 24 can access a database 24a of map information constructed in a memory, and the ECU 24 performs route searches from the current location to a destination, etc. The ECU 24, the map database 24a, and the GPS sensor 24b constitute a so-called navigation device.

[0022] The ECU 25 includes a communication device 25a for vehicle-to-vehicle communication. The communication device 25a performs wireless communication with other surrounding vehicles and exchanges information between the vehicles.

[0023] The ECU 26 controls the power plant 6. The power plant 6 is a mechanism that outputs a driving force to rotate the drive wheels of the vehicle 1, and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine in response to a driving operation (accelerator operation or acceleration operation) of the driver detected by an operation detection sensor 7a provided on the accelerator pedal 7A, for example, and switches the gear stage of the transmission based on information such as the vehicle speed detected by a vehicle speed sensor 7c. When the driving state of the vehicle 1 is autonomous, the ECU 26 automatically controls the power plant 6 in response to an instruction from the ECU 20, and controls the acceleration and deceleration of the vehicle 1.

[0024] The ECU 27 controls lighting devices (headlights, taillights, etc.) including turn signals 8 (blinkers). In the example of Fig. 1, the turn signals 8 are provided at the front, door mirrors, and rear of the vehicle 1.

[0025] The ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and accepts information input from the driver. The audio output device 91 notifies the driver of information by audio. The display device 92 notifies the driver of information by displaying an image. The display device 92 is arranged on the surface of the driver's seat, for example, and constitutes an instrument panel or the like. Note that, although audio and display are exemplified here, information may be notified by vibration or light. Information may be notified by a combination of audio, display, vibration, or light. Furthermore, the combination or the notification mode may be changed depending on the level of the information to be notified (for example, urgency). The input device 93 is a group of switches arranged in a position operable by the driver and used to give instructions to the vehicle 1, but may also include an audio input device.

[0026] The ECU 29 controls the brake device 10 and the parking brake (not shown). The brake device 10 is, for example, a disk brake device provided on each wheel of the vehicle 1, and applies resistance to the rotation of the wheel to decelerate or stop the vehicle 1. The ECU 29 controls the operation of the brake device 10 in response to the driver's driving operation (brake operation) detected by, for example, an operation detection sensor 7b provided on the brake pedal 7B. When the driving state of the vehicle 1 is automatic driving, the ECU 29 automatically controls the brake device 10 in response to an instruction from the ECU 20, and controls the deceleration and stop of the vehicle 1. The brake device 10 and the parking brake can also be operated to maintain the stopped state of the vehicle 1. In addition, when the transmission of the power plant 6 is equipped with a parking lock mechanism, this can also be operated to maintain the stopped state of the vehicle 1.

[0027] <Shooting range> Next, the shooting ranges of standard camera 40 and fisheye cameras 41 to 44 will be described with reference to Fig. 2. Fig. 2(a) shows the horizontal shooting range of each camera, Fig. 2(b) shows the vertical shooting range of fisheye camera 42 attached to the right side of vehicle 1, and Fig. 2(c) shows the vertical shooting range of fisheye camera 43 attached to the rear of vehicle 1.

[0028] First, the imaging range in a plan view of the vehicle 1 (i.e., the horizontal direction of the vehicle 1) will be described with reference to Fig. 2(a). The standard camera 40 captures the scenery included in the imaging range 200. The imaging center 200C of the standard camera 40 faces directly ahead of the vehicle 1. The horizontal angle of view of the standard camera 40 may be less than 90°, and may be, for example, about 45° or about 30°.

[0029] Fisheye camera 41 captures the scenery included in shooting range 201. The shooting center 201C of fisheye camera 41 faces directly in front of vehicle 1. Fisheye camera 42 captures the scenery included in shooting range 202. The shooting center 202C of fisheye camera 42 faces directly to the right of vehicle 1. Fisheye camera 43 captures the scenery included in shooting range 203. The shooting center 203C of fisheye camera 43 faces directly behind vehicle 1. Fisheye camera 44 captures the scenery included in shooting range 204. The shooting center 204C of fisheye camera 44 faces directly to the left of vehicle 1. The horizontal angle of view of fisheye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, or greater than 180°, and may be, for example, approximately 180°. FIG. 2(a) shows an example in which the horizontal angle of view of fish-eye cameras 41 to 44 is 180°.

[0030] The shooting range 201 can be divided into an area 201L diagonally forward to the left of the vehicle 1, an area 201F directly in front of the vehicle 1, and an area 201R diagonally forward to the right of the vehicle 1. The shooting range 202 can be divided into an area 202L diagonally forward to the right of the vehicle 1, an area 202F directly to the right of the vehicle 1, and an area 202R diagonally rearward to the right of the vehicle 1. The shooting range 203 can be divided into an area 203L diagonally rearward to the right of the vehicle 1, an area 203F directly behind the vehicle 1, and an area 203R diagonally rearward to the left of the vehicle 1. The shooting range 204 can be divided into an area 204L diagonally rearward to the right of the vehicle 1, an area 204F directly to the left of the vehicle 1, and an area 204R diagonally forward to the left of the vehicle 1. The imaging range 201 may be divided equally into three areas 201L, 201F, and 201R (that is, so that the angle of view of each area is equal). The other imaging ranges 202 to 204 may also be divided equally into three.

[0031] Since the standard camera 40 and the fisheye cameras 41-44 have the above-mentioned shooting ranges 200-204, the direct front and the four diagonal directions of the vehicle 1 are included in the shooting ranges of the two separate cameras. Specifically, the direct front of the vehicle 1 is included in both the shooting range 200 of the standard camera 40 and the area 201F of the shooting range 201 of the fisheye camera 41. The right diagonal front of the vehicle 1 is included in both the area 201R of the shooting range 201 of the fisheye camera 41 and the area 202L of the shooting range 202 of the fisheye camera 42. The same is true for the other three diagonal directions of the vehicle 1.

[0032] Next, the vertical imaging range of the vehicle 1 will be described with reference to Fig. 2(b) and Fig. 2(c). Fig. 2(b) describes the vertical imaging range of the fisheye camera 42, and Fig. 2(c) describes the vertical imaging range of the fisheye camera 43. The vertical imaging ranges of the other fisheye cameras 41 and 44 may be similar.

[0033] The vertical angle of view of the fish-eye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, or greater than 180°, for example, about 180°. FIGS. 2(b) and (c) show an example in which the vertical angle of view of the fish-eye cameras 41 to 44 is 180°. In the example shown, the imaging center 203C of the fish-eye camera 43 faces downward (toward the ground) from the direction parallel to the ground. Alternatively, the imaging center 203C of the fish-eye camera 43 may face in a direction parallel to the ground, or may face upward (opposite the ground) from the direction parallel to the ground. Furthermore, the imaging centers 201C to 204C of the fish-eye cameras 41 to 44 may face in different directions from each other in the vertical direction.

[0034] The distortion correction process for images captured by fisheye cameras 41 to 44 will be described with reference to Fig. 3. Image 300 is an image of the scenery to the right of vehicle 1 captured by fisheye camera 42. As shown, image 300 has large distortion, especially in the peripheral area.

[0035] The ECU 22 connected to the fisheye camera 42 performs distortion correction processing (processing for converting a fisheye image into a planar image) on the image 300. Specifically, the ECU 22 sets one point in the image 300 as a correction center point 301. The ECU 22 cuts out a rectangular area 302 centered on the correction center point 301 from the image 300. The ECU 22 performs distortion correction processing on this area 302 to generate an image 303 in which distortion is reduced. In the distortion correction processing, the distortion is reduced in a position closer to the correction center point 301, and the distortion is not reduced or is increased in a position farther from the correction center point 301. Therefore, in some embodiments, the ECU 22 sets the correction center point 301 in an area to be focused on in the environment around the vehicle 1, and generates an image in which distortion is reduced for this area.

[0036] <Processing> Next, the procedure of the process executed by the control device 2 according to this embodiment will be described with reference to the flowchart of FIG.

[0037] In S401, the ECU 22 and the ECU 23 acquire speed information of the vehicle 1 detected by the vehicle speed sensor 7c. The speed acquired here is the current traveling speed of the vehicle 1.

[0038] In S402, the ECU 22 and the ECU 23 control the conversion center position for converting the fisheye images taken by each fisheye camera (fisheye cameras 41 to 44) into a planar image based on the speed information of the vehicle 1 acquired in S401. The ECU 22 sets the conversion center position for converting the right side fisheye image taken by the fisheye camera 42 that takes the right side (right direction) of the vehicle 1 and the left side fisheye image taken by the fisheye camera 44 that takes the left side (left direction) of the vehicle 1 into a planar image. In addition, the ECU 23 sets the conversion center position for converting the front fisheye image taken by the fisheye camera 41 that takes the front of the vehicle 1 and the rear fisheye image taken by the fisheye camera 43 that takes the rear of the vehicle 1 into a planar image. A detailed example of the processing of this step will be described later with reference to FIG. 5. Note that image conversion is performed based on the conversion center position set in this step, and various operations such as acquisition of surrounding information and driving assistance are performed using the converted images. This ends the series of processing in FIG. 4.

[0039] Next, a procedure of processing executed by the control device 2 according to this embodiment will be described with reference to a flowchart in Fig. 5. The processing in Fig. 5 shows a detailed example of the processing in S402 in Fig. 4.

[0040] In S4021, the ECU 22 and the ECU 23 determine whether the speed of the vehicle 1 acquired in S401 is equal to or greater than a threshold (e.g., 20 km / h). If the speed of the vehicle 1 is equal to or greater than the threshold, the process proceeds to S4023. On the other hand, if the speed of the vehicle 1 is less than the threshold, the process proceeds to S4022.

[0041] In S4022, the ECU 22 sets, to predetermined positions, the conversion center position for converting the right-side fisheye image taken by the fisheye camera 42 into a planar image, and the conversion center position for converting the left-side fisheye image taken by the fisheye camera 44 into a planar image.

[0042] Here, the image 600 in FIG. 6 is an image of the scenery on the right side of the vehicle 1 captured by the fish-eye camera 42. The ECU 22 connected to the fish-eye camera 42 sets one point in the image 600 as a correction center point 601. The ECU 22 cuts out a rectangular area 602 centered on the correction center point 601 from the image 600, and performs distortion correction processing on the area 602 to generate an image (planar image) with reduced distortion. Here, the correction center point 601 is a predetermined position when the speed of the vehicle 1 is less than a threshold. The example in FIG. 6 is an example of the transformation center position for the image on the right side, but the same is true for the image on the left side. The transformation center positions can be set to the same height in the image (a predetermined height from the bottom end), but they may each be different predetermined positions.

[0043] Furthermore, in S4022, ECU 23 sets the conversion center position for converting the forward fisheye image taken by fisheye camera 41 into a planar image, and the conversion center position for converting the rearward fisheye image taken by fisheye camera 43 into a planar image, to predetermined positions.

[0044] Here, the image 700 in FIG. 7(a) is an image of the scenery ahead of the vehicle 1 captured by the fish-eye camera 41. The image captures the rear of the preceding vehicle. The ECU 23 connected to the fish-eye camera 41 sets one point in the image 700 as the correction center point 701. The ECU 23 cuts out a rectangular area 702 centered on the correction center point 701 from the image 700, and performs distortion correction processing on the area 702 to generate an image (planar image) with reduced distortion. Here, the correction center point 701 is a predetermined position when the speed of the vehicle 1 is less than a threshold. The example in FIG. 7(a) is an example of the transformation center position for the front image, but the same is true for the rear image. The transformation center positions can be set to the same height in the image (a predetermined height from the bottom end), but they may each be different predetermined positions.

[0045] In S4023, the ECU 22 sets the transformation center position for transforming the right side fisheye image captured by the fisheye camera 42 into a planar image and the transformation center position for transforming the left side fisheye image captured by the fisheye camera 44 into a planar image to a position higher than a predetermined position when the speed of the vehicle 1 is less than a threshold. For example, the transformation center position of the right side fisheye image is set to the position of the correction center point 301 shown in FIG. 3 (i.e., a position higher than the position of the correction center point 601). The example in FIG. 3 is an example of the transformation center position for the right side image, but the same is true for the left side image. The transformation center positions can be set to the same height in the image (a predetermined height from the bottom end), but they may each be at a different position.

[0046] Furthermore, in S4023, the ECU 23 sets the transformation center position for transforming the front fisheye image captured by the fisheye camera 41 into a planar image and the transformation center position for transforming the rear fisheye image captured by the fisheye camera 43 into a planar image to a position higher than a predetermined position when the speed of the vehicle 1 is less than a threshold. For example, the transformation center position of the front fisheye image is set to the position of the correction center point 751 in the rectangular area 752 of the image 750 shown in FIG. 7(b) (i.e., a position higher than the position of the correction center point 701). The example of FIG. 7(b) is an example of the transformation center position for the front image, but the same applies to the rear image. The transformation center positions can be set to the same height in the image (a predetermined height from the bottom end), but they may each be different predetermined positions. This ends the processing of FIG. 5.

[0047] As described above, in this embodiment, the conversion center position for converting fisheye images captured by each fisheye camera into a planar image is controlled based on vehicle speed information. For example, depending on whether the vehicle speed is equal to or higher than a threshold, the conversion center position is changed to be set to a predetermined position or to be set above the predetermined position. This makes it possible to accurately obtain appropriate surrounding information according to the vehicle speed.

[0048] When the vehicle speed is low (e.g., less than 20 km / h), the vehicle is often traveling on narrow roads in residential areas, etc., and by setting the conversion center position downward, it is possible to obtain accurate information about the surroundings near the vehicle. On the other hand, when the vehicle speed is high, the vehicle is often traveling on main roads, etc., and by shifting the conversion center position upward, it is possible to obtain accurate information about the surroundings farther away.

[0049] [Variations] In the above embodiment, an example was described in which the transformation center position was switched depending on whether the vehicle speed was above a threshold value or not, but when the vehicle speed was above a threshold value, the transformation center position may be changed continuously depending on the vehicle speed.

[0050] For example, when the vehicle speed is equal to or higher than a threshold, the higher the speed of the vehicle 1, the higher the transformation center position of the fisheye camera 42 capturing the right side may be set relative to a predetermined position (the position of the correction center point 601 shown in FIG. 6). That is, when the speed is less than the threshold, the transformation center position of the fisheye camera 42 is the position of the correction center point 601. When the speed is equal to or higher than the threshold but is low, the transformation center position of the fisheye camera 42 may be set to be closer to the position of the correction center point 601, and to continuously change (move) upward from the correction center point 601 to the position of the correction center point 301 shown in FIG. 3 (or to a position even higher than that) as the speed increases. The same applies to the fisheye camera 44 capturing the left side. This makes it possible to accurately acquire more appropriate surrounding information according to the vehicle speed.

[0051] Similarly, when the vehicle speed is equal to or higher than a threshold, the higher the speed of the vehicle 1, the higher the transformation center position of the fish-eye camera 41 that captures the forward view may be set relative to a predetermined position (the position of the correction center point 701 shown in FIG. 7). That is, when the speed is less than the threshold, the transformation center position of the fish-eye camera 41 is the position of the correction center point 701. When the speed is equal to or higher than the threshold but is low, the transformation center position of the fish-eye camera 41 may be set to be closer to the position of the correction center point 701, and to continuously change (move) upward from the correction center point 701 to the position of the correction center point 751 shown in FIG. 7(b) (or to a position even higher than that) as the speed increases. The same applies to the fish-eye camera 43 that captures the rearward view. This makes it possible to accurately acquire more appropriate surrounding information according to the vehicle speed.

[0052] In addition, in the above embodiment, an example was described in which processing was switched depending on the relative magnitude relationship between the vehicle speed and the threshold value, but the system may be configured so that, regardless of the vehicle speed threshold value, the transformation center position of each fisheye camera is set lower the lower the speed, and higher the speed the higher the speed.

[0053] In addition, the frequency of image conversion of the fisheye cameras 42, 44 arranged on the left and right sides of the vehicle 1 may be controlled to be lower than the frequency of image conversion of the fisheye cameras 41, 43 arranged at the front and rear of the vehicle 1. For example, when driving on a one-lane road or when lane changes are not performed, changes in the environment (surrounding information) in the front-rear direction often have a greater impact on driving than changes in the environment (surrounding information) in the left-right direction. Therefore, by increasing the frequency of image conversion in the front-rear direction (frequency of image acquisition) compared to the left-right direction, it is possible to acquire more accurate information in a timely manner, and by suppressing the frequency of conversion in the left-right direction, it is possible to reduce the processing load.

[0054] In the above embodiment, the ECU 22 and the ECU 23 perform processes separately. However, they may be configured as a single ECU, and the single ECU may perform the processes of the above embodiment or the modified examples.

[0055] In addition, the shape, position, and size of the rectangular area for cutting out shown in the above embodiment are merely examples, and are not limited to the illustrated examples. A rectangular area larger or smaller than the illustrated rectangle may be used. For example, in Figs. 3 and 6, a rectangular area is set on the right side of the fisheye image, but the rectangular area may be set in the center as in Figs. 7(a) and 7(b), or may be set on the left side. Various modifications are possible.

[0056] (Other embodiments) Furthermore, a program for realizing one or more functions described in each embodiment can be supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device can read and execute the program. The present invention can also be realized in such an embodiment.

[0057] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0058] <Summary of the embodiment> The control device (e.g. 2) according to the first aspect comprises: A control device that controls photography using fisheye cameras (e.g., 41 to 44) arranged in the front, rear, and sides of a vehicle (e.g., 1), Acquisition means (e.g., 22, 23) for acquiring information on the speed of the vehicle; A control means (e.g., 22, 23) for controlling a center position for converting a fisheye image of each fisheye camera into a planar image based on the speed; Equipped with.

[0059] This makes it possible to accurately obtain appropriate surrounding information according to the vehicle speed.

[0060] In the control device according to the second aspect (e.g. 2), The control means sets the center position of each fish-eye camera lower as the speed decreases (for example, 601, 701).

[0061] This makes it possible to obtain accurate information about the surroundings near the vehicle when the vehicle is traveling at low speeds, such as when passing through narrow roads, during parking assistance, or during automatic parking.

[0062] In the control device according to the third aspect (e.g. 2), The control means sets the center position of each fish-eye camera higher as the speed increases (for example, 301, 751).

[0063] This makes it possible to obtain accurate information about the surroundings far from the vehicle when traveling at high speeds. Therefore, when traveling at high speeds, such as when traveling on a main road or an expressway, it is possible to obtain accurate information about the surroundings far from the vehicle.

[0064] In the control device according to the fourth aspect (e.g. 2), When the speed is less than a threshold value (e.g., 20 km / h), the control means sets the center position of each fish-eye camera to a predetermined position (e.g., 601, 701), and when the speed is equal to or greater than the threshold value, the control means sets the center position of each fish-eye camera higher than the predetermined position the higher the speed (e.g., 301, 751).

[0065] In this way, the conversion center position is set to a predetermined lower position when traveling at low speeds, and the conversion center position gradually moves upward when the vehicle speed exceeds a certain level. Therefore, when traveling at low speeds such as when passing through narrow roads, using parking assistance, or during automatic parking, peripheral information near the vehicle can be obtained with high accuracy at the predetermined conversion center position, and when traveling at high speeds such as when traveling on main roads or expressways, peripheral information distant from the vehicle can be obtained with high accuracy.

[0066] In the control device (e.g. 2) according to the fifth aspect, When the speed is equal to or greater than the threshold value, the control means sets the center position of each fisheye camera higher relative to the specified position the higher the speed, and sets the center positions of each fisheye camera arranged at the front and rear of the vehicle higher than the center positions of each fisheye camera arranged to the sides of the vehicle.

[0067] At high speeds, changes in peripheral information in the longitudinal direction have a greater impact on driving than changes in the lateral direction. Therefore, by setting the transformation center position in the longitudinal direction higher than the transformation center position in the lateral direction, it is possible to obtain peripheral information that is more appropriate to the situation.

[0068] In the control device (e.g. 2) according to the sixth aspect, The control means sets the center position of each fish-eye camera to a predetermined position when the speed is less than a threshold, and sets the center position of each fish-eye camera above the predetermined position when the speed is equal to or greater than the threshold.

[0069] By switching the transformation center position depending on the relationship between the vehicle speed and a threshold value, it is possible to accurately obtain information about the surroundings near the vehicle when traveling at low speeds, and to accurately obtain information about the surroundings far from the vehicle when traveling at high speeds.

[0070] In the control device (e.g. 2) according to the seventh aspect, The control means controls the frequency of image conversion by each fish-eye camera disposed on the side of the vehicle so that it is lower than the frequency of image conversion by each fish-eye camera disposed in front of and behind the vehicle.

[0071] When driving on a one-lane road or when lane changes are not required, changes in the front-rear environment (surrounding information) often have a greater impact on driving than changes in the left-right environment (surrounding information). By increasing the frequency of image conversion in the front-rear direction (image acquisition frequency) compared to the left-right direction, it is possible to obtain more accurate information in a timely manner, and by reducing the frequency of image conversion in the left-right direction, it is possible to reduce the processing load.

[0072] A method of operating a control device (e.g., 2) according to an eighth aspect includes: A method for controlling a control device (e.g., 2) that controls photography using fisheye cameras installed in the front, rear, and sides of a vehicle (e.g., 1), comprising: An acquisition step (e.g., S401) of acquiring information on the speed of the vehicle; A control step (e.g., S402) of controlling a center position for converting a fisheye image of each fisheye camera into a planar image based on the speed; has.

[0073] This makes it possible to accurately obtain appropriate surrounding information according to the vehicle speed.

[0074] A program according to a ninth aspect comprises: A program for causing a computer to function as the control device according to any one of the first to seventh aspects.

[0075] This makes it possible to realize the operation of the vehicle control device by a computer. [Explanation of symbols]

[0076] 1: vehicle, 22, 23: ECU, 41-44: fisheye camera, 7c: vehicle speed sensor

Claims

1. A control device that controls photography using fisheye cameras arranged at the front, rear, and sides of a vehicle, An acquisition means for acquiring information on the speed of the vehicle; a control means for controlling a center position for converting a fisheye image of each fisheye camera into a planar image based on the speed; Equipped with the control means sets the center position of each fish-eye camera to a predetermined position when the speed is less than a threshold value, and sets the center position of each fish-eye camera higher relative to the predetermined position as the speed is higher when the speed is equal to or greater than the threshold value; The control device is characterized in that, when the speed is equal to or greater than the threshold value, the control means sets the center position of each fisheye camera higher relative to the specified position the higher the speed, and sets the center positions of each fisheye camera arranged at the front and rear of the vehicle higher than the center positions of each fisheye camera arranged to the sides of the vehicle.

2. A control device for controlling photography using fisheye cameras arranged at the front, rear, and sides of a vehicle, comprising: An acquisition means for acquiring information on the speed of the vehicle; a control means for controlling a frequency of image conversion for converting a fisheye image of each fisheye camera into a planar image based on the speed; Equipped with A control device characterized in that the control means controls the frequency of image conversion of each fisheye camera arranged on the side of the vehicle to be lower than the frequency of image conversion of each fisheye camera arranged at the front and rear of the vehicle.

3. The control device described in Claim 2, characterized in that the control means controls the frequency of image conversion of each fisheye camera arranged on the side of the vehicle when traveling on a one-lane road so that it is lower than the frequency of image conversion of each fisheye camera arranged in front of and behind the vehicle.

4. A method for controlling an image capturing device that captures images using fisheye cameras disposed at the front, rear, and sides of a vehicle, comprising: acquiring information on the speed of the vehicle; a control step of controlling a center position for converting a fisheye image of each fisheye camera into a planar image based on the speed; having In the control step, when the speed is less than a threshold value, the center position of each fish-eye camera is set to a predetermined position, and when the speed is equal to or greater than the threshold value, the center position of each fish-eye camera is set higher relative to the predetermined position as the speed increases, A method for operating a control device, characterized in that, in the control process, when the speed is equal to or greater than the threshold value, the higher the speed, the higher the center position of each fisheye camera is set relative to the specified position, and the center positions of each fisheye camera arranged at the front and rear of the vehicle are set higher than the center positions of each fisheye camera arranged to the sides of the vehicle.

5. A method for controlling an image capturing apparatus using fisheye cameras disposed at the front, rear, and sides of a vehicle, comprising: acquiring information on the speed of the vehicle; a control step of controlling a frequency of image conversion for converting the fisheye images of each fisheye camera into planar images based on the speed; having A method for operating a control device, characterized in that in the control process, the frequency of image conversion of each fisheye camera arranged on the side of the vehicle is controlled to be lower than the frequency of image conversion of each fisheye camera arranged in front of and behind the vehicle.

6. A program for causing a computer to function as the control device according to any one of claims 1 to 3.

Images