Control device, method of operation of control device, and storage medium

Abstract

The present application relates to a control device, a method of operation of the control device, and a storage medium. Provided is a technique for obtaining appropriate surrounding information with good accuracy according to vehicle speed. A control device controls imaging by fisheye cameras arranged in front and rear and side directions of a vehicle, and in the control device, there are provided: an acquisition unit that acquires information of a speed of the vehicle; and a control unit that controls a center position for converting a fisheye image of each fisheye camera into a planar image based on the speed.

Description

Technical Field

[0001] This invention relates to a control device, a method of operating the control device, and a storage medium. Background Technology

[0002] Objects are detected from images captured around a vehicle for driver assistance and other control purposes. To expand the detection range, a fisheye camera with a large field of view is considered. However, because fisheye cameras produce distorted images, detection accuracy may decrease when using object detection techniques that rely on undistorted images from conventional cameras.

[0003] Patent document 1 discloses the following technology: performing distortion correction processing on a distorted image and using the corrected image for object detection.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2008-48443 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] However, the technology described in Patent Document 1 does not consider which region to crop from the fisheye image based on factors such as vehicle speed for correction and transformation into a planar image. Therefore, it is difficult to accurately obtain appropriate surrounding information based on vehicle speed.

[0009] The present invention was made in view of the above-mentioned problems, and provides a technique for accurately acquiring appropriate surrounding information based on vehicle speed.

[0010] Solution for solving the problem

[0011] The control device of the present invention, which achieves the above-mentioned objective, controls the shooting of fisheye cameras arranged at the front, rear, and sides of a vehicle. The control device includes:

[0012] The acquisition unit acquires information about the speed of the vehicle; and

[0013] The control unit controls the center position for transforming the fisheye images of each fisheye camera into planar images based on the speed.

[0014] Furthermore, the control device operation method of the present invention, which achieves the above-mentioned objective, controls the capturing images of fisheye cameras arranged at the front, rear, and sides of a vehicle. The operation method of the control device includes:

[0015] The acquisition process includes acquiring information about the vehicle's speed; and

[0016] The control process, based on the speed, controls the center position of the image used to transform the fisheye image of each fisheye camera into a planar image.

[0017] The effects of the invention

[0018] According to the present invention, appropriate surrounding information can be obtained with good accuracy based on vehicle speed. Attached Figure Description

[0019] Figure 1 This is a block diagram of a vehicle according to one embodiment of the present invention.

[0020] Figure 2 (a) shows the horizontal shooting range of each fisheye camera. Figure 2 (b) shows the vertical shooting range of the fisheye camera on the right side. Figure 2 (c) shows the vertical shooting range of the rear fisheye camera.

[0021] Figure 3 This is an illustration of distortion correction processing for fisheye images.

[0022] Figure 4 This is a flowchart illustrating the steps of a process implemented by a control device according to an embodiment of the present invention.

[0023] Figure 5 This is a flowchart illustrating the steps of a process implemented by a control device according to an embodiment of the present invention.

[0024] Figure 6 This is a diagram showing an example of the transformation center position (correction center point) relative to a given position in the fisheye image on the right.

[0025] Figure 7 (a) is a diagram showing an example of the transformation center position (correction center point) relative to a given position in the foreground fisheye image. Figure 7 (b) is a diagram showing an example of a transformation center position (correction center point) that is set above a predetermined position relative to the fisheye image in front. Detailed Implementation

[0026] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the following embodiments are not intended to limit the scope of the invention as claimed, and the present invention does not require a combination of all the features described in the embodiments. Alternatively, two or more features from the plurality of features described in the embodiments may be arbitrarily combined. Additionally, the same or identical structures will be given the same reference numerals, and repeated descriptions will be omitted.

[0027] (Implementation Method)

[0028] <Structure>

[0029] Figure 1 This is a block diagram of a vehicle 1 according to one embodiment of the present invention. Figure 1 The diagram shows a general outline of vehicle 1 using top and side views. Vehicle 1 is exemplified as a four-wheeled passenger car. Vehicle 1 can be such a four-wheeled vehicle, or it can be a two-wheeled vehicle or other types of vehicles.

[0030] Vehicle 1 includes a vehicle control device 2 (hereinafter referred to as control device 2) that controls vehicle 1. Control device 2 includes multiple ECUs (Electronic Control Units) 20-29 that are communicatively connected via an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit), a memory such as a semiconductor memory, and an interface to external devices. The memory stores the program executed by the processor, the data used by the processor in processing, etc. Each ECU may also have multiple processors, memories, and interfaces. For example, ECU 20 has one or more processors 20a and one or more memories 20b. The processor 20a executes commands, including the program stored in the memory 20b, thereby ECU 20 performs processing. Alternatively, ECU 20 may have a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for executing the processing of ECU 20. The same applies to other ECUs.

[0031] The functions of each ECU 20 to 29 are explained below. Furthermore, the number of ECUs and their functions can be appropriately designed, allowing for a more detailed or integrated approach than this embodiment.

[0032] ECU 20 performs controls related to the autonomous driving of vehicle 1. In autonomous driving, it automatically controls at least one of the following: wheel turning, acceleration, and deceleration of vehicle 1. Alternatively, the autonomous driving performed by ECU 20 may include: autonomous driving that does not require driver intervention (also known as autonomous driving) and autonomous driving that assists the driver in performing driving operations (also known as driving assistance).

[0033] ECU 21 controls the electric power steering system 3. The electric power steering system 3 includes a mechanism that turns the front wheels according to the driver's driving operation (wheel operation) on the steering wheel 31. Additionally, the electric power steering system 3 includes: a motor that generates driving force to assist in wheel operation or to automatically turn the front wheels; and sensors that detect the wheel angle, etc. When the vehicle 1 is in automatic driving mode, ECU 21 automatically controls the electric power steering system 3 according to instructions from ECU 20 to control the direction of travel of the vehicle 1.

[0034] ECU 22 and ECU 23 control the detection unit that detects the surrounding environment of the vehicle and process the detection results. Vehicle 1 includes a standard camera 40 and four fisheye cameras 41-44 as detection units for detecting the surrounding environment of the vehicle. Standard camera 40, fisheye cameras 42 and 44 are connected to ECU 22. Fisheye cameras 41 and 43 are connected to ECU 23. ECU 22 and ECU 23 analyze the images captured by the standard camera 40 and the fisheye cameras 41-44, thereby extracting the outlines of body markers, lane markings (white lines, etc.) on the road.

[0035] Fisheye cameras 41-44 are cameras equipped with fisheye lenses. The structure of fisheye camera 41 will be described below. Other fisheye cameras 42-44 may also have the same structure. The field of view of fisheye camera 41 is larger than that of standard camera 40. Therefore, compared to standard camera 40, fisheye camera 41 can capture a larger area. Compared to images captured by standard camera 40, images captured by fisheye camera 41 have greater distortion. Therefore, it is possible that ECU 23 performs distortion reduction transformation processing (hereinafter referred to as "distortion correction processing") on the image captured by fisheye camera 41 before analyzing it. Alternatively, it is possible that ECU 22 does not perform distortion correction processing on the image captured by standard camera 40 before analyzing it. Thus, standard camera 40 is an imaging device that captures images that are not subject to distortion correction processing, while fisheye camera 41 is an imaging device that captures images that are subject to distortion correction processing. Alternatively, instead of the standard camera 40, other shooting devices that capture images that are not subject to distortion correction processing can be used, such as a camera equipped with a wide-angle lens or a telephoto lens.

[0036] A standard camera 40 is mounted at the center of the front of vehicle 1 to capture the surrounding environment in front of vehicle 1. A fisheye camera 41 is mounted at the center of the front of vehicle 1 to capture the surrounding environment in front of vehicle 1. Figure 1The diagram shows a standard camera 40 and a fisheye camera 41 arranged horizontally. However, the configuration of the standard camera 40 and the fisheye camera 41 is not limited to this; for example, these cameras can also be arranged vertically. Alternatively, at least one of the standard camera 40 and the fisheye camera 41 can be mounted on the front of the roof of the vehicle 1 (e.g., inside the front window). The fisheye camera 42 is mounted on the center of the right side of the vehicle 1 to capture the surrounding environment on the right side of the vehicle 1. The fisheye camera 43 is mounted on the center of the rear of the vehicle 1 to capture the surrounding environment at the rear of the vehicle 1. The fisheye camera 44 is mounted on the center of the left side of the vehicle 1 to capture the surrounding environment on the left side of the vehicle 1.

[0037] The types, number, and installation locations of cameras in vehicle 1 are not limited to the examples described above. Alternatively, vehicle 1 may include lidar (Light Detection and Ranging) or millimeter-wave radar as detection units used to detect objects around vehicle 1 or to measure distances to objects.

[0038] ECU 22 controls the standard camera 40, fisheye camera 42, and fisheye camera 44, and processes the detection results. ECU 23 controls the fisheye camera 41 and fisheye camera 43, and processes the detection results. By dividing the detection unit that detects the vehicle's surroundings into two systems, the reliability of the detection results can be improved.

[0039] ECU 24 controls gyroscope sensor 5, GPS sensor 24b, and communication device 24c, and processes the detection or communication results. Gyroscope sensor 5 detects the rotational motion of vehicle 1. Based on the detection results of gyroscope sensor 5, wheel speed, etc., the direction of travel of vehicle 1 can be determined. GPS sensor 24b detects the current position of vehicle 1. Communication device 24c wirelessly communicates with a server providing map information and traffic information to obtain this information. ECU 24 can access a map information database 24a built in memory, and ECU 24 performs path exploration from its current location to its destination. ECU 24, map database 24a, and GPS sensor 24b constitute a navigation device.

[0040] The ECU 25 is equipped with a communication device 25a for vehicle-to-vehicle communication. The communication device 25a wirelessly communicates with other vehicles in the vicinity and exchanges information between the vehicles.

[0041] ECU 26 controls powertrain 6. Powertrain 6 is a mechanism that outputs driving force to rotate the drive wheels of vehicle 1, and includes, for example, an engine and a transmission. ECU 26 controls the engine output in response to driver operations (accelerator operation or acceleration operation) detected by operation detection sensor 7a located on accelerator pedal 7A, and switches transmission gears based on information such as vehicle speed detected by vehicle speed sensor 7c. When vehicle 1 is in autonomous driving mode, ECU 26 controls powertrain 6 in response to instructions from ECU 20 to control the acceleration and deceleration of vehicle 1.

[0042] ECU 27 controls lighting devices (headlights, taillights, etc.), including the turn indicator 8. Figure 1 In the example, the direction indicator 8 is located at the front of the vehicle 1, the door rearview mirror, and the rear.

[0043] ECU 28 controls input / output device 9. Input / output device 9 outputs information to the driver and accepts information input from the driver. Voice output device 91 notifies the driver of information by sound. Display device 92 notifies the driver of information by displaying images. Display device 92 is, for example, disposed in front of the driver's seat, forming an instrument panel, etc. Moreover, although sound and display are exemplified here, information can also be notified by vibration or light. In addition, multiple combinations of sound, display, vibration, or light can be used to notify information. Furthermore, the combination or notification method can be different depending on the level of information to be notified (e.g., urgency). Input device 93 is disposed in a position that the driver can operate, and may be a group of switches for instructing the vehicle 1, or may include a voice input device.

[0044] ECU 29 controls the braking device 10 and the parking brake (not shown). The braking device 10 is, for example, a disc brake, installed on each wheel of the vehicle 1, applying resistance to the rotation of the wheels, thereby decelerating or stopping the vehicle 1. ECU 29 controls the operation of the braking device 10 in accordance with the driver's driving operation (brake operation) detected by the operation detection sensor 7b installed on the brake pedal 7B. When the vehicle 1 is in automatic driving mode, ECU 29 automatically controls the braking device 10 in accordance with the instructions from ECU 20 to control the deceleration and stopping of the vehicle 1. The braking device 10 and the parking brake can also be activated to maintain the vehicle 1 in a stopped state. In addition, if the transmission of the powertrain 6 is equipped with a parking lock mechanism, it can also be activated to maintain the vehicle 1 in a stopped state.

[0045] <Shooting Area>

[0046] Then refer to Figure 2 This describes the shooting range of the standard camera 40 and the fisheye cameras 41-44. Figure 2 (a) shows the horizontal shooting range of each camera. Figure 2 (b) shows the vertical shooting range of the fisheye camera 42 mounted on the right side of vehicle 1. Figure 2 (c) shows the vertical shooting range of the fisheye camera 43 mounted at the rear of vehicle 1.

[0047] First, refer to Figure 2 (a) illustrates the shooting range of the top view of vehicle 1 (i.e., the horizontal direction of vehicle 1). Standard camera 40 captures the scenery included in the shooting range 200. The shooting center 200C of standard camera 40 faces directly forward of vehicle 1. The horizontal field of view of standard camera 40 can be less than 90°, for example, it can be 45° or 30°.

[0048] Fisheye camera 41 captures the scenery within the shooting range 201. The shooting center 201C of fisheye camera 41 faces directly in front of vehicle 1. Fisheye camera 42 captures the scenery within the shooting range 202. The shooting center 202C of fisheye camera 42 faces the right side of vehicle 1. Fisheye camera 43 captures the scenery within the shooting range 203. The shooting center 203C of fisheye camera 43 faces directly behind vehicle 1. Fisheye camera 44 captures the scenery within the shooting range 204. The shooting center 204C of fisheye camera 44 faces the left side of vehicle 1. The horizontal field of view of fisheye cameras 41-44 can be greater than 0°, greater than 150°, or greater than 180°, for example, 180°. Figure 2 (a) shows an example of a fisheye camera 41-44 with a horizontal field of view of 180°.

[0049] The shooting range 201 can be divided into an area 201L located diagonally forward to the left of vehicle 1, an area 201F located directly in front of vehicle 1, and an area 201R located diagonally forward to the right of vehicle 1. The shooting range 202 can be divided into an area 202L located diagonally forward to the right of vehicle 1, an area 202F located to the right of vehicle 1, and an area 202R located diagonally behind to the right of vehicle 1. The shooting range 203 can be divided into an area 203L located to the right of vehicle 1, an area 203F located directly behind vehicle 1, and an area 203R located diagonally behind to the left of vehicle 1. The shooting range 204 can be divided into an area 204L located diagonally behind to the left of vehicle 1, an area 204F located to the left of vehicle 1, and an area 204R located diagonally forward to the left of vehicle 1. The shooting range 201 can also be equally divided into three areas 201L, 201F, and 201R (i.e., in a manner that makes the field of view of each area equal). The other shooting ranges 202 to 204 can also be equally divided into three.

[0050] The standard camera 40 and the fisheye cameras 41-44 have shooting ranges 200-204 as described above, thus the shooting ranges of the two cameras respectively cover the front of the vehicle 1 and four diagonal directions. Specifically, the shooting range 200 of the standard camera 40 and the area 201F of the shooting range 201 of the fisheye camera 41 cover the front of the vehicle 1 on both sides. The area 201R of the shooting range 201 of the fisheye camera 41 and the area 202L of the shooting range 202 of the fisheye camera 42 cover the right diagonal front of the vehicle 1 on both sides. The same applies to the other three diagonal directions of the vehicle 1.

[0051] Then, refer to Figure 2 (b) and Figure 2 (c) indicates the vertical shooting range of vehicle 1. Figure 2 In (b), the vertical shooting range of the fisheye camera 42 is described. Figure 2 In section (c), the vertical shooting range of fisheye camera 43 is described. The vertical shooting range of other fisheye cameras 41 and 44 can also be the same.

[0052] The vertical field of view of the fisheye camera 41 to 44 can be greater than 90°, greater than 150°, or greater than 180°, for example, it can be 180°. Figure 2 (b) and Figure 2(c) shows an example where the vertical field of view of fisheye cameras 41-44 is 180°. In the illustrated example, the shooting center 203C of fisheye camera 43 is oriented downwards (towards the ground) relative to a direction parallel to the ground. Alternatively, the shooting center 203C of fisheye camera 43 may be oriented parallel to the ground, or it may be oriented upwards (towards the opposite side of the ground) relative to a direction parallel to the ground. Furthermore, the shooting centers 201C-204C of fisheye cameras 41-44 may also be oriented in different directions in the vertical direction.

[0053] Reference Figure 3 This describes the distortion correction processing applied to the images captured by fisheye cameras 41-44. Image 300 is an image of the scenery to the right of vehicle 1 captured by fisheye camera 42. As shown, image 300 exhibits significant distortion, particularly in the peripheral areas.

[0054] The ECU 22, connected to the fisheye camera 42, performs distortion correction processing (transformation from a fisheye image to a planar image) on the image 300. Specifically, the ECU 22 sets a point within the image 300 as a correction center point 301. The ECU 22 cuts out a rectangular region 302 centered on the correction center point 301 from the image 300. The ECU 22 performs distortion correction processing on this region 302, thereby generating a distortion-reduced image 303. The distortion correction process reduces distortion as the image approaches the correction center point 301, while distortion does not decrease or even increases at locations far from the correction center point 301. Therefore, in one embodiment, the ECU 22 sets the correction center point 301 in a region of interest within the environment surrounding the vehicle 1 and generates a distortion-reduced image for that region.

[0055] <Processing>

[0056] Next refer to Figure 4 The flowchart illustrates the processing steps implemented by the control device 2 in this embodiment.

[0057] In S401, ECU 22 and ECU 23 acquire the speed information of vehicle 1 detected by vehicle speed sensor 7c. The speed acquired here is the current driving speed of vehicle 1.

[0058] In S402, ECU 22 and ECU 23, based on the vehicle 1 speed information acquired in S401, control the transformation center positions for converting the fisheye images captured by each fisheye camera (fisheye cameras 41-44) into planar images. ECU 22 sets the transformation center positions for converting the right-side fisheye image captured by fisheye camera 42 (capturing the right side of vehicle 1) and the left-side fisheye image captured by fisheye camera 44 (capturing the left side of vehicle 1) into planar images. Similarly, ECU 23 sets the transformation center positions for converting the frontal fisheye image captured by fisheye camera 41 (capturing the front of vehicle 1) and the rearal fisheye image captured by fisheye camera 43 (capturing the rear of vehicle 1) into planar images. (See reference...) Figure 5 A detailed example of this step's processing will be described later. Furthermore, image transformation is performed based on the transformation center position set in this step, and the transformed image is used to perform various actions such as acquiring surrounding information and providing driving assistance. That concludes the explanation. Figure 4 The series of processes have concluded.

[0059] Then, refer to Figure 5 The flowchart illustrates the processing steps implemented by the control device 2 in this embodiment. Figure 5 The processing shows Figure 4 Detailed example of the processing of S402.

[0060] In S4021, ECU 22 and ECU 23 determine whether the speed of vehicle 1 obtained in S401 is above a threshold (e.g., 20 km / h). If the speed of vehicle 1 is above the threshold, proceed to S4023. On the other hand, if the speed of vehicle 1 is below the threshold, proceed to S4022.

[0061] In S4022, ECU 22 sets the transformation center position for transforming the fisheye image on the right side captured by fisheye camera 42 into a planar image and the transformation center position for transforming the fisheye image on the left side captured by fisheye camera 44 into a planar image to predetermined positions.

[0062] here, Figure 6 Image 600 is an image of the scenery to the right of vehicle 1 captured by fisheye camera 42. ECU 22, connected to fisheye camera 42, sets a point within image 600 as a correction center point 601. ECU 22 cuts out a rectangular region 602 centered on the correction center point 601 from image 600 and performs distortion correction processing on region 602, thereby generating a distortion-reduced image (planar image). Here, the correction center point 601 is a predetermined position when the speed of vehicle 1 is less than a threshold. Figure 6 The example is the transformation center position associated with the image on the right, and the same applies to the image on the left. The transformation center positions can be set at the same height in the image (a predetermined height from the bottom), or they can be at their own different predetermined positions.

[0063] Furthermore, in S4022, ECU 23 sets the transformation center position for transforming the fisheye image captured by fisheye camera 41 in front into a planar image and the transformation center position for transforming the fisheye image captured by fisheye camera 43 in the rear into a planar image to predetermined positions.

[0064] here, Figure 7 Image 700 in (a) is an image of the scenery in front of vehicle 1 captured by fisheye camera 41. The image shows the rear of the leading vehicle. ECU 23, connected to fisheye camera 41, sets a point within image 700 as a correction center point 701. ECU 23 cuts out a rectangular region 702 centered on the correction center point 701 from image 700 and performs distortion correction processing on region 702, thereby generating a distortion-reduced image (planar image). Here, the correction center point 701 is a predetermined position when the speed of vehicle 1 is less than a threshold. Figure 7 Example (a) is an example of the transformation center position related to the foreground image, and the same applies to the background image. The transformation center positions can be set at the same height in the image (a predetermined height from the bottom), but they can also be at their own different predetermined positions.

[0065] In S4023, ECU 22 sets the transformation center positions for transforming the right-side fisheye image captured by fisheye camera 42 into a planar image and the transformation center positions for transforming the left-side fisheye image captured by fisheye camera 44 into a planar image, respectively, to be higher than the predetermined positions when the speed of vehicle 1 is less than a threshold. For example, the transformation center position of the right-side fisheye image is set to... Figure 3 The position of the correction center point 301 shown is (i.e., above the position of the correction center point 601). Figure 3 The example is the transformation center position associated with the image on the right, and the same applies to the image on the left. The transformation center positions can be set at the same height in the image (a predetermined height from the bottom), or they can be at different locations.

[0066] Furthermore, in S4023, ECU 23 sets the transformation center positions for converting the forward fisheye image captured by fisheye camera 41 into a planar image and the transformation center positions for converting the rear fisheye image captured by fisheye camera 43 into a planar image, respectively, to be higher than the predetermined positions when the speed of vehicle 1 is less than a threshold. For example, the transformation center position of the forward fisheye image is set to... Figure 7 The position of the correction center point 751 in the rectangular region 752 of the image 750 shown in (b) (i.e., above the position of the correction center point 701). Figure 7 Example (b) illustrates the transformation center location relative to the foreground image; the same applies to the background image. The transformation center locations can be set at the same height in the image (a predetermined height from the bottom), or they can be at their own different predetermined positions. (The above...) Figure 5 The processing is now complete.

[0067] As explained above, in this embodiment, the position of the transformation center used to transform the fisheye images captured by each fisheye camera into a planar image is controlled based on the vehicle's speed information. For example, the transformation center position is set at a predetermined position or set higher than the predetermined position, depending on whether the vehicle speed is above a threshold. This allows for the accurate acquisition of appropriate surrounding information based on the vehicle speed.

[0068] When the vehicle speed is low (e.g., less than 20 km / h), it is mostly traveling on narrow roads such as residential areas. Setting the change center position lower allows for accurate acquisition of surrounding information near the vehicle. On the other hand, when the vehicle speed is high, it is mostly traveling on main roads. Shifting the change center position higher allows for accurate acquisition of surrounding information further away.

[0069] [Variation Example]

[0070] Furthermore, in the above embodiments, an example of switching the transformation center position based on whether the vehicle speed is above a threshold was described. However, it is also possible to continuously change the transformation center position when the vehicle speed is above a threshold based on the vehicle speed.

[0071] For example, it can be configured such that, when the vehicle speed is above a threshold, the higher the speed of vehicle 1, the higher the position of the transformation center of the fisheye camera 42 that captures images on the right side will be relative to a predetermined position. Figure 6The position of the correction center point 601 shown is higher up. That is, when the speed is less than the threshold, the transformation center position of the fisheye camera 42 is the position of the correction center point 601. Moreover, it can also be set as follows: when the speed is above the threshold but still low, the transformation center position of the fisheye camera 42 is close to the correction center point 601, and as the speed increases, the transformation center position of the fisheye camera 42 moves from the correction center point 601 towards... Figure 3 The position of the correction center point 301 shown (or a position higher than it) changes (moves) continuously upwards. The same applies to the fisheye camera 44 that captures images to the left. Thus, appropriate surrounding information can be obtained with good accuracy based on the vehicle speed.

[0072] Similarly, it can be configured such that, when the vehicle speed is above a threshold, the higher the speed of vehicle 1, the higher the position of the transformation center of the fisheye camera 41 that captures images ahead is relative to a predetermined position. Figure 7 The position of the correction center point 701 shown is higher up. That is, when the speed is less than the threshold, the transformation center position of the fisheye camera 41 is the position of the correction center point 701. Moreover, it can also be set as follows: when the speed is above the threshold but still low, the transformation center position of the fisheye camera 41 is close to the position of the correction center point 701, and as the speed increases, the transformation center position of the fisheye camera 41 moves from the correction center point 701 towards... Figure 7 The position of the correction center point 751 shown in (b) changes (moves) continuously upwards. The same applies to the fisheye camera 43 that captures images of the rear. Thus, appropriate surrounding information can be obtained with good accuracy based on the vehicle speed.

[0073] Furthermore, in the above embodiment, an example of switching processing based on the relationship between the vehicle speed and the threshold was described. However, it can also be configured such that, regardless of the threshold for vehicle speed, the lower the speed, the lower the transformation center position of each fisheye camera is set, and the higher the speed, the higher the transformation center position of each fisheye camera is set.

[0074] Alternatively, control can be implemented such that the image transformation frequency of the fisheye cameras 42 and 44 positioned on the left and right sides of vehicle 1 is lower than the image transformation frequency of the fisheye cameras 41 and 43 positioned at the front and rear of vehicle 1. For example, when driving on a one-way road without changing lanes, changes in the environment (surrounding information) in the front and rear directions have a greater impact on driving than changes in the environment (surrounding information) in the left and right directions. Therefore, by increasing the image transformation frequency (image acquisition frequency) in the front and rear directions compared to the left and right directions, more accurate information can be acquired in real time, and the processing load can be reduced by suppressing the transformation frequency in the left and right directions.

[0075] Furthermore, in the above embodiments, examples of ECU 22 and ECU 23 performing processing separately have been described. However, it is also possible to configure these ECUs into a single ECU, and have one ECU perform the processing of the above embodiments or variations.

[0076] Furthermore, the shape, position, and size of the rectangular region for cutting shown in the above embodiments are merely examples and are not limited to the illustrated example. A rectangular region larger than the illustrated rectangle can be used, or a smaller rectangle can be used. For example, in Figure 3 as well as Figure 6 In the image, a rectangular area is defined on the right side, but it can also be like... Figure 7 (a) and Figure 7 As shown in (b), a rectangular area can be set in the center or on the left side. Various changes can be made.

[0077] (Other implementation methods)

[0078] Furthermore, a program that implements one or more functions described in each embodiment is provided to the system or device via a network or storage medium, and one or more processors in the computer of the system or device can read and execute the program. The present invention can be implemented in this manner.

[0079] The invention is not limited to the above-described embodiments, and various modifications and alterations can be made within the scope of the invention's intent.

[0080] <Summary of Implementation Methods>

[0081] The control device (e.g., 2) of the first type controls the shooting of fisheye cameras (e.g., 41-44) arranged in front of, behind, and to the sides of the vehicle (e.g., 1), and the control device includes:

[0082] Acquisition units (e.g., 22, 23) acquire information about the speed of the vehicle; and

[0083] A control unit (e.g., 22, 23) controls the center position for transforming the fisheye images from each fisheye camera into planar images based on the speed.

[0084] Therefore, it is possible to obtain appropriate surrounding information with good accuracy based on vehicle speed.

[0085] In the control device of the second method (e.g. 2), the lower the speed, the lower the center position (e.g. 601, 701) of each fisheye camera is set.

[0086] Therefore, at low speeds, it can accurately acquire information about the vehicle's surroundings. This is particularly useful when navigating narrow roads, during assisted parking, or during automatic parking, where the vehicle's surroundings can be acquired with high accuracy.

[0087] In the third type of control device (e.g., 2), the higher the speed, the higher the control unit sets the center position (e.g., 301, 751) of each fisheye camera.

[0088] Therefore, at high speeds, it can accurately acquire information about the vehicle's surroundings in the distance. Consequently, when driving on main roads or highways at high speeds, it can accurately acquire information about the vehicle's surroundings in the distance.

[0089] In the control device of the fourth method (e.g., 2), when the speed is less than a threshold (e.g., 20 km / h), the control unit sets the center position of each fisheye camera to a predetermined position (e.g., 601, 701). When the speed is above the threshold, the higher the speed, the higher the center position of each fisheye camera (e.g., 301, 751) is set relative to the predetermined position.

[0090] In this way, at low speeds, the predetermined lower position is set as the transformation center position. Once the vehicle reaches a certain speed, the transformation center position is gradually moved upwards. Therefore, at low speeds, such as when passing through narrow roads, during assisted parking, or during automatic parking, the predetermined transformation center position can be used to accurately obtain surrounding information near the vehicle. At high speeds, such as when driving on main roads or highways, the surrounding information far away from the vehicle can be accurately obtained.

[0091] In the control device of the fifth method (e.g. 2), when the speed is above the threshold, the higher the speed, the higher the control unit sets the center position of each fisheye camera to be relative to the predetermined position, and sets the center position of each fisheye camera arranged at the front and rear of the vehicle to be higher than the center position of each fisheye camera arranged on the side of the vehicle.

[0092] At high speeds, changes in surrounding information in the front-back direction have a greater impact on driving than changes in the left-right direction. Setting the center of change for the front-back direction higher than that for the left-right direction allows for the acquisition of more appropriate surrounding information that is relevant to the situation.

[0093] In the control device of the sixth method (e.g. 2), when the speed is less than a threshold, the control unit sets the center position of each fisheye camera to a predetermined position, and when the speed is above the threshold, the control unit sets the center position of each fisheye camera to be higher than the predetermined position.

[0094] The center position is switched based on the relationship between vehicle speed and threshold value, thus enabling accurate acquisition of surrounding information near the vehicle at low speeds and accurate acquisition of surrounding information far from the vehicle at high speeds.

[0095] In the control device of the seventh method (e.g. 2), the control unit controls the image transformation frequency of each fisheye camera arranged on the side of the vehicle to be lower than the image transformation frequency of each fisheye camera arranged at the front and rear of the vehicle.

[0096] When driving on a one-way, single-lane road without changing lanes, changes in the front-to-back environment (surrounding information) have a greater impact on driving than changes in the left-to-right environment (surrounding information). A higher frequency of image transformations (image acquisition frequency) in the front-to-back direction compared to the left-to-right direction allows for the immediate acquisition of more accurate information, while suppressing the frequency of image transformations in the left-to-right direction reduces the processing load.

[0097] The operation method of the control device (e.g., 2) of the eighth type, which controls the shooting of fisheye cameras arranged in front of, behind, and to the side of a vehicle (e.g., 1), includes:

[0098] The acquisition process (e.g., S401) acquires information about the speed of the vehicle; and

[0099] The control process (e.g., S402) controls the center position of the fisheye images from each fisheye camera to transform them into planar images based on the speed.

[0100] Therefore, it is possible to obtain appropriate surrounding information with good accuracy based on vehicle speed.

[0101] The ninth method's storage medium stores a program that enables the computer to function as a control device in any of the first to seventh methods.

[0102] Therefore, the actions of vehicle control devices can be implemented through computers.

[0103] Explanation of reference numerals in the attached figures

[0104] 1: Vehicle; 22, 23: ECU; 41-44: Fisheye camera; 7c: Vehicle speed sensor.

Claims

1. A control device that controls imaging of fisheye cameras arranged in front and rear and side of a vehicle, in the control device, comprising: an acquisition unit that acquires information of a speed of the vehicle; and a control unit that controls a center position for transforming a fisheye image of each fisheye camera into a planar image based on the speed, the control unit setting the center position of each fisheye camera at a predetermined position in a case where the speed is less than a threshold value, and setting the center position of each fisheye camera higher with respect to the predetermined position as the speed is higher in a case where the speed is equal to or more than the threshold value, and setting the center position of each fisheye camera arranged in front and rear of the vehicle higher than the center position of each fisheye camera arranged in side of the vehicle.

2. A control device that controls imaging of fisheye cameras arranged in front and rear and side of a vehicle, in the control device, comprising: an acquisition unit that acquires information of a speed of the vehicle; and a control unit that controls frequency of image transformation for transforming a fisheye image of each fisheye camera into a planar image based on the speed, the control unit controlling the frequency of image transformation of each fisheye camera arranged in side of the vehicle to be lower than the frequency of image transformation of each fisheye camera arranged in front and rear of the vehicle.

3. The control device according to claim 2, wherein the control unit controls the frequency of image transformation of each fisheye camera arranged in side of the vehicle to be lower than the frequency of image transformation of each fisheye camera arranged in front and rear of the vehicle in a case where the vehicle travels on a one-way one-lane road.

4. A method of operation of a control device that controls imaging of fisheye cameras arranged in front and rear and side of a vehicle, in the method of operation of the control device, comprising: an acquisition step of acquiring information of a speed of the vehicle; and a control step of controlling a center position for transforming a fisheye image of each fisheye camera into a planar image based on the speed, the control step setting the center position of each fisheye camera at a predetermined position in a case where the speed is less than a threshold value, and setting the center position of each fisheye camera higher with respect to the predetermined position as the speed is higher in a case where the speed is equal to or more than the threshold value, and setting the center position of each fisheye camera arranged in front and rear of the vehicle higher than the center position of each fisheye camera arranged in side of the vehicle.

5. A method of operation of a control device that controls imaging of fisheye cameras arranged in front and rear and side of a vehicle, in the method of operation of the control device, comprising: an acquisition step of acquiring information of a speed of the vehicle; and a control step of controlling frequency of image transformation for transforming a fisheye image of each fisheye camera into a planar image based on the speed, the control step controlling the frequency of image transformation of each fisheye camera arranged in side of the vehicle to be lower than the frequency of image transformation of each fisheye camera arranged in front and rear of the vehicle. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ In the control process, the frequency of image transformation of each fisheye camera arranged on the side of the vehicle is lower than the frequency of image transformation of each fisheye camera arranged on the front and rear of the vehicle.

6. A storage medium storing a program that causes a computer to function as the control device according to any one of claims 1 to 3.

Images