Vehicle, control device and control method

Abstract

This invention relates to a vehicle, a control device, and a control method. The control device is for a vehicle equipped with a camera, and includes: an image acquisition mechanism that periodically acquires images of the vehicle's external environment from the camera; a first identification mechanism that identifies targets outside the vehicle's environment based on the images acquired from the camera; a region control mechanism that controls the setting of a predetermined region for the images acquired from the camera; and a processing mechanism that performs distortion reduction processing on the predetermined region to reduce image distortion. Here, when a specific target identified by the first identification mechanism is included within the predetermined region, the region control mechanism changes the predetermined region in such a way that the region other than the specific target included within the predetermined region is increased.

Description

Technical Field

[0001] This invention relates to vehicles, control devices, and control methods. Background Technology

[0002] It is known that technology uses cameras to capture images of the exterior of a vehicle for driver assistance. Patent Document 1 proposes a technique for capturing a wide area behind a vehicle using a camera with a wide-angle lens, such as a fisheye lens. In this technique, the image captured by the camera with the wide-angle lens undergoes a coordinate transformation centered on the optical axis of the lens, and the central area is cropped for display, allowing the driver to focus on the central area of ​​the image when the vehicle is parked.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-171964 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, when performing coordinate transformation on images captured by a camera equipped with a wide-angle lens and cropping that portion to identify the outside of a vehicle, if there are obstructions (such as guardrails) in the captured image, the desired identification result may not be obtained for areas outside the obstructions where relatively important identification objects may exist, due to insufficient area.

[0008] The present invention was made in view of the above-mentioned problems, and its object is to realize a technique that can reduce the influence of occlusions in target object recognition using images captured by a camera equipped with a wide-angle lens.

[0009] means for solving problems

[0010] According to the present invention, a control device is provided.

[0011] It is a control device for a vehicle equipped with a camera, characterized in that,

[0012] The control device has:

[0013] An image acquisition mechanism that periodically acquires images of the outside world of the vehicle from the capturing device;

[0014] The first identification mechanism identifies external targets of the vehicle based on images acquired from the imaging device;

[0015] A region control mechanism that controls the setting of a predetermined region for images acquired from the imaging device; and

[0016] The processing unit performs distortion reduction processing on the predetermined area to reduce image distortion.

[0017] If a specific target object identified by the first identification agency is contained within the predetermined area, the area control agency changes the predetermined area in such a way that the area outside the specific target object contained within the predetermined area is increased.

[0018] Furthermore, according to the present invention, a vehicle is provided.

[0019] It is a vehicle equipped with a camera, characterized in that,

[0020] The vehicle has:

[0021] An image acquisition mechanism that periodically acquires images of the outside world of the vehicle from the capturing device;

[0022] The first identification mechanism identifies external targets of the vehicle based on images acquired from the imaging device;

[0023] A region control mechanism that controls the setting of a predetermined region for images acquired from the imaging device; and

[0024] The processing unit performs distortion reduction processing on the predetermined area to reduce image distortion.

[0025] If a specific target object identified by the first identification agency is contained within the predetermined area, the area control agency changes the predetermined area in such a way that the area outside the specific target object contained within the predetermined area is increased.

[0026] Furthermore, according to the present invention, a control device is provided.

[0027] It is a control device for a vehicle equipped with a camera, characterized in that,

[0028] The control device has:

[0029] An image acquisition mechanism that periodically acquires images of the outside world of the vehicle from the capturing device;

[0030] An identification mechanism that identifies external targets of the vehicle based on images acquired from the capturing device;

[0031] A region control mechanism that controls the setting of a predetermined region for images acquired from the imaging device; and

[0032] The processing unit performs distortion reduction processing on the predetermined area to reduce image distortion.

[0033] The area control mechanism sets the predetermined area in areas where the identification mechanism has not identified a specific target.

[0034] Furthermore, according to the present invention, a vehicle control method is provided.

[0035] It is a method for controlling a vehicle equipped with a camera, characterized in that,

[0036] The control method has the following characteristics:

[0037] The image acquisition step involves periodically acquiring images of the external environment of the vehicle from the capturing device.

[0038] In the first identification step, based on the image acquired from the imaging device, the target object outside the vehicle is identified;

[0039] A region control step, wherein the setting of a predetermined region for an image acquired from the capturing device is controlled; and

[0040] The processing step involves performing distortion reduction processing on the predetermined area to mitigate image distortion.

[0041] In the area control step, if a specific target object identified in the first identification step is included in the predetermined area, the predetermined area is changed in such a way that the area other than the specific target object included in the predetermined area is increased.

[0042] Furthermore, according to the present invention, a control method is provided.

[0043] It is a method for controlling a vehicle equipped with a camera, characterized in that,

[0044] The control method has the following characteristics:

[0045] The image acquisition step involves periodically acquiring images of the external environment of the vehicle from the capturing device.

[0046] The identification step involves identifying external targets of the vehicle based on images acquired from the imaging device.

[0047] A region control step, wherein the setting of a predetermined region for an image acquired from the capturing device is controlled; and

[0048] The processing step involves performing distortion reduction processing on the predetermined area to mitigate image distortion.

[0049] In the area control step, the predetermined area is set in the area where no specific target object was identified in the identification step.

[0050] Invention Effects

[0051] According to the present invention, in object recognition using images captured by a camera equipped with a wide-angle lens, the influence of obstructions can be reduced. Attached Figure Description

[0052] Figure 1 This is a diagram illustrating an example of the configuration of a vehicle according to an embodiment of the present invention.

[0053] Figures 2A to 2C This diagram illustrates the shooting range captured by the various cameras on the vehicle.

[0054] Figure 3 This is an illustration of an image captured by a fisheye camera, and an example of how that image has undergone distortion reduction processing.

[0055] Figures 4A to 4B This is a diagram illustrating the area surrounding the conversion center.

[0056] Figures 5A to 5B This diagram illustrates an example of changing the area around the conversion center in the presence of obstructions.

[0057] Figure 6 This diagram illustrates an example of changes to the area surrounding the conversion center, and an example of an image generated through distortion reduction processing.

[0058] Figure 7 This diagram illustrates other examples of changing the area around the conversion center in the presence of obstructions.

[0059] Figure 8 This diagram illustrates an example of changes to the area surrounding the conversion center, and an example of an image generated through distortion reduction processing.

[0060] Figure 9 This is a flowchart illustrating a series of actions for identifying the external environment of the vehicle involved in this embodiment.

[0061] Explanation of reference numerals in the attached figures

[0062] 1: Vehicle; 2: Control device; 41-44: Fisheye camera; 21-29: ECU. Detailed Implementation

[0063] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the following embodiments are not intended to limit the invention to which the technical solution pertains. Additionally, not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more features from the plurality of features described in the embodiments may be arbitrarily combined. Furthermore, identical or identical components are labeled with the same reference numerals, and repeated descriptions are omitted.

[0064] (Vehicle Composition)

[0065] Figure 1 This is a block diagram of a vehicle 1 according to one embodiment of the present invention. Figure 1 The diagram shows a general representation of vehicle 1 using top and side views. As an example, vehicle 1 is a four-wheeled passenger car of sedan type. Vehicle 1 can be a four-wheeled vehicle as described above, or it can be a two-wheeled vehicle or other types of vehicles.

[0066] Vehicle 1 includes a vehicle control device 2 (hereinafter referred to as control device 2) for controlling vehicle 1. Control device 2 includes multiple ECUs (Electronic Control Units) 20 to ECU 29 that are connected and able to communicate via an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), a memory such as semiconductor memory, and an interface for external devices. The memory stores the program executed by the processor, the data used by the processor in processing, etc. Each ECU may have multiple processors, memories, and interfaces. For example, ECU 20 has a processor 20a and a memory 20b. Processing based on ECU 20 is performed by the processor 20a executing instructions, including the program stored in memory 20b. Instead, ECU 20 may have a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for performing processing based on ECU 20. The same applies to other ECUs.

[0067] The functions of each ECU 20 to 29 will be explained below. Furthermore, the number of ECUs and their functions can be appropriately designed, and can be further refined or integrated than in this embodiment.

[0068] ECU 20 performs controls related to autonomous driving of vehicle 1. In autonomous driving, it automatically controls at least one of the steering, acceleration, and deceleration of vehicle 1. The autonomous driving performed by ECU 20 may include autonomous driving that does not require driving operations by the driver (also known as autonomous driving) and autonomous driving that assists the driver in driving operations (also known as driving assistance).

[0069] ECU 21 controls the electric power steering system 3. The electric power steering system 3 includes a mechanism for steering the front wheels according to the driver's driving operation (steering operation) on the steering wheel 31. Additionally, the electric power steering system 3 includes a motor that provides driving force for assisting steering operation or automatically steering the front wheels, a sensor for detecting the steering angle, etc. When the vehicle 1 is in automatic driving mode, ECU 21 automatically controls the electric power steering system 3 in accordance with instructions from ECU 20 to control the direction of travel of the vehicle 1.

[0070] ECU 22 and ECU 23 control the detection unit that detects the surrounding conditions of the vehicle and process the information of the detection results. As the detection unit for detecting the surrounding conditions of the vehicle, vehicle 1 includes one standard camera 40 and four fisheye cameras 41-44. Standard camera 40 and fisheye cameras 42 and 44 are connected to ECU 22. Fisheye cameras 41 and 43 are connected to ECU 23. ECU 22 and 23 can extract the outlines of objects and lane markings (white lines, etc.) on the road by analyzing the images captured by standard camera 40 and fisheye cameras 41-44. Furthermore, they can identify the type and area of ​​objects that are relatively close to the vehicle.

[0071] Fisheye cameras 41-44 refer to cameras equipped with fisheye lenses. The configuration of fisheye camera 41 will be described below. Other fisheye cameras 41-44 may also have the same configuration. The field of view of fisheye camera 41 is wider than that of standard camera 40. Therefore, fisheye camera 41 can capture a wider range than standard camera 40. The image captured by fisheye camera 41 has greater distortion compared to the image captured by standard camera 40. Therefore, although ECU 23 can use the image captured by fisheye camera 41 for simple object recognition, it can perform distortion reduction processing (hereinafter referred to as "distortion reduction processing") on the image before performing high-precision analysis. On the other hand, ECU 22 can perform distortion reduction processing on the image captured by standard camera 40 without performing distortion reduction processing on the image before analyzing it. Thus, standard camera 40 is an imaging device that captures images that are not subject to distortion reduction processing, while fisheye camera 41 is an imaging device that captures images that are subject to distortion reduction processing. Instead of the standard camera 40, other shooting devices that capture images of objects that do not undergo distortion reduction processing can be used, such as cameras equipped with wide-angle lenses or telephoto lenses.

[0072] A standard camera 40 is mounted at the center of the front of vehicle 1 to capture images of the surrounding environment in front of vehicle 1. A fisheye camera 41 is mounted at the center of the front of vehicle 1 to capture images of the surrounding environment in front of vehicle 1. Figure 1 In this design, the standard camera 40 and the fisheye camera 41 are shown arranged horizontally. However, the configuration of the standard camera 40 and the fisheye camera 41 is not limited to this; for example, they can also be arranged vertically. Furthermore, at least one of the standard camera 40 and the fisheye camera 41 can be mounted on the front of the roof of the vehicle 1 (e.g., inside the front window). The fisheye camera 42 is mounted on the center of the right side of the vehicle 1 to capture images of the surrounding environment on the right side. The fisheye camera 43 is mounted on the center of the rear of the vehicle 1 to capture images of the surrounding environment at the rear of the vehicle 1. The fisheye camera 44 is mounted on the center of the left side of the vehicle 1 to capture images of the surrounding environment on the left side of the vehicle 1.

[0073] The types, number, and installation locations of cameras in vehicle 1 are not limited to the examples described above. Furthermore, as a detection unit for detecting targets around vehicle 1 or measuring distances to targets, vehicle 1 may also include optical radar (Light Detection and Ranging) or millimeter-wave radar.

[0074] ECU 22 controls the standard camera 40 and fisheye cameras 42 and 44, and processes the information from the detection results. ECU 23 controls the fisheye cameras 41 and 43, and processes the information from the detection results. By dividing the detection unit that detects the vehicle's surroundings into two systems, the reliability of the detection results can be improved.

[0075] ECU 24 controls the gyroscope sensor 5, GPS sensor 24b, and communication device 24c, and processes the detection or communication results. The gyroscope sensor 5 detects the rotational motion of vehicle 1. It can determine the vehicle 1's route based on the detection results of the gyroscope sensor 5, wheel speed, etc. The GPS sensor 24b detects the current position of vehicle 1. The communication device 24c wirelessly communicates with a server providing map and traffic information to obtain this information. ECU 24 can access a map information database 24a built in memory to perform route searches from the current location to the destination, etc. ECU 24, map database 24a, and GPS sensor 24b constitute a navigation device.

[0076] The ECU 25 is equipped with a communication device 25a for vehicle-to-vehicle communication. The communication device 25a communicates wirelessly with other vehicles in the vicinity and exchanges information between vehicles.

[0077] ECU 26 controls the power unit 6. The power unit 6 is a mechanism that outputs driving force to rotate the drive wheels of vehicle 1, and includes, for example, an engine and a transmission. ECU 26 controls the engine output, for example, in response to driver operations (accelerator or acceleration) detected by the operation detection sensor 7a located on the accelerator pedal 7A, or switches transmission gears based on information such as vehicle speed detected by the vehicle speed sensor 7c. When vehicle 1 is in automatic driving mode, ECU 26 automatically controls the power unit 6 in response to instructions from ECU 20, controlling the acceleration and deceleration of vehicle 1.

[0078] ECU27 controls lighting devices (headlights, taillights, etc.), including the turn indicator 8 (turn signal). Figure 1 In the example, the direction indicator 8 is located at the front, door rearview mirror and rear of the vehicle 1.

[0079] ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and receives information from the driver. The sound output device 91 notifies the driver of information via sound. The display device 92 notifies the driver of information via image display. The display device 92 is, for example, positioned in front of the driver's seat, forming an instrument panel, etc. Furthermore, while sound and display are exemplified here, information can also be notified via vibration or light. Additionally, multiple combinations of sound, display, vibration, or light can be used to report information. Furthermore, the combination or reporting method can be different depending on the level of the information to be reported (e.g., urgency). The input device 93 is a switch assembly positioned within the driver's reach to provide instructions for the vehicle 1, but it may also include a sound input device.

[0080] ECU 29 controls the braking device 10 and the parking brake (not shown). The braking device 10 is, for example, a disc brake, installed on each wheel of the vehicle 1, which decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheels. ECU 29 controls the operation of the braking device 10 in accordance with the driver's driving operation (braking operation) detected by the operation detection sensor 7b installed on the brake pedal 7B. When the vehicle 1 is in automatic driving mode, ECU 29 automatically controls the braking device 10 in accordance with the instructions from ECU 20, controlling the deceleration and stopping of the vehicle 1. The braking device 10 and the parking brake can also operate to maintain the vehicle 1 in a stopped state. In addition, if the transmission of the power unit 6 is equipped with a parking lock mechanism, it can also be operated to maintain the vehicle 1 in a stopped state.

[0081] Reference Figures 2A to 2C The shooting range of the standard camera 40 and the fisheye cameras 41-44 is explained. Figure 2A This indicates the horizontal shooting range of each camera. Figure 2B The right side of vehicle 1 indicates the vertical shooting range of the fisheye camera 42. Figure 2C This indicates the vertical shooting range of the fisheye camera 43 installed at the rear of vehicle 1.

[0082] First, refer to Figure 2A The shooting range in the top view (i.e., the horizontal direction of vehicle 1) of vehicle 1 is described below. Standard camera 40 captures the scenery encompassed by the shooting range 200. The shooting center 200C of standard camera 40 faces directly forward of vehicle 1. The horizontal field of view of standard camera 40 can be less than 90°, for example, approximately 45° or 30°.

[0083] Fisheye camera 41 captures the scenery within the shooting range 201. The shooting center 201C of fisheye camera 41 faces directly in front of vehicle 1. Fisheye camera 42 captures the scenery within the shooting range 202. The shooting center 202C of fisheye camera 42 faces the right side of vehicle 1. Fisheye camera 43 captures the scenery within the shooting range 203. The shooting center 203C of fisheye camera 43 faces directly behind vehicle 1. Fisheye camera 44 captures the scenery within the shooting range 204. The shooting center 204C of fisheye camera 44 faces the left side of vehicle 1. The horizontal field of view of fisheye cameras 41-44 can be greater than 90°, greater than 150°, or greater than 180°, for example, approximately 180°. Figure 2A Examples of fisheye cameras 41-44 with a horizontal field of view of 180° are shown.

[0084] The shooting range 201 can be divided into an area 201L located diagonally forward to the left of vehicle 1, an area 201F located directly in front of vehicle 1, and an area 201R located diagonally forward to the right of vehicle 1. The shooting range 202 can be divided into an area 202L located diagonally forward to the right of vehicle 1, an area 202F located on the right side of vehicle 1, and an area 202R located diagonally backward to the right of vehicle 1. The shooting range 203 can be divided into an area 203L located diagonally backward to the right of vehicle 1, an area 203F located directly behind vehicle 1, and an area 203R located diagonally backward to the left of vehicle 1. The shooting range 204 can be divided into an area 204L located diagonally backward to the right of vehicle 1, an area 204F located on the left side of vehicle 1, and an area 204R located diagonally forward to the left of vehicle 1. The shooting range 201 can also be divided equally (i.e., with equal field of view in each area) into three areas: 201L, 201F, and 201R. Similarly, the other shooting ranges 202–204 can also be divided equally into three parts.

[0085] The standard camera 40 and the fisheye cameras 41-44 have shooting ranges 200-204 as described above, thus the front of vehicle 1 and the four oblique directions are included in the shooting range of two separate cameras. Specifically, the front of vehicle 1 is included in the shooting range 200 of the standard camera 40 and the area 201F of the shooting range 201 of the fisheye camera 41. The right oblique front of vehicle 1 is included in the area 201R of the shooting range 201 of the fisheye camera 41 and the area 202L of the shooting range 202 of the fisheye camera 42. The same applies to the other three oblique directions of vehicle 1.

[0086] Next, refer to Figure 2B as well as Figure 2CThe vertical shooting range of vehicle 1 will be explained. Figure 2B The section describes the vertical shooting range of the fisheye camera 42. Figure 2C The vertical shooting range of fisheye camera 43 is described below. The vertical shooting range of other fisheye cameras 41 and 44 can also be the same.

[0087] The vertical field of view of the fisheye camera 41-44 can be greater than 90°, greater than 150°, or greater than 180°, for example, it can be around 180°. Figure 2B as well as Figure 2C Examples are shown where the vertical field of view of fisheye cameras 41-44 is 180°. The shooting center 203C of fisheye camera 43 faces a direction parallel to the ground. Alternatively, the shooting center 203C of fisheye camera 43 may face downwards (towards the ground) or upwards (towards the opposite side of the ground) than the direction parallel to the ground. Furthermore, the shooting centers 201C-204C of fisheye cameras 41-44 may also face different directions vertically.

[0088] (Overview of distortion reduction processing)

[0089] Next, refer to Figures 3-8 A summary of distortion reduction processing for images captured by fisheye cameras 41-44, taking into account occlusion, is provided. Figure 3 As shown, image 300 is an image of the scenery to the right of vehicle 1, captured by fisheye camera 42. As shown, image 300 has significant distortion, particularly in the peripheral areas.

[0090] Figure 4A This diagram shows the view of vehicle 1 from above. Figure 4B This image shows a view of vehicle 1 from the rear. Image 300 is an image of the right side of vehicle 1 captured by fisheye camera 43.

[0091] The ECU22, connected to the fisheye camera 43, performs distortion reduction processing on the image 300. Specifically, as... Figure 3 As shown, ECU22 determines a point within image 300 as the conversion center 301. (As...) Figure 4A , Figure 4B As shown, the conversion center 301 is located on the right side of the shooting area 202 in the horizontal direction as viewed from the fisheye camera 42, and faces a direction parallel to the ground in the vertical direction. Furthermore, Figure 4A , Figure 4BThe example shown illustrates the application of this embodiment to 202R, but the embodiments shown below can also be applied to any of 202L, 202F, and 202R.

[0092] ECU22 extracts a region 302 centered on the conversion center 301 from image 300. Furthermore, for simplicity, region 302 is illustrated as a rectangle, but its shape can be different as long as it corresponds to image 303 described later. Figure 3 The shape shown. (As shown) Figure 4A As shown, region 302 corresponds to region 202T in region 202R, which is located on the right side of the shooting range 202 as viewed from the fisheye camera 42. ECU 22 generates a distortion-reduced image 303 by performing distortion reduction processing on region 302 centered on the conversion center 301. This image 303 is... Figure 4A The width of region 202T and Figure 4B The image corresponding to the height of region 202T.

[0093] As a result of distortion reduction processing, the closer to the conversion center 301, the more distortion is reduced; conversely, in areas far from the conversion center 301, distortion is not reduced or even increases. When the entire image 300 is considered as the object of distortion reduction processing, distortion increases in regions far from the conversion center 301. Therefore, even when using regions far from the conversion center 301 to analyze the external environment of the vehicle 1, high-precision analysis is not possible. Therefore, the ECU 22 sets the conversion center 301 as the region to be analyzed, performs distortion reduction processing on the region surrounding the conversion center 301, and uses the processed image to analyze the condition of the region to be analyzed.

[0094] Furthermore, when the ECU 22 wants to analyze the situation at the right front of the vehicle 1, it sets a conversion center 301 within the area 202L (e.g., the center of area 202L) encompassed by the shooting range 202 of the fisheye camera 42, performs distortion reduction processing on the area surrounding the conversion center 301, and uses the processed image to analyze the situation at the right front. When the ECU 22 wants to analyze the situation at the right front of the vehicle 1, it sets a conversion center 301 within the area 202F (e.g., the center of area 202F) encompassed by the shooting range 202 of the fisheye camera 42, performs distortion reduction processing on the area surrounding the conversion center 301, and uses the processed image to analyze the situation at the right front. When the ECU22 wants to analyze the situation to the right rear of the vehicle 1, it sets a conversion center 301 within the area 202R contained in the shooting range 202 of the fisheye camera 42 (for example, the center of the area 202R), performs distortion reduction processing on the area around the conversion center 301, and uses the processed image to analyze the situation to the right rear.

[0095] Thus, if the ECU22 is analyzing the situation on the right side of the vehicle and acquires an image from the fisheye camera 42, it sets the conversion center 301 to the direction required for analyzing the situation and performs distortion reduction processing on the area around the conversion center 301.

[0096] On the other hand, when acquiring images captured by the fisheye camera 42, sometimes there are obstructions 501, such as guardrails, in the area surrounding the set conversion center 301, and the area outside the obstruction becomes smaller (e.g., see reference). Figure 7 (See the image above). When an obstruction occupies a certain area of ​​the region 302 surrounding the conversion center 301, it is impossible to perform sufficient analysis of the area outside the obstruction, or the analysis processing of the area containing the obstruction is wasted. The obstruction 501 may exist, for example, between a roadway and a sidewalk, obstructing image recognition of objects or road users that may be present on the sidewalk. Besides guardrails, obstructions also include grass, trees, etc., with a height of 700mm or more. Obstructions include those extending along the direction of vehicle travel (e.g., horizontally).

[0097] When the region 302 contains an obstruction extending in the horizontal direction (such an obstruction is identified from the image 300 as described later), the ECU22 can change the area around the conversion center 301 to increase the area outside the obstruction.

[0098] For example, if ECU22 contains a horizontally extending obstruction in region 302, such as Figures 5A-5BAs shown, by changing the area surrounding the conversion center 301, the area outside of obstructions included in the area surrounding the conversion center 301 can be increased. For example, the ECU 22 changes the area surrounding the conversion center 301 so that the image 303 becomes a longer rectangle in the vertical direction. Figure 5A As shown, the area 202T surrounding the conversion center 301 in the horizontal plane of the vehicle 1, when viewed from above, is narrowed, as... Figure 5B As shown, the area 202T surrounding the conversion center 301 in the vertical plane of the vehicle 1 when viewed from the rear is widened.

[0099] Figure 6 Indicatively representing according to Figure 3 The example of image 300 shown is an example of changing the area surrounding the conversion center 301 from region 302 to region 302b. Region 302 corresponds to image 303, which is a rectangle longer in the horizontal direction, and conversely, region 302b corresponds to image 303b, which is a rectangle longer in the vertical direction. When a horizontally extending obstruction 501 is included on the lower side of region 302, the ECU 22 changes region 302 to region 302b, thereby increasing the area outside the obstruction included in region 302b. Thus, when the ECU 22 performs distortion reduction processing on region 302b surrounding the conversion center 301, the area outside the obstruction also increases in the processed image 303b. In this way, by changing the area 302 surrounding the conversion center to increase the area outside the obstruction, the influence of obstructions can be reduced in object recognition using images captured by a fisheye camera.

[0100] This process of changing the area around the transformation center 301 in such a way that the image 303 becomes a rectangle that is longer in the vertical direction is also useful when resolving the presence of traffic participants, etc., in the area where there is an obstruction extending in the horizontal direction.

[0101] Alternatively, other methods can be used to increase the area outside the obstruction by changing the area 302 surrounding the conversion center.

[0102] For example, such as Figure 7 As shown, when region 302 contains an obstruction extending in the horizontal direction (e.g., such an obstruction is identified from image 300 as described later), ECU 22 can change the position of conversion center 301 to increase the area outside the obstruction. Specifically, ECU 22 sets conversion center 301a so that its position in image 300 is higher than the position of conversion center 301, and performs distortion reduction processing on the area surrounding conversion center 301a.

[0103] Figure 8It schematically shows the following based on Figure 3 The example of image 300 shown is a case where the position of conversion center 301 is changed to the position of conversion center 301a (and the surrounding area is also changed). ECU 22 changes region 302 to region 302a, thereby increasing the area outside the occlusion 501 contained in region 302a. Therefore, since ECU 22 performs distortion reduction processing on the region 302a surrounding conversion center 301a, the area outside the occlusion also increases in the processed image 303a. Thus, by changing the position of conversion center 301 to increase the area outside the occlusion and changing the region 302 surrounding the conversion center, ECU 22 can reduce the influence of occlusions in object recognition using images captured by a fisheye camera.

[0104] (A series of actions to identify and process the vehicle's external environment)

[0105] Next, a series of actions for identifying the external environment of the vehicle will be explained. A series of actions for distortion reduction will also be explained. This processing is implemented by the processor 20a of each ECU 20 to ECU 29 of the control device 2 executing a program stored in memory 20b.

[0106] Furthermore, in the process of recognizing the external environment of the vehicle, it is possible to determine one or more regions from the images acquired from each camera that will be the targets of distortion reduction processing, based on the driving scenario of vehicle 1 (e.g., entering a T-junction, driving on a narrow road, making a right or left turn at an intersection, etc.). However, in the following description, for example, the case of performing distortion reduction processing on the 204F region on the side (left side) of vehicle 1 will be used as an example. In addition, in this process, the orientation of the conversion center viewed from the rear of vehicle 1 is set to the horizontal direction, and the area around the conversion center is set such that the image corresponding to that area becomes a rectangle extending in the horizontal direction after distortion reduction processing.

[0107] In S901, the control device 2 acquires the image captured by fisheye camera 44 among fisheye cameras 41-44. The image captured by fisheye camera 44 is also a highly distorted image, similar to the image 300 described above.

[0108] In S902, the control device 2 instructs the ECU 22 to perform processing to identify external targets (obstacles) of the vehicle 1 based on images captured by the fisheye camera 44. The external targets of the vehicle that are to be identified are obstructions such as guardrails and grass. In this embodiment, the obstructions (e.g., located between the road and the sidewalk) are assumed to be relatively close to the vehicle 1, so their presence can be identified even when using images captured by the fisheye camera 44. Obstruction identification can be achieved using a model that learns by using images from the fisheye camera containing the obstructions as training data, for example, employing known deep learning techniques. If an unknown fisheye camera image is input, the model, after learning, outputs the presence or absence of an obstruction in the image and the area of ​​that obstruction. The model can output more than one area where an obstruction exists.

[0109] In S903, the control device 2 determines whether the area surrounding the conversion center contains an occluder extending in a specific direction (e.g., horizontal). If the control device 2 determines that the area surrounding the conversion center contains an occluder, the process proceeds to S904; otherwise, the process proceeds to S906. The control device 2 determines whether the shape of the occluder identified by the learning model is an occluder extending in a specific direction (e.g., horizontal). Whether it is an occluder extending in a specific direction can be determined, for example, based on whether the horizontal length of the occluder's region in the image is greater than a predetermined length.

[0110] In S904, the control device 2 determines, based on the area of ​​obstruction identified in S902, whether there is an area without obstructions in the area surrounding the conversion center. For example, if the control device 2 determines that the area outside the obstructions in the area surrounding the conversion center is larger than a predetermined proportion (e.g., 15%), then an area without obstructions exists. If the control device 2 determines that an area without obstructions exists, the process proceeds to S905; otherwise, the process proceeds to S908.

[0111] In S905, control device 2 changes the area surrounding the conversion center. Control device 2, for example, by referring to... Figure 5A , Figure 5B as well as Figure 6 The method described here alters the area surrounding the conversion center. Specifically, when the area surrounding the conversion center contains obstructions, the method alters the area surrounding the conversion center by increasing the area outside the obstructions. This is achieved by using a reference... Figure 5A , Figure 5B as well as Figure 6 The method described herein can easily secure the area outside the obstruction in a vertical direction relative to the obstruction extending in the horizontal direction (especially above the obstruction).

[0112] At this time, the control device 2 changes the area surrounding the conversion center in such a way that the size of the image corresponding to the area surrounding the conversion center before the change is the same as the size of the image corresponding to the area surrounding the conversion center after the change. In this way, the size of the image after distortion reduction processing is the same before and after the change, thus keeping the load constant in the image recognition processing using that image. That is, the processing time for each image can be controlled within the required time, or the power consumption can be controlled within a predetermined level.

[0113] In S906, the control device 2 performs distortion reduction processing on the area surrounding the conversion center. Through the processing in S905, for example, the image corresponding to the area surrounding the conversion center is changed from a rectangle that is longer in the horizontal direction (image 303) to a rectangle that is longer in the vertical direction (image 303b).

[0114] In S907, the control device 2 identifies external targets based on an image that has undergone distortion reduction processing. Specifically, the identification of external targets is performed to identify the situation in areas other than obstructions (for the purpose of identifying the presence of traffic participants, etc.). The identification of traffic participants and targets can be achieved using a model trained on images containing the people or targets to be identified (equivalent to the distortion-corrected image) as training data, for example, employing well-known deep learning techniques. If an unknown image (the distortion-reduced image) is input, the model, after learning, outputs the presence or absence of the identified object within the image and the region of that object. The model can output more than one region where the identified object exists.

[0115] In S908, the control device 2 reduces the frequency of identifying external targets in the area surrounding the conversion center (due to the image being occupied by an obstruction). For example, the control device 2 reduces the frequency of identifying targets in the 204F area on the side (left side of the vehicle) of the vehicle 1 while increasing the frequency in the 204L area, or reduces the frequency of using the image of the side (left side of the vehicle) of the vehicle 1 for targets.

[0116] In S909, the control device 2 determines whether to end the action. If the action continues, the process returns to S901; otherwise, the process ends.

[0117] Furthermore, in the above description, this is used when changing the area around the conversion center in S905. Figure 5A , Figure 5B as well as Figure 6 The example shown illustrates the change method. However, it is also possible to use... Figure 7 as well as Figure 8The method shown changes the area surrounding the conversion center. That is, by moving the position of the conversion center in a direction perpendicular to the extension direction of the obstruction, the area surrounding the conversion center changes.

[0118] In addition to the methods described above, other methods can be used to change the area surrounding the conversion center. For example, control device 2 can also increase the area excluding obstructions included in the area surrounding the conversion center by setting the location of the conversion center to an area without obstructions. Alternatively, control device 2 can set the area surrounding the conversion center to an area in the image captured by the fisheye camera where no obstructions were detected. Even so, in images where distortion reduction processing is applied (the area of ​​obstructions is reduced or disappears), higher accuracy image resolution is possible.

[0119] Furthermore, in the above embodiment, the processing of images captured by cameras (fisheye cameras 42, 44) on the side of the vehicle 1 was described as an example. However, the above processing can also be applied to images captured from the front and rear of the vehicle 1. That is, if the images captured from the front and rear of the vehicle 1 contain obstructions extending in the horizontal direction, the area to which distortion reduction processing is applied can be controlled in such a way that the area outside the obstruction is enlarged.

[0120] Furthermore, the aforementioned external processing of the vehicle may not all be performed in vehicle 1. For example, when acquiring an image captured by a fisheye camera on vehicle 1, the acquired image may be sent to an external server (not shown), where all or part of the processing in S902 to S907 may be performed, and vehicle 1 may receive the processing result.

[0121] As explained above, in this embodiment, images of the outside world of the vehicle are periodically acquired from a wide-angle camera (fisheye camera), and targets (obstacles) outside the vehicle are identified based on the images acquired from the wide-angle camera. Furthermore, in this embodiment, for the images from the wide-angle camera, a predetermined region (the region surrounding the conversion center) is controlled, and distortion reduction processing is performed on this predetermined region to reduce image distortion. In this embodiment, if an obstruction is contained within the predetermined region (the region surrounding the conversion center), the predetermined region (the region surrounding the conversion center) is changed by increasing the area outside the obstruction. Therefore, in the image where distortion reduction processing is applied, (the area of ​​the obstruction is reduced) allows for more accurate image resolution. In other words, in target identification using images captured by a camera equipped with a wide-angle lens, the influence of obstructions can be reduced.

[0122] <Summary of Implementation Methods>

[0123] 1. The control device (e.g., 2) described above is a control device for a vehicle (e.g., 1) equipped with a shooting device (e.g., 41-44).

[0124] The control device has:

[0125] An image acquisition mechanism (e.g., 22, 23, S901) periodically acquires images of the outside world of the vehicle from the capturing device;

[0126] The first identification mechanism (e.g., 2, 22, 23, S902) identifies external targets of the vehicle based on images acquired from the imaging device;

[0127] A zone control mechanism (e.g., 2, 22, 23, S905) controls the setting of a predetermined zone for an image acquired from the imaging device; and

[0128] The processing unit (e.g., 2, 22, 23, S906) performs distortion reduction processing on a predetermined area to reduce image distortion.

[0129] If a specific target object identified by the first identification agency is contained within a predetermined area, the area control agency changes the predetermined area in such a way that the area outside the specific target object contained within the predetermined area is increased (e.g., S905).

[0130] According to this embodiment, the influence of obstructions can be reduced in target object recognition using images captured by a camera equipped with a wide-angle lens.

[0131] 2. In the above embodiments,

[0132] If a specific target object (e.g., an obstruction) is not contained in the first region (e.g., 302) of the image acquired by the imaging device (e.g., S903), the region control mechanism sets the first region as a predetermined region. If a specific target object is contained in the first region (e.g., S903 to S905), the region control mechanism changes the predetermined region from the first region (e.g., 302) to a second region (e.g., 302b, 302a) in a manner that increases the area other than the specific target object contained in the predetermined region.

[0133] According to this embodiment, distortion reduction processing can be performed directly when there are no obstructions, while the influence of obstructions can be reduced when there are obstructions.

[0134] 3. In the above embodiments,

[0135] When a specific target object is an obstruction extending along a first direction, the area control mechanism changes the predetermined area from a first area (e.g., 302) to a second area (e.g., 302b) in such a way that the image corresponding to the predetermined area (e.g., 303b) becomes an image that is longer along a second direction that is perpendicular to the first direction.

[0136] According to this embodiment, in a region extending in a direction perpendicular to the shield extending in a specific direction, the area outside the shield can be easily secured.

[0137] 4. In the above embodiments,

[0138] When a specific target object is an obstruction extending in the horizontal direction, the area control mechanism changes the predetermined area from a first area (e.g., 302) to a second area (e.g., 302b) in such a way that the image that is longer in the horizontal direction (e.g., 303b) becomes an image that is longer in the vertical direction.

[0139] According to this embodiment, in the area extending in a direction perpendicular to the horizontally extending barrier (especially above the barrier), the area outside the barrier can be easily secured.

[0140] 5. In the above embodiments,

[0141] The region control mechanism changes the predetermined region from the first region to the second region in such a way that the size of the image corresponding to the first region (e.g., 303) and the size of the image corresponding to the second region (e.g., 303b) become the same.

[0142] According to this embodiment, the size of the image after distortion reduction processing is the same before and after the change, thus the load in image recognition processing using this image remains constant. That is, the processing time for each image can be controlled within the required time, or the power consumption can be controlled within a predetermined level.

[0143] 6. In the above embodiments,

[0144] The predetermined area (e.g., 302) is the area defined by a transition center (e.g., 301) determined within the image acquired from the imaging device.

[0145] The regional control agency controls the location of the transfer center (e.g., Figure 8 301, 301a) to make the predetermined area change (e.g., 302a).

[0146] According to this embodiment, by setting the region near the conversion center as the conversion target, it is possible to suppress the distortion of the image that is the object of distortion reduction processing while changing the region that is the object of distortion reduction processing.

[0147] 7. In the above embodiments,

[0148] The regional control agency changes the predetermined area by setting the conversion center in an area where no specific target object exists.

[0149] According to this embodiment, in the area where distortion reduction processing is applied, high-precision image resolution can be performed (by reducing or eliminating the area of ​​the occluder).

[0150] 8. In the above embodiments,

[0151] The shooting device is equipped with a fisheye lens (e.g., 41-44).

[0152] According to this embodiment, a wide range of external conditions around a vehicle can be analyzed using a single camera.

[0153] 9. In the above embodiments,

[0154] The shooting device (e.g., 41-44) is configured to shoot the outside world in at least one of the front, side and rear directions of the vehicle.

[0155] According to this embodiment, it is possible to analyze the conditions of various directions outside the vehicle.

[0156] 10. In the above embodiments,

[0157] The control device also has a second identification mechanism (e.g., 2, 21-24) that identifies the vehicle’s external environment based on an image with reduced distortion.

[0158] According to this implementation method, it is possible to perform higher-precision image analysis to identify the presence of traffic participants and targets at greater distances.

[0159] 11. The control device (e.g., 2) of the above embodiments, which is a control device for a vehicle (e.g., 1) equipped with a shooting device (e.g., 41-44), is characterized in that,

[0160] The control device has:

[0161] An image acquisition mechanism (e.g., 22, 23, S901) periodically acquires images of the outside world of the vehicle from the imaging device;

[0162] The identification mechanism (e.g., 2, 22, 23, S902) identifies external targets of the vehicle based on images acquired from the imaging device;

[0163] A zone control mechanism (e.g., 2, 22, 23, S905) controls the setting of a predetermined zone for an image acquired from the imaging device; and

[0164] The processing unit (e.g., 2, 22, 23, S906) performs distortion reduction processing on a predetermined area to reduce image distortion.

[0165] The area control agency sets up a predetermined area in an area where the identification agency has not identified a specific target (e.g., S905).

[0166] According to this embodiment, the influence of obstructions can be reduced in target object recognition using images captured by a camera equipped with a wide-angle lens.

[0167] 12. The vehicle of the above embodiment is a vehicle that includes the control device described in any one of 1 to 11 above (for example, 1).

[0168] According to this embodiment, in a vehicle, the influence of obstructions can be reduced in target object recognition using images captured by a camera equipped with a wide-angle lens.

[0169] This invention is not limited to the embodiments described above, and various modifications and alterations can be made within the scope of the spirit of this invention.

Claims

1. A control device that is a control device of a vehicle having an imaging device, characterized by comprising: an image acquisition mechanism that acquires periodically an image of an outside of the vehicle from the imaging device; a first recognition mechanism that recognizes a target object of the outside of the vehicle including an obstruction based on the image acquired from the imaging device; a region control mechanism that controls setting of a predetermined region with respect to the image acquired from the imaging device; and a processing mechanism that performs distortion alleviation processing for alleviating distortion of the image with respect to the set predetermined region, wherein the region control mechanism changes the predetermined region in a manner that increases a region other than the specific obstruction included in the predetermined region, in a case where a specific obstruction recognized by the first recognition mechanism is included in the predetermined region, the region control mechanism sets a first region where the specific obstruction is not included in the predetermined region, in a case where the specific obstruction is included in the first region, the region control mechanism changes the predetermined region from the first region to a second region in a manner that increases a region other than the specific obstruction included in the predetermined region, and in a case where the specific obstruction is an obstruction extending in a first direction, the region control mechanism changes the predetermined region from the first region to the second region in a manner that the image corresponding to the predetermined region becomes an image that is longer in a second direction perpendicular to the first direction.

2. The control device according to claim 1, wherein in a case where the specific obstruction is an obstruction extending in a horizontal direction, the region control mechanism changes the predetermined region from the first region to the second region in a manner that an image longer in the horizontal direction becomes an image longer in a vertical direction.

3. The control device according to claim 1, wherein the region control mechanism changes the predetermined region from the first region to the second region in a manner that a size of the image corresponding to the first region and a size of the image corresponding to the second region become the same.

4. The control device according to claim 1, wherein the predetermined region is a region defined by a conversion center determined within the image acquired from the imaging device, the region control mechanism changes the predetermined region by controlling a position of the conversion center, the region control mechanism changes the predetermined region in a manner that the conversion center is set in a region where the specific obstruction is not present, the imaging device has a fisheye lens, the imaging device is configured to image the outside of at least one of a front, a side, and a rear of the vehicle, and the control device further comprises a second recognition mechanism that recognizes the outside of the vehicle based on the image whose distortion is alleviated. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 5. The control device of claim 4, wherein ​ 6. The control device of claim 1, wherein ​ 7. The control device of claim 1, wherein ​ 8. The control device of claim 1, wherein ​ 9. A vehicle that is a vehicle having a photographing device, characterized by comprising: the vehicle having: an image acquisition mechanism that acquires periodically an image of an outside of the vehicle from the photographing device; a first recognition mechanism that recognizes a target object of the outside of the vehicle including an obstruction based on the image acquired from the photographing device; a region control mechanism that controls setting of a predetermined region with respect to the image acquired from the photographing device; and a processing mechanism that performs distortion alleviation processing for alleviating distortion of an image with respect to the predetermined region set by the region control mechanism, in a case where a specific obstruction recognized by the first recognition mechanism is included in the predetermined region, the region control mechanism changes the predetermined region in a manner that a region other than the specific obstruction included in the predetermined region is increased, in a case where the specific obstruction is not included in a first region of the image acquired by the photographing device, the region control mechanism sets the first region as the predetermined region, in a case where the specific obstruction is included in the first region, the region control mechanism changes the predetermined region from the first region to a second region in a manner that a region other than the specific obstruction included in the predetermined region is increased, in a case where the specific obstruction is an obstruction extending in a first direction, the region control mechanism changes the predetermined region from the first region to the second region in a manner that an image corresponding to the predetermined region becomes an image longer in a second direction perpendicular to the first direction.

10. A control method that is a control method of a vehicle having a photographing device, characterized by comprising: the control method having: an image acquisition step of acquiring periodically an image of an outside of the vehicle from the photographing device; a first recognition step of recognizing a target object of the outside of the vehicle including an obstruction based on the image acquired from the photographing device; a region control step of controlling setting of a predetermined region with respect to the image acquired from the photographing device; and a processing step of performing distortion alleviation processing for alleviating distortion of an image with respect to the predetermined region set by the region control step, in the region control step, in a case where a specific obstruction recognized in the first recognition step is included in the predetermined region, the predetermined region is changed in a manner that a region other than the specific obstruction included in the predetermined region is increased, in the region control step, in a case where the specific obstruction is not included in a first region of the image acquired by the photographing device, the first region is set as the predetermined region, in a case where the specific obstruction is included in the first region, the predetermined region is changed from the first region to a second region in a manner that a region other than the specific obstruction included in the predetermined region is increased, ​ ​ In the region control step, in a case where the specific barrier is a barrier extending in a first direction, the predetermined region is changed from the first region to a second region in a manner such that an image corresponding to the predetermined region becomes an image that is longer in a second direction perpendicular to the first direction.

Images