Image processing device and image processing method, vehicle control device, and storage medium

Abstract

The present invention relates to an image processing device and an image processing method, a vehicle control device, and a storage medium, and is intended to provide a technology for effectively using a wide range of captured images obtained from a fisheye camera for recognition processing for driving assistance or automatic driving. An image processing device that performs external recognition for driving assistance or automatic driving of a vehicle moves and sets a partial area that becomes an object of distortion correction processing according to a rule based on a captured image obtained from a capturing device that captures an image that requires distortion correction, and obtains a partial image after distortion correction by applying distortion correction processing to the set partial area of ​​the captured image. When the image processing device detects a curved mirror around the vehicle, the above-mentioned rule is changed in such a way that a partial area corresponding to a predetermined direction of the vehicle is preferentially set as an object of distortion correction processing.

Description

Technical Field

[0001] The present invention relates to an image processing device and an image processing method, a vehicle control device, and a storage medium. Background Art

[0002] It is proposed to install a fisheye camera in a vehicle and use the images of a wider range captured by the fisheye camera for driving assistance. For example, Patent Document 1 describes a structure in which a panoramic view is generated using images captured by an omnidirectional camera (fisheye camera) and displayed on a monitor, thereby performing parking assistance. In addition, Patent Document 2 describes a system that uses images from an on-vehicle sensor such as a fisheye camera to detect the relative distance and relative speed to other vehicles, bicycles, pedestrians, obstacles, etc. that exist around the vehicle.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application No. 2021-004017

[0006] Patent Document 2: Japanese Patent Application No. 2021-003906 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, the images captured by a fisheye camera are generally highly distorted and cannot be directly used for advanced recognition processing for driving assistance control and automatic driving control (for example, recognition processing using AI). Therefore, it is considered to perform distortion correction on the images captured by the fisheye camera, and use the images after distortion correction for recognition processing. For example, consider using the center of the captured image as the center of distortion correction for distortion correction, and using the images after distortion correction for recognition processing. However, even for images after distortion correction, the distortion will increase in the parts away from the center of the captured image. Therefore, in the recognition process, even if a wider range of captured images is obtained through a fisheye camera, only a part of it (the image of a predetermined range centered on the distortion correction) can be used, and the wide range of captured images cannot be effectively utilized.

[0009] The present invention provides a technology for effectively using a wide-range captured image obtained from a fisheye camera for recognition processing for driving assistance or automatic driving.

[0010] Means used to solve problems

[0011] According to one aspect of the present invention, there is provided an image processing device for performing external recognition for driving assistance or automatic driving of a vehicle based on a captured image obtained from a capturing device that captures an image requiring distortion correction.

[0012] The image processing device comprises:

[0013] a correction unit that moves and sets a partial area to be subjected to a distortion correction process according to a rule, and acquires a partial image after distortion correction by applying the distortion correction process to the set partial area of ​​the captured image;

[0014] a detection unit that detects a curved mirror around the vehicle; and

[0015] A changing unit changes the rule so that a partial area corresponding to a predetermined direction of the vehicle is preferentially set as a target of the distortion correction process when the curved mirror is detected by the detecting unit.

[0016] According to another aspect of the present invention, there is provided a vehicle control device,

[0017] The vehicle control device comprises:

[0018] An image processing device that performs external recognition for driving assistance or automatic driving of a vehicle based on a captured image obtained from a capturing device that captures an image that requires distortion correction, wherein the image processing device comprises: a correction unit that moves and sets a partial area that becomes an object of distortion correction processing according to a rule, and obtains a partial image after distortion correction by applying the distortion correction processing to the set partial area of ​​the captured image; a detection unit that detects a curved mirror around the vehicle; and a change unit that changes the rule in a manner such that a partial area corresponding to a predetermined direction of the vehicle is preferentially set as an object of the distortion correction processing when the curved mirror is detected by the detection unit;

[0019] a recognition unit that recognizes a surrounding environment based on the distortion-corrected partial image acquired by the correction unit; and

[0020] A control unit performs control for driving assistance or automatic driving based on the surrounding environment recognized by the recognition unit.

[0021] In addition, according to another embodiment of the present invention, there is provided an image processing method for performing external recognition for driving assistance or automatic driving of a vehicle based on a captured image obtained from a capturing device that captures an image that requires distortion correction.

[0022] The image processing method comprises:

[0023] a correction step, in which a partial area to be subjected to a distortion correction process is moved and set according to a rule, and a partial image after distortion correction is acquired by applying the distortion correction process to the set partial area of ​​the captured image;

[0024] a detection step, in which a curved mirror is detected around the vehicle; and

[0025] A changing step, in which, when a curved mirror is detected in the detecting step, the rule is changed so that a partial area corresponding to a predetermined direction of the vehicle is preferentially set as an object of the distortion correction process.

[0026] In addition, according to another embodiment of the present invention, there is provided a storage medium storing a program for causing a computer to execute an image processing method, wherein the image processing method performs external environment recognition for driving assistance or automatic driving of a vehicle.

[0027] The image processing method comprises:

[0028] a correction step, in which a partial area to be subjected to a distortion correction process is moved and set according to a rule, and a partial image after distortion correction is acquired by applying the distortion correction process to the set partial area of ​​the captured image;

[0029] a detection step, in which a curved mirror is detected around the vehicle; and

[0030] A changing step, in which, when a curved mirror is detected in the detecting step, the rule is changed so that a partial area corresponding to a predetermined direction of the vehicle is preferentially set as an object of the distortion correction process.

[0031] Effects of the Invention

[0032] According to the present invention, it is possible to effectively use a wide-range captured image obtained by a fisheye camera for recognition processing for driving assistance or automatic driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a block diagram showing a configuration example of a vehicle according to the embodiment.

[0034] Figure 2 This figure explains the imaging range of a standard camera and a fisheye camera.

[0035] Figure 3 The diagram explains the distortion correction process of an image captured by a fisheye camera.

[0036] Figure 4This is a block diagram showing an example of a functional configuration for surrounding environment recognition using a captured image.

[0037] Figure 5 This is a diagram for explaining the rules for moving (selecting) a partial area.

[0038] Figure 6 This is a flowchart for explaining the procedure of surrounding environment recognition processing for driving control.

[0039] Figure 7 This is a flowchart showing a process for setting a rule for moving a partial area.

[0040] Figure 8 The diagrams illustrate the left and right front and left and right rear of a vehicle.

[0041] Fig. 9 This is a diagram for explaining an example of a rule in the case where priority is given to a partial area corresponding to the left front of the vehicle.

[0042] Fig.10 This is a diagram for explaining a method for detecting the direction of a curved mirror.

[0043] Description of Reference Numerals

[0044] 1: vehicle; 2: vehicle control device; 20-29 ECU; 40: standard camera; 41-44: fisheye camera; 421: distortion correction unit; 441: external recognition unit; 401: driving control unit; 411: rule setting unit; 412: partial area setting unit. DETAILED DESCRIPTION

[0045] Hereinafter, the embodiments are described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the invention involved in the technical solution. In addition, all combinations of features described in the embodiments are not limited to the content necessary for the invention. Two or more of the multiple features described in the embodiments may also be combined arbitrarily. In addition, the same or similar structures are marked with the same figure numerals, and repeated descriptions are omitted.

[0046] Figure 1 FIG. 1 is a block diagram of a vehicle 1 according to an embodiment of the present invention. Figure 1 , a top view and a side view are used to show the outline of the vehicle 1. As an example, the vehicle 1 is a sedan-type four-wheeled passenger vehicle. The vehicle 1 may be such a four-wheeled vehicle, or may be a two-wheeled vehicle or other types of vehicles.

[0047] The vehicle 1 of this embodiment is, for example, a parallel hybrid vehicle. In this case, the power device 6 as a travel drive unit that outputs a driving force for rotating the drive wheels of the vehicle 1 may include an internal combustion engine, a motor, and an automatic transmission. The motor can be used as a driving source for accelerating the vehicle 1, and can also be used as a generator (regenerative braking) during deceleration.

[0048] The vehicle 1 includes a vehicle control device 2 (hereinafter referred to as the control device 2) for controlling the vehicle 1. The control device 2 includes a plurality of ECUs (Electronic Control Units) 20 to 29 that are communicatively connected via an in-vehicle network. Each ECU includes a processor (computer) such as a CPU (Central Processing Unit), a memory such as a semiconductor memory, an interface with external devices, and the like. The memory stores programs executed by the processor, data used by the processor in processing, and the like. Each ECU may also include a plurality of processors, memories, interfaces, and the like. For example, ECU20 includes a processor 20a and a memory 20b. The processing of ECU20 is performed by the processor 20a executing commands containing a program stored in the memory 20b. Alternatively, ECU20 may also include a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for performing the processing of ECU20. The same is true for other ECUs.

[0049] The following describes the functions and the like that are in charge of each of the ECUs 20 to 29. It should be noted that the number of ECUs and the functions that they are in charge of can be designed appropriately, and can be further subdivided or integrated than in the present embodiment.

[0050] ECU 20 performs control related to automatic driving of vehicle 1. In automatic driving, at least one of the steering and acceleration / deceleration of vehicle 1 is automatically controlled. Automatic driving based on ECU 20 may also include automatic driving that does not require driving operation by the driver (also referred to as automatic driving) and automatic driving for assisting the driving operation of the driver (also referred to as driving assistance).

[0051] ECU21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism for steering the front wheels according to the driver's driving operation (steering operation) of the steering wheel 31. In addition, the electric power steering device 3 includes a motor that exerts a driving force for assisting the steering operation or automatically steering the front wheels, a sensor for detecting the steering angle, etc. When the driving state of the vehicle 1 is automatic driving, the ECU21 automatically controls the electric power steering device 3 in accordance with the instruction from the ECU20 to control the travel direction of the vehicle 1.

[0052] ECU22 and ECU23 control the detection unit for detecting the surrounding conditions of the vehicle and process the information of the detection results. Vehicle 1, as a detection unit for detecting the surrounding conditions of the vehicle, includes a standard camera 40 and four fisheye cameras 41 to 44. The standard camera 40 and the fisheye cameras 42 and 44 are connected to ECU22. The fisheye camera 41 and the fisheye camera 43 are connected to ECU23. ECU22 and ECU23 analyze the images captured by the standard camera 40 and the fisheye cameras 41 to 44, thereby being able to extract the outline of the target and the lane dividing lines (white lines, etc.) on the road.

[0053] The fisheye cameras 41 to 44 are cameras equipped with fisheye lenses. The structure of the fisheye camera 41 will be described below. The other fisheye cameras 41 to 44 may also have the same structure. The field of view of the fisheye camera 41 is wider than that of the standard camera 40. Therefore, the fisheye camera 41 can capture a wider range than the standard camera 40. The image captured by the fisheye camera 41 has greater distortion than the image captured by the standard camera 40. Therefore, the ECU 23 may also perform a transformation process (hereinafter referred to as "distortion correction process") for reducing the distortion of the image captured by the fisheye camera 41 before analyzing the image. On the other hand, the ECU 22 may not perform a distortion correction process on the image captured by the standard camera 40 before analyzing the image. In this way, the standard camera 40 is a shooting device that captures an image that is not the object of the distortion correction process, and the fisheye camera 41 is a shooting device that captures an image that is the object of the distortion correction process. Instead of the standard camera 40 , another imaging device that captures an image that is not the subject of the distortion correction process, for example, a camera equipped with a wide-angle lens or a telephoto lens may be used.

[0054] The standard camera 40 is installed at the front center of the vehicle 1 to capture the surrounding conditions in front of the vehicle 1. The fisheye camera 41 is installed at the front center of the vehicle 1 to capture the surrounding conditions in front of the vehicle 1. Figure 1 , the standard camera 40 and the fisheye camera 41 are shown to be arranged in the horizontal direction. However, the configuration of the standard camera 40 and the fisheye camera 41 is not limited thereto, and for example, they may be arranged in the vertical direction. In addition, at least one of the standard camera 40 and the fisheye camera 41 may be installed at the front of the roof of the vehicle 1 (for example, inside the cabin of the front window). The fisheye camera 42 is installed at the center of the right side of the vehicle 1 to capture the surrounding conditions on the right side of the vehicle 1. The fisheye camera 43 is installed at the center of the rear part of the vehicle 1 to capture the surrounding conditions on the rear side of the vehicle 1. The fisheye camera 44 is installed at the center of the left side of the vehicle 1 to capture the surrounding conditions on the left side of the vehicle 1.

[0055] The type, number and installation position of the camera of the vehicle 1 are not limited to the above examples. In addition, the vehicle 1 may also include an optical radar (Light Detection and Ranging) or a millimeter wave radar as a detection unit for detecting objects around the vehicle 1 or measuring the distance to the object.

[0056] ECU 22 controls the standard camera 40 and the fisheye cameras 42 and 44 and processes the information of the detection results. ECU 23 controls the fisheye cameras 41 and 43 and processes the information of the detection results. By dividing the detection unit for detecting the surrounding conditions of the vehicle into two systems, the reliability of the detection results can be improved.

[0057] ECU24 controls the gyro sensor 5, GPS sensor 24b, and communication device 24c, and performs information processing of detection results or communication results. Gyro sensor 5 detects the rotational movement of vehicle 1. The travel route of vehicle 1 can be determined by the detection results of gyro sensor 5, wheel speed, etc. GPS sensor 24b detects the current position of vehicle 1. Communication device 24c wirelessly communicates with a server that provides map information and traffic information, and obtains this information. ECU24 can access a database 24a of map information constructed in a memory, and ECU24 searches for routes from the current location to the destination, etc. ECU24, map database 24a, and GPS sensor 24b constitute a so-called navigation device.

[0058] The ECU 25 includes a communication device 25a for inter-vehicle communication. The communication device 25a wirelessly communicates with other vehicles in the vicinity to exchange information between the vehicles.

[0059] ECU26 controls the power unit 6. The power unit 6 is a mechanism that outputs a driving force that rotates the driving wheels of the vehicle 1, and includes, for example, an engine and a transmission. For example, ECU26 controls the output of the engine in accordance with the driver's driving operation (accelerator operation or acceleration operation) detected by the operation detection sensor 7a provided on the accelerator pedal 7A, or switches the gear of the transmission based on information such as the vehicle speed detected by the vehicle speed sensor 7c. When the driving state of the vehicle 1 is automatic driving, ECU26 automatically controls the power unit 6 in accordance with the instruction from ECU20 to control the acceleration and deceleration of the vehicle 1.

[0060] The ECU 27 controls the lighting devices (headlights, taillights, etc.) including the direction indicator 8 (turn indicator). Figure 1 In the example of FIG. 1 , the direction indicators 8 are provided at the front portion, the door mirrors, and the rear portion of the vehicle 1 .

[0061] ECU28 controls the input-output device 9. The input-output device 9 outputs information to the driver and receives information from the driver. The sound output device 91 reports information to the driver by sound. The display device 92 reports information to the driver by displaying an image. The display device 92 is, for example, arranged on the surface of the driver's seat to constitute a dashboard, etc. It should be noted that, here, sound and display are illustrated, but information can also be reported by vibration or light. In addition, it is also possible to combine multiple of sound, display, vibration or light to report information. Moreover, it is also possible to make different combinations or different reporting methods according to the level of information to be reported (for example, urgency). The input device 93 is arranged at a position that the driver can operate, and is a switch group for indicating the vehicle 1, but it may also include a sound input device.

[0062] ECU29 controls the braking device 10 and the parking brake (not shown). The braking device 10 is, for example, a disc braking device, which is provided on each wheel of the vehicle 1 and decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheel. ECU29 controls the operation of the braking device 10 in accordance with the driver's driving operation (brake operation) detected by the operation detection sensor 7b provided on the brake pedal 7B. When the driving state of the vehicle 1 is automatic driving, ECU29 automatically controls the braking device 10 in accordance with the instructions from ECU20 to control the deceleration and stopping of the vehicle 1. The braking device 10 and the parking brake can also be operated to maintain the stopped state of the vehicle 1. In addition, when the transmission of the power unit 6 is equipped with a parking lock mechanism, it can also be operated to maintain the stopped state of the vehicle 1.

[0063] Reference Figure 2 , the imaging ranges of the standard camera 40 and the fisheye cameras 41 to 44 are described. Figure 2 (a) in the figure represents the horizontal shooting range of each camera. Figure 2 (b) in FIG. 1 shows the vertical imaging range of the fisheye camera 42 on the right side of the vehicle 1 . Figure 2 (c) in the figure shows the imaging range in the vertical direction of the fisheye camera 43 installed at the rear part of the vehicle 1 .

[0064] First, refer to Figure 2 (a) in the figure explains the shooting range in the top view of the vehicle 1 (i.e., the horizontal direction of the vehicle 1). The standard camera 40 shoots the scenery included in the shooting range 200. The shooting center 200C of the standard camera 40 faces directly in front of the vehicle 1. The field of view angle of the standard camera 40 in the horizontal direction can be less than 90°, for example, about 45° or about 30°.

[0065] The fisheye camera 41 captures the scenery included in the capturing range 201. The capturing center 201C of the fisheye camera 41 faces directly in front of the vehicle 1. The fisheye camera 42 captures the scenery included in the capturing range 202. The capturing center 202C of the fisheye camera 42 faces directly to the right side of the vehicle 1. The fisheye camera 43 captures the scenery included in the capturing range 203. The capturing center 203C of the fisheye camera 43 faces directly behind the vehicle 1. The fisheye camera 44 captures the scenery included in the capturing range 204. The capturing center 204C of the fisheye camera 44 faces directly to the left side of the vehicle 1. The field of view angles of the fisheye cameras 41 to 44 in the horizontal direction may be, for example, greater than 90°, greater than 150°, or greater than 180°, and may be, for example, about 180°. Figure 2 (a) shows an example in which the horizontal field angle of the fisheye cameras 41 to 44 is 180°.

[0066] The photographing range 201 may be divided into an area 201L located in the left oblique front of the vehicle 1, an area 201F located directly in front of the vehicle 1, and an area 201R located in the right oblique front of the vehicle 1. The photographing range 202 may be divided into an area 202L located in the right oblique front of the vehicle 1, an area 202F located directly on the right side of the vehicle 1, and an area 202R located in the right oblique rear of the vehicle 1. The photographing range 203 may be divided into an area 203L located in the right oblique rear of the vehicle 1, an area 203F located directly behind the vehicle 1, and an area 203R located in the left oblique rear of the vehicle 1. The photographing range 204 may be divided into an area 204L located in the right oblique rear of the vehicle 1, an area 204F located directly on the left side of the vehicle 1, and an area 204R located in the left oblique front of the vehicle 1. The photographing range 201 may also be divided equally (i.e., in such a way that the field of view angles of the respective areas are equal) into the three areas 201L, 201F, and 201R. The other imaging ranges 202 to 204 may also be equally divided into three parts.

[0067] The standard camera 40 and the fisheye cameras 41 to 44 have the above-mentioned shooting ranges 200 to 204, so that the front of the vehicle 1 and the four oblique directions are included in the shooting ranges of the two separate cameras. Specifically, the front of the vehicle 1 is included in both the shooting range 200 of the standard camera 40 and the area 201F of the shooting range 201 of the fisheye camera 41. The right oblique front of the vehicle 1 is included in both the area 201R of the shooting range 201 of the fisheye camera 41 and the area 202L of the shooting range 202 of the fisheye camera 42. The same is true for the other three oblique directions of the vehicle 1.

[0068] Next, refer to Figure 2 (b) and Figure 2(c) in FIG. 1 illustrates the shooting range in the vertical direction of the vehicle 1. Figure 2 In (b), the vertical shooting range of the fisheye camera 42 is described. Figure 2 In (c) of FIG. 1 , the vertical imaging range of the fisheye camera 43 is described. The vertical imaging ranges of the other fisheye cameras 41 and 44 may be the same.

[0069] The vertical field angle of the fisheye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, or greater than 180°, and may be, for example, approximately 180°. Figure 2 (b) and (c) in the figure represent examples in which the field of view of the fisheye cameras 41 to 44 in the vertical direction is 180°. In addition, the shooting center 202C of the fisheye camera 42 and the shooting center 203C of the fisheye camera 43 are oriented to a position lower than the direction parallel to the ground (on the ground side). Alternatively, the shooting center 203C of the fisheye camera 43 may be oriented to a direction parallel to the ground or to a position higher than the direction parallel to the ground (on the side opposite to the ground). In addition, the shooting centers 201C to 204C of the fisheye cameras 41 to 44 may be oriented to different directions in the vertical direction.

[0070] Reference Figure 3 , the distortion correction process of the images captured by the fisheye cameras 41 to 44 will be described. Image 300 is an image of the scenery to the right of the vehicle 1 captured by the fisheye camera 43. Figure 3 As shown, the image 300 has large distortion particularly in the peripheral portion.

[0071] ECU22 connected to the fisheye camera 43 performs distortion correction processing on the image 300. Specifically, ECU22 sets a point in the image 300 as the correction center point 301. ECU22 cuts out a rectangular area 302 centered on the correction center point 301 from the image 300. ECU22 generates an image 303 with reduced distortion by performing distortion correction processing on the area 302. In the distortion correction process, the closer the position is to the correction center point 301, the more the distortion is reduced, and at a position far away from the correction center point 301, the distortion is not reduced or the distortion increases. Therefore, in some embodiments, ECU22 sets the correction center point 301 in an area of ​​the surrounding environment of the vehicle 1 that you want to pay attention to, and generates an image with reduced distortion for this area.

[0072] Next, a functional configuration for performing surrounding environment recognition processing using the captured images from the standard camera 40 and the fisheye cameras 41 to 44 will be described. It should be noted that the recognition result of the surrounding environment recognition processing can be used for automatic driving control or driving assistance control. Figure 4This is a block diagram showing an example of a functional configuration for performing image processing for surrounding environment recognition processing using a captured image according to the present embodiment. Figure 4 Each function shown can be realized by executing a program stored in a memory by one or more processors constituting each of ECU20, ECU22, and ECU23, or by dedicated hardware, or by cooperation between them. In addition, the structure of the functional modules, which ECU implements each functional module, etc. are of course not limited to Figure 4 For example, part or all of the processing performed by the external recognition units 441 a to 441 e may be implemented by the driving control unit 401 .

[0073] The images captured by the standard camera 40 and the left and right fisheye cameras 42 and 44 are provided to the ECU 22. In the ECU 22, the external recognition unit 441e performs external recognition processing on the captured images from the standard camera 40, and extracts various objects such as lane dividing lines, road signs, other vehicles, curved mirrors (road reflectors, traffic convex mirrors), and people. The extraction of various objects in the external recognition unit 441e can use, for example, judgment processing based on artificial intelligence (AI) and machine learning. As described above, the captured images from the standard camera 40 are less distorted, so there is no need to perform distortion correction before the external recognition unit 441e performs recognition processing.

[0074] In the ECU 22, the distortion correction unit 421a performs the following operations on the image captured by the fisheye camera 42: Figure 3 The distortion correction unit 421a performs distortion correction processing on a rectangular area (hereinafter referred to as a designated partial area) centered on the correction center point designated by the partial area designation unit 412, and outputs the partial image after the distortion correction processing to the external recognition unit 441a. The external recognition unit 441a performs the same external recognition processing as the external recognition unit 441e on the partial image whose distortion has been corrected by the distortion correction unit 421a, and extracts various objects and logos.

[0075] The fisheye camera 42 can capture a wide range of space, but the captured images contain large distortions and are not suitable for external recognition processing using AI or the like directly. Therefore, the captured images from the fisheye camera 42 are subjected to distortion correction processing by the distortion correction unit 421a before being provided to the external recognition unit 441a. The external recognition unit 441a uses the image from which the distortion has been removed by the distortion correction unit 421a to perform external recognition processing, so that various objects and logos can be extracted with high precision. The distortion correction unit 421a moves the correction center (i.e., the partial area) in sequence for each frame of the image captured by the fisheye camera 42 according to the instruction from the partial area designation unit 412 to perform distortion correction, and outputs the partial image after the distortion is corrected. In this way, by moving the partial area in the image of a wide range and acquiring the partial image in which the distortion is corrected, it is possible to obtain an image without distortion for the wider area in the wider shooting range. Therefore, it is possible to perform high-precision external recognition processing on the image of the wider area captured by the fisheye camera 42.

[0076] Figure 5 (a) and (b) in FIG. 1 are diagrams for explaining an example of controlling the movement of a partial area in normal times. Normal times refer to, for example, a state where, as a result of the recognition of the surrounding environment, it is determined that there is no direction that should be paid special attention to. Figure 5 As shown in (a) of FIG. 5 , in the present embodiment, in the image 500 of the fisheye camera 42, by sequentially selecting any one of the three partial areas 511 to 513 determined by the three correction center points 501 to 503, the partial area in the captured image that is the correction target is moved. Hereinafter, such information indicating the order of selecting the partial areas is referred to as a rule. In addition, in the present embodiment, as Figure 2 The partial areas are set corresponding to the three-divided shooting range shown in (a) in FIG. 4 . For example, the ranges of the areas 201L, 201F, and 201R in the shooting range of the fisheye camera 41 correspond to the partial areas 511, 512, and 513, respectively. Similarly, for example, the ranges of the areas 204L, 204F, and 204R in the shooting range of the fisheye camera 44 correspond to the partial areas 511, 512, and 513, respectively.

[0077] It should be noted that the settings of some areas are not limited to Figure 5 The illustrated setting may be, for example, divided into four or more partial regions in the horizontal direction, or may be set so that there are two or more partial regions in the vertical direction, and a part of a partial region may overlap a part of another partial region. Figure 5In the figure, for the sake of convenience, a partial area in an image captured by a fisheye camera that is the object of distortion correction processing is schematically shown. Generally, when a rectangular image is desired as an image after distortion correction processing, the corresponding partial area in the captured image is not rectangular.

[0078] Normally, the movement of the partial area of ​​the calibration object is performed by selecting three partial areas 511 to 513 equally. Figure 5 As shown in (b) in FIG. 1 , the selected partial area is switched as the correction object in the manner of partial area 511 → partial area 512 → partial area 513 → partial area 511 → ... for each frame. According to such a rule, partial areas 511 to 513 are selected once every three frames. That is, each partial area is selected as the correction object with an equal frequency, and the partial image after distortion correction is provided to the external recognition processing. It should be noted that which partial area is selected is indicated by the partial area designation unit 412 described later. In addition, in the present embodiment, each partial area is selected equally at normal times, but it is not limited to this. In addition, it is not necessary to apply the same rule to all captured images from the fisheye cameras 41 to 44, and the rules can also be applied independently. For example, with respect to the front fisheye camera 41, the normal rule can also be set in a manner that the selection frequency of the front partial area 512 becomes higher, and with respect to the other fisheye cameras 42 to 44, the normal rule can also be set in a manner that each partial area is selected equally.

[0079] return Figure 4 The distortion correction unit 421b and the external recognition unit 441b perform the above-mentioned distortion correction processing and external recognition processing on the image captured by the fisheye camera 44. That is, the distortion correction unit 421b performs the distortion correction processing on the partial area specified by the partial area specifying unit 412 in the image captured by the fisheye camera 44. The external recognition unit 441b performs the external recognition processing on the partial image whose distortion is corrected by the distortion correction unit 421b, and extracts various objects and marks.

[0080] In addition, in ECU 23, the distortion correction unit 421c performs distortion correction processing on the partial area specified by the partial area specifying unit 412 in the image obtained by the fisheye camera 41. The external recognition unit 441c performs the above-mentioned external recognition processing on the image corrected by the distortion correction unit 421c. Similarly, the distortion correction unit 421d performs the above-mentioned distortion correction processing on the image obtained by the fisheye camera 43, and the external recognition unit 441d performs the above-mentioned external recognition processing on the image corrected by the distortion correction unit 421d. Hereinafter, the distortion correction units 421a to 421d are collectively referred to as the distortion correction unit 421, and the external recognition units 441a to 441d are collectively referred to as the external recognition unit 441.

[0081] In the ECU 20, the driving control unit 401 controls driving assistance and automatic driving based on the recognition result provided by the external recognition unit 441. The rule setting unit 411 determines the direction to be looked at based on the recognition result provided by the external recognition unit 441, and changes and sets the rule for selecting a partial area from the image captured by the fisheye camera. The partial area specifying unit 412 specifies the partial area to be selected as the object of distortion correction processing to the distortion correction unit 421 according to the rule set by the rule setting unit 411.

[0082] As described above, in the control device 2 of this embodiment, by moving the partial area to be the object of distortion correction while performing the distortion correction process, it is possible to use a wider range of images obtained from the fisheye camera for external recognition processing using AI or the like. On the other hand, as described above, the image to be the object of external recognition processing is not the entire frame of the fisheye camera, but a partial image selected from the image of the frame. For example, in Figure 4 In the example of the rule shown, the frequency at which each of the three partial areas becomes the object of the external recognition processing is once in three frames. In contrast, depending on the surrounding environment, sometimes a specific direction of the vehicle 1 should be looked at. In such a case, it is preferable to increase the frequency at which the partial area corresponding to the direction to be looked at becomes the object of the external recognition processing. For example, when entering an intersection, the direction to be looked at changes depending on which operation is performed at this time, such as going straight, turning right, or turning left (which can be determined by the driver's direction indicator operation or handle operation). Therefore, it is preferable to increase the frequency at which the partial area corresponding to the direction to be looked at based on such surrounding environment and driving operation becomes the object of the external recognition processing. Therefore, the rule setting unit 411 of the present embodiment changes the rule in a manner that gives priority to a specific partial area according to the surrounding environment, thereby increasing the frequency of selecting a specific partial area in the distortion correction unit 421. As a result, an image of the direction to be looked at according to the surrounding environment is obtained preferentially, so that more appropriate driving assistance and automatic driving control can be achieved.

[0083] Usually, the curved mirror is set to reflect a place where it is difficult for the driver to confirm the presence of an object, etc. Therefore, in a place where a curved mirror is set, it is necessary to pay more attention to the direction in which the curved mirror is facing. Therefore, when the rule setting unit 411 of this embodiment detects the presence of a curved mirror through the external recognition process, the rule indicating the order of movement in the above-mentioned partial area is changed in a manner to increase the frequency of the partial area corresponding to the direction of the curved mirror becoming the object of the distortion correction process.

[0084] Figure 64 is a flowchart illustrating the steps of the external recognition processing of this embodiment. In S601, the rule setting unit 411 sets a rule for selecting a partial area based on the external recognition result obtained from the external recognition unit 441 and / or the driver's driving operation (for example, the direction indicator operation / handle operation). For details of the processing of S601, refer to Figure 7 The flowchart of the process is described below. In S602, the partial region specifying unit 412 specifies the partial regions of the correction target in sequence according to the rule set in S601 for the distortion correction unit 421.

[0085] In S611, the distortion correction unit 421a performs distortion correction processing on the image of the partial area specified by the partial area specifying unit 412 in the image obtained from the fisheye camera 42, and obtains a partial image (plane image) with reduced distortion in the partial area. In S612, the external recognition unit 441a performs external recognition processing on the partial image obtained by the distortion correction unit 421a, and extracts various objects such as lane dividing lines, road signs, other vehicles, curved mirrors, and people. Here, in the detection of curved mirrors, AI can be used, and judgment processing based on machine learning can also be used. In S613, the external recognition unit 441a provides the information of the extracted object to the driving control unit 401 for driving assistance control or automatic driving control. It should be noted that the processing of the distortion correction unit 421a and the external recognition unit 441a is described above, but the distortion correction units 421b~421d and the external recognition units 441b~441d also perform the same processing.

[0086] In S603 , the driving control unit 401 recognizes the surrounding environment of the vehicle 1 based on the external environment recognition result obtained from the external environment recognition unit 441 , and performs automatic driving or driving assistance control based on the recognition result.

[0087] Figure 7 4 is a flowchart showing the processing performed by the rule setting unit 411 for setting the rules for moving a partial area. In S701, the rule setting unit 411 determines whether it is possible to set a normal rule as the rule for moving a partial area based on the external recognition result and the driving operation. The situation in which the normal rule can be set refers to a state in which, based on the recognition result of the external recognition unit 441, it is determined that there is no direction to be particularly focused on, and it is sufficient to check all directions around the vehicle equally. On the other hand, in the case in which it is determined based on the recognition result of the external recognition unit 441 that there is a need to further focus on a specific direction, the rule setting unit 411 determines that it is impossible to set the normal rule. The situation in which the need to further focus on a specific direction is generated can be listed as, for example, the situation in which, based on the recognition result from the external recognition unit 441, an intersection, a curved mirror, a passing car on a narrow road, the driver's operation of the turn signal lever, etc. are detected.

[0088] If it is determined in S701 that a normal rule can be set, the process proceeds to S711. In S711, the rule setting unit 411 sets, for example, Figure 5 As a result, in S602, the partial area designation unit 412 selects each of the distortion correction units 421a to 421d connected to the fisheye cameras 41 to 44 according to the general rule (rule of selecting each partial area with uniform frequency) described in (b) of FIG. Figure 5 To specify a partial area, follow the rules described in .

[0089] On the other hand, if it is determined in S701 that the usual rules cannot be set (when it is determined that there is a direction to be looked at), the process enters S702. In S702, the rule setting unit 411 determines whether a curved mirror is extracted in the external recognition process of the external recognition unit 441. For example, when there is an object whose probability of being a curved mirror is greater than a predetermined value among the objects detected by the external recognition unit 441, the rule setting unit 411 determines that a curved mirror is detected. In the case where a curved mirror is not extracted (No in S702), it is determined that the rule needs to be changed due to a main reason other than the detection of the curved mirror. Therefore, the process enters S712, and the rule setting unit 411 changes the rule based on the surrounding environment and driving operation detected by the driving control unit 401 based on the external recognition process. For example, in S712, the rule is changed in a manner that prioritizes the setting of a partial area corresponding to the direction of looking determined based on the surrounding environment recognition result and the driver's driving operation, that is, in a manner that the selection frequency of the partial area corresponding to the direction of looking becomes higher.

[0090] When it is determined that a curved mirror is detected (Yes in S702), in S703, the rule setting unit 411 determines the direction of the mirror surface of the curved mirror based on the position and shape of the curved mirror extracted by the external recognition process. For example, (1) the rule setting unit 411 determines the direction of the mirror surface based on the setting position of the curved mirror. Specifically, Fig.10 As shown in (a) in the figure, if the detected setting position of the curved mirror 901 is located to the right of the center of the lane 902 in which the vehicle 1 is traveling, it can be determined that the mirror surface is facing left relative to the direction of travel of the vehicle 1. Similarly, if the curved mirror is located to the left of the center of the lane, it can be determined that the mirror surface is facing right relative to the direction of travel of the vehicle. Or, for example, (2) the direction of the mirror surface is determined based on the positional relationship between the outer shape (ellipse) representing the mirror surface and the support supporting the mirror surface. In this method, for example, Fig.10As shown in (b) in FIG. 1 , in the curved mirror 911, if the support 913 is located to the right of the center line of the ellipse 912 corresponding to the mirror surface, it can be determined that the mirror surface is facing the left. Similarly, if the support 923 is located to the left of the center line of the ellipse 922 corresponding to the mirror surface as in the curved mirror 921, it can be determined that the mirror surface is facing the right.

[0091] Of course, the method for determining the direction of the curved mirror is not limited to the above-mentioned methods (1) and (2), and various determination methods can be used. In addition, multiple determination methods can also be combined. For example, in each of the determination methods (1) and (2), a score indicating the probability of the mirror facing left and a score indicating the probability of the mirror facing right can be calculated, and the direction of the side with the higher total score is determined as the direction of the mirror. The total score is obtained, for example, in the following manner: the farther the position of the curved mirror is from the center of the lane, the higher the score for the determined direction is, the greater the distance between the center of the ellipse corresponding to the mirror and the pillar, the higher the score for the determined direction is, and the scores for each direction are summed up. In addition, the direction of the curved mirror does not need to be one. When a curved mirror is detected on both sides of the road, it is determined that the mirror surface of the curved mirror is facing the left and right sides.

[0092] It should be noted that it is not necessary to use the images captured by all cameras in determining the curved mirror. For example, even if the curved mirror is detected in the image captured by the fisheye camera 43 from the rear, the direction to be looked at will not be affected. Therefore, only the images captured by the standard camera 40 that captures the front of the vehicle 1 and the images captured by the fisheye camera 41 from the front can be used for detecting the curved mirror.

[0093] In S704 to S706, the rule setting unit 411 changes the rule according to the orientation of the curved mirror determined in S703. First, when it is determined that the orientation of the curved mirror is to the left, in S704, the rule setting unit 411 sets a rule that increases the priority of the partial area corresponding to the left front of the vehicle 1. Figure 8The left and right front and the left and right rear of the vehicle 1 of the present embodiment are described. Four areas A to D are formed by a line 811 that roughly divides the width of the vehicle 1 into two equal parts and a line 812 that roughly divides the length of the vehicle 1 into two equal parts. In the present embodiment, area A is defined as the right front, area B is defined as the right rear, area C is defined as the left rear, and area D is defined as the left front. And, for example, the partial area with the largest area including area D among the multiple partial areas set for the shooting range of each fisheye camera is used as the partial area corresponding to the left front. For example, the partial area corresponding to the left front with respect to the fisheye camera 41 is area 201L, and the partial area corresponding to the left front with respect to the fisheye camera 44 is area 204R. It should be noted that the definitions of the left and right front and the left and right rear are not limited to the above definitions, and for example, the position of the line 812 may be forward or backward than the position that divides the length of the vehicle into two equal parts.

[0094] In S704, the rule setting unit 411 sets the following rule, for example, so that the partial area set in the image captured by the fisheye camera 41 from the front is Fig. 9 Move as shown in (a) in the figure. Fig. 9 In the example (a) of FIG. 1 , a rule for specifying partial areas in the order of partial area 511 → partial area 512 → partial area 511 → partial area 513 → ... is shown. According to this rule, the frequency of partial area 511 (area 201L) corresponding to the left front of the vehicle 1 in the shooting range of the front fisheye camera 41 being the target of distortion correction processing is twice in four frames. That is, Figure 5 Compared with the frequency in the rule shown in FIG. 1 (once in three frames), the frequency of partial area 511 being the target of distortion correction processing increases. Similarly, for the fisheye camera 44 on the left side of the vehicle body, the rule is set in such a way that the frequency of specifying the partial area corresponding to the left front of the vehicle 1 becomes higher. In the image captured by the fisheye camera 44 on the left side, the partial area 513 on the right side corresponds to the left front of the vehicle 1. Therefore, Fig. 9 As shown in (b) in FIG. 1 , the rule setting unit 411 sets the rule for the distortion correction unit 421c. As a result, the frequency of the partial area 513 (area 204R) being the object of the distortion correction processing increases. That is, the partial image corresponding to the front of the vehicle 1 in the image captured by the fisheye camera 44 is preferentially selected. It should be noted that the normal rules are applied to the right fisheye camera 42 and the rear fisheye camera 43 that do not include the partial area that helps to watch the left front of the vehicle 1.

[0095] When it is determined in S703 that the direction of the mirror surface of the curved mirror is rightward, in S705, the rule setting unit 411 sets a rule that increases the priority of the partial area corresponding to the right front of the vehicle 1. For example, the rule setting unit 411 sets a rule in a manner that increases the frequency of specifying the partial area corresponding to the right front of the vehicle 1 for the distortion correction unit 421c of the fisheye camera 41 for the front and the distortion correction unit 421d of the fisheye camera 42 for the right. As a specific example, Fig. 9 The rule shown in (b) in FIG. 4 is applied to the front fisheye camera 41 (distortion correction unit 421c), Fig. 9 The rule shown in (a) in FIG. 1 is applied to the right fisheye camera 42 (distortion correction unit 421a). In this case, the general rule can be applied to the left fisheye camera 44 and the rear fisheye camera 43 which do not include a partial area that helps to watch the right front of the vehicle 1.

[0096] In the case where both left-facing and right-facing curved mirrors are detected, or in the case where the direction of the mirror cannot be determined even though a curved mirror is detected, the processing enters S706. In S706, the rule setting unit 411 sets rules using both the rule of looking at the left front described in S705 and the rule of looking at the right front described in S706. In this case, in the rule corresponding to the front fisheye camera 41, it is necessary to increase the frequency with which partial area 511 and partial area 513 become the objects of distortion correction processing. Therefore, for example, partial area 511→partial area 513→partial area 511→partial area 513→partial area 512→… can be repeated. In this way, the frequency of specifying partial area 511 and the frequency of specifying partial area 513 are respectively twice / five frames, which is more than the frequency of specifying partial area 511 and the frequency of specifying partial area 513 by the usual rule ( Figure 5 ) The frequency specified by the camera (once / three frames) is high. It should be noted that the usual rules apply to the rear fisheye camera 43 which does not contribute to the left front gaze or the right front gaze.

[0097] According to the above-mentioned embodiment, a partial area of ​​the image captured by the fisheye camera is subjected to distortion correction processing and provided as a partial image. Since the distortion correction processing is performed in an area smaller than the entire image, the distortion of the provided partial image is removed in its entire area, and high-precision recognition processing can be achieved. As a result, high-precision recognition processing can be performed on the entire circumference of the vehicle 1 captured by the fisheye camera. In addition, since the partial area corresponding to the direction of the set curved mirror is preferentially used for recognition processing, faster recognition can be achieved for the direction to be watched, and more effective and appropriate driving assistance control and automatic driving control can be achieved. That is, the rules for controlling movement are based on the detection of the curved mirror, thereby making it possible to easily infer the conditions of the crosswalk that needs to be watched, and set the area that should be paid attention to.

[0098] It should be noted that the above embodiment shows a specific example for realizing the following idea: determining the direction to be looked at based on the detection of the mirror direction of the curved mirror, and increasing the priority of the partial area used in peripheral recognition based on the determined direction. Therefore, the definition of the left and right front and the corresponding partial areas, and the selection rules for applying them to peripheral recognition are not limited to the contents described in the embodiment. For example, in the above embodiment, as a rule for selecting a partial area, the partial area is moved according to each frame, but it is not limited to this. It is also possible to select the same partial area more than twice in a row to increase the frequency of selecting the partial area.

[0099] In addition, in the embodiment, the same combination of partial regions (for example, partial regions 511 to 513) is used in the general rule and the rule for looking at a specific direction, but the present invention is not limited thereto. Different combinations of partial regions may be used in the general rule and the rule for looking at a specific direction. For example, any partial region corresponding to the direction of looking may be added. Specifically, in the case of looking at the left front of the vehicle 1, the combination of partial regions 511 to 513 may be used in the rule for looking at a specific direction. Fig. 9 In (a) of FIG. 1 , a new partial area is set between partial area 511 and partial area 512, and a group consisting of four partial areas is used to set the rule. Alternatively, along with the addition of such a new partial area, an existing partial area (e.g., partial area 513) that is away from the gaze direction may be deleted from the group. That is, the new partial area and the existing partial area may be swapped. Further, for example, Figure 2 As in (b) and (c) in FIG. 1 , when the shooting center is facing downward, a partial area moving in the vertical direction (upward direction) may be set as a rule for focusing on the left front of the vehicle 1. Thus, the left front of the vehicle 1 can be recognized as being farther away.

[0100] In addition, in the above-mentioned embodiment, the direction of the curved mirror is determined to determine the gaze direction, but it is not limited to this. Generally, when a curved mirror is provided, at least one of the left direction or the right direction thereof becomes the gaze area. Therefore, when a curved mirror is detected (in the case of yes in S702), the determination of the direction in S703 may be omitted, and the left front and the right front of the vehicle 1 are immediately used as the directions to be gazed and the processing in S706 is executed.

[0101] It should be noted that by repeatedly Figure 7 Flowchart, the rule changed due to the detection of the curved mirror continues until the curved mirror is no longer detected. But, it is not limited to this, for example, the rule can also be returned to the usual rule according to the situation that the curved mirror disappears from the field angle of the standard camera 40 in front.

[0102] <Summary of Implementation Methods>

[0103] The above-mentioned embodiment discloses at least the following embodiments.

[0104] 1. An image processing device, which is an image processing device (421, 441) for performing external recognition for driving assistance or automatic driving of a vehicle based on a captured image obtained from a capturing device (41-44) for capturing an image requiring distortion correction, wherein:

[0105] The image processing device comprises:

[0106] A correction unit (421) moves and sets a partial area to be subjected to distortion correction processing according to a rule, and obtains a partial image after distortion correction by applying the distortion correction processing to the set partial area of ​​the captured image;

[0107] a detection unit (441, 411) that detects a curved mirror around the vehicle; and

[0108] A changing unit (411, 412) changes the rule in such a way that a partial area corresponding to a predetermined direction of the vehicle is preferentially set as an object of the distortion correction processing when the curved mirror is detected by the detecting unit.

[0109] According to the above embodiment, it is possible to preferentially acquire an image of a partial area corresponding to the gaze direction obtained by detection of the curved mirror, so that objects can be detected with high frequency and high accuracy in the gaze direction suitable for the road conditions. Therefore, appropriate control can be performed in driving assistance control and automatic driving control.

[0110] 2. In addition, in the above embodiment, the detection unit detects the curved mirror based on the image of the partial area after the distortion is corrected by the correction unit.

[0111] According to this embodiment, a captured image obtained from a capturing device that captures an image that requires distortion correction can be used for detecting a curved mirror, and the captured image can be effectively utilized.

[0112] 3. In the above embodiment, the detection unit detects the curved mirror based on an image captured by an image capturing device that captures an image that is not a target of distortion correction.

[0113] According to this embodiment, for example, an image captured by a standard camera can be used for detecting the curved mirror, and the curved mirror can be detected more frequently.

[0114] 4. In the above embodiment, the changing unit changes the rule so that the frequency of selecting at least one partial area corresponding to at least one of the left front and the right front of the vehicle as the object of the distortion correction becomes higher.

[0115] According to this embodiment, when the road condition of the curved mirror is detected, an image of the area corresponding to the direction to be watched can be obtained at a high frequency, and more appropriate and rapid driving assistance control and automatic driving control can be performed.

[0116] 5. In addition, in the above embodiment,

[0117] The detection unit further detects the direction reflected by the detected curved mirror.

[0118] The changing unit changes the rule so that the frequency of selecting a partial area corresponding to the detected direction reflected by the curved mirror as the object of the distortion correction becomes higher.

[0119] According to this embodiment, images can be acquired at a high frequency for the area corresponding to the direction reflected by the curved mirror, and other areas are not affected by the detection of the curved mirror. Therefore, it is possible to avoid unnecessarily increasing the selection frequency of a partial area.

[0120] 6. In addition, in the above embodiment, the detection unit detects the direction reflected by the curved mirror based on whether the curved mirror is located to the left or to the right of the center of the lane in which the vehicle is traveling.

[0121] 7. In the above embodiment, the detection unit detects the direction reflected by the curved mirror based on the positional relationship between an ellipse corresponding to the mirror surface of the curved mirror and a support supporting the mirror surface.

[0122] According to these embodiments, the direction of the curved mirror can be detected relatively simply and accurately.

[0123] 8. In addition, in the above embodiment,

[0124] The rule not changed by the changing unit is a rule indicating an order of selecting one from each of a plurality of pre-set partial regions at different positions.

[0125] The changing unit changes the rule so that the frequency of selecting at least one partial region corresponding to the predetermined direction among the plurality of partial regions becomes higher.

[0126] According to this embodiment, even if the rule is changed, selection can be made from the same set of partial regions, so that the processing cost can be suppressed.

[0127] 9. In addition, in the above embodiment,

[0128] The rule not changed by the changing unit is a rule indicating an order of selecting one from each of a plurality of pre-set partial regions at different positions.

[0129] The rule changed by the changing unit is a rule that selects partial regions from a group of partial regions including partial regions corresponding to the predetermined direction and different from any of the plurality of partial regions in such a manner that a frequency of selection of at least one partial region corresponding to the predetermined direction becomes higher.

[0130] According to this embodiment, the partial area can be set more flexibly with respect to the gaze direction corresponding to the detection of the curved mirror.

[0131] 10. In addition, in the above embodiment, the group of partial areas is formed by adding a partial area corresponding to the predetermined direction and different from any of the multiple partial areas to the multiple partial areas, or by exchanging any of the multiple partial areas with the different partial area.

[0132] According to this embodiment, the partial area can be set more flexibly with respect to the gaze direction corresponding to the detection of the curved mirror.

[0133] 11. In addition, according to the above embodiment, the photographing device for photographing the image requiring the distortion correction is a fisheye camera.

[0134] According to the above embodiment, the wide imaging range of the fisheye camera can be effectively utilized, and the number of cameras to be installed and the image processing systems associated therewith can be reduced, which can contribute to power saving of the vehicle control device.

[0135] 12. In addition, according to the above embodiment,

[0136] The image processing device is connected to a plurality of photographing devices, and the plurality of photographing devices are configured to photograph the entire circumference of the vehicle and to photograph the image requiring the distortion correction.

[0137] The changing unit changes the rule for an imaging device, among the plurality of imaging devices, whose imaging range includes the predetermined direction.

[0138] According to the above embodiment, the surrounding environment based on the captured image can be recognized for the entire circumference of the vehicle, and when detecting the curved mirror, the image of the area corresponding to the direction determined to be the direction to be looked at can be acquired with high frequency.

[0139] 13. In addition, the above embodiment discloses the following vehicle control device. That is,

[0140] The vehicle control device comprises:

[0141] The image processing device according to any one of items 1 to 12 above;

[0142] a recognition unit that recognizes a surrounding environment based on the distortion-corrected partial image acquired by the correction unit; and

[0143] A control unit performs control for driving assistance or automatic driving based on the surrounding environment recognized by the recognition unit.

[0144] According to this embodiment, a vehicle control device is provided that realizes more rapid and appropriate driving assistance and automatic driving control based on detection of a curved mirror.

[0145] 14. In addition, the above embodiment discloses an image processing method, which performs external recognition for driving assistance or automatic driving of a vehicle based on a captured image obtained from a capturing device that captures an image that requires distortion correction, wherein:

[0146] The image processing method comprises:

[0147] a correction step in which a partial area to be subjected to distortion correction processing is moved and set according to a rule, and a partial image after distortion correction is acquired by applying the distortion correction processing to the set partial area of ​​the captured image,

[0148] a detection step, in which a curved mirror is detected around the vehicle; and

[0149] A changing step, in which, when a curved mirror is detected in the detecting step, the rule is changed so that a partial area corresponding to a predetermined direction of the vehicle is preferentially set as an object of the distortion correction process.

[0150] According to the above embodiment, since it is possible to preferentially acquire an image of a partial area corresponding to the gaze direction obtained by detection of the curved mirror, it is possible to detect objects with high frequency and high accuracy in the gaze direction suitable for the road conditions. Therefore, appropriate control can be performed in driving assistance control and automatic driving control.

[0151] 15. Furthermore, the above-mentioned embodiments disclose a program for causing a computer to function as each unit of the above-mentioned image processing device or vehicle control device.

[0152] For example, by executing the above program by the ECU possessed by the vehicle control device, the image of the partial area corresponding to the gaze direction obtained by the detection of the curved mirror is preferentially acquired, so that the object can be detected with high frequency and high accuracy in the gaze direction suitable for the road conditions.

[0153] As mentioned above, although embodiment of the invention was described, the invention is not limited to the above-mentioned embodiment, Various deformation|transformation and change are possible within the range of the summary of the invention.

Claims

1. An image processing device for performing external recognition for driving assistance or automatic driving of a vehicle based on a captured image obtained from a capturing device that captures an image requiring distortion correction, It is characterized in that The image processing device comprises: a correction unit that sequentially moves and sets a partial area to be subjected to distortion correction processing in accordance with a rule and in each frame, and acquires a partial image after distortion correction by applying the distortion correction processing to the set partial area of ​​the captured image; a detection unit that detects a curved mirror around the vehicle; as well as A changing unit changes the rule so that a partial area corresponding to a predetermined direction of the vehicle is selected as a target of the distortion correction process more frequently when the curved mirror is detected by the detecting unit.

2. The image processing device according to claim 1, characterized in that The detection unit detects the curved mirror based on the image of the partial area after the distortion correction by the correction unit.

3. The image processing device according to claim 1 or 2, characterized in that: The detection unit detects the curved mirror based on an image captured by an imaging device that captures an image that is not a target of distortion correction.

4. The image processing device according to claim 1 or 2, characterized in that: The changing unit changes the rule so that the frequency of selecting at least one partial region corresponding to at least one of the left front and the right front of the vehicle as a target of the distortion correction process becomes higher.

5. The image processing device according to claim 1 or 2, characterized in that: The detection unit further detects the direction reflected by the detected curved mirror. The changing unit changes the rule so that the frequency of selecting a partial area corresponding to the detected direction reflected by the curved mirror as a target of the distortion correction process becomes higher.

6. The image processing device according to claim 5, characterized in that: The detection unit detects the direction reflected by the curved mirror based on whether the curved mirror is located to the left or to the right of the center of the lane in which the vehicle is traveling.

7. The image processing device according to claim 5, characterized in that: The detection unit detects a direction reflected by the curved mirror based on a positional relationship between an ellipse corresponding to a mirror surface of the curved mirror and a support supporting the mirror surface.

8. The image processing device according to claim 1 or 2, characterized in that: The rule not changed by the changing unit is a rule indicating the order of selecting one by one from a plurality of pre-set partial regions at different positions. The changing unit changes the rule so that the frequency of selecting at least one partial region corresponding to the predetermined direction among the plurality of partial regions becomes higher.

9. The image processing device according to claim 1 or 2, characterized in that: The rule not changed by the changing unit is a rule indicating the order of selecting one by one from a plurality of pre-set partial regions at different positions. The rule changed by the changing unit is a rule that selects partial regions from a group of partial regions including partial regions corresponding to the predetermined direction and different from any of the plurality of partial regions in such a manner that a frequency of selection of at least one partial region corresponding to the predetermined direction becomes higher.

10. The image processing device according to claim 9, characterized in that: The group of partial regions is formed by adding a partial region corresponding to the predetermined direction and different from any of the plurality of partial regions to the plurality of partial regions, or replacing any of the plurality of partial regions with the different partial region.

11. The image processing device according to claim 1 or 2, characterized in that: The photographing device for photographing the image requiring the distortion correction is a fisheye camera.

12. The image processing device according to claim 1 or 2, characterized in that: The image processing device is connected to a plurality of photographing devices, and the plurality of photographing devices are configured to photograph the entire circumference of the vehicle and to photograph the image requiring the distortion correction. The changing unit changes the rule for an imaging device, among the plurality of imaging devices, whose imaging range includes the predetermined direction.

13. A vehicle control device, characterized in that: The vehicle control device comprises: An image processing device for performing external recognition for driving assistance or automatic driving of a vehicle based on a captured image obtained from a capturing device that captures an image requiring distortion correction, wherein the image processing device comprises: a correction unit that sequentially moves and sets a partial area to be a target of distortion correction processing in accordance with a rule and for each frame, and acquires a partial image after distortion correction by applying the distortion correction processing to the set partial area of ​​the captured image; a detection unit that detects a curved mirror around the vehicle; and a changing unit that changes the rule in such a manner that a frequency of selecting a partial area corresponding to a predetermined direction of the vehicle as an object of the distortion correction processing becomes higher when the curved mirror is detected by the detecting unit; an identification unit, which identifies the surrounding environment based on the partial image obtained by the correction unit and after distortion correction; as well as A control unit performs control for driving assistance or automatic driving based on the surrounding environment recognized by the recognition unit.

14. An image processing method for performing external environment recognition for driving assistance or automatic driving of a vehicle based on an image obtained from a shooting device that shoots an image that requires distortion correction, characterized in that: The image processing method comprises: a correction step, in which a partial area to be subjected to distortion correction processing is sequentially moved and set in accordance with a rule and frame by frame, and a partial image after distortion correction is acquired by applying the distortion correction processing to the set partial area of ​​the captured image; a detection step, in which a curved mirror is detected around the vehicle; and A changing step, in which, when the curved mirror is detected in the detecting step, the rule is changed so that the frequency of selecting a partial area corresponding to a predetermined direction of the vehicle as a target of the distortion correction processing becomes higher.

15. A storage medium storing a program for causing a computer to execute an image processing method for performing external recognition for driving assistance or automatic driving of a vehicle based on an image obtained from a photographing device that photographs an image that requires distortion correction, characterized in that: The image processing method comprises: a correction step, in which a partial area to be subjected to distortion correction processing is sequentially moved and set in accordance with a rule and frame by frame, and a partial image after distortion correction is acquired by applying the distortion correction processing to the set partial area of ​​the captured image; a detection step, in which a curved mirror is detected around the vehicle; and A changing step, in which, when the curved mirror is detected in the detecting step, the rule is changed so that the frequency of selecting a partial area corresponding to a predetermined direction of the vehicle as a target of the distortion correction processing becomes higher.

Images