Vehicle, control device and control method thereof

Abstract

To reduce the influence of a shield in target recognition using an image photographed by a camera to which a wide-angle lens is attached.SOLUTION: A control device for a vehicle having a photographing device includes image acquisition means for periodically acquiring the image of the external field of the vehicle from the photographing device, first recognition means for recognizing the target of the external field of the vehicle on the basis of the image acquired from the photographing device, area control means for controlling the setting of a predetermined area with respect to the image acquired from the photographing device, and processing means for performing distortion reduction processing for reducing the distortion of the image in the set predetermined area. When a specific target recognized by the first recognition means is included in the predetermined area, the area control means changes the predetermined area, so as to increase an area other than the specific target included in the predetermined area.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a vehicle and a control device and control method thereof. [Background technology]

[0002] There is a known technology that uses a camera to capture the outside world of a vehicle and uses it to assist the driver. Patent Document 1 proposes a technology that captures a wide area behind a vehicle using a camera with a wide-angle lens such as a fisheye lens. In this technology, a coordinate transformation is performed on an image captured by a camera equipped with a wide-angle lens, with the optical axis position of the lens as the center, and the central area is cut out and displayed so that the driver can focus on the central area of ​​the image when parking. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-171964 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when an image captured by a camera equipped with a wide-angle lens is subjected to coordinate transformation and a portion of the image is cut out to recognize the outside world of the vehicle, if an obstruction (e.g., a guard rail) is present in the captured image, the desired recognition result may not be obtained because the area other than the obstruction where a more important recognition target may exist is insufficient.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that can reduce the influence of obstructions in target recognition using images captured by a camera equipped with a wide-angle lens. [Means for solving the problem]

[0006] According to the present invention, A control device for a vehicle having an imaging device, an image acquisition means for periodically acquiring an image of the outside world of the vehicle from the imaging device; A first recognition means for recognizing an object in an external environment of the vehicle based on an image acquired from the imaging device; an area control means for controlling setting of a predetermined area in an image acquired from the photographing device; a processing means for performing a distortion reduction process for reducing image distortion in the set predetermined area, The area control means A first region of an image acquired from the imaging device, The specific target recognized by the first recognition means If the specific target is not included, the first area is set as the predetermined area, and the specific target is not included in the first area. If the specific target is included, the specific area is expanded so that the area other than the specific target included in the specific area is increased. From the first region to the second region A control device is provided that varies the Effect of the Invention

[0007] According to the present invention, it is possible to reduce the influence of obstructions in target recognition using images captured by a camera equipped with a wide-angle lens. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a vehicle according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a diagram illustrating the range of photography by each camera of a vehicle. [Diagram 3] FIG. 1 is a diagram illustrating an example of an image captured by a fisheye camera and a case where distortion reduction processing is applied to the image. [Figure 4] Diagram explaining the region around the transformation center [Diagram 5] FIG. 1 is a diagram illustrating an example of changing the region around the transformation center when an occluder is present. [Figure 6] FIG. 1 illustrates an example of modification of the region around the transformation center and an example of an image generated by a distortion reduction process. [Figure 7]FIG. 13 is a diagram illustrating another example of changing the region around the transformation center when an occluder is present. [Figure 8] FIG. 1 illustrates an example of modification of the region around the transformation center and an example of an image generated by a distortion reduction process. [Figure 9] A flowchart showing a series of operations in a process for recognizing the external environment of a vehicle according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] (Vehicle configuration) Fig. 1 is a block diagram of a vehicle 1 according to an embodiment of the present invention. In Fig. 1, the vehicle 1 is generally shown in a plan view and a side view. As an example, the vehicle 1 is a sedan-type four-wheeled passenger car. The vehicle 1 may be such a four-wheeled vehicle, or may be a two-wheeled vehicle or another type of vehicle.

[0011] The vehicle 1 includes a vehicle control device 2 (hereinafter, simply referred to as the control device 2) that controls the vehicle 1. The control device 2 includes a plurality of ECUs (Electronic Control Units) 20 to 29 that are communicatively connected via an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), a memory such as a semiconductor memory, an interface with an external device, and the like. The memory stores a program executed by the processor and data used by the processor for processing, and the like. Each ECU may include a plurality of processors, memories, interfaces, and the like. For example, the ECU 20 includes a processor 20a and a memory 20b. The processor 20a executes instructions including a program stored in the memory 20b, whereby the ECU 20 executes processing. Alternatively, the ECU 20 may include a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for executing processing by the ECU 20. The same applies to the other ECUs.

[0012] The following describes the functions and the like handled by each of the ECUs 20 to 29. Note that the number of ECUs and the functions handled by them can be designed appropriately, and they can be divided more finely or integrated than in this embodiment.

[0013] The ECU 20 executes control related to the automatic driving of the vehicle 1. In the automatic driving, at least one of the steering and the acceleration / deceleration of the vehicle 1 is automatically controlled. The automatic driving by the ECU 20 may include automatic driving that does not require driving operation by the driver (which may also be called automatic driving) and automatic driving that assists the driving operation by the driver (which may also be called driving assistance).

[0014] The ECU 21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism for steering the front wheels in response to a driver's driving operation (steering operation) on a steering wheel 31. The electric power steering device 3 also includes a motor that generates a driving force for assisting the steering operation and automatically steering the front wheels, a sensor that detects the steering angle, and the like. When the driving state of the vehicle 1 is autonomous driving, the ECU 21 automatically controls the electric power steering device 3 in response to an instruction from the ECU 20, and controls the traveling direction of the vehicle 1.

[0015] The ECUs 22 and 23 control the detection units that detect the surrounding conditions of the vehicle and process information on the detection results. The vehicle 1 includes one standard camera 40 and four fisheye cameras 41 to 44 as detection units that detect the surrounding conditions of the vehicle. The standard camera 40 and the fisheye cameras 42 and 44 are connected to the ECU 22. The fisheye cameras 41 and 43 are connected to the ECU 23. The ECUs 22 and 23 can extract the contours of targets and lane markings (white lines, etc.) on the road by analyzing images captured by the standard camera 40 and the fisheye cameras 41 to 44. In addition, it is also possible to recognize the type and area of ​​targets that are relatively close by.

[0016] The fisheye cameras 41 to 44 are cameras to which a fisheye lens is attached. The configuration of the fisheye camera 41 will be described below. The other fisheye cameras 41 to 44 may have the same configuration. The angle of view of the fisheye camera 41 is wider than that of the standard camera 40. Therefore, the fisheye camera 41 can capture a wider range than the standard camera 40. The image captured by the fisheye camera 41 has a large distortion compared to the image captured by the standard camera 40. Therefore, the ECU 23 can perform simple target recognition using the image captured by the fisheye camera 41, but may perform a conversion process (hereinafter referred to as "distortion reduction process") to reduce distortion on the image before performing a highly accurate analysis. On the other hand, the ECU 22 may not perform distortion reduction process on the image captured by the standard camera 40 before analyzing the image. In this way, the standard camera 40 is an imaging device that captures an image that is not subject to distortion reduction process, and the fisheye camera 41 is an imaging device that captures an image that is subject to distortion reduction process. Instead of the standard camera 40, other imaging devices that capture images that are not subject to distortion reduction processing, such as cameras equipped with wide-angle or telephoto lenses, may be used.

[0017] The standard camera 40 is attached to the center of the front of the vehicle 1 and photographs the surroundings in front of the vehicle 1. The fisheye camera 41 is attached to the center of the front of the vehicle 1 and photographs the surroundings in front of the vehicle 1. In FIG. 1, the standard camera 40 and the fisheye camera 41 are shown aligned horizontally. However, the arrangement of the standard camera 40 and the fisheye camera 41 is not limited to this, and for example, they may be aligned vertically. In addition, at least one of the standard camera 40 and the fisheye camera 41 may be attached to the front of the roof of the vehicle 1 (for example, on the inside of the passenger compartment of the front window). The fisheye camera 42 is attached to the center of the right side of the vehicle 1 and photographs the surroundings to the right of the vehicle 1. The fisheye camera 43 is attached to the center of the rear of the vehicle 1 and photographs the surroundings behind the vehicle 1. The fisheye camera 44 is attached to the center of the left side of the vehicle 1 and photographs the surroundings to the left of the vehicle 1.

[0018] The type, number and mounting positions of the cameras of the vehicle 1 are not limited to the above examples. The vehicle 1 may also include a lidar (Light Detection and Ranging) or a millimeter wave radar as a detection unit for detecting targets around the vehicle 1 and measuring the distance to the targets.

[0019] The ECU 22 controls the standard camera 40 and the fisheye cameras 42 and 44 and processes information on the detection results. The ECU 23 controls the fisheye cameras 41 and 43 and processes information on the detection results. By dividing the detection unit that detects the surrounding conditions of the vehicle into two systems, the reliability of the detection results can be improved.

[0020] The ECU 24 controls the gyro sensor 5, the GPS sensor 24b, and the communication device 24c, and processes information on the detection results or communication results. The gyro sensor 5 detects the rotational motion of the vehicle 1. The path of the vehicle 1 can be determined based on the detection results of the gyro sensor 5, the wheel speed, etc. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c wirelessly communicates with a server that provides map information and traffic information to obtain this information. The ECU 24 can access a database 24a of map information constructed in a memory, and the ECU 24 performs route searches from the current location to a destination, etc. The ECU 24, the map database 24a, and the GPS sensor 24b constitute a so-called navigation device.

[0021] The ECU 25 includes a communication device 25a for vehicle-to-vehicle communication. The communication device 25a performs wireless communication with other surrounding vehicles and exchanges information between the vehicles.

[0022] The ECU 26 controls the power plant 6. The power plant 6 is a mechanism that outputs a driving force to rotate the drive wheels of the vehicle 1, and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine in response to a driving operation (accelerator operation or acceleration operation) of the driver detected by an operation detection sensor 7a provided on the accelerator pedal 7A, for example, and switches the gear stage of the transmission based on information such as the vehicle speed detected by a vehicle speed sensor 7c. When the driving state of the vehicle 1 is autonomous, the ECU 26 automatically controls the power plant 6 in response to an instruction from the ECU 20, and controls the acceleration and deceleration of the vehicle 1.

[0023] The ECU 27 controls lighting devices (headlights, taillights, etc.) including turn signals 8 (blinkers). In the example of Fig. 1, the turn signals 8 are provided at the front, door mirrors, and rear of the vehicle 1.

[0024] The ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and accepts information input from the driver. The audio output device 91 notifies the driver of information by audio. The display device 92 notifies the driver of information by displaying an image. The display device 92 is arranged on the surface of the driver's seat, for example, and constitutes an instrument panel or the like. Note that, although audio and display are exemplified here, information may be notified by vibration or light. Information may be notified by a combination of audio, display, vibration, or light. Furthermore, the combination or the notification mode may be changed depending on the level of the information to be notified (for example, urgency). The input device 93 is a group of switches arranged in a position operable by the driver and used to give instructions to the vehicle 1, but may also include an audio input device.

[0025] The ECU 29 controls the brake device 10 and the parking brake (not shown). The brake device 10 is, for example, a disk brake device provided on each wheel of the vehicle 1, and applies resistance to the rotation of the wheel to decelerate or stop the vehicle 1. The ECU 29 controls the operation of the brake device 10 in response to the driver's driving operation (brake operation) detected by, for example, an operation detection sensor 7b provided on the brake pedal 7B. When the driving state of the vehicle 1 is automatic driving, the ECU 29 automatically controls the brake device 10 in response to an instruction from the ECU 20, and controls the deceleration and stop of the vehicle 1. The brake device 10 and the parking brake can also be operated to maintain the stopped state of the vehicle 1. In addition, when the transmission of the power plant 6 is equipped with a parking lock mechanism, this can also be operated to maintain the stopped state of the vehicle 1.

[0026] The shooting ranges of standard camera 40 and fisheye cameras 41 to 44 will be described with reference to Fig. 2. Fig. 2(a) shows the horizontal shooting range of each camera, Fig. 2(b) shows the vertical shooting range of fisheye camera 42 on the right side of vehicle 1, and Fig. 2(c) shows the vertical shooting range of fisheye camera 43 attached to the rear of vehicle 1.

[0027] First, the imaging range in a plan view of the vehicle 1 (i.e., the horizontal direction of the vehicle 1) will be described with reference to Fig. 2(a). The standard camera 40 captures the scenery included in the imaging range 200. The imaging center 200C of the standard camera 40 faces directly ahead of the vehicle 1. The horizontal angle of view of the standard camera 40 may be less than 90°, and may be, for example, about 45° or about 30°.

[0028] Fisheye camera 41 captures the scenery included in shooting range 201. The shooting center 201C of fisheye camera 41 faces directly in front of vehicle 1. Fisheye camera 42 captures the scenery included in shooting range 202. The shooting center 202C of fisheye camera 42 faces directly to the right of vehicle 1. Fisheye camera 43 captures the scenery included in shooting range 203. The shooting center 203C of fisheye camera 43 faces directly behind vehicle 1. Fisheye camera 44 captures the scenery included in shooting range 204. The shooting center 204C of fisheye camera 44 faces directly to the left of vehicle 1. The horizontal angle of view of fisheye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, or greater than 180°, and may be, for example, approximately 180°. FIG. 2(a) shows an example in which the horizontal angle of view of fish-eye cameras 41 to 44 is 180°.

[0029] The shooting range 201 can be divided into an area 201L diagonally forward to the left of the vehicle 1, an area 201F directly in front of the vehicle 1, and an area 201R diagonally forward to the right of the vehicle 1. The shooting range 202 can be divided into an area 202L diagonally forward to the right of the vehicle 1, an area 202F directly to the right of the vehicle 1, and an area 202R diagonally rearward to the right of the vehicle 1. The shooting range 203 can be divided into an area 203L diagonally rearward to the right of the vehicle 1, an area 203F directly behind the vehicle 1, and an area 203R diagonally rearward to the left of the vehicle 1. The shooting range 204 can be divided into an area 204L diagonally rearward to the right of the vehicle 1, an area 204F directly to the left of the vehicle 1, and an area 204R diagonally forward to the left of the vehicle 1. The imaging range 201 may be divided equally into three areas 201L, 201F, and 201R (that is, so that the angle of view of each area is equal). The other imaging ranges 202 to 204 may also be divided equally into three.

[0030] Since the standard camera 40 and the fisheye cameras 41-44 have the above-mentioned shooting ranges 200-204, the direct front and the four diagonal directions of the vehicle 1 are included in the shooting ranges of the two separate cameras. Specifically, the direct front of the vehicle 1 is included in both the shooting range 200 of the standard camera 40 and the area 201F of the shooting range 201 of the fisheye camera 41. The right diagonal front of the vehicle 1 is included in both the area 201R of the shooting range 201 of the fisheye camera 41 and the area 202L of the shooting range 202 of the fisheye camera 42. The same is true for the other three diagonal directions of the vehicle 1.

[0031] Next, the vertical imaging range of the vehicle 1 will be described with reference to Fig. 2(b) and Fig. 2(c). Fig. 2(b) describes the vertical imaging range of the fisheye camera 42, and Fig. 2(c) describes the vertical imaging range of the fisheye camera 43. The vertical imaging ranges of the other fisheye cameras 41 and 44 may be similar.

[0032] The vertical angle of view of the fish-eye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, or greater than 180°, for example, about 180°. FIGS. 2(b) and (c) show an example in which the vertical angle of view of the fish-eye cameras 41 to 44 is 180°. The imaging center 203C of the fish-eye camera 43 faces in a direction parallel to the ground. Alternatively, the imaging center 203C of the fish-eye camera 43 may face downward (toward the ground) from the direction parallel to the ground, or upward (opposite the ground) from the direction parallel to the ground. Furthermore, the imaging centers 201C to 204C of the fish-eye cameras 41 to 44 may face in different directions from each other in the vertical direction.

[0033] (Overview of distortion reduction processing) Next, an overview of the distortion reduction process that takes into account obstructions for images captured by the fisheye cameras 41 to 44 will be described with reference to Fig. 3 to Fig. 8. As shown in Fig. 3, image 300 is an image of the scenery to the right of vehicle 1 captured by fisheye camera 42. As shown, image 300 has large distortion, especially in the peripheral area.

[0034] 4(a) shows a view of the vehicle 1 seen from above, and FIG. 4(b) shows a view of the vehicle 1 seen from behind. Image 300 is an image of the situation to the right of the vehicle 1 captured by fisheye camera 43.

[0035] The ECU 22 connected to the fisheye camera 43 performs distortion reduction processing on the image 300. Specifically, as shown in Fig. 3, the ECU 22 determines one point in the image 300 as a transformation center 301. As shown in Figs. 4(a) and (b), the transformation center 301 is located on the right side of the shooting area 202 as seen from the fisheye camera 42 in the horizontal direction, and faces in a direction parallel to the ground in the vertical direction. Note that the example shown in Fig. 4 describes a case where this embodiment is applied to 202R, but the embodiment described below can be applied to any of 202L, 202F, and 202R.

[0036] The ECU 22 cuts out an area 302 centered on a transformation center 301 from an image 300. For ease of explanation, the area 302 is illustrated as a rectangle, but the shape of the area 302 does not have to be the shape shown in FIG. 3 as long as the area 302 corresponds to an image 303 described later. As shown in FIG. 4(a), the area 302 corresponds to an area 202T in an area 202R located on the right side of the shooting range 202 as seen from the fisheye camera 42. The ECU 22 performs distortion reduction processing on the area 302 centered on the transformation center 301 to generate an image 303 with reduced distortion. The image 303 corresponds to the width of the area 202T in FIG. 4(a) and the height of the area 202T in FIG. 4(b).

[0037] As a result of the distortion reduction process, the distortion is reduced in positions closer to the transformation center 301, and is not reduced or is increased in positions farther from the transformation center 301. When the entire image 300 is subjected to the distortion reduction process, the distortion is increased in areas far from the transformation center 301. Therefore, even if the external environment of the vehicle 1 is analyzed using an area far from the transformation center 301, an accurate analysis cannot be performed. Therefore, the ECU 22 sets the transformation center 301 in the area to be analyzed, performs the distortion reduction process on the area around the transformation center 301, and analyzes the condition of the area to be analyzed using the processed image.

[0038] When the ECU 22 wants to analyze the situation diagonally to the right of the vehicle 1, the ECU 22 sets the transformation center 301 in the area 202L (for example, the center of the area 202L) included in the shooting range 202 of the fisheye camera 42, performs distortion reduction processing on the area around the transformation center 301, and analyzes the situation diagonally to the right of the vehicle 1 using the processed image. When the ECU 22 wants to analyze the situation on the right front of the vehicle 1, the ECU 22 sets the transformation center 301 in the area 202F (for example, the center of the area 202F) included in the shooting range 202 of the fisheye camera 42, performs distortion reduction processing on the area around the transformation center 301, and analyzes the situation on the right front of the vehicle 1 using the processed image. When the ECU 22 wants to analyze the situation diagonally to the right of the vehicle 1, the ECU 22 sets the transformation center 301 in the area 202R (for example, the center of the area 202R) included in the shooting range 202 of the fisheye camera 42, performs distortion reduction processing on the area around the transformation center 301, and analyzes the situation on the right front of the vehicle 1 using the processed image.

[0039] In this way, when ECU 22 analyzes the situation to the right of the vehicle, for example, upon acquiring an image from fisheye camera 42, ECU 22 sets transformation center 301 in the direction required to analyze the situation and performs distortion reduction processing on the area around transformation center 301.

[0040] On the other hand, when acquiring an image captured by the fisheye camera 42, there may be cases where an obstruction 501 such as a guard rail exists within the region around the set transformation center 301, and the region other than the obstruction becomes small (see, for example, the upper diagram of FIG. 7). When the obstruction occupies a certain range in the region 302 around the transformation center 301, analysis of the region other than the obstruction cannot be performed sufficiently, or analysis processing of the obstruction region is performed in vain. The obstruction 501 exists, for example, between the road and the sidewalk, and blocks image recognition of targets and traffic participants that may exist on the sidewalk side. The obstruction includes not only guardrails, but also grass and trees that are 700 mm or more in height. The obstruction includes an obstruction that extends in the traveling direction of the vehicle (for example, horizontally).

[0041] If the ECU 22 detects a horizontally extending obstruction in the region 302 (e.g., if such an obstruction is recognized from the image 300 as described below), the ECU 22 can modify the region around the transformation center 301 so as to increase the area other than the obstruction.

[0042] For example, when the region 302 includes an obstruction extending in the horizontal direction, the ECU 22 can increase the region other than the obstruction included in the region around the transformation center 301 by changing the region around the transformation center 301 as shown in Fig. 5. For example, the ECU 22 changes the region around the transformation center 301 so that the image 303 becomes a rectangle elongated in the vertical direction. As shown in Fig. 5(a), the ECU 22 narrows the region 202T around the transformation center 301 in the horizontal plane when the vehicle 1 is viewed from above, and as shown in Fig. 5(b), the ECU 22 widens the region 202T around the transformation center 301 in the vertical plane when the vehicle 1 is viewed from behind.

[0043] FIG. 6 is a schematic diagram showing an example in which the region around the transformation center 301 is changed from the region 302 to the region 302b according to the example of the image 300 shown in FIG. 3. The region 302 corresponds to the horizontally long rectangular image 303, whereas the region 302b corresponds to the vertically long rectangular image 303b. When the region 302 includes a horizontally extending obstruction 501 at the lower side, the ECU 22 changes the region 302 to the region 302b, so that the region other than the obstruction included in the region 302b increases. In this way, when the ECU 22 performs a distortion reduction process on the region 302b around the transformation center 301, the region other than the obstruction increases in the processed image 303b as well. In this way, the ECU 22 changes the region 302 around the transformation center so that the region other than the obstruction increases, thereby reducing the influence of the obstruction in target recognition using an image captured by a fisheye camera.

[0044] This process of changing the area around the transformation center 301 so that the image 303 becomes a rectangle that is long in the vertical direction is also useful when analyzing the presence of traffic participants, etc. in an area where a horizontally extending obstruction ends.

[0045] Additionally, other methods may be used to modify the region 302 around the transformation center to increase the non-occluded area.

[0046] 7, when the region 302 includes an obstruction extending in the horizontal direction (for example, when such an obstruction is recognized from the image 300 as described later), the ECU 22 can change the position of the transformation center 301 so as to increase the region other than the obstruction. Specifically, the ECU 22 sets the transformation center 301a so that the position in the image 300 is higher than the position of the transformation center 301, and performs distortion reduction processing on the region around the transformation center 301a.

[0047] FIG. 8 shows a schematic diagram of a case where the position of the transformation center 301 is changed to the position of the transformation center 301a (and the surrounding area is also changed) according to the example of the image 300 shown in FIG. 3. The ECU 22 changes the area 302 to the area 302a, so that the area other than the obstruction 501 included in the area 302a increases. In this way, when the ECU 22 performs a distortion reduction process on the area 302a around the transformation center 301a, the area other than the obstruction increases in the processed image 303a as well. In this way, the ECU 22 changes the position of the transformation center 301 so that the area other than the obstruction increases, and changes the area 302 around the transformation center, so that the influence of the obstruction can be reduced in target recognition using an image captured by a fisheye camera.

[0048] (A series of operations for processing the vehicle's external environment) Next, a series of operations in the process of recognizing the outside world of the vehicle will be described. A series of operations in the distortion reduction process will be described. This process is realized by the processor 20a of each of the ECUs 20 to 29 of the control device 2 executing a program in the memory 20b.

[0049] In addition, the process of recognizing the outside world of the vehicle may be performed while determining one or more areas to be subjected to the distortion reduction process among the images acquired from each camera according to the driving scene of the vehicle 1 (for example, a scene of entering a two-way intersection, a scene of driving on a narrow road, a scene of turning right or left at an intersection, etc.). However, in the following explanation, for example, a case where the distortion reduction process is performed on the area 204F on the side of the vehicle 1 (the left side of the vehicle) will be explained as an example. In this process, the orientation of the transformation center as viewed from the rear of the vehicle 1 is set to the horizontal direction, and the area around the transformation center is set so that the image corresponding to the area becomes a rectangle extending in the horizontal direction after the distortion reduction process.

[0050] In S901, the control device 2 acquires an image captured by the fish-eye camera 44 among the fish-eye cameras 41 to 44. The image captured by the fish-eye camera 44 is an image with large distortion, similar to the image 300 described above.

[0051] In S902, the control device 2 instructs the ECU 22 to execute a process of recognizing targets (obstructions) in the outside world of the vehicle 1 based on the image captured by the fisheye camera 44. The targets in the outside world of the vehicle to be recognized are obstructions such as the guard rail and grass described above. Since the obstructions assumed in this embodiment are relatively close to the vehicle 1 (for example, between the road and the sidewalk), the presence of the obstructions can be recognized even when an image captured by the fisheye camera 44 is used. The recognition of the obstructions can be realized by using a model using, for example, a known deep learning technique, which is trained using images of a fisheye camera including the obstructions as teacher data. When an unknown image of the fisheye camera is input, the trained model outputs the presence or absence of the obstruction in the image and the area of ​​the obstruction. The model can output one or more areas where the obstruction exists.

[0052] In S903, the control device 2 determines whether an obstruction extending in a specific direction (e.g., horizontal direction) is included in the region around the transformation center. If the control device 2 determines that an obstruction is included in the region around the transformation center, the process proceeds to S904, otherwise the process proceeds to S906. The control device 2 determines whether the shape of the obstruction recognized by the learning model is an obstruction extending in a specific direction (e.g., horizontal direction). Whether an obstruction extends in a specific direction may be determined, for example, depending on whether the horizontal length of the obstruction region in the image is equal to or greater than a predetermined length.

[0053] In S904, the control device 2 determines whether there is an area free of obstructions in the area around the transformation center based on the area of ​​obstructions determined in S902. For example, the control device 2 determines that there is an area free of obstructions when the area free of obstructions in the area around the transformation center is greater than a predetermined ratio (e.g., 15%). If the control device 2 determines that there is an area free of obstructions, it advances the process to S905, and if not, it advances the process to S908.

[0054] In S905, the control device 2 changes the area around the transformation center. The control device 2 changes the area around the transformation center, for example, by the method described with reference to Fig. 5 and Fig. 6. That is, when an obstruction is included in the area around the transformation center, the area around the transformation center is changed so that the area other than the obstruction increases. By using the method described with reference to Fig. 5 and Fig. 6, the area other than the obstruction can be easily secured in the area extending vertically relative to the obstruction extending horizontally (particularly above the obstruction).

[0055] At this time, the control device 2 changes the area around the transformation center so that the size of the image corresponding to the area around the transformation center before the change is the same as the size of the image corresponding to the area around the transformation center after the change. In this way, the size of the image after the subsequent distortion reduction process is the same before and after the change, so the load in the image recognition process using the image is constant. In other words, the processing time for each image can be kept within the required time, and power consumption can be kept within a predetermined level.

[0056] In S906, the control device 2 executes distortion reduction processing for the area around the transformation center. By the processing of S905, for example, the image corresponding to the area around the transformation center changes from a rectangle long in the horizontal direction (image 303) to a rectangle long in the vertical direction (image 303b).

[0057] In S907, the control device 2 recognizes targets in the outside world based on the image that has been subjected to distortion reduction processing. In particular, recognition of targets in the outside world is performed to recognize the situation in areas other than obstructions (to recognize the presence of traffic participants, etc.). Recognition of traffic participants and targets can be realized by using a model using, for example, a known deep learning technique, in which an image (corresponding to an image after distortion correction) containing a person or target to be recognized is trained as teacher data. When an unknown image (image after distortion reduction processing) is input, the trained model outputs the presence or absence of a recognition target in the image and the area of ​​that target. The model can output one or more areas in which the recognition target exists.

[0058] In S908, the control device 2 reduces the frequency of recognizing external targets in the area around the target transformation center (because the image is occupied by an obstruction). For example, the control device 2 reduces the frequency of recognizing targets in the area 204F on the side of the vehicle 1 (left side of the vehicle) and increases the frequency in the area 204L, or reduces the frequency of using the image on the side of the vehicle 1 (left side of the vehicle) for target recognition.

[0059] In S909, the control device 2 determines whether to end the operation, and if the operation is to be continued, returns the process to S901, and if not, ends this process.

[0060] In the above description, the case where the region around the transformation center is changed in S905 using the change method shown in Fig. 5 and Fig. 6 has been described as an example. However, the region around the transformation center may be changed using the method shown in Fig. 7 and Fig. 8. That is, the region around the transformation center is changed by moving the position of the transformation center in a direction perpendicular to the extension direction of the obstruction.

[0061] Furthermore, the area around the transformation center may be changed using other methods, not limited to the above-mentioned method. For example, the control device 2 may increase the area other than the obstruction included in the area around the transformation center by setting the position of the transformation center to an area where no obstruction exists. Alternatively, the control device 2 may set the area around the transformation center to within an area where no obstruction is recognized in the image captured by the fisheye camera. Even in this way, more accurate image analysis can be performed in the image to which the distortion reduction process is applied (as the area of ​​the obstruction is reduced or eliminated).

[0062] Furthermore, in the above embodiment, an example has been described in which processing is performed on images captured by cameras (fisheye cameras 42, 44) that capture the sides of the vehicle 1. However, the above processing can also be applied to images captured in front of or behind the vehicle 1. In other words, when an obstruction extending in the horizontal direction is included in an image captured in front of or behind the vehicle 1, the area to which the distortion reduction processing is applied can be controlled so that the area other than the obstruction becomes larger.

[0063] Further, all of the above-mentioned processes for recognizing the outside world of the vehicle do not have to be executed in the vehicle 1. For example, when an image captured by a fisheye camera is acquired on the vehicle 1 side, the acquired image may be transmitted to an external server (not shown), and all or part of the processes from S902 to S907 may be performed in the external server, and the vehicle 1 may receive the processing results.

[0064] As described above, in this embodiment, images of the outside world of the vehicle are periodically acquired from a wide-angle camera (fisheye camera), and targets (obstructions) in the outside world of the vehicle are recognized based on the images acquired from the wide-angle camera. In addition, in this embodiment, the setting of a predetermined area (area around the transformation center) is controlled for the image from the wide-angle camera, and distortion reduction processing is performed to reduce image distortion for the set predetermined area. At this time, in this embodiment, if an obstruction is included in the predetermined area (area around the transformation center), the predetermined area (area around the transformation center) is changed so that the area other than the obstruction increases. In this way, in the image to which the distortion reduction processing is applied, (the area of ​​the obstruction is reduced), more accurate image analysis can be performed. In other words, the influence of the obstruction can be reduced in target recognition using an image taken by a camera equipped with a wide-angle lens.

[0065] <Summary of the embodiment> 1. The control device (e.g., 2) of the above embodiment is a control device of a vehicle (e.g., 1) having an imaging device (e.g., 41 to 44), Image acquisition means (e.g., 22, 23, S901) for periodically acquiring images of the outside world of the vehicle from an image capture device; A first recognition means (e.g., 2, 22, 23, S902) for recognizing an object in the external environment of the vehicle based on an image acquired from the photographing device; A region control means (e.g., 2, 22, 23, S905) for controlling the setting of a predetermined region for an image acquired from an image capturing device; A processing means (e.g., 2, 22, 23, S906) for performing a distortion reduction process for reducing image distortion in the set predetermined area, When the specific target recognized by the first recognition means is included in the predetermined area, the area control means changes the predetermined area so that the area other than the specific target included in the predetermined area increases (e.g., S905).

[0066] According to this embodiment, it is possible to reduce the influence of obstructions in target recognition using images captured by a camera equipped with a wide-angle lens.

[0067] 2. In the above embodiment, The area control means sets the first area (e.g., 302) of an image acquired from the photographing device as a predetermined area when the first area does not include a specific target (e.g., an obstruction) (e.g., S903), and changes the predetermined area from the first area (e.g., 302) to a second area (e.g., 302b, 302a) when the first area includes a specific target (e.g., S903 to S905) so that the area other than the specific target included in the predetermined area increases.

[0068] According to this embodiment, when no obstruction is present, the distortion reduction process can be performed as is, whereas when an obstruction is present, the influence of the obstruction can be reduced.

[0069] 3. In the above embodiment, When the specific target is an obstruction extending in a first direction, the area control means changes the predetermined area from a first area (e.g., 302) to a second area (e.g., 302b) so that an image (e.g., 303b) corresponding to the predetermined area becomes an image elongated in a second direction perpendicular to the first direction.

[0070] According to this embodiment, it is possible to easily secure an area other than the obstruction in a region extending in a direction perpendicular to the obstruction extending in a specific direction.

[0071] 4. In the above embodiment, When a specific target is an obstruction extending horizontally, the area control means changes the predetermined area from a first area (e.g., 302) to a second area (e.g., 302b) so that an image long in the horizontal direction (e.g., 303b) becomes an image long in the vertical direction.

[0072] According to this embodiment, it is possible to easily secure an area other than the shielding object in an area extending in a vertical direction to the horizontally extending shielding object (particularly above the shielding object).

[0073] 5. In the above embodiment, The area control means changes the specified area from a first area to a second area so that the size of an image corresponding to the first area (e.g., 303) and the size of an image corresponding to the second area (e.g., 303b) become the same.

[0074] According to this embodiment, the size of the image after the distortion reduction process is the same before and after the change, so the load in the image recognition process using the image is constant, which means that the processing time for each image can be kept within the required time and power consumption can be kept within a predetermined level.

[0075] 6. In the above embodiment, The predetermined region (e.g., 302) is a region defined by a transformation center (e.g., 301) defined in an image acquired from an imaging device, The area control means changes a predetermined area (eg, 302a) by controlling the position of the transformation center (eg, 301, 301a in FIG. 8).

[0076] According to this embodiment, by targeting an area close to the transformation center as the transformation target, it is possible to change the area to be subjected to the distortion reduction process while suppressing distortion of the image to be subjected to the distortion reduction process.

[0077] 7. In the above embodiment, The area control means changes the predetermined area so that the transformation center is set to an area where no specific target exists.

[0078] According to this embodiment, more accurate image analysis can be performed in the areas where distortion reduction processing is applied (as areas of occlusions are reduced or eliminated).

[0079] 8. In the above embodiment, The imaging device is equipped with a fisheye lens (eg, 41 to 44).

[0080] According to this embodiment, a wide range of conditions outside the vehicle can be analyzed using a single camera.

[0081] 9. In the above embodiment, The image capturing devices (eg, 41 to 44) are arranged to capture an image of the outside world in at least one direction among the front, side, and rear of the vehicle.

[0082] According to this embodiment, the situation in various directions outside the vehicle can be analyzed.

[0083] 10. In the above embodiment, The vehicle further includes a second recognition means (eg, 2, 21 to 24) for recognizing the outside world of the vehicle based on the image with reduced distortion.

[0084] According to this embodiment, more accurate image analysis can be performed, such as confirming the presence of traffic participants or targets at greater distances.

[0085] 11. The control device (e.g., 2) of the above embodiment is a control device for a vehicle (e.g., 1) having an imaging device (e.g., 41 to 44), Image acquisition means (e.g., 22, 23, S901) for periodically acquiring images of the outside world of the vehicle from an image capture device; A recognition means (e.g., 2, 22, 23, S902) for recognizing an object in the external environment of the vehicle based on an image acquired from the photographing device; A region control means (e.g., 2, 22, 23, S905) for controlling the setting of a predetermined region for an image acquired from an image capturing device; A processing means (e.g., 2, 22, 23, S906) for performing a distortion reduction process for reducing image distortion in the set predetermined area, The area control means sets a predetermined area in an area where a specific target is not recognized by the recognition means (for example, S905).

[0086] According to this embodiment, it is possible to reduce the influence of obstructions in target recognition using images captured by a camera equipped with a wide-angle lens.

[0087] 12. The vehicle of the above embodiment, A vehicle (for example, 1) including the control device according to any one of 1 to 11 above.

[0088] According to this embodiment, it is possible to reduce the influence of obstructions in target recognition using images captured by a camera equipped with a wide-angle lens in a vehicle.

[0089] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0090] 1...vehicle, 2...control device, 41-44...fisheye camera, 21-29...ECU

Claims

1. A control device for a vehicle having an imaging device, an image acquisition means for periodically acquiring an image of the outside world of the vehicle from the imaging device; A first recognition means for recognizing an object in an external environment of the vehicle based on an image acquired by the imaging device; an area control means for controlling setting of a predetermined area in an image acquired from the photographing device; a processing means for performing a distortion reduction process for reducing image distortion in the set predetermined area, The control device is characterized in that, when a first region of an image acquired from the photographing device does not include a specific target recognized by the first recognition means, the region control means sets the first region as the specified region, and, when the specific target is included in the first region, changes the specified region from the first region to a second region so that an area other than the specific target included in the specified region is increased.

2. 2. The control device according to claim 1, wherein the area control means changes the specified area from the first area to the second area when the specific target is an obstruction extending in a first direction so that the image corresponding to the specified area becomes an image that is long in a second direction perpendicular to the first direction.

3. The control device according to claim 1 or 2, characterized in that, when the specific target is an obstruction extending in the horizontal direction, the area control means changes the specified area from the first area to the second area so that an image that is long in the horizontal direction becomes an image that is long in the vertical direction.

4. 4. The control device according to claim 1, wherein the area control means changes the specified area from the first area to the second area so that a size of an image corresponding to the first area and a size of an image corresponding to the second area become the same.

5. the predetermined region is a region defined by a transformation center defined in an image acquired from the imaging device, 5. The control device according to claim 1, wherein the area control means changes the predetermined area by controlling a position of the transformation center.

6. 6. The control device according to claim 5, wherein the area control means changes the predetermined area so that the transformation center is set to an area in which the specific target does not exist.

7. The control device according to claim 1 , wherein the image capturing device includes a fisheye lens.

8. 8. The control device according to claim 1, wherein the image capturing device is disposed so as to capture an image of an external environment in at least one direction selected from the front, the side, and the rear of the vehicle.

9. 9. The control device according to claim 1, further comprising a second recognition means for recognizing an external environment of the vehicle based on the image with reduced distortion.

10. A vehicle comprising the control device according to any one of claims 1 to 9.

11. A program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 9.

12. A method for controlling a vehicle having an imaging device, comprising: an image acquisition step of periodically acquiring an image of the outside world of the vehicle from the imaging device; a first recognition step of recognizing an object in an external environment of the vehicle based on an image acquired by the imaging device; a region control step of controlling setting of a predetermined region in an image acquired from the photographing device; and performing a distortion reduction process for reducing image distortion in the set predetermined area, A control method characterized in that, in the area control process, if a first area of ​​an image acquired from the photographing device does not include a specific target recognized in the first recognition process, the first area is set as the specified area, and if the specific target is included in the first area, the specified area is changed from the first area to a second area so that an area other than the specific target included in the specified area is increased.

Images