Control Device, Control Method, and Program for a Moving Body

Abstract

To provide a technology for accurately acquiring information on an object around a moving object from an image obtained by an imaging device equipped with a wide-angle lens.SOLUTION: A control device acquires an image obtained by photographing the external environment of a moving object from an imaging device equipped with a wide-angle lens, and detects a target object by image recognition on the basis of the image acquired by the imaging device. Furthermore, according to the detection result of the target object, the control device targets a partial region in the image acquired from the imaging device, centered on the detection position of the target object or in the vicinity thereof to perform distortion reduction processing for reducing the distortion of the image. The control device recognizes the external environment of the vehicle on the basis of the image obtained by the distortion reduction processing.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a control device, a control method, and a program for a moving body.

Background Art

[0002] It has been practiced to recognize the external environment of a vehicle from an image captured around the vehicle and use the recognition result for control such as driving support. In this case, in order to expand the detection range, it is also assumed to use a wide-angle camera such as a fish-eye lens. However, although such a camera can acquire a wide-range image, the acquired image is distorted. Therefore, when applying an object detection technique premised on an undistorted image obtained from a normal camera, the detection accuracy may decrease. Patent Document 1 discloses a technique for performing distortion reduction processing on a distorted image and performing object detection using the corrected image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to realize appropriate driving support control or automatic driving control based on the recognition result of the external environment of a moving body such as a vehicle, it is required to acquire precise information about objects that are stationary or moving around the moving body. However, if appropriate distortion reduction processing is not performed on an image obtained from a photographing device equipped with a wide-angle lens such as a fish-eye lens, it may not be possible to appropriately acquire information about objects existing around the moving body.

[0005] The present invention provides a technique for accurately acquiring information about objects around a moving body from an image obtained by a photographing device equipped with a wide-angle lens.

Means for Solving the Problem

[0006] A control device according to an aspect of the present invention is a control device for a moving body having a photographing device to which a wide-angle lens is attached, the control device including: an image acquisition unit that acquires an image obtained by photographing the external environment of the moving body from the photographing device; a determination unit that determines a target object to be detected in a detection process based on the image, based on the operating state of the moving body; a detection unit that detects the target object by image recognition based on the image acquired from the photographing device; a processing unit that performs a distortion reduction process for reducing distortion of an image on a partial area in the image acquired from the photographing device, the partial area being centered on the detection position of the target object or its vicinity; and a recognition unit that recognizes the external environment of the moving body based on the image obtained by the distortion reduction process. , the detection means outputs information indicating the recognition accuracy of the target object by the image recognition, and when the recognition accuracy indicated by the information exceeds the threshold value corresponding to the target object determined by the determination means, the processing means performs the distortion reduction processing, and the threshold value is predetermined for each type of object to be detected by the detection means. It is characterized by this. A control device according to another aspect of the present invention is a control device for a moving body having a photographing device to which a wide-angle lens is attached, the control device including: an image acquisition means for acquiring, from the photographing device, an image obtained by photographing the outside of the moving body; a determination means for determining a target object to be detected in a detection process based on the image based on the operating state of the moving body; a detection means for detecting the target object by image recognition based on the image acquired from the photographing device; a processing means for performing distortion reduction processing for reducing distortion of an image on a partial area in the image acquired from the photographing device, the partial area being centered on the detection position of the target object or its vicinity, according to the detection result by the detection means; and a recognition means for recognizing the outside of the moving body based on the image obtained by the distortion reduction processing, wherein the detection means outputs information indicating the recognition accuracy of the target object by the image recognition, and when the recognition accuracy indicated by the information exceeds the threshold value corresponding to the operating state, the processing means performs the distortion reduction processing, and the threshold value is predetermined for each operating state of the moving body. In the present invention Furthermore A control device according to another aspect of the present invention is a control device for a moving body having a photographing device to which a wide-angle lens is attached, the control device including: an image acquisition unit that acquires an image obtained by photographing the external environment of the moving body from the photographing device; a detection unit that detects a target object by image recognition based on the image acquired from the photographing device; a processing unit that performs a distortion reduction process for reducing distortion of an image on a partial area in the image acquired from the photographing device, the partial area being centered on the detection position of the target object or its vicinity; and a recognition unit that recognizes the external environment of the moving body based on the image obtained by the distortion reduction process, wherein the detection unit outputs information indicating the recognition accuracy of the target object by the image recognition, and the processing unit performs the distortion reduction process when the recognition accuracy indicated by the information exceeds a threshold value corresponding to the target object, and the threshold value is predetermined for each type of object to be detected by the detection unit. A control device according to still another aspect of the present invention is a control device for a moving body having a photographing device to which a wide-angle lens is attached, the control device including: an image acquisition unit that acquires an image obtained by photographing the outside of the moving body from the photographing device; a detection unit that detects a target object by image recognition based on the image acquired from the photographing device; a processing unit that performs a distortion reduction process for reducing distortion of an image on a partial region in the image acquired from the photographing device, the partial region being centered on the detection position of the target object or the vicinity thereof, according to the detection result by the detection unit; and a recognition unit that recognizes the outside of the moving body based on the image obtained by the distortion reduction process, wherein the detection unit outputs information indicating the recognition accuracy of the target object by the image recognition, and the processing unit performs the distortion reduction process when the recognition accuracy indicated by the information exceeds a threshold value corresponding to the operation state of the moving body, and the threshold value is predetermined for each operation state of the moving body.

Effects of the Invention

[0007] According to the present invention, it is possible to accurately acquire information regarding objects around the moving body from an image obtained from a photographing device to which a wide-angle lens is attached.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of the features described in the embodiments are essential for the invention. Two or more of the plurality of features described in the embodiments may be arbitrarily combined. Also, the same or similar configurations are given the same reference numerals, and duplicate descriptions are omitted. In the following embodiments, the moving body is described as a vehicle, but the moving body is not limited to a vehicle and may be an aircraft, a robot, or the like.

[0010] <Configuration> FIG. 1 is a block diagram of a vehicle 1 according to an embodiment of the present invention. In FIG. 1, the outline of the vehicle 1 is shown in a plan view and a side view. The vehicle 1 is, as an example, a four-wheel sedan-type passenger car. The vehicle 1 may be such a four-wheel vehicle, or may be a two-wheel vehicle or another type of vehicle.

[0011] The vehicle 1 includes a vehicle control device 2 (hereinafter simply referred to as the control device 2) that controls the vehicle 1. The control device 2 includes a plurality of ECUs (Electronic Control Units) 20 to 29 communicably connected by an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit), a memory such as a semiconductor memory, and an interface with an external device. Programs executed by the processor and data used by the processor for processing are stored in the memory. Each ECU may include a plurality of processors, memories, interfaces, and the like. For example, the ECU 20 includes one or more processors 20a and one or more memories 20b. The processing by the ECU 20 is executed by the processor 20a executing an instruction including a program stored in the memory 20b. Instead of this, the ECU 20 may include a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for executing the processing by the ECU 20. The same applies to other ECUs.

[0012] The functions and the like that each of the ECUs 20 to 29 is in charge of will be described below. Note that the number of ECUs and the functions they are in charge of can be designed as appropriate, and it is possible to subdivide or integrate them more than in this embodiment.

[0013] ECU 20 executes control related to the automatic driving of the vehicle 1. In automatic driving, at least one of the steering and the acceleration / deceleration of the vehicle 1 is automatically controlled. The automatic driving by ECU 20 may include automatic driving that does not require a driving operation by the driver (which may also be called autonomous driving) and automatic driving for assisting the driving operation by the driver (which may also be called driving assistance).

[0014] ECU 21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism that steers the front wheels in response to the driver's driving operation (steering operation) on the steering wheel 31. Further, the electric power steering device 3 includes a motor that exerts a driving force for assisting the steering operation or automatically steering the front wheels, a sensor that detects the steering angle, and the like. When the driving state of the vehicle 1 is in autonomous driving, ECU 21 automatically controls the electric power steering device 3 in response to an instruction from ECU 20 to control the traveling direction of the vehicle 1.

[0015] ECUs 22 and 23 perform control of a detection unit that detects the surrounding situation of the vehicle and information processing of the detection results. The vehicle 1 includes one standard camera 40 and four fisheye cameras 41 to 44 as a detection unit that detects the surrounding situation of the vehicle. The standard camera 40 and the fisheye cameras 42 and 44 are connected to ECU 22. The fisheye cameras 41 and 43 are connected to ECU 23. ECUs 22 and 23 can extract the contour of an object and the lane dividing lines (such as white lines) on the road by analyzing the images taken by the standard camera 40 and the fisheye cameras 41 to 44.

[0016] The fish-eye cameras 41 to 44 are cameras equipped with fish-eye lenses. Although they can acquire images over a wide range, large distortions occur in the acquired images (compared to the images captured by a standard camera). They are an example of imaging devices equipped with wide-angle lenses. Hereinafter, the configuration of the fish-eye camera 41 will be described. The other fish-eye cameras 42 to 44 may have a similar configuration. The angle of view of the fish-eye camera 41 is wider than that of the standard camera 40. Therefore, the fish-eye camera 41 can capture a wider range than the standard camera 40. The image captured by the fish-eye camera 41 has large distortions compared to the image captured by the standard camera 40. Therefore, the ECU 23 may perform a conversion process (hereinafter referred to as "distortion reduction process") to reduce the distortion on the image before analyzing the image captured by the fish-eye camera 41. On the other hand, the ECU 22 does not necessarily have to perform the distortion reduction process on the image captured by the standard camera 40 before analyzing the image. Thus, the standard camera 40 is an imaging device that captures images that are not subject to the distortion reduction process, and the fish-eye camera 41 is an imaging device that captures images that are subject to the distortion reduction process. Instead of the standard camera 40, other imaging devices that capture images that are not subject to the distortion reduction process, such as cameras equipped with wide-angle lenses or telephoto lenses, may be used.

[0017] The standard camera 40 is mounted at the center of the front part of the vehicle 1 and captures the surrounding situation in front of the vehicle 1. The fisheye camera 41 is mounted at the center of the front part of the vehicle 1 and captures the surrounding situation in front of the vehicle 1. In FIG. 1, the standard camera 40 and the fisheye camera 41 are shown as being arranged side by side in the horizontal direction. However, the arrangements of the standard camera 40 and the fisheye camera 41 are not limited to this, and for example, they may be arranged side by side in the vertical direction. Also, at least one of the standard camera 40 and the fisheye camera 41 may be mounted on the front part of the roof of the vehicle 1 (for example, the inside of the vehicle compartment of the front window). The fisheye camera 42 is mounted at the center of the right side part of the vehicle 1 and captures the surrounding situation on the right side of the vehicle 1. The fisheye camera 43 is mounted at the center of the rear part of the vehicle 1 and captures the surrounding situation behind the vehicle 1. The fisheye camera 44 is mounted at the center of the left side part of the vehicle 1 and captures the surrounding situation on the left side of the vehicle 1.

[0018] The types, numbers, and mounting positions of the cameras possessed by the vehicle 1 are not limited to the above examples. Also, the vehicle 1 may include a lidar (Light Detection and Ranging) or a millimeter-wave radar as a detection unit for detecting object targets around the vehicle 1 and measuring the distance to the object targets.

[0019] The ECU 22 performs control of the standard camera 40 and the fisheye cameras 42 and 44 and information processing of the detection results. The ECU 23 performs control of the fisheye cameras 41 and 43 and information processing of the detection results. By dividing the detection units for detecting the surrounding situation of the vehicle into two systems, the reliability of the detection results can be improved.

[0020] The ECU 24 controls the gyro sensor 5, the GPS sensor 24b, and the communication device 24c, and performs information processing on the detection results or communication results. The gyro sensor 5 detects the rotational movement of the vehicle 1. The traveling route of the vehicle 1 can be determined based on the detection result of the gyro sensor 5 and the wheel speed or the like. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c performs wireless communication with a server that provides map information and traffic information, and acquires this information. The ECU 24 can access the database 24a of map information constructed in the memory, and the ECU 24 performs route search from the current location to the destination and the like. The ECU 24, the map database 24a, and the GPS sensor 24b constitute a so-called navigation device.

[0021] The ECU 25 is provided with a communication device 25a for vehicle-to-vehicle communication. The communication device 25a performs wireless communication with other surrounding vehicles and exchanges information between vehicles.

[0022] The ECU 26 controls the power plant 6. The power plant 6 is a mechanism that outputs a driving force for rotating the drive wheels of the vehicle 1, and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine in response to the driving operation (accelerator operation or acceleration operation) of the driver detected by the operation detection sensor 7a provided on the accelerator pedal 7A, or switches the gear position of the transmission based on information such as the vehicle speed detected by the vehicle speed sensor 7c. When the driving state of the vehicle 1 is in autonomous driving, the ECU 26 automatically controls the power plant 6 in response to an instruction from the ECU 20 and controls the acceleration and deceleration of the vehicle 1.

[0023] The ECU 27 controls a lighting device (headlight, taillight, etc.) including the direction indicator 8 (winker). In the case of the example in FIG. 1, the direction indicator 8 is provided at the front part, the door mirror, and the rear part of the vehicle 1.

[0024] The ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and receives input of information from the driver. The voice output device 91 notifies the driver of information by voice. The display device 92 notifies the driver of information by displaying an image. The display device 92 is arranged, for example, on the surface of the driver's seat and constitutes an instrument panel or the like. Here, voice and display are exemplified, but information may also be notified by vibration or light. Further, information may be notified by combining a plurality of voice, display, vibration, or light. Furthermore, the combination or the notification mode may be varied according to the level (e.g., urgency) of the information to be notified. The input device 93 is arranged at a position where the driver can operate and is a group of switches for giving instructions to the vehicle 1, but may also include a voice input device.

[0025] The ECU 29 controls the brake device 10 and the parking brake (not shown). The brake device 10 is, for example, a disk brake device, is provided on each wheel of the vehicle 1, and decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheel. The ECU 29 controls the operation of the brake device 10 in response to a driving operation (brake operation) of the driver detected by, for example, an operation detection sensor 7b provided on the brake pedal 7B. When the driving state of the vehicle 1 is in automatic driving, the ECU 29 automatically controls the brake device 10 in response to an instruction from the ECU 20 and controls the deceleration and stop of the vehicle 1. The brake device 10 and the parking brake can also operate to maintain the stopped state of the vehicle 1. Further, when the transmission of the power plant 6 is provided with a parking lock mechanism, this can also operate to maintain the stopped state of the vehicle 1.

[0026] <Shooting range> Next, with reference to FIG. 2, the shooting ranges of the standard camera 40 and the fisheye cameras 41 to 44 will be described. FIG. 2(a) shows the horizontal shooting range of each camera, FIG. 2(b) shows the vertical shooting range of the fisheye camera 42 attached to the right side of the vehicle 1, and FIG. 2(c) shows the vertical shooting range of the fisheye camera 43 attached to the rear of the vehicle 1.

[0027] First, with reference to FIG. 2(a), the imaging range in the plan view of vehicle 1 (i.e., the horizontal direction of vehicle 1) will be described. The standard camera 40 images the scenery included in the imaging range 200. The imaging center 200C of the standard camera 40 faces the true front of vehicle 1. The horizontal angle of view of the standard camera 40 may be less than 90°, for example, about 45° or about 30°.

[0028] The fisheye camera 41 images the scenery included in the imaging range 201. The imaging center 201C of the fisheye camera 41 faces the true front of vehicle 1. The fisheye camera 42 images the scenery included in the imaging range 202. The imaging center 202C of the fisheye camera 42 faces the true side on the right of vehicle 1. The fisheye camera 43 images the scenery included in the imaging range 203. The imaging center 203C of the fisheye camera 43 faces the true rear of vehicle 1. The fisheye camera 44 images the scenery included in the imaging range 204. The imaging center 204C of the fisheye camera 44 faces the true side on the left of vehicle 1. The horizontal angle of view of the fisheye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, greater than 180°, for example, about 180°. FIG. 2(a) shows an example in which the horizontal angle of view of the fisheye cameras 41 to 44 is 180°.

[0029] The imaging range 201 can be divided into an area 201L diagonally in front of the left of the vehicle 1, an area 201F directly in front of the vehicle 1, and an area 201R diagonally in front of the right of the vehicle 1. The imaging range 202 can be divided into an area 202L diagonally in front of the right of the vehicle 1, an area 202F directly to the right of the vehicle 1, and an area 202R diagonally behind the right of the vehicle 1. The imaging range 203 can be divided into an area 203L diagonally behind the right of the vehicle 1, an area 203F directly behind the vehicle 1, and an area 203R diagonally behind the left of the vehicle 1. The imaging range 204 can be divided into an area 204L diagonally behind the right of the vehicle 1, an area 204F directly to the left of the vehicle 1, and an area 204R diagonally in front of the left of the vehicle 1. The imaging range 201 may be evenly divided into three areas 201L, 201F, and 201R (that is, so that the viewing angles of each area are equal). The other imaging ranges 202 to 204 may also be evenly divided into three parts.

[0030] Since the standard camera 40 and the fisheye cameras 41 to 44 have the imaging ranges 200 to 204 as described above, the front directly in front of the vehicle 1 and the four diagonal directions are included in the imaging ranges of two separate cameras. Specifically, the front directly in front of the vehicle 1 is included in both the imaging range 200 of the standard camera 40 and the area 201F of the imaging range 201 of the fisheye camera 41. The diagonally front right of the vehicle 1 is included in both the area 201R of the imaging range 201 of the fisheye camera 41 and the area 202L of the imaging range 202 of the fisheye camera 42. The same applies to the other three diagonal directions of the vehicle 1.

[0031] Subsequently, referring to FIGS. 2(b) and 2(c), the imaging range in the vertical direction of the vehicle 1 will be described. In FIG. 2(b), the vertical imaging range of the fisheye camera 42 will be described, and in FIG. 2(c), the vertical imaging range of the fisheye camera 43 will be described. The same may apply to the vertical imaging ranges of the other fisheye cameras 41 and 44.

[0032] The vertical viewing angle of the fisheye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, greater than 180°, or may be about 180°, for example. FIGS. 2(b) and (c) show an example in which the vertical viewing angle of the fisheye cameras 41 to 44 is 180°. In the illustrated example, the shooting center 203C of the fisheye camera 43 faces downward (toward the ground) rather than in a direction parallel to the ground. Instead of this, the shooting center 203C of the fisheye camera 43 may face in a direction parallel to the ground, or may face upward (opposite to the ground) rather than in a direction parallel to the ground. Also, the shooting centers 201C to 204C of the fisheye cameras 41 to 44 may face different directions in the vertical direction.

[0033] Referring to FIG. 3, the distortion reduction process of the images captured by the fisheye cameras 41 to 44 will be described. Image 300 is an image of the scenery to the right of the vehicle 1 captured by the fisheye camera 42. As shown, the image 300 has a large distortion, particularly in the peripheral portion.

[0034] The ECU 22 connected to the fisheye camera 42 performs a distortion reduction process on the image 300. Specifically, the ECU 22 sets a point in the image 300 as the correction center point 301. The ECU 22 cuts out a partial area (rectangular area 302) centered on the correction center point 301 from the image 300. The ECU 22 generates an image 303 with reduced distortion by performing a distortion reduction process on this area 302. In the distortion reduction process, the distortion is reduced more at a position closer to the correction center point 301, and the distortion is not reduced or the distortion is increased at a position farther from the correction center point 301. Therefore, in some embodiments, the ECU 22 sets the correction center point 301 within the area of interest in the environment around the vehicle 1 and generates an image with reduced distortion for this area.

[0035] <Process> As described with reference to FIG. 3, in the images obtained by the fisheye cameras 41 to 44, which are examples of imaging devices equipped with wide-angle lenses, capturing the external environment of the vehicle 1, significant distortion occurs particularly in the peripheral portions. For such distorted images, it is not possible to directly apply the models for external environment recognition, such as object recognition and road recognition, prepared for the images obtained by a camera such as the standard camera 40. Therefore, it is necessary to convert the acquired images into images with reduced distortion (planar images) for use in external environment recognition.

[0036] In order to realize appropriate driving support control or autonomous driving control based on the recognition result of the external environment of the vehicle 1, it is required to acquire precise information about the objects (such as other vehicles, pedestrians, bicycles, signals, and road signs, etc.) that are stationary or moving around the vehicle 1. Therefore, in the following embodiments, the control device 2 discovers a specific object from which precise information should be acquired (i.e., determines where in the image the specific object exists) in the images obtained by the fisheye cameras 41 to 44. Further, the control device 2 performs a distortion reduction process for reducing the distortion of the image on a partial region (for example, the region 302 in FIG. 3) centered on the position of the discovered object or the vicinity thereof in the images obtained by the fisheye cameras 41 to 44. That is, the control device 2 sets a correction center point (for example, the correction center point 301 in FIG. 3) at the position of the discovered object or in its vicinity, and performs the distortion reduction process. The control device 2 uses the images obtained in this way for the recognition process of the external environment of the vehicle 1. This makes it possible to acquire more precise information about the environment around the vehicle 1 through the recognition process. Note that in this specification, the vicinity of the position of an object refers to a position to such an extent that desired external environment recognition processing for the target object is possible using the image obtained by the distortion reduction process.

[0037] Hereinafter, with reference to FIG. 4, an example of a method for the control device 2 to control the vehicle 1 in some embodiments will be described. This method may be performed by the processors 20a of the respective ECUs 20 to 29 of the control device 2 executing programs in the memories 20b. The method in FIG. 5 may be started in response to the driving support function or the autonomous driving function by the control device 2 being turned on.

[0038] In S401, the control device 2 acquires images of the external environment of the vehicle 1 from each of the standard camera 40 and the fisheye cameras 41 to 44. Each image includes the situation within the range described in FIG. 2 of the external environment of the vehicle 1. Note that it is not necessary to perform the processing of S402 to S404 on the image acquired from the standard camera 40.

[0039] In S402, the control device 2 performs a detection process of an object (target object) predetermined as a detection target based on the images acquired from each fisheye camera (fisheye cameras 41 to 44). This detection process corresponds to a process of discovering a specific object in the acquired image from which precise information should be obtained by external recognition. As an example, the target object to be detected may include one or more of other vehicles, pedestrians, bicycles, signals, and road signs.

[0040] As an example of the detection process, the control device 2 performs image recognition on the images acquired from each fisheye camera to detect the target object. This image recognition can be realized by using a model using, for example, a known deep learning technique, which is trained with an image containing the target object in the images obtained by the fisheye camera as teacher data. When an unknown image is input, the learned model outputs the presence or absence of the target object in the image and the region (position) of the target object. The learned model may be stored in the memory 20b in advance. Note that the learned model can output one or more of the regions where the target object exists.

[0041] As another example of the detection process, the control device 2 may detect a target object by performing image recognition on an image obtained by dividing the image acquired from each fisheye camera at a predetermined angle of view (for example, 120°). Thereby, it becomes possible to discover the target object in each of the divided images and individually apply distortion reduction processing (S404) and external world recognition processing (S405) to the regions where the discovered objects are located. This makes it possible to appropriately acquire information about objects existing in a specific direction around the vehicle. Further, as still another example, the control device 2 can also detect a target object by performing image recognition on an image obtained by once performing distortion reduction processing on the image acquired from each fisheye camera.

[0042] Next, in S403 and S404, the control device 2 performs distortion reduction processing on the images acquired from each fisheye camera according to the detection results of the detection process in S402. More specifically, in S403, the control device 2 determines whether a target object has been detected in the images acquired from each fisheye camera in the detection process of S402. For each image corresponding to each fisheye camera, if the target object has not been detected, the control device 2 returns the process to S401, and if the target object has been detected, it proceeds to S404.

[0043] In S404, the control device 2 performs distortion reduction processing on a partial region in the image acquired from each fisheye camera, which is a partial region centered on the detection position of the target object or its vicinity. For example, the control device 2 sets the position where the target object is detected or its vicinity position in the image acquired from each fisheye camera as a correction center point (conversion center position), cuts out a rectangular region centered on the correction center point, and performs distortion reduction processing on the cut-out image. Thereby, an image with reduced distortion is generated. Since existing techniques may be used for the distortion reduction processing, detailed description is omitted.

[0044] When the conversion process of S404 is completed, in S405, the control device 2 performs recognition processing for recognizing the external environment of the vehicle 1 based on the image acquired from the standard camera 40 and the image obtained by the conversion process (the image with reduced distortion). Similar to S402, the recognition processing can be realized by using a model, for example, a known deep learning technique, which is trained with the image containing the target object among the images obtained by the fisheye camera as teacher data. The trained model may be stored in the memory 20b in advance. Note that the trained model can output one or more regions where the target object exists.

[0045] When the recognition processing is completed, information about the object extracted by the recognition processing is provided for driving assistance control or autonomous driving control. That is, the control device 2 may perform control of the vehicle 1 (for example, automatic braking, notification to the driver, change of the autonomous driving level, etc.) according to the recognition result of the external environment. Since existing technologies may be applied to the control of the vehicle 1 according to the recognition result of the external environment, detailed description is omitted.

[0046] Thereafter, in S407, the control device 2 determines whether to end the operation. If it is determined to end the operation, the operation is ended. Otherwise, the process returns to S401 and the above-described process is repeated. The control device 2 may determine to end the operation, for example, in response to the driving assistance function or the autonomous driving function being turned off.

[0047] As described above, S401 to S507 are repeatedly executed. The control device 2 may execute the processes of S401 to S507 periodically. This execution period varies depending on the required time for the detection process of S402, the distortion reduction process of S404, and the recognition process of S405, and may be, for example, about 100 ms.

[0048] As described above, in the present embodiment, the control devices 2 (ECUs 22 and 23) acquire an image obtained by photographing the external environment of the vehicle 1 from a photographing device (fisheye camera), and based on the image acquired from the photographing device, detect a target object by image recognition. Further, the control device 2 performs a distortion reduction process for reducing the distortion of an image on a partial region in the image acquired from the photographing device, which is a partial region centered on the detection position of the target object or a position in the vicinity thereof, according to the detection result of the target object (for example, the presence or absence of detection). The control device 2 recognizes the external environment of the vehicle 1 based on the image obtained by the distortion reduction process. As a result, it becomes possible to accurately acquire information about the objects around the vehicle 1 from the image obtained by the photographing device (fisheye camera) to which the fisheye lens is attached.

[0049] <Modification example> In some of the above-described embodiments, it is premised that an object to be discovered (detected) in the image acquired from the fisheye camera is predetermined. In contrast, in other embodiments, the object to be detected may be determined based on the operating state of the vehicle 1 (for example, for each driving scene of the vehicle 1).

[0050] With reference to FIG. 5, an example of a method for the control device 2 to control the vehicle 1 in another embodiment will be described. This method may be performed by the processors 20a of the ECUs 20 to 29 of the control device 2 executing the programs in the memories 20b, similar to the method of FIG. 4. The method of FIG. 5 may be started in response to the driving support function or the autonomous driving function by the control device 2 being turned on. Hereinafter, for the sake of simplifying the description, the description of the processes similar to those in the method of FIG. 4 will be omitted.

[0051] In S501, the control device 2 determines an object to be detected by the object detection process (S402) based on the operation state of the vehicle 1. The operation state may be a vehicle driving scene, or may be a driving state of the vehicle (e.g., an autonomous driving level). For example, in a country where left-hand traffic is adopted, the control device 2 may determine an oncoming vehicle traveling in an oncoming lane as an object to be detected when the driving scene of the vehicle 1 corresponding to the operation state of the vehicle 1 is a scene of making a right turn at an intersection. Also, the control device 2 may determine a traffic participant (such as a pedestrian, bicycle, or other vehicle) that may cause an entrapment accident when turning left as an object to be detected when the driving scene of the vehicle 1 corresponding to the operation state of the vehicle 1 is a scene of making a left turn at an intersection.

[0052] After the target object is determined, in S401, the control device 2 acquires images of the outside world of the vehicle 1 from each of the standard camera 40 and the fish-eye cameras 41 to 44, similar to the method of FIG. 4, and proceeds to S402. In this example, the process of S501 is performed before the process of S401, but the process of S501 may be performed after the process of S401. In S402, the control device 2 performs a detection process of the target object based on the images acquired from each fish-eye camera (fish-eye cameras 41 to 44), similar to the method of FIG. 4. However, in this example, the object to be detected by the detection process is the object determined by the process of S501. Furthermore, for example, an output indicating the presence or absence of the target object in the image acquired by each fish-eye camera according to the above-mentioned trained model is output, and information indicating the recognition accuracy of the target object by image recognition is output.

[0053] After the detection process is completed, in S502, the control device 2 determines whether the recognition accuracy of the target object by the image recognition in the detection process executed in S402 exceeds the accuracy threshold. If the recognition accuracy exceeds the accuracy threshold, the control device 2 determines that the target object has been detected and proceeds to S404, otherwise, the control device 2 returns to S401.

[0054] Here, the confidence threshold may be determined in advance for each operating state of the vehicle 1, or may be determined in advance for each type of object to be detected by the detection process (S402). In S501, the control device 2 of the present embodiment determines (selects) the confidence threshold to be used in S502 according to the operating state of the vehicle 1 or the type of object determined as the detection target (based on the operating state). For example, for a type of object with high detection importance in a certain driving scene, the corresponding confidence threshold may be set low so that it is more likely to be determined as detected in S502 (so that the distortion reduction process and the recognition process are more likely to be performed). On the other hand, for a type of object with low detection importance, the corresponding confidence threshold may be set high so that the distortion reduction process and the recognition process are less likely to be performed. This makes it possible to efficiently reduce the computational amount associated with the distortion reduction process and the recognition process.

[0055] In S404 to S407, the control device 2 performs the same processing as the method in FIG. 4.

[0056] As described above, according to the present embodiment, by determining the object to be detected based on the operating state of the vehicle 1, it is possible to appropriately obtain information regarding the type of object that needs to be detected corresponding to the operating state of the vehicle from the image obtained by the fisheye camera. In addition, for an object with high detection importance, the distortion reduction process is executed to facilitate the execution of the recognition process, while for an object with low detection importance, it is possible to prevent the execution of the distortion reduction process and the recognition process. This makes it possible to reduce the computational amount associated with the distortion reduction process and the recognition process. Further, by changing the confidence threshold compared with the recognition accuracy of image recognition in the object detection process according to the operating state of the vehicle 1 or the type of object determined as the detection target, it is possible to efficiently reduce the computational amount associated with the distortion reduction process and the recognition process.

[0057] <Other Embodiments> In addition, a program for realizing one or more functions described in each embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device can read and execute this program. The present invention can also be realized in such a manner.

[0058] <Summary of Embodiments> The above embodiments disclose at least the following embodiments.

[0059] [Item 1] A control device (for example, 2) of a moving body (for example, 1) having a photographing device (for example, 41 to 44) to which a wide-angle lens is attached, an image acquisition means for acquiring an image (for example, 300) obtained by photographing the outside of the moving body from the photographing device; a detection means for detecting a target object by image recognition based on the image acquired from the photographing device; a processing means for performing a distortion reduction process for reducing the distortion of an image on a partial region in the image acquired from the photographing device and centered on the detection position of the target object or its vicinity (for example, 302) according to the detection result by the detection means; a recognition means for recognizing the outside of the vehicle based on the image (for example, 303) obtained by the distortion reduction process; A control device characterized by comprising the above. According to this item, it becomes possible to accurately acquire information about an object around the moving body from an image obtained by a photographing device to which a wide-angle lens is attached.

[0060] [Item 2] The control device according to Item 1, wherein the processing means performs the distortion reduction process when the target object is detected by the detection means. This item is characterized by the above. According to this item, it becomes possible to appropriately perform the distortion reduction process according to the detection result of the target object.

[0061] [Item 3] The detection means performs the image recognition on the image acquired from the imaging device to detect the target object. The control device according to claim 1 or 2, characterized in that. According to this item, the target object can be detected without requiring additional processing on the image obtained by the imaging device, and the processing load can be suppressed.

[0062] [Item 4] The detection means performs the image recognition on the image obtained by dividing the image acquired from the imaging device at a predetermined angle of view to detect the target object. The control device according to claim 1 or 2, characterized in that. According to this item, it becomes possible to appropriately acquire information regarding an object existing in a specific direction.

[0063] [Item 5] The detection means performs the image recognition on the image obtained by performing the distortion reduction process on the image acquired from the imaging device to detect the target object. The control device according to claim 1 or 2, characterized in that. According to this item, it becomes possible to realize the image recognition for detecting the target object using a model prepared for an image with less distortion, such as an image taken by a standard camera 40.

[0064] [Item 6] The control device further includes a determination means for determining an object to be detected by the detection means based on the operating state of the vehicle. The control device according to any one of claims 1 to 5, characterized in that. According to this item, it becomes possible to appropriately acquire information regarding types of objects for which detection is required corresponding to the operating state of the moving body.

[0065] [Item 7] The detection means outputs information indicating the recognition accuracy of the target object by the image recognition. When the recognition accuracy indicated by the information exceeds a threshold value, the processing means performs the distortion reduction process. The control device according to item 6, characterized in that. According to this item, based on the recognition accuracy of the target object by the detection process, it is possible to appropriately control whether to perform the distortion reduction process and the recognition process, and to efficiently reduce the required amount of calculation.

[0066] [Item 8] The threshold value is predetermined for each type of object to be detected by the detection means. When the recognition accuracy indicated by the information exceeds the threshold value corresponding to the object determined by the determination means, the processing means performs the distortion reduction process. The control device according to item 7, characterized in that. According to this item, it is possible to efficiently reduce the amount of calculation associated with the distortion reduction process and the recognition process.

[0067] [Item 9] The threshold value is predetermined for each operating state of the vehicle. When the recognition accuracy indicated by the information exceeds the threshold value corresponding to the operating state, the processing means performs the distortion reduction process. The control device according to item 7, characterized in that. According to this item, it is possible to efficiently reduce the amount of calculation associated with the distortion reduction process and the recognition process.

[0068] [Item 10] The control device according to any one of items 1 to 9, characterized in that the imaging device is an imaging device to which a fish-eye lens is attached. According to this item, it is possible to accurately acquire information about objects around the moving body from an image obtained by an imaging device to which a fish-eye lens is attached.

[0069] [Item 11] The vehicle includes a plurality of imaging devices respectively arranged at the front, rear, and sides of the vehicle. The image acquisition means acquires the image from each of the plurality of imaging devices. The control device according to any one of Items 1 to 10, characterized in that. According to this item, it becomes possible to accurately acquire information on surrounding objects in all directions around the vehicle.

[0070] [Item 12] The control device according to any one of Items 1 to 11, characterized in that the moving body is a vehicle. According to this item, it becomes possible to accurately acquire information on surrounding objects of the vehicle from an image obtained by an imaging device equipped with a fish-eye lens.

[0071] [Item 13] A control method for a moving body (for example, 1) having an imaging device (for example, 41 to 44) equipped with a wide-angle lens, an image acquisition step of acquiring, from the imaging device, an image (for example, 300) obtained by photographing the outside of the moving body; a detection step of detecting a target object by image recognition based on the image acquired from the imaging device; a processing step of performing a distortion reduction process for reducing the distortion of an image on a partial area (for example, 302) in the image acquired from the imaging device and centered on the detection position of the target object or its vicinity according to the detection result in the detection step; a recognition step of recognizing the outside of the moving body based on the image (for example, 303) obtained by the distortion reduction process; characterized by including. According to this item, it becomes possible to accurately acquire information on surrounding objects of the moving body from an image obtained by an imaging device equipped with a wide-angle lens.

[0072] [Item 14] A program for causing a computer to function as each means of the control device according to any one of Items 1 to 12. According to this item, the above effects can be obtained in the form of a program.

[0073] Although the embodiments of the invention have been described above, the invention is not limited to the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention.

Explanation of Reference Numerals

[0074] 1: Vehicle, 2: Control device, 20 - 29: ECU, 41 - 44: Fish-eye camera

Claims

1. A control device for a moving body having a photographing device to which a wide-angle lens is attached, comprising: image acquisition means for acquiring an image obtained by photographing the external environment of the moving body from the photographing device; determination means for determining a target object to be detected in a detection process based on the image, based on the operating state of the moving body; detection means for detecting the target object by image recognition based on the image acquired from the photographing device; processing means for performing distortion reduction processing for reducing distortion of an image on a partial region in the image acquired from the photographing device according to the detection result by the detection means, the partial region being centered on the detection position of the target object or its vicinity; recognition means for recognizing the external environment of the moving body based on the image obtained by the distortion reduction processing, wherein the detection means outputs information indicating the recognition accuracy of the target object by the image recognition; the processing means performs the distortion reduction processing when the recognition accuracy indicated by the information exceeds a threshold value corresponding to the target object determined by the determination means; the threshold value is predetermined for each type of object to be detected by the detection means; a control device characterized by the above.

2. A control device for a moving body having a photographing device to which a wide-angle lens is attached, comprising: image acquisition means for acquiring an image obtained by photographing the external environment of the moving body from the photographing device; determination means for determining a target object to be detected in a detection process based on the image, based on the operating state of the moving body; detection means for detecting the target object by image recognition based on the image acquired from the photographing device; processing means for performing distortion reduction processing for reducing distortion of an image on a partial region in the image acquired from the photographing device according to the detection result by the detection means, the partial region being centered on the detection position of the target object or its vicinity; recognition means for recognizing the external environment of the moving body based on the image obtained by the distortion reduction processing, wherein the detection means outputs information indicating the recognition accuracy of the target object by the image recognition; the processing means performs the distortion reduction processing when the recognition accuracy indicated by the information exceeds a threshold value corresponding to the operating state; the threshold value is predetermined for each operating state of the moving body; a control device characterized by the above.

3. The mobile body is a vehicle, and the operating state is a driving scene or an operating state of the vehicle. The control device according to claim 1 or 2, characterized in that.

4. When the target object is detected by the detection means, the processing means performs the distortion reduction processing The control device according to claim 1 or 2, characterized in that.

5. The detection means detects the target object by performing the image recognition on the image acquired from the imaging device The control device according to any one of claims 1 to 4, characterized in that.

6. The detection means detects the target object by performing the image recognition on an image obtained by dividing the image acquired from the imaging device at a predetermined viewing angle The control device according to any one of claims 1 to 4, characterized in that.

7. The detection means detects the target object by performing the image recognition on an image obtained by performing the distortion reduction processing on the image acquired from the imaging device The control device according to any one of claims 1 to 4, characterized in that.

8. The imaging device is an imaging device to which a fish-eye lens is attached. The control device according to any one of claims 1 to 7, characterized in that.

9. The mobile body includes a plurality of imaging devices respectively arranged in front, behind, and on the sides of the mobile body, The image acquisition means acquires images from each of the plurality of imaging devices The control device according to any one of claims 1 to 8, characterized in that.

10. The mobile body is a vehicle. The control device according to any one of claims 1 to 9, characterized in that.

11. A control device for a mobile body having an imaging device to which a wide-angle lens is attached, An image acquisition means for acquiring an image obtained by photographing the outside of the mobile body from the imaging device, A detection means for detecting a target object by image recognition based on the image acquired from the imaging device, According to the detection result by the detection means, a part of the region in the image acquired from the imaging device, which is a part of the region centered on the detection position of the target object or its vicinity, is targeted, and a distortion reduction process for reducing the distortion of the image is performed. Processing means, A recognition means for recognizing the outside of the mobile body based on the image obtained by the distortion reduction processing, and is provided with The detection means outputs information indicating the recognition accuracy of the target object by the image recognition, When the recognition accuracy indicated by the information exceeds a threshold value corresponding to the target object, the processing means performs the distortion reduction processing. The threshold value is determined in advance for each type of object to be detected by the detection means. A control device characterized by the above.

12. A control device for a moving body having a photographing device with a wide-angle lens, comprising: an image acquisition means for acquiring an image obtained by photographing the external environment of the moving body from the photographing device; a detection means for detecting a target object by image recognition based on the image acquired from the photographing device; a processing means for performing a distortion reduction process for reducing the distortion of an image on a partial region in the image acquired from the photographing device according to the detection result of the detection means, the partial region being centered on the detection position of the target object or its vicinity; a recognition means for recognizing the external environment of the moving body based on the image obtained by the distortion reduction process; the detection means outputs information indicating the recognition accuracy of the target object by the image recognition; when the recognition accuracy indicated by the information exceeds a threshold value corresponding to the operating state of the moving body, the processing means performs the distortion reduction processing; the threshold value is determined in advance for each operating state of the moving body; A control device characterized by the above.

13. A control method for a moving body having a photographing device with a wide-angle lens, comprising: an image acquisition step of acquiring an image obtained by photographing the external environment of the moving body from the photographing device; a determination step of determining a target object to be detected in the detection process based on the image based on the operating state of the moving body; a detection step of detecting the target object by image recognition based on the image acquired from the photographing device; a processing step of performing a distortion reduction process for reducing the distortion of an image on a partial region in the image acquired from the photographing device according to the detection result in the detection step, the partial region being centered on the detection position of the target object or its vicinity; a recognition step of recognizing the external environment of the moving body based on the image obtained by the distortion reduction process; in the detection step, information indicating the recognition accuracy of the target object by the image recognition is output; in the processing step, when the recognition accuracy indicated by the information exceeds a threshold value corresponding to the target object determined in the determination step, the distortion reduction processing is performed. The threshold value is determined in advance for each type of object to be detected in the detection step. A control method characterized by the above.

14. A control method for a moving body having a photographing device equipped with a wide-angle lens, comprising: An image acquisition step of acquiring an image obtained by photographing the external environment of the moving body from the photographing device; A determination step of determining a target object to be detected in the detection process based on the image based on the moving state of the moving body; A detection step of detecting the target object by image recognition based on the image acquired from the photographing device; A processing step of performing distortion reduction processing for reducing distortion of an image on a partial region in the image acquired from the photographing device, which is a partial region centered on the detection position of the target object or its vicinity, according to the detection result in the detection step; A recognition step of recognizing the external environment of the moving body based on the image obtained by the distortion reduction processing, In the detection step, information indicating the recognition accuracy of the target object by the image recognition is output; In the processing step, when the recognition accuracy indicated by the information exceeds a threshold value corresponding to the operation state, the distortion reduction processing is performed; The threshold value is determined in advance for each operation state of the moving body. A control method characterized by the above.

15. A control method for a moving body having a photographing device equipped with a wide-angle lens, comprising: An image acquisition step of acquiring an image obtained by photographing the external environment of the moving body from the photographing device; A detection step of detecting a target object by image recognition based on the image acquired from the photographing device; A processing step of performing distortion reduction processing for reducing distortion of an image on a partial region in the image acquired from the photographing device, which is a partial region centered on the detection position of the target object or its vicinity, according to the detection result in the detection step; A recognition step of recognizing the external environment of the moving body based on the image obtained by the distortion reduction processing, In the detection step, information indicating the recognition accuracy of the target object by the image recognition is output; In the processing step, when the recognition accuracy indicated by the information exceeds a threshold value corresponding to the target object, the distortion reduction processing is performed; The threshold value is determined in advance for each type of object to be detected in the detection step. A control method characterized by the above.

16. A control method for a moving body having a photographing device with a wide-angle lens, comprising: an image acquisition step of acquiring, from the photographing device, an image obtained by photographing the external environment of the moving body; a detection step of detecting a target object by image recognition based on the image acquired from the photographing device; a processing step of performing distortion reduction processing for reducing distortion of an image on a partial area in the image acquired from the photographing device, the partial area being centered on the detection position of the target object or its vicinity, according to the detection result in the detection step; a recognition step of recognizing the external environment of the moving body based on the image obtained by the distortion reduction processing, wherein in the detection step, information indicating the recognition accuracy of the target object by the image recognition is output; in the processing step, when the recognition accuracy indicated by the information exceeds a threshold value corresponding to the operating state of the moving body, the distortion reduction processing is performed; the threshold value is predetermined for each operating state of the moving body; A control method characterized by the above.

17. A program for causing a computer to function as each means of the control device according to any one of Claims 1 to 12.

Images