Control device, control method, and storage medium for a moving body

Abstract

The present invention relates to a control device, a control method, and a storage medium for a moving body, and provides a technique for accurately obtaining information related to an object around the moving body from an image obtained by a photographing device equipped with a wide-angle lens. The control device acquires an image obtained by photographing the outside of the moving body from a photographing device equipped with a wide-angle lens, and based on the image acquired from the photographing device, detects an object by image recognition. Further, the control device performs distortion reduction processing for reducing the distortion of the image on a partial area in the image acquired from the photographing device, that is, a partial area centered on the detection position of the object or its vicinity, according to the detection result of the object. The control device recognizes the outside of the vehicle based on the image obtained by the distortion reduction processing.

Description

Technical Field

[0001] The present invention relates to a control device, a control method, and a storage medium for a moving body. Background Art

[0002] The outside of a vehicle is recognized based on an image obtained by photographing the surroundings of the vehicle, and the recognition result is used for control such as driving assistance. At this time, in order to expand the detection range, it is also conceivable to use a camera with a wide viewing angle such as a fish-eye lens. However, in such a camera, although a wide-range image can be obtained, since the obtained image is distorted, if an object detection technique premised on an undistorted image obtained from a normal camera is applied, the detection accuracy may decrease. Patent Document 1 discloses a technique of performing distortion reduction processing on a distorted image and using the corrected image for object detection.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2008-48443 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] In order to implement appropriate driving assistance control or autonomous driving control based on the recognition result of the outside of a moving body such as a vehicle, it is required to obtain precise information about an object that is stationary or moving around the moving body. However, if the distortion reduction processing is not appropriately performed on an image obtained from a photographing device equipped with a wide viewing angle lens such as a fish-eye lens, it may sometimes be impossible to appropriately obtain information related to an object existing around the moving body.

[0008] The present invention provides a technique for highly accurately obtaining information related to an object around a moving body from an image obtained by a photographing device equipped with a wide viewing angle lens.

[0009] Means for Solving the Problems

[0010] According to one aspect of the present invention, there is provided a control device which is a control device for a moving body having a photographing device equipped with a wide-angle lens. The control device is characterized in that it includes: an image acquisition unit that acquires an image obtained by photographing the outside of the moving body from the photographing device; a detection unit that detects an object based on the image acquired from the photographing device through image recognition; a processing unit that performs distortion reduction processing for reducing distortion of the image on a partial area in the image acquired from the photographing device, that is, a partial area centered on the detection position of the object or its vicinity; and an identification unit that identifies the outside of the moving body based on the image obtained through the distortion reduction processing.

[0011] According to another aspect of the present invention, there is provided a control method which is a control method for a moving body having a photographing device equipped with a wide-angle lens. The control method is characterized in that it includes: an image acquisition step in which an image obtained by photographing the outside of the moving body is acquired from the photographing device; a detection step in which an object is detected based on the image acquired from the photographing device through image recognition; a processing step in which distortion reduction processing for reducing distortion of the image is performed on a partial area in the image acquired from the photographing device, that is, a partial area centered on the detection position of the object or its vicinity, according to the detection result in the detection step; and an identification step in which the outside of the moving body is identified based on the image obtained through the distortion reduction processing.

[0012] Advantages of the Invention

[0013] According to the present invention, information related to the objects around the moving body can be accurately obtained from the image obtained by the photographing device equipped with a wide-angle lens. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a block diagram showing a configuration example of a vehicle according to one embodiment.

[0015] Figure 2 It is a schematic diagram for explaining the photographing range of a camera according to one embodiment.

[0016] Figure 3 It is a schematic diagram for explaining the distortion reduction processing according to one embodiment.

[0017] Figure 4It is a flowchart showing the process of the processing performed by the control device according to an embodiment.

[0018] Figure 5 It is a flowchart showing the process of the processing performed by the control device according to another embodiment.

[0019] Explanation of reference numerals

[0020] 1: Vehicle; 2: Control device; 20 - 29: ECU; 41 - 44: Fish-eye camera. Detailed implementation mode

[0021] Hereinafter, the embodiments will be described in detail with reference to the drawings. In addition, the following embodiments do not limit the invention related to the technical solution. Also, the combinations of the features described in the embodiments are not necessarily all essential for the invention. Any two or more of the multiple features described in the embodiments can be arbitrarily combined. In addition, the same or similar components are labeled with the same reference numerals, and repeated descriptions are omitted. Further, in the following embodiments, the case where the moving body is a vehicle is described, but the moving body is not limited to a vehicle and may also be an aircraft, a robot, or the like.

[0022] <Configuration>

[0023] Figure 1 It is a block diagram of the vehicle 1 according to an embodiment of the present invention. In Figure 1 it, the outline of the vehicle 1 is shown in a top view and a side view. As an example, the vehicle 1 is a sedan-type four-wheel passenger vehicle. The vehicle 1 can be such a four-wheel vehicle, or a two-wheel vehicle, or other types of vehicles.

[0024] Vehicle 1 includes a vehicle control device 2 (hereinafter simply referred to as control device 2) that controls vehicle 1. Control device 2 includes a plurality of ECUs (Electronic Control Units) 20 to 29 that are connected via an in-vehicle network for communication. Each ECU includes a processor such as a CPU (Central Processing Unit), a memory such as a semiconductor memory, and an interface with external devices. Programs executed by the processor, data used by the processor in processing, etc. are stored in the memory. Each ECU may also include a plurality of processors, memories, and interfaces, etc. For example, ECU 20 includes one or more processors 20a and one or more memories 20b. By executing commands including programs stored in memory 20b by processor 20a, the processing performed by ECU 20 is executed. Alternatively, ECU 20 may include an application-specific integrated circuit (ASIC) or other dedicated integrated circuit for executing the processing performed by ECU 20. The same applies to other ECUs.

[0025] Hereinafter, the functions and the like responsible for each of ECUs 20 to 29 will be described. In addition, regarding the number of ECUs and the functions responsible, appropriate design can be carried out, and it can be more refined or integrated than this embodiment.

[0026] ECU 20 executes control related to the autonomous driving of vehicle 1. In autonomous driving, at least one of the steering and acceleration / deceleration of vehicle 1 is automatically controlled. The autonomous driving performed by ECU 20 may include autonomous driving that does not require driving operations by the driver (which may also be referred to as autonomous driving) and autonomous driving for assisting driving operations performed by the driver (which may also be referred to as driving assistance).

[0027] ECU 21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism that steers the front wheels according to the driving operation (steering operation) of the driver on the steering wheel 31. In addition, the electric power steering device 3 includes a motor that generates a driving force for assisting the steering operation or automatically steering the front wheels, a sensor that detects the steering angle, etc. When the driving state of vehicle 1 is autonomous driving, ECU 21 automatically controls the electric power steering device 3 in correspondence with an instruction from ECU 20 to control the traveling direction of vehicle 1.

[0028] The ECU22 and the ECU23 control the detection unit for detecting the surrounding conditions of the vehicle and process the information of the detection results. The vehicle 1 includes a standard camera 40 and four fisheye cameras 41 to 44 as the detection unit for detecting the surrounding conditions of the vehicle. The standard camera 40, the fisheye camera 42, and the fisheye camera 44 are connected to the ECU22. The fisheye camera 41 and the fisheye camera 43 are connected to the ECU23. The ECU22 and the ECU23 can extract the outline of the target object and the lane dividing lines (such as white lines) on the road by analyzing the images captured by the standard camera 40 and the fisheye cameras 41 to 44.

[0029] The fisheye cameras 41 to 44 are cameras equipped with fisheye lenses and are an example of a photographing device equipped with a wide-angle lens that can obtain a wide range of images but has a large distortion in the obtained images (compared with the images captured by the standard camera). Hereinafter, the configuration of the fisheye camera 41 will be described. The other fisheye cameras 42 to 44 may have the same configuration. The field of view angle of the fisheye camera 41 is wider than that of the standard camera 40. Therefore, the fisheye camera 41 can capture a larger range than the standard camera 40. The image captured by the fisheye camera 41 has a larger distortion compared with the image captured by the standard camera 40. Therefore, the ECU23 can also perform conversion processing for reducing distortion (hereinafter referred to as "distortion reduction processing") on the image captured by the fisheye camera 41 before analyzing the image. On the other hand, the ECU22 may not perform distortion reduction processing on the image captured by the standard camera 40 before analyzing the image. In this way, the standard camera 40 is a photographing device that captures images that are not the object of distortion reduction processing, and the fisheye camera 41 is a photographing device that captures images that are the object of distortion reduction processing. Other photographing devices that capture images that are not the object of distortion reduction processing, such as cameras equipped with wide-angle lenses or telephoto lenses, may be used instead of the standard camera 40.

[0030] The standard camera 40 is installed at the center of the front of the vehicle 1 and captures the surrounding conditions in front of the vehicle 1. The fisheye camera 41 is installed at the center of the front of the vehicle 1 and captures the surrounding conditions in front of the vehicle 1. Figure 1In [the figure], the standard camera 40 and the fisheye camera 41 are arranged horizontally. However, the configurations of the standard camera 40 and the fisheye camera 41 are not limited thereto. For example, they may also be arranged vertically. In addition, at least one of the standard camera 40 and the fisheye camera 41 may be installed at the front part of the roof of the vehicle 1 (for example, the inner side of the vehicle compartment of the front window). The fisheye camera 42 is installed at the center of the right side part of the vehicle 1 and captures the surrounding conditions on the right side of the vehicle 1. The fisheye camera 43 is installed at the center of the rear part of the vehicle 1 and captures the surrounding conditions behind the vehicle 1. The fisheye camera 44 is installed at the center of the left side part of the vehicle 1 and captures the surrounding conditions on the left side of the vehicle 1. <(

[0031] The types, numbers, and installation positions of the cameras provided in the vehicle 1 are not limited to the above examples. In addition, the vehicle 1 may also include a lidar (Light Detection and Ranging), a millimeter-wave radar as a detection unit for detecting the surrounding objects of the vehicle 1 or measuring the distance to the objects.

[0032] The ECU 22 controls the standard camera 40, the fisheye camera 42, and the fisheye camera 44 and processes the information of the detection results. The ECU 23 controls the fisheye camera 41 and the fisheye camera 43 and processes the information of the detection results. By dividing the detection units for detecting the surrounding conditions of the vehicle into two systems, the reliability of the detection results can be improved.

[0033] The ECU 24 controls the gyro sensor 5, the GPS sensor 24b, and the communication device 24c and processes the information of the detection results or the communication results. The gyro sensor 5 detects the rotational movement of the vehicle 1. The traveling route of the vehicle 1 can be determined based on the detection results of the gyro sensor 5, the wheel speed, etc. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c performs wireless communication with a server that provides map information and traffic information and acquires this information. The ECU 24 can access the database 24a of the map information constructed in the memory, and the ECU 24 performs route search from the current location to the destination, etc. The ECU 24, the map database 24a, and the GPS sensor 24b constitute a so-called navigation device.

[0034] The ECU 25 is equipped with a communication device 25a for vehicle-to-vehicle communication. The communication device 25a performs wireless communication with other surrounding vehicles and exchanges information between vehicles.

[0035] The ECU 26 controls the power unit 6. The power unit 6 is a mechanism that outputs a driving force for rotating the drive wheels of the vehicle 1 and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine corresponding to, for example, the driving operation (throttle operation or acceleration operation) of the driver detected by the operation detection sensor 7a provided on the accelerator pedal 7A, or switches the gear position of the transmission based on information such as the vehicle speed detected by the vehicle speed sensor 7c. When the driving state of the vehicle 1 is autonomous driving, the ECU 26 automatically controls the power unit 6 corresponding to the instruction from the ECU 20 and controls the acceleration and deceleration of the vehicle 1.

[0036] The ECU 27 controls lighting devices (headlights, taillights, etc.) including the direction indicator 8 (turn signal). In Figure 1 the case of the example, the direction indicator 8 is provided at the front part of the vehicle 1, on the door mirrors, and at the rear.

[0037] The ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and receives input of information from the driver. The sound output device 91 reports information to the driver by sound. The display device 92 reports information to the driver by displaying an image. The display device 92 is arranged, for example, in front of the driver's seat and forms a dashboard or the like. In addition, here, sound and display are exemplified, but information can also be reported by vibration or light. Further, multiple of sound, display, vibration, or light can be combined to report information. Furthermore, the combination can be made different or the reporting method can be made different according to the level (for example, the degree of urgency) of the information to be reported. The input device 93 is a set of switches arranged at a position where the driver can operate to give an instruction to the vehicle 1 and can also include a sound input device.

[0038] The ECU 29 controls the braking device 10 and the parking brake (not shown). The braking device 10 is, for example, a disc brake device provided on each wheel of the vehicle 1 and decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheels. The ECU 29 controls the operation of the braking device 10 corresponding to, for example, the driving operation (braking operation) of the driver detected by the operation detection sensor 7b provided on the brake pedal 7B. When the driving state of the vehicle 1 is autonomous driving, the ECU 29 automatically controls the braking device 10 corresponding to the instruction from the ECU 20 and controls the deceleration and stop of the vehicle 1. The braking device 10 and the parking brake can also operate to maintain the stopped state of the vehicle 1. In addition, when the transmission of the power unit 6 has a parking lock mechanism, the parking lock mechanism can also operate to maintain the stopped state of the vehicle 1.

[0039] <Shooting range>

[0040] Next, with reference to Figure 2 , the shooting ranges of the standard camera 40 and the fisheye cameras 41 to 44 will be described. Figure 2 (a) in Figure 2 shows the horizontal shooting range of each camera, Figure 2 (b) in

[0041] shows the vertical shooting range of the fisheye camera 42 installed on the right side of the vehicle 1, Figure 2 and (c) in

[0042] shows the vertical shooting range of the fisheye camera 43 installed at the rear of the vehicle 1. Figure 2 (a) in

[0043] The shooting range 201 can be divided into an area 201L located at the left front diagonal of the vehicle 1, an area 201F located directly in front of the vehicle 1, and an area 201R located at the right front diagonal of the vehicle 1. The shooting range 202 can be divided into an area 202L located at the right front diagonal of the vehicle 1, an area 202F located on the right side directly beside the vehicle 1, and an area 202R located at the right rear diagonal of the vehicle 1. The shooting range 203 can be divided into an area 203L located at the right rear diagonal of the vehicle 1, an area 203F located directly behind the vehicle 1, and an area 203R located at the left rear diagonal of the vehicle 1. The shooting range 204 can be divided into an area 204L located at the left rear diagonal of the vehicle 1, an area 204F located on the left side directly beside the vehicle 1, and an area 204R located at the left front diagonal of the vehicle 1. The shooting range 201 can be equally divided (i.e., in a way that the field of view angles of each area are equal) into three areas 201L, 201F, and 201R. Regarding the other shooting ranges 202 to 204, they can also be equally divided into three parts.

[0044] The standard camera 40 and the fisheye cameras 41 to 44 have the shooting ranges 200 to 204 as described above, so that the area directly in front of the vehicle 1 and the four diagonal directions are included in the shooting ranges of two separate cameras. Specifically, the area directly in front of the vehicle 1 is included in both the shooting range 200 of the standard camera 40 and the area 201F of the shooting range 201 of the fisheye camera 41. The right front diagonal of the vehicle 1 is included in both the area 201R of the shooting range 201 of the fisheye camera 41 and the area 202L of the shooting range 202 of the fisheye camera 42. The same applies to the other three diagonal directions of the vehicle 1.

[0045] Next, referring to Figure 2 in (b) and Figure 2 in (c), the shooting range in the vertical direction of the vehicle 1 will be described. In Figure 2 in (b), the shooting range in the vertical direction of the fisheye camera 42 is described, and in Figure 2 in (c), the shooting range in the vertical direction of the fisheye camera 43 is described. The same can apply to the shooting ranges in the vertical direction of the other fisheye cameras 41 and 44.

[0046] The field of view angles in the vertical direction of the fisheye cameras 41 to 44 can be, for example, greater than 90°, can also be greater than 150°, can also be greater than 180°, and can be, for example, about 180°. Figure 2 in (b) and Figure 2In (c), an example is shown where the vertical field of view angle of the fish-eye cameras 41 to 44 is 180°. In the illustrated example, the shooting center 203C of the fish-eye camera 43 faces a direction lower than the direction parallel to the ground (the ground side). Instead, the shooting center 203C of the fish-eye camera 43 may face a direction parallel to the ground or a direction higher than the direction parallel to the ground (the side opposite to the ground). In addition, the shooting centers 201C to 204C of the fish-eye cameras 41 to 44 may face different directions in the vertical direction.

[0047] Refer to Figure 3 , and the distortion reduction process for the images captured by the fish-eye cameras 41 to 44 will be described. The image 300 is an image of the scenery on the right side of the vehicle 1 captured by the fish-eye camera 42. As shown in the figure, the image 300 has a large distortion, especially in the peripheral part.

[0048] The ECU 22 connected to the fish-eye camera 42 performs a distortion reduction process on the image 300. Specifically, the ECU 22 sets a point in the image 300 as the correction center point 301. The ECU 22 intercepts a part of the area (the rectangular area 302) centered on the correction center point 301 from the image 300. The ECU 22 generates a distortion-reduced image 303 by performing a distortion reduction process on this area 302. In the distortion reduction process, the closer the position is to the correction center point 301, the more the distortion is reduced, and the farther the position is from the correction center point 301, the less the distortion is reduced or the distortion increases. Therefore, in some embodiments, the ECU 22 sets the correction center point 301 within the area of interest in the surrounding environment of the vehicle 1 and generates a distortion-reduced image for this area.

[0049] <Process>

[0050] As referred to Figure 3 As described, in the images obtained by the fish-eye cameras 41 to 44, which are examples of shooting devices equipped with wide-angle lenses, for shooting the exterior of the vehicle 1, a large distortion is generated, especially in the peripheral part. For the images with such distortion, it is impossible to directly apply the external recognition models such as object recognition and road recognition prepared for the images obtained by a normal camera like the standard camera 40. Therefore, it is necessary to convert the acquired images into distortion-reduced images (planar images) for external recognition.

[0051] In order to implement appropriate driving assistance control or automatic driving control based on the recognition result of the outside of the vehicle 1, it is required to obtain precise information about objects that are stationary or moving around the vehicle 1 (for example, other vehicles, pedestrians, bicycles, signals, and road signs, etc.). Therefore, in the following embodiment, the control device 2 finds the specific object for which precise information is to be obtained from the images obtained by the fisheye cameras 41 to 44 (that is, determines where the specific object exists in the image). Furthermore, the control device 2 will take a part of the area centered on the position of the discovered object or its vicinity in the image obtained by the fisheye cameras 41 to 44 (for example, Figure 3 The control device 2 sets a correction center point (for example, Figure 3 The image obtained in this manner is used to identify the exterior of vehicle 1. This allows for more precise information about the surrounding environment of vehicle 1 to be acquired through the identification process. In this specification, the term "near an object" refers to a location where the desired exterior identification process can be performed using the image obtained through the distortion reduction process.

[0052] Below, refer to Figure 4 In one embodiment, an example of a method in which the control device 2 controls the vehicle 1 will be described. This method may also be performed by the processor 20a of each of the ECUs 20 to 29 of the control device 2 executing a program in the memory 20b. Figure 4 The method may be started based on the fact that the driving assistance function or the automatic driving function of the control device 2 is already activated.

[0053] In S401, the control device 2 obtains images of the exterior of the vehicle 1 from the standard camera 40 and the fisheye cameras 41 to 44. Each image includes the exterior of the vehicle 1. Figure 2 Furthermore, it is not necessary to perform the processing of S402 to S404 on the image acquired from the standard camera 40.

[0054] In S402, the control device 2 performs detection processing on objects predetermined as detection targets (target objects) based on the images captured by each fisheye camera (fisheye cameras 41 to 44). This detection processing is equivalent to finding specific objects in the captured images, for which precise information is to be obtained through external recognition. Target objects for detection may include, for example, one or more of other vehicles, pedestrians, bicycles, traffic lights, and road signs.

[0055] As an example of the detection process, the control device 2 performs detection of an object by performing image recognition on the images acquired from the respective fish-eye cameras. This image recognition can be implemented by using a model that has been learned using, as training data, an image containing the object in the image obtained by the fish-eye camera, for example, a model using a known deep learning technique. When an unknown image is input, the learned model outputs the presence or absence of the object in the image and the region (position) of the object. The learned model can be stored in the memory 20b in advance. In addition, the learned model can output one or more regions where the object exists.

[0056] As another example of the detection process, the control device 2 can also perform detection of an object by performing image recognition on the images obtained by dividing the images acquired from the respective fish-eye cameras at a predetermined field of view angle (for example, 120°). Thereby, an object can be detected in each of the divided images, and the distortion reduction process (S404) and the external recognition process (S405) can be separately applied to the regions where the separately detected objects are located. Thereby, information related to the objects existing in a specific direction around the vehicle can be appropriately obtained. Further, as still another example, the control device 2 can also perform detection of an object by performing image recognition on the images obtained by temporarily performing distortion reduction processing on the images acquired from the respective fish-eye cameras.

[0057] Next, in S403 and S404, the control device 2 performs distortion reduction processing on the images acquired from the respective fish-eye cameras according to the detection result of the detection process in S402. More specifically, in S403, the control device 2 determines whether an object is detected in the images acquired from the respective fish-eye cameras in the detection process in S402. For each image corresponding to each fish-eye camera, the control device 2 returns the process to S401 when no object is detected, and enters S404 when an object is detected.

[0058] In S404, the control device 2 performs distortion reduction processing on a part of the region in the images acquired from the respective fish-eye cameras, that is, a part of the region centered on the detection position of the object or its vicinity. For example, the control device 2 sets the position of the detected object or its vicinity in the images acquired from the respective fish-eye cameras as the correction center point (transformation center position), intercepts a rectangular region centered on the correction center point, and performs distortion reduction processing on the intercepted image. Thereby, an image with reduced distortion is generated. Since existing techniques can be used in the distortion reduction processing, detailed description is omitted.

[0059] When the conversion process in S404 is completed, in S405, the control device 2 performs an identification process for identifying the exterior of the vehicle 1 based on the image obtained from the standard camera 40 and the image obtained through the conversion process (the image with reduced distortion). The identification process can be implemented, for example, by using a well-known deep learning technique that has been learned using the image containing the object in the image obtained by the fisheye camera as training data, in the same manner as in S402. The learned model can be stored in the memory 20b in advance. In addition, the learned model can output one or more regions where the object exists.

[0060] When the identification process is completed, the information related to the object extracted through the identification process is provided to the driving assistance control or the autonomous driving control. That is, the control device 2 can control the vehicle 1 based on the identification result of the exterior (for example, automatic braking, notification to the driver, change of the autonomous driving level, etc.). Existing techniques can be applied to the control of the vehicle 1 based on the identification result of the exterior, so detailed description is omitted.

[0061] After that, in S407, the control device 2 determines whether to end the operation. If it is determined that the operation is to be ended, the operation ends. Otherwise, the process returns to S401, and the above-described process is repeated. The control device 2 can also determine that the operation is to be ended, for example, based on the fact that the driving assistance function or the autonomous driving function has been turned off.

[0062] As described above, S401 to S407 are repeatedly executed. The control device 2 can also periodically execute the processes of S401 to S407. The execution cycle varies depending on the required time of the detection process in S402, the distortion reduction process in S404, and the identification process in S405, and can be about 100 ms, for example.

[0063] As described above, in the present embodiment, the control device 2 (ECU22 and ECU23) obtains an image obtained by photographing the exterior of the vehicle 1 from the photographing device (fisheye camera), and based on the image obtained from the photographing device, detects an object through image recognition. Further, the control device 2 takes, as an object, a part of the region in the image obtained from the photographing device, that is, a part of the region centered on the detection position of the object or its vicinity, according to the detection result of the object (for example, whether the object is detected), and performs a distortion reduction process for reducing the distortion of the image. The control device 2 identifies the exterior of the vehicle 1 based on the image obtained through the distortion reduction process. Thus, information related to the objects around the vehicle 1 can be obtained with high accuracy based on the image obtained by the photographing device (fisheye camera) equipped with a fisheye lens.

[0064] <Modification Example>

[0065] In some of the above-described embodiments, it is premised that an object to be detected (detected) in an image obtained from a fish-eye camera is predetermined. In contrast, in other embodiments, an object to be detected may be determined based on the operating state of the vehicle 1 (e.g., for each driving scenario of the vehicle 1).

[0066] Refer to Figure 5 , and an example of a method for the control device 2 to control the vehicle 1 in other embodiments will be described. This method can be performed in the same manner as the method of Figure 4 by the processor 20a of each ECU20 - ECU29 of the control device 2 executing a program in the memory 20b. Figure 5 The method of Figure 4 can also be started when the driving assistance function or the autonomous driving function of the control device 2 is turned on. In addition, hereinafter, for the sake of simplicity of description, the description of the same processing as the processing in the method of

[0067] In S501, the control device 2 determines an object to be the detection target of the object detection process (S402) based on the operating state of the vehicle 1. The operating state may be the driving scenario of the vehicle, or may also be the driving state of the vehicle (e.g., the autonomous driving level). For example, in a country where left-hand traffic is adopted, when the driving scenario of the vehicle 1 corresponding to the operating state of the vehicle 1 is a scenario of turning right at an intersection, the control device 2 may determine an oncoming vehicle traveling in the oncoming lane as an object to be the detection target. In addition, when the driving scenario of the vehicle 1 corresponding to the operating state of the vehicle 1 is a scenario of turning left at an intersection, the control device 2 may determine traffic participants (pedestrians, bicycles, and other vehicles, etc.) that are the cause of being involved in an accident during a left turn as an object to be the detection target.

[0068] After determining the target object, in S401, in the same manner as the method of Figure 4 , the control device 2 acquires images of the outside of the vehicle 1 from the standard camera 40 and the fish-eye cameras 41 - 44 respectively, and proceeds to S402. In addition, in this example, the process of S501 is performed before the process of S401, but the process of S501 may also be performed after the process of S401. In S402, in the same manner as Figure 4Similarly, the control device 2 performs object detection processing based on the images acquired from each fisheye camera (fisheye cameras 41 to 44). However, in this example, the object to be detected in the detection processing is the object determined in the process of S501. Further, for example, information indicating the presence or absence of an object in the image obtained by each fisheye camera based on the learned model described above is output, and information indicating the recognition accuracy of the object based on image recognition is output.

[0069] After the detection processing is completed, in S502, the control device 2 determines whether the recognition accuracy of the object based on image recognition in the detection processing performed in S402 exceeds the accuracy threshold. When the recognition accuracy exceeds the accuracy threshold, the control device 2 determines that an object has been detected and causes the process to proceed to S404, and in other cases, causes the process to return to S401.

[0070] Here, the accuracy threshold can be determined in advance for each motion state of the vehicle 1, or can be determined in advance for each type of object that is the detection target of the detection processing (S402). The control device 2 of the present embodiment determines (selects) the accuracy threshold used in S502 according to the motion state of the vehicle 1 or the type of object determined as the detection target (based on the motion state). For example, for a type of object with a high importance detected in a certain driving scenario, the corresponding accuracy threshold can be set lower so that it is easy to determine that an object has been detected in S502 (so that distortion reduction processing and recognition processing can be easily performed). On the other hand, for a type of object with a low detection importance, the corresponding accuracy threshold can be set higher so that distortion reduction processing and recognition processing are not easily performed. Thereby, the computational amount associated with distortion reduction processing and recognition processing can be efficiently reduced.

[0071] In S404 to S407, the control device 2 performs the same processing as Figure 4 the method of

[0072] Thus, according to the present embodiment, by determining an object to be detected based on the operation state of the vehicle 1, information on an object of a type that needs to be detected can be appropriately obtained from the image obtained by the fisheye camera in correspondence with the operation state of the vehicle. In addition, it is possible to easily perform distortion reduction processing and recognition processing on an object with a relatively high detection importance, while on the other hand, it is possible not to perform distortion reduction processing and recognition processing on an object with a relatively low detection importance. Thereby, the computational load associated with distortion reduction processing and recognition processing can be reduced. Further, by varying the accuracy threshold for comparison with the recognition accuracy of image recognition in the object detection process according to the operation state of the vehicle 1 or the type of the object determined to be the detection target, the computational load associated with distortion reduction processing and recognition processing can be efficiently reduced.

[0073] <Other Embodiments>

[0074] In addition, a program that implements one or more functions described in each embodiment is provided to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device can read and execute the program. The present invention can also be implemented in this way.

[0075] <Summary of Embodiments>

[0076] The above embodiments disclose at least the following embodiments.

[0077] [Item 1]

[0078] A control device (e.g., 2), which is a control device of a moving body (e.g., 1), the moving body having a photographing device (e.g., 41 to 44) equipped with a lens having a wide field of view angle, is characterized in that

[0079] the control device includes:

[0080] an image acquisition unit that acquires an image (e.g., 300) obtained by photographing the outside of the moving body from the photographing device;

[0081] a detection unit that detects an object by image recognition based on the image acquired from the photographing device;

[0082] a processing unit that, according to the detection result of the detection unit, performs distortion reduction processing for reducing distortion of an image on a partial area in the image acquired from the photographing device, i.e., a partial area centered on the detection position of the object or its vicinity (e.g., 302); and

[0083] An identification unit that identifies the exterior of the moving body based on an image (e.g., 303) obtained through the distortion reduction process.

[0084] According to this item, it is possible to accurately obtain information related to objects around the moving body from an image obtained by a photographing device equipped with a wide-angle lens.

[0085] [Item 2]

[0086] The control device according to Item 1, characterized in that

[0087] When the object is detected by the detection unit, the processing unit performs the distortion reduction process.

[0088] According to this item, it is possible to appropriately perform the distortion reduction process according to the detection result of the object.

[0089] [Item 3]

[0090] The control device according to Item 1 or 2, characterized in that

[0091] The detection unit detects the object by performing the image recognition on the image obtained from the photographing device.

[0092] According to this item, it is possible to detect the object without performing additional processing on the image obtained by the photographing device, and the processing load can be suppressed.

[0093] [Item 4]

[0094] The control device according to Item 1 or 2, characterized in that

[0095] The detection unit detects the object by performing the image recognition on an image obtained by dividing the image obtained from the photographing device at a predetermined field of view angle.

[0096] According to this item, it is possible to appropriately obtain information related to an object existing in a specific direction.

[0097] [Item 5]

[0098] The control device according to Item 1 or 2, characterized in that

[0099] The detection unit detects the object by performing the image recognition on an image obtained by performing the distortion reduction process on the image obtained from the photographing device.

[0100] According to this item, it is possible to implement image recognition for detecting an object by using a model prepared for an image with less distortion such as an image captured by the standard camera 40.

[0101] [Item 6]

[0102] The control device according to any one of Items 1 to 5, characterized in that

[0103] the control device further includes a determination unit that determines an object to be the object of the detection performed by the detection unit based on the action state of the moving body.

[0104] According to this item, it is possible to appropriately obtain information about an object of a type that needs to be detected corresponding to the action state of the moving body.

[0105] [Item 7]

[0106] The control device according to Item 6, characterized in that

[0107] the detection unit outputs information indicating the recognition accuracy of the object detected based on the image recognition,

[0108] when the recognition accuracy indicated by the information exceeds a threshold value, the processing unit performs the distortion reduction process.

[0109] According to this item, it is possible to appropriately control whether to perform the distortion reduction process and the recognition process based on the recognition accuracy of the object of the detection process, and effectively reduce the required amount of calculation.

[0110] [Item 8]

[0111] The control device according to Item 7, characterized in that

[0112] the threshold value is determined in advance for each type of object that is the object of the detection performed by the detection unit,

[0113] when the recognition accuracy indicated by the information exceeds the threshold value corresponding to the object determined by the determination unit, the processing unit performs the distortion reduction process.

[0114] According to this item, it is possible to efficiently reduce the amount of calculation associated with the distortion reduction process and the recognition process.

[0115] [Item 9]

[0116] The control device according to Item 7, characterized in that

[0117] the threshold value is determined in advance for each action state of the moving body,

[0118] When the recognition accuracy indicated by the information exceeds the threshold value corresponding to the operation state, the processing unit performs the distortion reduction process.

[0119] According to this item, it is possible to efficiently reduce the computational amount associated with the distortion reduction process and the recognition process.

[0120] [Item 10]

[0121] The control device according to any one of Items 1 to 9, characterized in that the photographing device is a photographing device equipped with a fish-eye lens.

[0122] According to this item, it is possible to highly accurately obtain information related to the objects around the moving body from the image obtained by the photographing device equipped with a fish-eye lens.

[0123] [Item 11]

[0124] The control device according to any one of Items 1 to 10, characterized in that

[0125] the moving body includes a plurality of photographing devices respectively arranged at the front, rear, and sides of the moving body,

[0126] and the image acquisition unit acquires the images from the plurality of photographing devices respectively.

[0127] According to this item, it is possible to highly accurately obtain information related to the surrounding objects in all directions around the vehicle.

[0128] [Item 12]

[0129] The control device according to any one of Items 1 to 11, characterized in that the moving body is a vehicle.

[0130] According to this item, it is possible to highly accurately obtain information related to the objects around the vehicle from the image obtained by the photographing device equipped with a fish-eye lens.

[0131] [Item 13]

[0132] A control method, which is a control method of a moving body (for example, 1), the moving body having a photographing device (for example, 41 to 44) equipped with a wide-angle lens, characterized in that

[0133] the control method includes:

[0134] an image acquisition step in which an image (for example, 300) obtained by photographing the outside of the moving body is acquired from the photographing device;

[0135] Detection step, in which object detection is performed by image recognition based on the image obtained from the imaging device;

[0136] Processing step, in which, according to the detection result in the detection step, distortion reduction processing for reducing the distortion of the image is performed on a partial area (e.g., 302) in the image obtained from the imaging device, i.e., a partial area centered on the detection position of the object or its vicinity; and

[0137] Recognition step, in which the exterior of the moving body is recognized based on the image (e.g., 303) obtained by the distortion reduction processing.

[0138] According to this item, information related to the objects around the moving body can be accurately obtained from the image obtained by the imaging device equipped with a wide-angle lens.

[0139] [Item 14]

[0140] A program for causing a computer to function as each unit of the control device according to any one of Items 1 to 12.

[0141] According to this item, the above effects can be achieved in the form of a program.

[0142] The embodiments of the present invention have been described above, but the present invention is not limited to the above embodiments, and various modifications and changes can be made within the scope of the gist of the present invention.

Claims

1. A control device, which is a control device for a moving body, and the moving body is provided with a photographing device having a wide-angle lens, characterized in that: The control device includes: An image acquisition unit that acquires an image obtained by photographing the outside of the moving body from the photographing device; A determination unit that determines an object to be detected in the detection process based on the image based on the motion state of the moving body; A detection unit that detects the object by image recognition based on the image acquired from the photographing device; A processing unit that performs distortion reduction processing for reducing the distortion of the image on a partial area in the image acquired from the photographing device, that is, a partial area centered on the detection position of the object or its vicinity; And An identification unit that identifies the outside of the moving body based on the image obtained by the distortion reduction processing, The detection unit outputs information indicating the recognition accuracy of the object based on the image recognition, When the recognition accuracy indicated by the information exceeds a threshold corresponding to the object determined by the determination unit, the processing unit performs the distortion reduction processing, The threshold is predetermined for each type of object that is the object of the detection performed by the detection unit.

2. The control device according to claim 1, characterized in that: When the detection unit detects the object, the processing unit performs the distortion reduction processing.

3. The control device according to claim 1 or 2, characterized in that: The detection unit detects the object by performing the image recognition on the image acquired from the photographing device.

4. The control device according to claim 1 or 2, characterized in that: The detection unit detects the object by performing the image recognition on the image obtained by dividing the image acquired from the photographing device at a predetermined field of view angle.

5. The control device according to claim 1 or 2, characterized in that: The detection unit detects the object by performing the image recognition on the image obtained by performing the distortion reduction processing on the image acquired from the photographing device.

6. The control device according to claim 1 or 2, characterized in that, The photographing device is a photographing device equipped with a fish-eye lens.

7. The control device according to claim 1 or 2, characterized in that: The moving body is provided with a plurality of photographing devices respectively arranged at the front, rear and sides of the moving body, The image acquisition unit acquires images from the plurality of photographing devices respectively.

8. The control device according to claim 1 or 2, characterized in that, The moving body is a vehicle.

9. A storage medium that stores a program for causing a computer to function as each unit of the control device according to claim 1 or 2.

10. A control method, which is a control method for a moving body, and the moving body is provided with a photographing device having a wide-angle lens, characterized in that: The control method includes: An image acquisition step, in which an image obtained by photographing the exterior of the moving body is acquired from the photographing device; A determination step, in which an object to be detected in the detection process based on the image is determined based on the motion state of the moving body; A detection step, in which the object is detected by image recognition based on the image acquired from the photographing device; A processing step, in which, according to the detection result in the detection step, a part of the image acquired from the photographing device, that is, a part of the area centered on the detection position of the object or its vicinity, is subjected to distortion reduction processing for reducing image distortion; And A recognition step, in which the exterior of the moving body is recognized based on the image obtained by the distortion reduction processing; In the detection step, information indicating the recognition accuracy of the object based on the image recognition is output; In the processing step, when the recognition accuracy indicated by the information exceeds a threshold corresponding to the object determined in the determination step, the distortion reduction processing is performed; The threshold is predetermined for each type of object to be detected in the detection step.

Images