Control device and control method for mobile body, storage medium, and vehicle

Abstract

The present application relates to a mobile body control device and control method, storage medium, and vehicle that appropriately recognize the outside of a mobile body according to a movement scene. A mobile body control device having one or more imaging devices includes: an image acquisition unit that acquires images of the outside of the mobile body from the one or more imaging devices; a correction unit that performs distortion reduction processing for reducing distortion of the images for one or more regions included in the images acquired from the one or more imaging devices; and a recognition unit that recognizes the outside of the mobile body based on the images after the distortion reduction processing, the correction unit determining the one or more regions that are the targets of the distortion reduction processing in accordance with a predetermined rule corresponding to the movement scene of the mobile body.

Description

TECHNICAL FIELD

[0001] The present application relates to a control device and a control method for a moving body, a storage medium, and a vehicle. BACKGROUND

[0002] A technique of recognizing an outside of a vehicle using a plurality of cameras has been put into practical use. The recognition result of the outside is used for driving assistance, automatic driving. In Patent Literature 1, a technique of capturing a periphery of a vehicle in a wide range by a camera of a wide-angle lens is proposed. In order to reduce distortion of an image captured by the camera of the wide-angle lens, coordinate conversion is performed.

[0003] PRIOR ART DOCUMENT

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2018-171964 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] A process for reducing distortion of an image captured by a camera in which a wide-angle lens or a fish-eye lens is installed consumes electric power. Therefore, if the process for reducing distortion is excessively performed in order to recognize an outside of a vehicle, the amount of electric power consumption increases. The increase in the amount of electric power consumption as described above is not limited to a vehicle, but is also applicable to other moving bodies. An object of an aspect of a part of the present application is to provide a technique for appropriately recognizing an outside of a moving body in accordance with a moving scenario.

[0008] MEANS FOR SOLVING THE PROBLEMS

[0009] In view of the above problem, according to one aspect of the embodiments, there is provided a control device of a mobile body having one or more imaging devices, the control device including: an image acquisition unit configured to acquire images of an outside of the mobile body from the one or more imaging devices; a correction unit configured to perform distortion reduction processing for reducing distortion of the images, individually for one or more regions included in the images acquired from the one or more imaging devices; and an identification unit configured to identify the outside of the mobile body based on the images after the distortion reduction processing, the correction unit determining the one or more regions as targets of the distortion reduction processing in accordance with a predetermined rule corresponding to a movement scenario of the mobile body. According to another aspect of the embodiments, there is provided a control method of a mobile body having one or more imaging devices, the control method including: an image acquisition step of acquiring images of an outside of the mobile body from the one or more imaging devices; a correction step of performing distortion reduction processing for reducing distortion of the images, individually for one or more regions included in the images acquired from the one or more imaging devices; and an identification step of identifying the outside of the mobile body based on the images after the distortion reduction processing, the one or more regions being determined as targets of the distortion reduction processing in the correction step in accordance with a predetermined rule corresponding to a movement scenario of the mobile body.

[0010] Effects of Invention

[0011] According to the above means, the outside of the mobile body can be appropriately identified in accordance with the movement scenario. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a block diagram illustrating a configuration example of a vehicle according to the embodiments.

[0013] Figure 2 is a schematic view illustrating an imaging range of a camera according to the embodiments.

[0014] Figure 3 is a schematic view illustrating distortion reduction processing according to the embodiments.

[0015] Figure 4 is a schematic view illustrating a target region of the distortion reduction processing according to the embodiments.

[0016] Figure 5 is a flowchart illustrating an example of an operation of a control device of a vehicle according to the embodiments.

[0017] Figure 6 is a timing chart illustrating a target candidate of the distortion reduction processing according to the embodiments.

[0018] Figure 7 is a schematic diagram illustrating a general driving scenario involved in the embodiment.

[0019] Figure 8 is a schematic diagram illustrating a state transition of a specific rule involved in the embodiment.

[0020] Figure 9 is a schematic diagram illustrating a position of a vertical direction of a region of a specific rule involved in the embodiment.

[0021] Figure 10 is a schematic diagram illustrating a specific driving scenario involved in the embodiment.

[0022] Figure 11 is a schematic diagram illustrating a state transition of a specific rule involved in the embodiment.

[0023] Figure 12 is a schematic diagram illustrating a state transition of a specific rule involved in the embodiment.

[0024] Figure 13 is a schematic diagram illustrating a specific driving scenario involved in the embodiment.

[0025] Figure 14 is a schematic diagram illustrating a state transition of a specific rule involved in the embodiment.

[0026] Figure 15 is a schematic diagram illustrating a state transition of a specific rule involved in the embodiment.

[0027] Figure 16 is a schematic diagram illustrating a position of a vertical direction of a region of a specific rule involved in the embodiment.

[0028] Figure 17 is a schematic diagram illustrating a specific driving scenario involved in the embodiment.

[0029] Figure 18 is a schematic diagram illustrating a state transition of a specific rule involved in the embodiment.

[0030] Figure 19 is a schematic diagram illustrating a specific driving scenario involved in the embodiment.

[0031] Figure 20 is a schematic diagram illustrating a state transition of a specific rule involved in the embodiment.

[0032] Figure 21 is a schematic diagram illustrating a specific driving scenario involved in the embodiment.

[0033] Figure 22 is a schematic diagram illustrating a state transition according to a specific rule involved in the embodiment.

[0034] Figure 23 is a schematic diagram illustrating a state transition according to a specific rule involved in the embodiment.

[0035] Figure 24 is a schematic diagram illustrating a specific driving scenario involved in the embodiment.

[0036] Figure 25 is a schematic diagram illustrating a state transition according to a specific rule involved in the embodiment. DETAILED DESCRIPTION

[0037] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the technical scope of the present application, and the combinations of characteristics described in the embodiments are not necessarily essential to the application. Two or more characteristics described in the embodiments can be combined arbitrarily. In addition, the same reference numerals are used to designate the same or similar components, and repetitive description is omitted. Note that in the following embodiments, a mobile body is described as a vehicle, and the mobile body is not limited to a vehicle but can be an aircraft, a robot, or the like.

[0038] Figure 1 is a block diagram of a vehicle 1 according to one embodiment of the present application. In Figure 1 , an outline of the vehicle 1 is shown in a plan view and a side view. As one example, the vehicle 1 is a four-wheeled passenger car of a sedan type. The vehicle 1 can be a four-wheeled vehicle as described above, or can be a two-wheeled vehicle or another type of vehicle.

[0039] The vehicle 1 includes a vehicle control device 2 (hereinafter, simply referred to as a control device 2) that controls the vehicle 1. The control device 2 includes a plurality of ECUs (Electronic Control Units) 20 to 29 that are communicable by a vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit), a memory such as a semiconductor memory, an interface of an external device, and the like. In the memory, a program executed by the processor, data used by the processor in processing, and the like are stored. Each ECU can have a plurality of processors, memories, interfaces, and the like. For example, the ECU 20 has a processor 20a and a memory 20b. By the processor 20a executing an instruction including a program stored in the memory 20b, processing based on the ECU 20 is executed. Instead, the ECU 20 can have a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for executing processing based on the ECU 20. The same applies to the other ECUs.

[0040] Hereinafter, functions and the like assumed by each ECU 20 to 29 will be described. Note that the number of ECUs and the functions assumed thereby can be appropriately designed, can be more detailed than in the present embodiment, or can be integrated.

[0041] The ECU 20 executes control related to automatic travel of the vehicle 1. In automatic driving, at least any one of steering and acceleration / deceleration of the vehicle 1 is automatically controlled. The automatic travel by the ECU 20 can include automatic travel (also referred to as automatic driving) that does not require a travel operation by a driver, and automatic travel (also referred to as driving assistance) for assisting a travel operation by a driver.

[0042] The ECU 21 controls an electric power steering device 3. The electric power steering device 3 includes a mechanism that steers front wheels in accordance with a steering operation (steering operation) of a driver on a steering wheel 31. In addition, the electric power steering device 3 includes a motor that functions as a driving force for assisting a steering operation or automatically steering the front wheels, a sensor that detects a steering angle, and the like. In a case where a driving state of the vehicle 1 is automatic driving, the ECU 21 automatically controls the electric power steering device 3 in correspondence with an instruction from the ECU 20, and controls a traveling direction of the vehicle 1.

[0043] The ECU 22 and the ECU 23 perform control of a detection unit that detects a situation around the vehicle and information processing of a detection result. As the detection unit that detects a situation around the vehicle, the vehicle 1 includes one standard camera 40 and four fisheye cameras 41 to 44. The standard camera 40 and the fisheye cameras 42 and 44 are connected to the ECU 22. The fisheye cameras 41 and 43 are connected to the ECU 23. The ECU 22 and the ECU 23 are able to extract an outline of an object and a lane division line (white line or the like) on a road by analyzing images captured by the standard camera 40 and the fisheye cameras 41 to 44.

[0044] The fisheye cameras 41 to 44 refer to cameras in which a fisheye lens is installed. Hereinafter, the configuration of the fisheye camera 41 will be described. The other fisheye cameras 42 to 44 can have the same configuration. The fisheye camera 41 has a wider field of view than the standard camera 40. Therefore, the fisheye camera 41 is able to capture a wider range than the standard camera 40. An image captured by the fisheye camera 41 has a larger distortion than an image captured by the standard camera 40. Therefore, the ECU 23 can perform conversion processing for reducing distortion (hereinafter, referred to as "distortion reduction processing") on an image captured by the fisheye camera 41 before analyzing the image. On the other hand, the ECU 22 can not perform distortion reduction processing on an image captured by the standard camera 40 before analyzing the image. In this way, the standard camera 40 is an imaging device that captures an image that is not a target of distortion reduction processing, and the fisheye camera 41 is an imaging device that captures an image that is a target of distortion reduction processing. Instead of the standard camera 40, another imaging device that captures an image that is not a target of distortion reduction processing, such as a camera in which a wide-angle lens or a telephoto lens is installed, can be used.

[0045] The standard camera 40 is installed at the center of the front portion of the vehicle 1 and captures a situation in front of the vehicle 1. The fisheye camera 41 is installed at the center of the front portion of the vehicle 1 and captures a situation in front of the vehicle 1. In Figure 1 In the embodiment, the standard camera 40 and the fisheye camera 41 are arranged in the horizontal direction. However, the arrangement of the standard camera 40 and the fisheye camera 41 is not limited to this, and they can be arranged in the vertical direction, for example. In addition, at least one of the standard camera 40 and the fisheye camera 41 can be installed at the front portion of the roof of the vehicle 1 (for example, the inner side of the front window of the vehicle 1). For example, the fisheye camera 41 can be installed at the center of the front portion of the vehicle 1 (for example, the bumper), and the standard camera 40 can be installed at the front portion of the roof of the vehicle 1. The fisheye camera 42 is installed at the center of the right portion of the vehicle 1 and captures a situation in the right direction of the vehicle 1. The fisheye camera 43 is installed at the center of the rear portion of the vehicle 1 and captures a situation in the rear direction of the vehicle 1. The fisheye camera 44 is installed at the center of the left portion of the vehicle 1 and captures a situation in the left direction of the vehicle 1.

[0046] The kind, number, and mounting position of the cameras possessed by the vehicle 1 are not limited to the above-described examples. In addition, as a detection unit for detecting a target object around the vehicle 1 or measuring the distance to the target object, the vehicle 1 can also include a Light Detection and Ranging (LiDAR), a millimeter wave radar.

[0047] The ECU 22 performs control of the standard camera 40 and the fisheye cameras 42, 44, and information processing of the detection results. The ECU 23 performs control of the fisheye cameras 41, 43, and information processing of the detection results. By dividing the detection unit that detects the state of the vehicle into two systems, the reliability of the detection results can be improved.

[0048] The ECU 24 performs control of the gyro sensor 5, the GPS sensor 24b, the communication device 24c, and information processing of the detection results or communication results. The gyro sensor 5 detects the rotational motion of the vehicle 1. The route of travel of the vehicle 1 can be determined from the detection result of the gyro sensor 5, the wheel speed, and the like. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c performs wireless communication with a server that provides map information, traffic information, and acquires these information. The ECU 24 can access the database 24a of the map information constructed in the memory and perform route search from the current position to the destination, and the like. The ECU 24, the map database 24a, and the GPS sensor 24b constitute a so-called navigation device.

[0049] The ECU 25 is provided with a communication device 25a for vehicle-to-vehicle communication. The communication device 25a performs wireless communication with other vehicles in the vicinity and performs information exchange between vehicles.

[0050] The ECU 26 controls the power device 6. The power device 6 is a mechanism that outputs a driving force that rotates the driving wheels of the vehicle 1, and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine in correspondence with the driving operation (accelerator operation or acceleration operation) of the driver detected by the operation detection sensor 7a provided to the accelerator pedal 7A, for example, or switches the gear of the transmission based on information such as the vehicle speed detected by the vehicle speed sensor 7c. In the case where the driving state of the vehicle 1 is in automatic driving, the ECU 26 automatically controls the power device 6 in correspondence with the instruction from the ECU 20, and controls the acceleration and deceleration of the vehicle 1.

[0051] The ECU 27 controls the illuminating devices (headlamps, tail lamps, and the like) including the direction indicator 8 (turn signal). In the example of FIG. 1, the direction indicator 8 is provided to the front portion of the vehicle 1, the door mirror, and the rear portion. Figure 1 The ECU 27 controls the illuminating devices (headlamps, tail lamps, and the like) including the direction indicator 8 (turn signal). In the example of FIG. 1, the direction indicator 8 is provided to the front portion of the vehicle 1, the door mirror, and the rear portion.

[0052] ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and receives information from the driver. The sound output device 91 notifies the driver of information via sound. The display device 92 notifies the driver of information via image display. The display device 92 may be located in front of the driver's seat, forming part of the instrument panel, etc. It should be noted that sound and display are exemplified here, but information can also be notified via vibration or light. Furthermore, multiple combinations of sound, display, vibration, or light can be used to report information. Further, the combination or reporting method can be different depending on the level of the information to be reported (e.g., urgency). The input device 93 is a switch group configured in a position operable by the driver to provide instructions for the vehicle 1, but it may also include a sound input device.

[0053] ECU 29 controls the braking device 10 and the parking brake (not shown). The braking device 10 is, for example, a disc brake, installed on each wheel of the vehicle 1, which decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheels. ECU 29 controls the operation of the braking device 10 in accordance with the driver's driving operation (braking operation) detected by the operation detection sensor 7b installed on the brake pedal 7B. When the vehicle 1 is in automatic driving mode, ECU 29 automatically controls the braking device 10 in accordance with the instructions from ECU 20, controlling the deceleration and stopping of the vehicle 1. The braking device 10 and the parking brake can also operate to maintain the vehicle 1 in a stopped state. In addition, if the transmission of the power unit 6 is equipped with a parking lock mechanism, it can also be operated to maintain the vehicle 1 in a stopped state.

[0054] Reference Figure 2 The shooting range of the standard camera 40 and the fisheye cameras 41-44 is explained. Figure 2 In the diagram, (a) represents the horizontal shooting range of each camera. Figure 2 (b) in the figure represents the vertical shooting range of the fisheye camera 41 at the front of vehicle 1. Figure 2 (c) in the figure represents the vertical shooting range of the fisheye camera 42 on the right side of vehicle 1. Figure 2 In this specification, (d) indicates the vertical shooting range of the fisheye camera 43 at the rear of vehicle 1. In this specification, the horizontal and vertical directions are based on the body of vehicle 1. The vertical shooting range of the fisheye camera 42 on the left side of vehicle 1 can be compared with... Figure 2 (c) is the same as in the original text, so it is omitted.

[0055] First, refer to Figure 2(a) of FIG. 10, the shooting range in the plan view (i.e., the horizontal direction of the vehicle 1) of the vehicle 1 is described. The standard camera 40 shoots the situation included in the shooting range 200. The shooting center 200C of the standard camera 40 faces the front of the vehicle 1. The shooting center 200C can also be defined by the direction of the optical axis of the lens. The horizontal field of view of the standard camera 40 can be less than 90°, for example, it can be about 45° or about 30°.

[0056] The fisheye camera 41 shoots the situation included in the shooting range 201. The shooting center 201C of the fisheye camera 41 faces the front of the vehicle 1. The fisheye camera 42 shoots the situation included in the shooting range 202. The shooting center 202C of the fisheye camera 42 faces the right of the vehicle 1. The fisheye camera 43 shoots the situation included in the shooting range 203. The shooting center 203C of the fisheye camera 43 faces the rear of the vehicle 1. The fisheye camera 44 shoots the situation included in the shooting range 204. The shooting center 204C of the fisheye camera 44 faces the left of the vehicle 1. The horizontal field of view of the fisheye cameras 41 to 44 can be greater than 90°, greater than 150°, or greater than 180°, for example, it can be about 180°. Figure 2 (a) of FIG. 10 shows an example in which the horizontal field of view of the fisheye cameras 41 to 44 is 180°.

[0057] Next, with reference to Figure 2 (b) to Figure 2 (d) of FIG. 10, the shooting range in the vertical direction of the vehicle 1 is described. In Figure 2 (b) of FIG. 10, the shooting range in the vertical direction of the fisheye camera 41 is described, in Figure 2 (c) of FIG. 10, the shooting range in the vertical direction of the fisheye camera 42 is described, in Figure 2 (d) of FIG. 10, the shooting range in the vertical direction of the fisheye camera 43 is described. The shooting range in the vertical direction of the other fisheye camera 44 can be the same as that in Figure 2 (c) of FIG. 10.

[0058] The vertical field of view of the fisheye cameras 41 to 44 can be greater than 90°, greater than 150°, or greater than 180°, for example, it can be about 180°. Figure 2 (b) to Figure 2In the text, (d) indicates an example where the vertical field of view of fisheye cameras 41-44 is 180°. The shooting center 203C of fisheye camera 43 faces downwards (towards the ground) compared to a direction parallel to the ground. Alternatively, the shooting center 203C of fisheye camera 43 can face either a direction parallel to the ground or a direction higher than parallel to the ground (opposite to the ground). Furthermore, the shooting centers 201C-204C of fisheye cameras 41-44 can also face a single direction in the vertical direction.

[0059] The standard camera 40 and the fisheye cameras 41-44 have shooting ranges 200-204 as described above, thus the frontal view and the four oblique directions of the vehicle 1 are included in the shooting range of two independent cameras. Specifically, the frontal view of the vehicle 1 is included in both the shooting range 200 of the standard camera 40 and the shooting range 201 of the fisheye camera 41. The right oblique front view of the vehicle 1 is included in both the shooting range 201 of the fisheye camera 41 and the shooting range 202 of the fisheye camera 42. The same applies to the other three oblique directions of the vehicle 1.

[0060] Reference Figure 3 as well as Figure 4 The distortion reduction processing of images captured by fisheye cameras 41-44 is explained. Figure 3 This represents the images before and after distortion reduction processing. Figure 4 This indicates the region that will be the target of distortion reduction processing. Figure 4 (a) in the diagram represents a view of vehicle 1 from above. Figure 4 Image (b) shows a view of vehicle 1 from the rear. Image 300 is an image of the right side of vehicle 1 taken by fisheye camera 42. Figure 3 As shown, image 300 exhibits significant distortion, particularly in the peripheral regions.

[0061] The ECU22, connected to the fisheye camera 42, performs distortion reduction processing on the image 300. Specifically, as... Figure 3 As shown, ECU22 determines a point within image 300 as the conversion center 301. (As...) Figure 4 As shown, the conversion center 301 is located on the right side of the shooting range 202 when viewed from the fisheye camera 42 in the horizontal direction, and faces a direction parallel to the ground in the vertical direction.

[0062] ECU22 extracts a rectangular region 302 from image 300, centered at conversion center 301. For example... Figure 4As shown, the region 302 corresponds to the region 202R located on the right side as viewed from the fisheye camera 42 in the photographing range 202. The ECU 22 generates an image 303 in which distortion is reduced by performing distortion reduction processing on the region 302. The image 303 is an image representing the situation of the region 202R.

[0063] As a result of the distortion reduction processing, the closer to the conversion center 301, the more the distortion is reduced, and the farther from the conversion center 301, the distortion is not reduced or the distortion increases. In a case where the entire image 300 is the object of the distortion reduction processing, the distortion increases in the region located at a position farther from the conversion center 301. Therefore, even if the region located at the farther position is used to analyze the outside of the vehicle 1, high-precision analysis cannot be performed. Therefore, the control device 2 sets the conversion center 301 as the region of the object of analysis, performs distortion reduction processing on the region around the conversion center 301, and uses the processed image to analyze the situation of the region of the object of analysis.

[0064] The photographing range 201 includes the region 201L photographing the left oblique front of the vehicle 1, the region 201F photographing the front of the vehicle 1, and the region 201R photographing the right oblique front of the vehicle 1 as the region of the object of analysis. The photographing range 202 includes the region 202L photographing the right oblique front of the vehicle 1, the region 202F photographing the right front of the vehicle 1, and the region 202R photographing the right oblique rear of the vehicle 1 as the region of the object of analysis. The photographing range 203 includes the region 203L photographing the right oblique rear of the vehicle 1, the region 203F photographing the rear of the vehicle 1, and the region 203R photographing the left oblique rear of the vehicle 1 as the region of the object of analysis. The photographing range 204 includes the region 204L photographing the left oblique rear of the vehicle 1, the region 204F photographing the left front of the vehicle 1, and the region 204R photographing the left oblique front of the vehicle 1 as the region of the object of analysis. The photographing range 201 can also be divided in such a manner that the three regions 201L, 201F, and 201R are equal in the horizontal direction (i.e., in such a manner that the field angles of the regions in the horizontal direction are equal). The other photographing ranges 202 to 204 can also be divided into three equal parts.

[0065] The control device 2 sets the conversion center 301 within the region 202L (for example, the center of the region 202L) included in the imaging range 202 of the fisheye camera 42, performs the distortion reduction process on the region around the conversion center 301, and uses the processed image to analyze the situation in the right oblique front of the vehicle 1 when the situation in the right oblique front of the vehicle 1 is intended to be analyzed. The control device 2 sets the conversion center 301 within the region 202F (for example, the center of the region 202F) included in the imaging range 202 of the fisheye camera 42, performs the distortion reduction process on the region around the conversion center 301, and uses the processed image to analyze the situation in the right front of the vehicle 1 when the situation in the right front of the vehicle 1 is intended to be analyzed. The control device 2 sets the conversion center 301 within the region 202R (for example, the center of the region 202R) included in the imaging range 202 of the fisheye camera 42, performs the distortion reduction process on the region around the conversion center 301, and uses the processed image to analyze the situation in the right oblique rear of the vehicle 1 when the situation in the right oblique rear of the vehicle 1 is intended to be analyzed.

[0066] Referring to Figure 5 An example of a method in which the control device 2 controls the vehicle 1 in some embodiments will be described. The method can also be performed by the processor 20a of each ECU 20 to 29 of the control device 2 executing a program stored in the memory 20b. Figure 5 The method can also be started in accordance with the driving assist function or the automatic driving function of the control device 2 being turned on.

[0067] In step S501, the control device 2 acquires images of the outside of the vehicle 1 from the standard camera 40 and the fisheye cameras 41 to 44, respectively. Each image includes the situation in the range described in the above-described embodiment in the outside of the vehicle 1. Figure 2 In step S501, the control device 2 acquires images of the outside of the vehicle 1 from the standard camera 40 and the fisheye cameras 41 to 44, respectively. Each image includes the situation in the range described in the above-described embodiment in the outside of the vehicle 1.

[0068] In step S502, the control device 2 determines the current driving scene of the vehicle 1. In the example described below, as the driving scene of the vehicle, the following is processed: (1) a scene in which the vehicle enters a T-shaped intersection or a scene in which the vehicle temporarily stops and then starts, (2) a scene in which the vehicle travels on a narrow road, (3) a scene in which the vehicle makes a right turn or a left turn at an intersection, (4) a scene in which the vehicle backs up, and (5) a scene in which the vehicle changes lanes. Scenes other than these are processed as a general (default) scene. The general scene includes, for example, a scene in which the vehicle 1 travels along a road. In this specification, a case in which the vehicle 1 travels in a country in which left-hand traffic is adopted is processed. In a country in which right-hand traffic is adopted, the left turn and the right turn in the following description are exchanged.

[0069] In step S503, the control device 2 decides one or more regions in the image acquired in step S501 as the object of the distortion alleviation processing in accordance with a rule corresponding to the current driving scene of the vehicle 1. Hereinafter, the rule is referred to as a region decision rule. The region decision rule is decided in advance, for example, stored in the memory 20b. Specific examples of the region decision rule will be described later.

[0070] In step S504, as shown in Figure 3 , the control device 2 performs the distortion alleviation processing on the one or more regions decided as the object of the distortion alleviation processing, respectively. The distortion alleviation processing is processing for alleviating the distortion of the image acquired from the fisheye cameras 41 to 44. Existing techniques can also be used in the distortion alleviation processing, and thus detailed description is omitted. The image acquired from the standard camera 40 does not need to be subjected to the distortion alleviation processing.

[0071] In step S505, the control device 2 recognizes the outside of the vehicle 1 on the basis of the image acquired from the standard camera 40 and the images acquired from the fisheye cameras 41 to 44 and subjected to the distortion alleviation processing. For example, the control device 2 can also determine the target objects around the vehicle 1 by applying the corrected images to a model learned in advance and stored in the memory 20b. Further, the control device 2 can also perform the control of the vehicle 1 (for example, automatic braking, notification to the driver, change of the automatic driving level, and the like) in accordance with the recognition result of the outside. Existing techniques can also be applied to the control of the vehicle 1 corresponding to the recognition result of the outside, and thus detailed description is omitted.

[0072] In step S506, the control device 2 determines whether or not to end the operation. The control device 2 ends the operation in the case where it is determined to end the operation (YES in step S506), and returns the operation to step S501 in the case where it is not determined to end the operation (NO in step S506). The control device 2 can also determine to end the operation, for example, in the case where the driving assistance function or the automatic driving function has been turned off.

[0073] As described above, steps S501 to S505 are repeatedly executed. The control device 2 can also periodically execute the operation of steps S501 to S505. The execution period differs depending on the required time of the distortion alleviation processing of S504 and the recognition processing of S505, and can be, for example, about 100 ms.

[0074] With reference to Figure 6 , the periodic operation of the control device 2 will be described. Figure 6 The circles indicate candidates of the image used in the recognition processing of step S505. Figure 6Δt indicates a period in which steps S501 to S505 are executed. Since the distortion of the image acquired from the standard camera 40 is small, the distortion mitigation process of step S504 can not be performed and the image can be used in the recognition process of step S505. The images acquired from the fisheye cameras 41 to 44 are subjected to the distortion mitigation process in step S504 and then used in the recognition process of step S505. As described above, the images acquired from the fisheye cameras can generate images in which three regions divided in the horizontal direction are each subjected to the distortion mitigation process. Therefore, in one action timing of the periodic action, the control device 2 can execute the recognition process using at most thirteen images. Since twelve of the thirteen images are acquired from the fisheye cameras, the distortion mitigation process is performed before the recognition process. If the distortion mitigation process is performed on all of the twelve regions, the processing load becomes large and the power consumption also becomes high. Therefore, in the following embodiments, based on the travel scene of the vehicle 1, it is decided which of the twelve regions is subjected to the distortion mitigation process and used for the recognition process in each action timing.

[0075] Reference Signs List Figure 7 to Figure 9 A region decision rule for a normal scene will be described. Figure 7 An example of a travel scene is shown. In the example shown in FIG. 8, the vehicle 1 is traveling along a road on a straight road. Figure 7

[0076] Figure 8 Regions that become the analysis target in each action timing are shown. The control device 2 repeatedly the states 800 to 802 in order. That is, if the action timing at time t is the state 800, the action timing at time t + Δt (Δt indicates a period as described above) becomes the state 801, the action timing at time t + 2 x Δt becomes the state 802, and the action timing at time t + 3 x Δt returns to the state 800. The same applies to the region decision rules in other travel scenes described later.

[0077] In the state 800, the photographed range 200 of the standard camera 40, the region 201L of the fisheye camera 41, the region 202R of the fisheye camera 42, the region 203L of the fisheye camera 43, and the region 204R of the fisheye camera 44 become the analysis target. In the state 801, the photographed range 200 of the standard camera 40, the region 201F of the fisheye camera 41, the region 202F of the fisheye camera 42, the region 203F of the fisheye camera 43, and the region 204F of the fisheye camera 44 become the analysis target. In the state 802, the photographed range 200 of the standard camera 40, the region 201R of the fisheye camera 41, the region 202L of the fisheye camera 42, the region 203R of the fisheye camera 43, and the region 204L of the fisheye camera 44 become the analysis target.

[0078] ​In a case where a region included in the photographing range of the fisheye camera becomes a resolution target, the control device 2 performs the distortion reduction process on the region as described above. Therefore, the region determination rule specifies a horizontal position of a region that becomes a target of the distortion reduction process and a timing at which the region located at the position becomes a target of the distortion reduction process. In addition, the region determination rule specifies a rule for each of the plurality of fisheye cameras 41 to 44.

[0079] As described above, by changing the state at each operation timing, the control device 2 sets the front face of the vehicle 1 as a resolution target every cycle (i.e., every time), sets the right oblique front face, the right face, the right oblique rear face, the rear face, the left oblique rear face, the left face, and the left oblique front face of the vehicle 1 as resolution targets every three cycles, respectively. In addition, in order to distribute the load of the control device 2 at a specific operation timing, the region that is not a resolution target is distributed to a plurality of operation timings. Furthermore, for the front face of the vehicle 1, resolution using both the image of the standard camera 40 and the image of the fisheye camera is performed every three cycles, and for each of the four oblique directions of the vehicle 1, resolution using the images of two fisheye cameras is performed every three cycles. In this way, at each operation timing, by making a part of the images of the fisheye cameras 41 to 44 that perform the distortion correction process a resolution target, the processing load of the control device 2 is reduced, and the power consumption is reduced.

[0080] Figure 9 represents a position in the vertical direction of the region of the fisheye camera 41. Figure 9 (a) in (a) represents a position in the vertical direction of the region 201F of the fisheye camera 41. Figure 9 (b) in (b) represents a position in the vertical direction of the region 202F of the fisheye camera 42. Figure 9 (c) in (c) represents a position in the vertical direction of the region 203F of the fisheye camera 43. The positions in the vertical direction of the regions of the fisheye camera 44 can be the same as (b) in (b), and thus the description is omitted. Figure 9

[0081] In the region determination rule, as described above, the region determination rule specifies a horizontal position of a region that becomes a target of the distortion reduction process and a timing at which the region located at the position becomes a target of the distortion reduction process. Figure 9 ​As shown in (a), the angle between the conversion center 301 of region 201F and the vertical direction of vehicle 1 is defined as θ1. θ1 can be 90 degrees or a value less than 90 degrees (e.g., 80 degrees). For regions 201R and 201L, the angle between the conversion center 301 and the vertical direction of vehicle 1 can also be defined as θ1. Similarly, for the regions of fisheye cameras 42-44, the angle between the conversion center 301 and the vertical direction of vehicle 1 can also be defined as θ1. In this way, the region determination rule defines the vertical position of the region to which distortion reduction processing is applied. By setting the angle between the conversion center 301 and the vertical direction of vehicle 1 to θ1 (e.g., 90 degrees), it is possible to evenly resolve the distance and proximity of vehicle 1.

[0082] Reference Figure 10 to Figure 12 The rules for determining the area are explained for scenarios where vehicle 1 enters a T-junction or where vehicle 1 stops temporarily and then starts again. Figure 10 This is an example illustrating such a scenario. Figure 10 In the example shown, vehicle 1 is about to enter a T-junction. Because there is a stop sign before the T-junction, vehicle 1 stops temporarily there before starting again. Figure 10 The example illustrates scenarios involving both entering a T-junction and restarting after a temporary stop, but the following area determination rules can also be applied to scenarios involving only one of these actions. The vertical position of the parsed object's area in the scenario of vehicle 1 entering a T-junction or the scenario of vehicle 1 temporarily stopping and then restarting can also be compared with... Figure 9 The descriptions are the same as those in the previous text, so repeated descriptions are omitted.

[0083] Figure 11 This illustrates an example of a region-determination rule, where the region becomes the object of analysis at each action timing. Control device 2 repeats state 1100. In state 1100, the shooting range 200 of standard camera 40, regions 201L and 201R of fisheye camera 41, region 202L of fisheye camera 42, and region 204R of fisheye camera 44 become the objects of analysis. In this example, control device 2 sets the front, right-side front, and left-side front of vehicle 1 as the objects of analysis for each cycle (i.e., each time). On the other hand, the right-side front, right-side rear, rear-side front, left-side rear, and left-side front of vehicle 1 are not set as objects of analysis. Furthermore, for the right-side front and left-side front of vehicle 1, analysis is performed using images from both fisheye cameras each time. By using images from two different fisheye cameras for analysis, stereoscopic distance measurement can be performed, thus enabling more accurate analysis.

[0084] Figure 12In another example of the region determination rule, the regions that become the analysis targets at each operation timing are indicated. The control device 2 repeatedly the states 1200 to 1202 in order. In the state 1200, the imaging range 200 of the standard camera 40, the regions 201L and 201R of the fisheye camera 41, and the regions 203L and 203R of the fisheye camera 43 become the analysis targets. In the state 1201, the imaging range 200 of the standard camera 40, the regions 201L and 201R of the fisheye camera 41, the region 202F of the fisheye camera 42, and the region 204F of the fisheye camera 44 become the analysis targets. In the state 1202, the imaging range 200 of the standard camera 40, the regions 201L and 201R of the fisheye camera 41, the region 202L of the fisheye camera 42, and the region 204R of the fisheye camera 44 become the analysis targets.

[0085] By shifting the states at each operation timing as described above, the control device 2 sets the front face, the right oblique front face, and the left oblique front face of the vehicle 1 as the analysis targets at each cycle (i.e., each time), and sets the right face, the right oblique rear face, the rear face, the left oblique rear face, and the left face of the vehicle 1 as the analysis targets at every three cycles, respectively. In addition, in order to distribute the load to the control device 2 at specific operation timings, the regions that are not the analysis targets are distributed to a plurality of operation timings. Furthermore, for the right oblique front face and the left oblique front face of the vehicle 1, the analysis of both the images using two fisheye cameras is performed every three cycles.

[0086] In any of the above examples, the region determination rule specifies that the left oblique front face and the right oblique front face of the vehicle 1 are the targets of the distortion reduction processing at a higher frequency than the right face, the left face, the right oblique rear face, the left oblique rear face, and the rear face of the vehicle 1. In a scenario in which the vehicle 1 enters a T-shaped intersection or a scenario in which the vehicle 1 temporarily stops and then starts, the possibility that another road user (pedestrian, bicycle, other vehicle) appears from the left oblique front face and the right oblique front face of the vehicle 1 is high. Therefore, by setting the left oblique front face and the right oblique front face of the vehicle 1 as the analysis targets at a high frequency, it is possible to reduce the processing load of the control device 2 and to perform appropriate analysis corresponding to the driving scenario.

[0087] Referring to Figure 13 to Figure 16 The region determination rule for a scenario in which the vehicle 1 drives on a narrow road will be described. The narrow road can also be a driving scenario in which the distance between the vehicle 1 and an obstacle is equal to or less than a threshold value (for example, 50 cm or less). Figure 13 An example of such a scenario is shown in (a) of FIG. 12. Figure 13 An example of such a scenario is shown in (a) of FIG. 12. Figure 13 An example of such a scenario is shown in (b) of FIG. 12. Figure 13 An example of such a scenario is shown in (c) of FIG. 12. Figure 13In the diagram, (d) indicates a scenario where vehicle 1 passes alongside a vehicle that is about to turn right. For example... Figure 13 (a) and Figure 13 As shown in (b), narrow roads are sometimes created based on the shape of the road, such as... Figure 13 (c) and Figure 13 As shown in (d) in the diagram, narrow roads sometimes arise depending on traffic conditions.

[0088] Figure 14 This represents an example of a region-determining rule, where the region becomes the object of analysis at each action timing. Control device 2 sequentially repeats states 1400 to 1406. In state 1400, the shooting range 200 of standard camera 40, region 201F of fisheye camera 41, regions 202L and 202F of fisheye camera 42, and region 204R of fisheye camera 44 become the objects of analysis. In state 1401, the shooting range 200 of standard camera 40, region 201F of fisheye camera 41, regions 202L and 202R of fisheye camera 42, and region 204R of fisheye camera 44 become the objects of analysis. In state 1402, the shooting range 200 of standard camera 40, region 201F of fisheye camera 41, region 202L of fisheye camera 42, region 203L of fisheye camera 43, and region 204R of fisheye camera 44 become the objects of analysis. In state 1403, the shooting range 200 of the standard camera 40, area 201F of the fisheye camera 41, area 202L of the fisheye camera 42, area 203F of the fisheye camera 43, and area 204R of the fisheye camera 44 become the objects of analysis. In state 1404, the shooting range 200 of the standard camera 40, area 201F of the fisheye camera 41, area 202L of the fisheye camera 42, area 203R of the fisheye camera 43, and area 204R of the fisheye camera 44 become the objects of analysis. In state 1405, the shooting range 200 of the standard camera 40, area 201F of the fisheye camera 41, area 202L of the fisheye camera 42, and areas 204R and 204L of the fisheye camera 44 become the objects of analysis. In state 1406, the shooting range 200 of the standard camera 40, the area 201F of the fisheye camera 41, the area 202L of the fisheye camera 42, and the areas 204R and 204F of the fisheye camera 44 become the objects of analysis.

[0089] By shifting the state at each operation timing as described above, the control device 2 sets the front face, the right oblique front, and the left oblique front of the vehicle 1 as the analysis target at each cycle (i.e., each time), sets the right face, the rear face, and the left face of the vehicle 1 as the analysis target at every seven cycles, respectively, and sets the right oblique rear and the left oblique rear of the vehicle 1 as the analysis target at two times out of seven cycles, respectively. In addition, in order to prevent the load from concentrating on the control device 2 at a specific operation timing, the regions not as the analysis target are dispersed to a plurality of operation timings.

[0090] Figure 15 In one example of the region determination rule, the regions that become the analysis target at each operation timing are shown. The control device 2 sequentially repeats the states 1500 to 1502. In the state 1500, the photographing range 200 of the standard camera 40, the region 201F of the fisheye camera 41, the region 202L of the fisheye camera 42, the region 203F of the fisheye camera 43, and the region 204R of the fisheye camera 44 become the analysis target. In the state 1501, the photographing range 200 of the standard camera 40, the regions 202L and 202F of the fisheye camera 42, and the regions 204F and 204R of the fisheye camera 44 become the analysis target. In the state 1502, the photographing range 200 of the standard camera 40, the region 202L of the fisheye camera 42, the regions 203L and 203R of the fisheye camera 43, and the region 204R of the fisheye camera 44 become the analysis target.

[0091] By shifting the state at each operation timing as described above, the control device 2 sets the front face, the right oblique front, and the left oblique front of the vehicle 1 as the analysis target at each cycle (i.e., each time), sets the right face, the right oblique rear, the rear face, the left oblique rear, and the left face of the vehicle 1 as the analysis target at every three cycles, respectively. In addition, in order to prevent the load from concentrating on the control device 2 at a specific operation timing, the regions not as the analysis target are dispersed to a plurality of operation timings.

[0092] Figure 16 The position in the vertical direction of the region that is the analysis target is shown. Figure 16 (a) in FIG. 20 shows the position in the vertical direction of the region 201F of the fisheye camera 41. Figure 16 (b) in FIG. 20 shows the position in the vertical direction of the region 202F of the fisheye camera 42. Figure 16 (c) in FIG. 20 shows the position in the vertical direction of the region 203F of the fisheye camera 43. The positions in the vertical direction of the regions of the fisheye camera 44 can be the same as (b) in FIG. 20, and thus the description is omitted. Figure 16

[0093] In the region determination rule, as shown in Figure 16 ​As shown in (a) of FIG. 20, the angle of the conversion center 301 of the region 201F with respect to the vertical downward direction of the vehicle 1 is defined as θ2. The θ2 is a value smaller than the θ1, and for example, can also be 70 degrees. For the regions 201R and 201L, the angle of the conversion center 301 with respect to the vertical downward direction of the vehicle 1 can also be defined as θ2. As for the regions of the fisheye cameras 42 to 44, the angle of the conversion center 301 with respect to the vertical downward direction of the vehicle 1 is also defined as θ3. The θ3 is a value smaller than the θ2, and for example, can also be 45 degrees. Figure 16

[0094] Thus, in any of the regions of the front straight ahead, the right oblique front, the right straight ahead, the right oblique rear, the rear straight ahead, the left oblique rear, the left straight ahead, and the left oblique front of the vehicle 1, the position in the vertical direction of the region of the analysis target in the case where the vehicle 1 is traveling on a narrow road is on the lower side (for example, the conversion center 301 is downward) compared to the case where the vehicle 1 is traveling outside of a narrow road (for example, the case of the above-described normal scene). In the case where the vehicle 1 is traveling on a narrow road, the wheels of the vehicle 1 can hit a curbstone or fall into a side ditch. By locating the region of the analysis target on the lower side, the analysis accuracy of the situation near the ground is improved. In addition, in the region determination rule in the case where the vehicle 1 is traveling on a narrow road, the region 201F of the front straight ahead of the fisheye camera 41 becomes the analysis target. Thus, as shown in (a) of FIG. 20, it is possible to analyze the vicinity of the front straight ahead of the vehicle 1 that is not included in the imaging range 200 of the standard camera 40. In addition, the situation of the right oblique front of the vehicle 1 is analyzed based on the region 202L of the fisheye camera 42. Thus, it is possible to analyze the region of the vicinity of the front wheel of the vehicle 1. The same is true for the situation of the left oblique front. Figure 9

[0095] In the case where the vehicle 1 is traveling on a narrow road, the side straight ahead of the vehicle is located near an obstacle, and thus the necessity of analyzing a distant region is low, but it is better to analyze a certain degree of a distant region in the front of the vehicle (including the right oblique front and the left oblique front). Thus, in the above-described example, the position in the vertical direction of the region including the right oblique front and the left oblique front of the vehicle 1 is located on the upper side than the position in the vertical direction of the region including the right straight ahead and the left straight ahead of the vehicle 1 (that is, θ2 > θ3).

[0096] In any of the above-described examples, the region determination rule defines the right oblique front and the left oblique front of the vehicle 1 as the analysis target at a higher frequency than the right straight ahead, the left straight ahead, the right oblique rear, the left oblique rear, and the rear straight ahead of the vehicle 1. In the scene where the vehicle 1 is traveling on a narrow road, the possibility of the vehicle 1 contacting an object located in the left oblique front and the right oblique front is high. Thus, by defining the left oblique front and the right oblique front of the vehicle 1 as the analysis target at a high frequency, it is possible to reduce the processing load of the control device 2, and to perform appropriate analysis corresponding to the traveling scene. ​​

[0097] Referring to Figure 16 The area determination rule for the scenario in which the vehicle 1 turns at the intersection will be described. Figure 17 to Figure 20 and Figure 17 An example of such a scenario is shown in Fig. 18. In the example shown in Fig. 18, the vehicle 1 turns left at the intersection. In a left-hand traffic country, a left turn means that the vehicle 1 turns at the intersection in a direction that does not cross the opposite lane. In the example shown in Fig. 18, the vehicle 1 turns left at the intersection. In a left-hand traffic country, a left turn means that the vehicle 1 turns at the intersection in a direction that does not cross the opposite lane. The position of the area of the analysis target in the vertical direction in the scenario in which the vehicle 1 turns at the intersection can also be the same as described in Figure 19 In the example shown in Fig. 18, the vehicle 1 turns right at the intersection. In a left-hand traffic country, a right turn means that the vehicle 1 turns at the intersection in a direction that crosses the opposite lane. The position of the area of the analysis target in the vertical direction in the scenario in which the vehicle 1 turns at the intersection can also be the same as described in Figure 17 Figure 19

[0098] Figure 9 An example of the area determination rule in the scenario in which the vehicle 1 turns left at the intersection is shown in Figs. 18A to 18C. In the example shown in Figs. 18A to 18C, the control device 2 repeatedly transitions the states 1800 to 1802 in order. In the state 1800, the photographing range 200 of the standard camera 40, the area 201L of the fisheye camera 41, the area 202R of the fisheye camera 42, the area 203R of the fisheye camera 43, and the area 204F of the fisheye camera 44 become the analysis targets. In the state 1801, the photographing range 200 of the standard camera 40, the area 201L of the fisheye camera 41, the area 202F of the fisheye camera 42, the areas 203F and 203R of the fisheye camera 43, and the area 204F of the fisheye camera 44 become the analysis targets. In the state 1802, the photographing range 200 of the standard camera 40, the area 201L of the fisheye camera 41, the area 202L of the fisheye camera 42, the areas 203L and 203R of the fisheye camera 43, and the area 204F of the fisheye camera 44 become the analysis targets.

[0099] By transitioning the states as described above at each action timing, the control device 2 sets the front face, the left oblique front face, the left oblique front face, and the left oblique rear face of the vehicle 1 as the analysis targets at each cycle (i.e., each time), and sets the right oblique front face, the right oblique front face, the right oblique rear face, and the rear face of the vehicle 1 as the analysis targets at every three cycles, respectively. In addition, in order to distribute the load on the control device 2 at a specific action timing, the areas that are not the analysis targets are distributed to a plurality of action timings.

[0100] Figure 18 ​​In one example of the region determination rule representing the scenario in which the vehicle 1 makes a right turn at the intersection, the regions that become the analysis targets at each action timing. The control device 2 repeatedly the states 2000 to 2002 in order. In the state 2000, the imaging range 200 of the standard camera 40, the regions 201F and 201R of the fisheye camera 41, the region 203L of the fisheye camera 43, and the region 204F of the fisheye camera 44 become the analysis targets. In the state 2001, the imaging range 200 of the standard camera 40, the regions 201F and 201R of the fisheye camera 41, the region 203F of the fisheye camera 43, and the region 204R of the fisheye camera 44 become the analysis targets. In the state 2002, the imaging range 200 of the standard camera 40, the regions 201F and 201R of the fisheye camera 41, the region 202F of the fisheye camera 42, and the region 203R of the fisheye camera 43 become the analysis targets.

[0101] By shifting the states at each action timing as described above, the control device 2 sets the front face of the vehicle 1 and the right oblique front face as the analysis targets at each cycle (i.e., each time), and sets the right face of the vehicle 1, the right oblique rear face, the rear face of the vehicle 1, the left oblique front face, the left face of the vehicle 1, and the left oblique rear face as the analysis targets at every three cycles, respectively. In addition, in order to distribute the load on the control device 2 at a specific action timing, the regions that are not the analysis targets are distributed to a plurality of action timings.

[0102] In any of the above examples, the region determination rule prescribes the oblique front face of the vehicle 1 in the direction in which the vehicle 1 turns as the object of the distortion reduction processing at a higher frequency than the rear face of the vehicle 1 and the side face (including the oblique front face, the side face, and the oblique rear face) of the vehicle 1 on the side opposite to the direction in which the vehicle 1 turns. In the scenario in which the vehicle 1 turns at the intersection, the possibility that another road user (pedestrian, bicycle, other vehicle) appears from the oblique front face in the direction in which the vehicle 1 turns is high. Therefore, by setting the oblique front face in the direction in which the vehicle 1 turns as the analysis target at a high frequency, it is possible to reduce the processing load on the control device 2 and perform appropriate analysis corresponding to the driving scenario.

[0103] In the above example, in the case where the vehicle 1 turns left, the left diagonal rear of the vehicle 1 is made the object of the distortion mitigation processing at a higher frequency than the side (including the diagonal front, the side front, and the diagonal rear) of the vehicle 1 on the side opposite to the direction in which the vehicle 1 turns. In this way, by making the left diagonal rear of the vehicle 1 the analysis object at a high frequency, the recognition accuracy of the traffic participant who becomes the cause of the right turn accident is improved. The region determination rule prescribes that, in the case where the vehicle 1 turns right, the front face of the vehicle 1 is made the object of the distortion mitigation processing at a higher frequency than the side (including the diagonal front, the side front, and the diagonal rear) of the vehicle 1 on the side opposite to the direction in which the vehicle 1 turns. In this way, by making the front face of the vehicle 1 the analysis object at a high frequency, the recognition accuracy of the oncoming vehicle at the time of the right turn is improved.

[0104] Referring to Figure 20 The region determination rule for the scene in which the vehicle 1 backs up is described. Figure 21 to Figure 23 An example of such a scene is shown. In Figure 21 In the example shown in FIG. 22, the vehicle 1 backs up in order to start from the parking space. The position in the vertical direction of the region of the analysis object in the scene in which the vehicle 1 backs up can also be the same as described in Figure 21 In the example shown in FIG. 22, the vehicle 1 backs up in order to start from the parking space. The position in the vertical direction of the region of the analysis object in the scene in which the vehicle 1 backs up can also be the same as described in

[0105] Figure 16 Regions that become the analysis object at each action timing in one example of the region determination rule are shown. The control device 2 repeatedly the states 2200 to 2204 in order. In the state 2200, the photographing range 200 of the standard camera 40, the regions 203L, 203F, and 203R of the fisheye camera 43, and the region 204F of the fisheye camera 44 become the analysis objects. In the state 2201, the photographing range 200 of the standard camera 40, the region 201L of the fisheye camera 41, the regions 203L, 203F, and 203R of the fisheye camera 43 become the analysis objects. In the state 2202, the photographing range 200 of the standard camera 40, the region 201F of the fisheye camera 41, the regions 203L, 203F, and 203R of the fisheye camera 43 become the analysis objects. In the state 2203, the photographing range 200 of the standard camera 40, the region 201R of the fisheye camera 41, the regions 203L, 203F, and 203R of the fisheye camera 43 become the analysis objects. In the state 2204, the photographing range 200 of the standard camera 40, the region 202F of the fisheye camera 42, the regions 203L, 203F, and 203R of the fisheye camera 43 become the analysis objects.

[0106] By shifting the state at each operation timing as described above, the control device 2 sets the front face, the right oblique rear, the rear face, and the left oblique rear of the vehicle 1 as the analysis targets at each cycle (i.e., each time), and sets the left oblique front, the right oblique front, the right face, the left face, and the left oblique front of the vehicle 1 as the analysis targets at every five cycles, respectively. In addition, in order to disperse the load to the control device 2 at a specific operation timing, the regions not as the analysis targets are dispersed to a plurality of operation timings.

[0107] Figure 22 In one example of the region determination rule, the regions that become the analysis targets at each operation timing are shown. The control device 2 sequentially repeats the states 2300 to 2205. In the state 2300, the photographing range 200 of the standard camera 40, the region 201L of the fisheye camera 41, the regions 203L, 203F, and 203R of the fisheye camera 43 become the analysis targets. In the state 2301, the photographing range 200 of the standard camera 40, the region 201F of the fisheye camera 41, the region 202F of the fisheye camera 42, the region 203F of the fisheye camera 43, and the region 204F of the fisheye camera 44 become the analysis targets. In the state 2302, the photographing range 200 of the standard camera 40, the region 201R of the fisheye camera 41, the regions 203L, 203F, and 203R of the fisheye camera 43 become the analysis targets. In the state 2303, the photographing range 200 of the standard camera 40, the region 201L of the fisheye camera 41, the region 202F of the fisheye camera 42, the region 203F of the fisheye camera 43, and the region 204F of the fisheye camera 44 become the analysis targets. In the state 2304, the photographing range 200 of the standard camera 40, the region 201F of the fisheye camera 41, the regions 203L, 203F, and 203R of the fisheye camera 43 become the analysis targets. In the state 2305, the photographing range 200 of the standard camera 40, the region 201R of the fisheye camera 41, the region 202F of the fisheye camera 42, the region 203F of the fisheye camera 43, and the region 204F of the fisheye camera 44 become the analysis targets.

[0108] By shifting the state at each operation timing as described above, the control device 2 sets the front face and the rear face of the vehicle 1 as the analysis targets at each cycle (i.e., each time), sets the right oblique rear and the left oblique rear of the vehicle 1 as the analysis targets at every two cycles, respectively, and sets the right oblique front, the right face, the left face, and the left oblique front of the vehicle 1 as the analysis targets at every three cycles, respectively. In addition, in order to disperse the load to the control device 2 at a specific operation timing, the regions not as the analysis targets are dispersed to a plurality of operation timings.

[0109] In any of the above examples, the region determination rule prescribes the rear face, the right oblique rear, and the left oblique rear of the vehicle 1 as the objects of the distortion mitigation processing at a higher frequency than the right oblique front and the left oblique front of the vehicle 1. In a scenario in which the vehicle 1 is backing up, the possibility of other road users (pedestrians, bicycles, other vehicles) appearing from the left oblique rear, the rear face, and the right oblique rear of the vehicle 1 is high. Therefore, by prescribing the rear face, the right oblique rear, and the left oblique rear of the vehicle 1 as the analysis targets at a high frequency, it is possible to reduce the processing load of the control device 2 and perform appropriate analysis corresponding to the driving scenario. Further, in the example of Figure 23 the rear face of the vehicle 1 as the object of the distortion mitigation processing at a higher frequency than the right oblique rear and the left oblique rear of the vehicle 1. The front face of the vehicle 1 is always an analysis target of the image from the standard camera 40, but there is no such standard camera in the rear face of the vehicle 1. Therefore, it is also possible to prescribe the image of the rear face taken by the fisheye camera 43 as an analysis target at a high frequency.

[0110] In the above example, the region determination rule prescribes that, in the case where the vehicle 1 is backing up, the position in the vertical direction of the region including the rear face of the vehicle 1 is set to the lower side compared to the case where the vehicle 1 is advancing. Thereby, it is possible to improve the analysis accuracy in the vicinity of the rear face of the vehicle 1, which becomes a blind spot from the driver's perspective.

[0111] With reference to Figure 23 and Figure 24 , the region determination rule for a scenario in which the vehicle 1 changes lanes is described. Figure 25 An example of such a scenario is shown. The lane change can be performed in accordance with a travel plan to a destination, or can be performed in order to avoid an obstacle in front of the vehicle 1 (for example, cut-in of another vehicle, or a pedestrian running out).

[0112] Figure 24 An example of a region that is an analysis target at each action timing in one example of the region determination rule is shown. Figure 25The case where the vehicle 1 moves to the right lane is handled. In the case where the vehicle 1 moves to the left lane, the left and right are reversed. The control device 2 sequentially repeats the states 2500 to 2505. In the state 2500, the photographing range 200 of the standard camera 40, the regions 201L and 201R of the fisheye camera 41, the region 202F of the fisheye camera 42, and the region 204F of the fisheye camera 44 become the analysis targets. In the state 2501, the photographing range 200 of the standard camera 40, the regions 201L and 201R of the fisheye camera 41, and the regions 203L and 203R of the fisheye camera 43 become the analysis targets. In the state 2502, the photographing range 200 of the standard camera 40, the regions 201L and 201R of the fisheye camera 41, the region 202F of the fisheye camera 42, and the region 203F of the fisheye camera 43 become the analysis targets. In the state 2503, the photographing range 200 of the standard camera 40, the regions 201L and 201R of the fisheye camera 41, the region 203L of the fisheye camera 43, and the region 204F of the fisheye camera 44 become the analysis targets. In the state 2504, the photographing range 200 of the standard camera 40, the regions 201L and 201R of the fisheye camera 41, the region 202F of the fisheye camera 42, and the region 203R of the fisheye camera 43 become the analysis targets. In the state 2505, the photographing range 200 of the standard camera 40, the regions 201L and 201R of the fisheye camera 41, and the regions 203L and 203F of the fisheye camera 43 become the analysis targets.

[0113] By shifting the state at each action timing as described above, the control device 2 sets the left oblique front, the front face, and the right oblique front of the vehicle 1 as the analysis targets at each cycle (i.e., each time), sets the right face and the right oblique rear of the vehicle 1 as the analysis targets at every two cycles, respectively, and sets the rear face, the left oblique rear, and the left face of the vehicle 1 as the analysis targets at every three cycles, respectively. In addition, in order to disperse the load at a specific action timing to the control device 2, the regions not as the analysis targets are dispersed to a plurality of action timings.

[0114] In the above example, the region determination rule prescribes that the right slanting front of the vehicle 1 and the left slanting front of the vehicle 1 are made the targets of the distortion alleviation processing with a higher frequency than the front of the moving direction of the vehicle 1 and the slanting rear of the moving direction of the vehicle 1. Further, the region determination rule prescribes that the front of the moving direction of the vehicle 1 and the slanting rear of the moving direction of the vehicle 1 are made the targets of the distortion alleviation processing with a higher frequency than the front of the side opposite to the moving direction of the vehicle 1 and the slanting rear of the side opposite to the moving direction of the vehicle 1. In the case where the vehicle 1 makes a lane change, the front (including the left slanting front, the front normal, and the right slanting front) in the direction of travel of the vehicle is made the analysis target with a high frequency, and the moving direction of the vehicle (for example, the right normal and the right slanting rear if the movement is to the right lane) is made the analysis target with a higher frequency than the opposite side thereof (for example, the left normal and the left slanting rear if the movement is to the right lane). Thereby, it is possible to reduce the processing load of the control device 2, and to perform appropriate analysis corresponding to the driving scene.

[0115] The position in the vertical direction of the region of the analysis target in the scene where the vehicle 1 makes a lane change can also be the same as that described in the above. Figure 25 Figure 9 Instead, the region determination rule can also prescribe that the position in the vertical direction of the region including the slanting front of the side opposite to the moving direction of the vehicle 1 is made lower than the position in the vertical direction of the region including the slanting front of the moving direction of the vehicle 1. For example, if the vehicle 1 is moving to the right lane, the position in the vertical direction of the region 204R based on the fisheye camera 44 can also be made lower than the position in the vertical direction of the region 204L based on the fisheye camera 42. For example, in the case where a lane change is made for emergency avoidance, the possibility that the object that becomes the cause of the emergency avoidance exists in the vicinity of the vehicle 1 is high. Therefore, by making the position in the vertical direction of the region including the slanting front of the side opposite to the moving direction of the vehicle 1 the lower side, it is possible to identify the object with high accuracy.

[0116] A modification example of the above embodiment will be described. The control device 2 decides one or more regions that are the object of the distortion alleviation processing based on the driver's line of sight direction of the vehicle 1 in addition to the rule corresponding to the current driving scene of the vehicle 1. The control device 2 can determine the driver's line of sight direction based on an image captured by a driver monitoring camera (not shown) mounted to the vehicle 1, for example. It is considered that even if the control device 2 does not perform the recognition processing, the driver's line of sight direction will be moved by the driver in accordance with the situation. Therefore, the control device 2 can reduce the frequency of the region of the driver's line of sight direction in the outside of the vehicle 1 as the object of the distortion alleviation processing. For example, in the case where the driver looks at the left oblique rear of the vehicle 1, the control device 2 can reduce the frequency of the region 201L of the fisheye camera 41 or the region 204R of the fisheye camera 44 as the resolution object.

[0117] <Summary of Embodiment>

[0118] <Project 1>

[0119] A control device that is a control device (2) of a moving body (1) having one or more photographing devices (41 to 44),

[0120] The control device includes:

[0121] an image acquisition unit that acquires an image (300) of the outside of the moving body from the one or more photographing devices;

[0122] a correction unit that performs distortion alleviation processing for alleviating distortion of an image for each of one or more regions (302) included in the image acquired from the one or more photographing devices; and

[0123] a recognition unit that recognizes the outside of the moving body based on the image (303) after the distortion alleviation processing,

[0124] The correction unit decides the one or more regions that are the object of the distortion alleviation processing in accordance with a predetermined rule corresponding to a moving scene of the moving body.

[0125] According to this project, the outside of the moving body can be appropriately recognized in accordance with the moving scene.

[0126] <Project 2>

[0127] The control device according to Project 1,

[0128] The correction unit performs the distortion alleviation processing for each predetermined period,

[0129] The predetermined rule is a rule that specifies a region that is an object of the distortion alleviation processing for each of the predetermined periods.

[0130] According to this item, it is possible to analyze a region that should be focused on at a specified frequency.

[0131] Item 3

[0132] The control device according to any one of items 1 or 2,

[0133] The predetermined rule specifies the following:

[0134] a position in a horizontal direction of a region that is an object of the distortion alleviation processing; and

[0135] a timing at which a region at the position is made an object of the distortion alleviation processing.

[0136] According to this item, it is possible to analyze a position that should be focused on in a horizontal direction at a specified frequency.

[0137] Item 4

[0138] The control device according to any one of items 1 to 3, the predetermined rule specifies a position in a vertical direction of a region that is an object of the distortion alleviation processing.

[0139] According to this item, it is possible to analyze a position that should be focused on in a vertical direction with emphasis.

[0140] Item 5

[0141] The control device according to any one of items 1 to 4,

[0142] The moving body has a plurality of photographing devices,

[0143] The image acquisition unit acquires images of the outside of the moving body from the plurality of photographing devices, respectively,

[0144] The predetermined rule specifies separate rules for the plurality of photographing devices, respectively.

[0145] According to this item, it is possible to select and analyze separate regions included in a photographing range of a plurality of photographing devices with emphasis.

[0146] Item 6

[0147] The control device according to item 5,

[0148] The plurality of photographing devices include:

[0149] a first photographing device (41) that photographs a front face, a right oblique front, and a left oblique front of the moving body;

[0150] a second photographing device (43) that photographs a right oblique rear side, a left oblique rear side, and a rear side of the moving body;

[0151] a third photographing device (42) that photographs a right side, a right oblique front side, and a right oblique rear side of the moving body; and

[0152] a fourth photographing device (44) that photographs a left side, a left oblique front side, and a left oblique rear side of the moving body.

[0153] According to this item, it is possible to analyze the surroundings of the moving body in all directions.

[0154] <ITEM 7>

[0155] According to the control device described in any one of items 1 to 6, the moving body is a vehicle (1), and the moving scene is a traveling scene of the vehicle.

[0156] According to this item, it is possible to analyze directions that should be focused on when the vehicle is traveling.

[0157] <ITEM 8>

[0158] According to the control device described in item 7,

[0159] in a case where the vehicle enters a T-shaped intersection or in a case where the vehicle temporarily stops and then starts,

[0160] the predetermined rule prescribes that the left oblique front side and the right oblique front side of the vehicle are made targets of the distortion alleviation processing with higher frequency than the right side, the left side, the right oblique rear side, the left oblique rear side, and the rear side of the vehicle.

[0161] According to this item, it is possible to analyze directions that should be focused on when the vehicle enters a T-shaped intersection or when the vehicle temporarily stops and then starts.

[0162] <ITEM 9>

[0163] According to the control device described in item 7 or 8,

[0164] in a case where the vehicle enters a T-shaped intersection or in a case where the vehicle temporarily stops and then starts,

[0165] the predetermined rule prescribes that two regions including the oblique front sides of the vehicle in two images acquired from two photographing devices are made targets of the distortion alleviation processing at the same timing.

[0166] According to this item, it is possible to analyze directions that should be focused on when the vehicle enters a T-shaped intersection or when the vehicle temporarily stops and then starts.

[0167] <PROJECT 10>

[0168] The control device according to any one of <PROJECTS> 7 to 9,

[0169] In a case where the vehicle is traveling on a narrow road,

[0170] The predetermined rule prescribes the right slanting front and the left slanting front of the vehicle as the objects of the distortion alleviation processing with a higher frequency than the right front face, the left front face, the right slanting rear, the left slanting rear, and the rear face of the vehicle.

[0171] According to this project, it is possible to analyze the directions that should be focused on when traveling on a narrow road with emphasis.

[0172] <PROJECT 11>

[0173] The control device according to any one of <PROJECTS> 7 to 10,

[0174] In a case where the vehicle is traveling on a narrow road,

[0175] The predetermined rule prescribes the right slanting front and the left slanting front of the vehicle as the objects of the distortion alleviation processing with a higher frequency than the right front face, the left front face, the right slanting rear, the left slanting rear, and the rear face of the vehicle.

[0176] According to this project, it is possible to analyze the directions that should be focused on when traveling on a narrow road with emphasis.

[0177] <PROJECT 12>

[0178] The control device according to any one of <PROJECTS> 7 to 11,

[0179] In a case where the vehicle is traveling on a narrow road, compared with a case where the vehicle is traveling on another road that is not a narrow road,

[0180] The predetermined rule prescribes a position in a vertical direction of a region including the right front face, the left front face, the right slanting rear, the left slanting rear, and the rear face of the vehicle as a lower side.

[0181] According to this project, it is possible to analyze the directions that should be focused on when traveling on a narrow road with emphasis.

[0182] <PROJECT 13>

[0183] The control device according to any one of <PROJECTS> 7 to 11,

[0184] In a case where the vehicle is traveling on a narrow road, compared with a case where the vehicle is traveling on another road that is not a narrow road,

[0185] The predetermined rule specifies a position in the vertical direction of a region including the front, right oblique front, and left oblique front of the vehicle as a lower side.

[0186] According to this item, the direction to be focused on when driving on a narrow road can be analyzed with emphasis.

[0187] Item 14

[0188] The control device according to any one of items 7 to 13,

[0189] In a case where the vehicle is driving on a narrow road,

[0190] The predetermined rule specifies a position in the vertical direction of a region including the right oblique front and left oblique front of the vehicle as a higher side than a position in the vertical direction of a region including the right front and left front of the vehicle.

[0191] According to this item, the direction to be focused on when driving on a narrow road can be analyzed with emphasis.

[0192] Item 15

[0193] The control device according to any one of items 7 to 14,

[0194] In a case where the vehicle is turning at an intersection,

[0195] The predetermined rule specifies the oblique front of the vehicle in the direction in which the vehicle is turning as a target of the distortion alleviation processing with a higher frequency than the rear front of the vehicle and the oblique rear of the vehicle on the side opposite to the direction in which the vehicle is turning.

[0196] According to this item, the direction to be focused on when turning at an intersection can be analyzed with emphasis.

[0197] Item 16

[0198] The control device according to item 15,

[0199] In a case where the vehicle is turning in a direction crossing an opposite lane at an intersection,

[0200] The predetermined rule specifies the front of the vehicle as a target of the distortion alleviation processing with a higher frequency than the rear front of the vehicle and the oblique rear of the vehicle on the side opposite to the direction in which the vehicle is turning.

[0201] According to this item, the direction to be focused on when turning at an intersection can be analyzed with emphasis.

[0202] Item 17

[0203] The control device according to any one of items 15 to 20,

[0204] In a case where the vehicle turns at an intersection in a direction that does not cross an opposite lane,

[0205] The predetermined rule specifies the oblique rear of the vehicle in the direction in which the vehicle turns as a target of the distortion alleviation processing with a higher frequency than the rear face of the vehicle and the oblique rear of the vehicle on a side opposite to the direction in which the vehicle turns.

[0206] According to this item, the direction that should be focused on when turning at an intersection can be analyzed with emphasis.

[0207] <ITEM 18>

[0208] The control device according to any one of items 7 to 17,

[0209] In a case where the vehicle backs up,

[0210] The predetermined rule specifies the rear face of the vehicle, the right oblique rear of the vehicle, and the left oblique rear of the vehicle as targets of the distortion alleviation processing with a higher frequency than the right oblique front of the vehicle and the left oblique front of the vehicle.

[0211] According to this item, the direction that should be focused on when backing up can be analyzed with emphasis.

[0212] <ITEM 19>

[0213] The control device according to item 18,

[0214] In a case where the vehicle backs up,

[0215] The predetermined rule specifies the rear face of the vehicle as a target of the distortion alleviation processing with a higher frequency than the right oblique rear of the vehicle and the left oblique rear of the vehicle.

[0216] According to this item, the direction that should be focused on when backing up can be analyzed with emphasis.

[0217] <ITEM 20>

[0218] The control device according to any one of items 7 to 19,

[0219] In a case where the vehicle backs up, compared to a case where the vehicle advances,

[0220] The predetermined rule specifies a position in a vertical direction of a region including the rear face of the vehicle as a lower side.

[0221] According to this item, the direction that should be focused on when backing up can be analyzed with emphasis.

[0222] <PROJECT 21>

[0223] The control device according to any one of <PROJECTS> 7 to 20,

[0224] In a case where the vehicle is making a lane change,

[0225] The predetermined rule prescribes the right and left oblique front of the vehicle as the object of the distortion alleviation processing with a higher frequency than the front in the moving direction of the vehicle among the right and left and the oblique rear in the moving direction of the vehicle among the right and left,

[0226] The predetermined rule prescribes the front in the moving direction of the vehicle among the right and left and the oblique rear in the moving direction of the vehicle among the right and left as the object of the distortion alleviation processing with a higher frequency than the front of the side opposite to the moving direction of the vehicle among the right and left and the oblique rear of the side opposite to the moving direction of the vehicle among the right and left.

[0227] According to this project, the direction to be focused on at the time of the lane change can be analyzed with emphasis.

[0228] <PROJECT 22>

[0229] The control device according to any one of <PROJECTS> 7 to 21,

[0230] In a case where the vehicle is making a lane change,

[0231] The predetermined rule prescribes the position in the vertical direction of the region including the oblique front of the side opposite to the moving direction of the vehicle among the right and left as lower than the position in the vertical direction of the region including the oblique front of the moving direction of the vehicle among the right and left.

[0232] According to this project, the direction to be focused on at the time of the lane change can be analyzed with emphasis.

[0233] <PROJECT 23>

[0234] The control device according to any one of <PROJECTS> 1 to 22, the one or more photographing devices are photographing devices on which a fisheye lens is mounted.

[0235] According to this project, the photographing range of the photographing device can be expanded.

[0236] <PROJECT 24>

[0237] The control device according to any one of <PROJECTS> 1 to 23, the moving body further has another photographing device (40) that photographs an image less distorted than the one or more photographing devices,

[0238] The image acquisition unit acquires an image of the outside of the moving body from the other photographing device,

[0239] The recognition unit recognizes the outside of the moving body based on the image from the other photographing device.

[0240] According to this item, the outside can be recognized using multiple types of photographing devices.

[0241] < Item 25 >

[0242] A vehicle provided with the control device described in any one of items 1 to 24.

[0243] According to this item, the above-described effects are obtained in the form of a vehicle.

[0244] < Item 26 >

[0245] A program that causes a computer to function as each unit of the control device described in any one of items 1 to 24.

[0246] According to this item, the above-described effects are obtained in the form of a program.

[0247] < Item 27 >

[0248] A control method of a moving body (1) having one or more photographing devices (41 to 44),

[0249] The control method includes:

[0250] An image acquisition step in which an image (300) of the outside of the moving body is acquired from the one or more photographing devices;

[0251] A correction step in which one or more regions (302) included in the image acquired from the one or more photographing devices are respectively subjected to distortion reduction processing for reducing distortion of the image; and

[0252] A recognition step in which the outside of the moving body is recognized based on the image (303) after the distortion reduction processing,

[0253] In the correction step, the one or more regions that are the objects of the distortion reduction processing are determined in accordance with a predetermined rule corresponding to a moving scenario of the moving body.

[0254] According to this item, the outside of the moving body can be appropriately recognized in accordance with the moving scenario.

[0255] The present application is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the gist of the present application.

Claims

1. A control device for a moving body having one or more imaging devices. Its features are, The control device includes: An image acquisition unit acquires images of the external environment of the moving object from one or more imaging devices; The correction unit performs distortion reduction processing on one or more regions contained in the images acquired from the one or more imaging devices to reduce image distortion. as well as The recognition unit identifies the external environment of the moving object based on the image after distortion reduction processing. The correction unit determines one or more regions as targets for distortion reduction processing according to predetermined rules corresponding to the movement scenario of the moving body. The predetermined rules apply in specific mobile scenarios. The distortion reduction process targets a first region contained in images acquired from the one or more imaging devices at a first frequency. The distortion reduction process targets a second region contained in an image acquired from one or more imaging devices at a second frequency higher than the first frequency.

2. The control device according to claim 1, characterized in that, The correction unit performs the distortion reduction process for each predetermined period. The predetermined rules are rules that define the regions that are the objects of the distortion reduction processing for each predetermined period.

3. The control device according to claim 1, characterized in that, The predetermined rules stipulate the following: The horizontal position of the region to which the distortion reduction process is applied; and The timing for using the region at the aforementioned location as the target of the distortion reduction process.

4. The control device according to claim 1, characterized in that, The predetermined rules define the vertical position of the region that is the object of the distortion reduction process.

5. The control device according to claim 1, characterized in that, The moving body has multiple shooting devices. The image acquisition unit acquires images of the external environment of the moving object from the plurality of shooting devices respectively. The predetermined rules specify separate rules for each of the multiple shooting devices.

6. The control device according to claim 5, characterized in that, The plurality of imaging devices include: The first imaging device captures images of the frontal, right-southeast, and left-southeast views of the moving body. The second shooting device captures images of the rear front, right rear, and left rear of the moving object. The third imaging device captures images of the right front, right diagonal front, and right diagonal rear of the moving object; and The fourth imaging device captures images of the left front, left diagonal front, and left diagonal rear of the moving object.

7. The control device according to claim 1, characterized in that, The moving object is a vehicle, and the moving scene is the driving scene of the vehicle.

8. The control device according to claim 7, characterized in that, When the vehicle enters a T-junction or when the vehicle starts moving again after a temporary stop. The predetermined rule stipulates that the left front and right front of the vehicle are targeted for distortion reduction processing at a higher frequency than the right front, left front, right rear, left rear, and rear front of the vehicle.

9. The control device according to claim 7, characterized in that, When the vehicle enters a T-junction or when the vehicle starts moving again after a temporary stop. The predetermined rules stipulate that two areas containing the oblique front of the vehicle in two images acquired by the two shooting devices will be used as the objects of the distortion reduction processing at the same time.

10. The control device according to claim 7, characterized in that, When the vehicle is traveling on a narrow road, The predetermined rule stipulates that the left front and right front of the vehicle are targeted for distortion reduction processing at a higher frequency than the right front, left front, right rear, left rear, and rear front of the vehicle.

11. The control device according to claim 7, characterized in that, When the vehicle is traveling on a narrow road, The predetermined rule stipulates that the right front and left front of the vehicle are targeted for distortion reduction processing at a higher frequency than the right front, left front, right rear, left rear, and rear front of the vehicle.

12. The control device according to claim 7, characterized in that, When the vehicle is traveling on a narrow road, compared to when the vehicle is traveling on other roads that are not narrow... The predetermined rule defines the vertical position of the area including the right front, left front, right rear, left rear, and rear front of the vehicle as the lower side.

13. The control device according to claim 7, characterized in that, When the vehicle is traveling on a narrow road, compared to when the vehicle is traveling on other roads that are not narrow... The predetermined rule defines the vertical position of the area including the front, right front, and left front of the vehicle as the lower side.

14. The control device according to claim 7, characterized in that, When the vehicle is traveling on a narrow road, The predetermined rule specifies that the vertical position of the area containing the right diagonal front and left diagonal front of the vehicle is higher than the vertical position of the area containing the right front and left front of the vehicle.

15. The control device according to claim 7, characterized in that, When the vehicle is turning at an intersection, The predetermined rule stipulates that the oblique front of the vehicle in the direction of the vehicle's turn should be the target of the distortion reduction process at a higher frequency than the rear front of the vehicle and the oblique rear of the vehicle on the side opposite to the direction of the vehicle's turn.

16. The control device according to claim 15, characterized in that, When the vehicle turns at an intersection in a direction intersecting with the oncoming lane. The predetermined rule stipulates that the front of the vehicle is the target of the distortion reduction process at a higher frequency than the rear front of the vehicle and the diagonal rear of the vehicle on the side opposite to the direction the vehicle is turning.

17. The control device according to claim 15, characterized in that, When the vehicle turns at an intersection in a direction that does not intersect with the oncoming lane. The predetermined rule specifies that the oblique rear of the vehicle in the direction of the vehicle's turn should be the target of the distortion reduction process at a higher frequency than the frontal rear of the vehicle and the oblique rear of the vehicle on the side opposite to the direction of the vehicle's turn.

18. The control device according to claim 7, characterized in that, When the vehicle is reversing, The predetermined rule stipulates that the rear front, right rear, and left rear of the vehicle are targeted for distortion reduction processing at a higher frequency than the right front and left rear of the vehicle.

19. The control device according to claim 18, characterized in that, When the vehicle is reversing, The predetermined rule treats the rear front of the vehicle as the target of the distortion reduction process at a higher frequency than the right and left rear sides of the vehicle.

20. The control device according to claim 7, characterized in that, When the vehicle is reversing, compared to when the vehicle is moving forward, The predetermined rule defines the vertical position of the area containing the rear front of the vehicle as the lower side.

21. The control device according to claim 7, characterized in that, When the vehicle changes lanes. The predetermined rule stipulates that the right front and left front of the vehicle in the direction of movement of the vehicle (either from the front or from the rear) should be the targets of the distortion reduction processing at a higher frequency than the frontal view and the rearward view in the direction of movement of the vehicle (either from the front or rear). The predetermined rule stipulates that the front view of the vehicle in the direction of movement in the left and right directions and the oblique rear view of the vehicle in the direction of movement in the left and right directions are considered as objects of the distortion reduction processing at a higher frequency than the front view of the side opposite to the direction of movement of the vehicle in the left and right directions and the oblique rear view of the side opposite to the direction of movement of the vehicle in the left and right directions.

22. The control device according to claim 7, characterized in that, When the vehicle changes lanes. The predetermined rule defines the vertical position of the area diagonally in front of the side opposite to the vehicle's direction of movement, which includes the left and right sides, as lower than the vertical position of the area diagonally in front of the vehicle's direction of movement, which includes the left and right sides.

23. The control device according to claim 1, characterized in that, The one or more shooting devices are all shooting devices equipped with fisheye lenses.

24. The control device according to claim 1, characterized in that, The moving body also has other imaging devices that capture images with less distortion than the one or more imaging devices mentioned above. The image acquisition unit acquires images of the external environment of the moving object from the other imaging devices. The recognition unit also identifies the external environment of the moving object based on images from the other imaging devices.

25. A vehicle comprising the control device according to any one of claims 1 to 24.

26. A storage medium storing a program for enabling a computer to function as a unit of the control device according to any one of claims 1 to 24.

27. A control method for a moving body having one or more imaging devices, characterized in that, The control method includes: The image acquisition step involves acquiring images of the external environment of the moving object from one or more imaging devices. The correction step involves performing distortion reduction processing on one or more regions contained in the images acquired from the one or more imaging devices to reduce image distortion; and In the identification step, the external environment of the moving object is identified based on the image after the distortion reduction processing. In the correction step, one or more regions are determined as the objects of the distortion reduction processing according to predetermined rules corresponding to the movement scene of the moving body. The predetermined rules apply in specific mobile scenarios. The distortion reduction process targets a first region contained in images acquired from the one or more imaging devices at a first frequency. The distortion reduction process targets a second region contained in an image acquired from one or more imaging devices at a second frequency higher than the first frequency.

Images