Image processing device, method for controlling image processing device, vehicle, program, and storage medium

Abstract

This image processing device processes a captured image acquired by an imaging unit provided on a side mirror of a vehicle, and is provided with an image conversion unit that converts the captured image on the basis of the state of the side mirror. When the captured image is acquired in a first state in which the side mirror is folded, the image conversion unit converts the captured image such that the captured image faces the same direction as when the captured image is acquired in a second state in which the side mirror is not folded.

Description

Image processing device, control method for image processing device, vehicle, program, and storage medium

[0001] The present invention relates to an image processing device, a control method for an image processing device, a vehicle, a program, and a storage medium.

[0002] Patent Document 1 discloses a technology for combining image data acquired by an imaging camera installed on the side mirror of a vehicle to capture images of the area behind the vehicle with image data acquired by another imaging camera and displaying the combined image data.

[0003] JP 2010-287163 A

[0004] However, Patent Document 1 does not take into consideration the state in which the side mirrors are folded. For example, when driving forward on narrow roads or parking in narrow parking lots, there is a need to be able to drive at low speeds while checking the image of the surroundings with the side mirrors folded. The technology described in Patent Document 1 has a problem in that the orientation of the image changes when the side mirrors are folded, making it difficult for the user to properly grasp the surrounding environment.

[0005] The present invention has been made in view of the above-mentioned problems, and aims to provide a technique that enables a user to appropriately grasp the surrounding environment.

[0006] An image processing device according to one aspect of the present invention that achieves the above-mentioned object is an image processing device that processes a captured image acquired by an image capturing unit provided on a side mirror of a vehicle, and is equipped with an image conversion unit that converts the captured image based on the state of the side mirror, and is characterized in that when the captured image is acquired in a first state in which the side mirror is folded, the image conversion unit converts the captured image so that it is oriented in the same direction as when the captured image is acquired in a second state in which the side mirror is not folded.

[0007] According to the present invention, it becomes possible for a user to appropriately grasp the surrounding environment.

[0008] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0009] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention, and together with the description are used to explain the principles of the present invention.

[0023] FIG. 1 is a block diagram of a vehicle according to one embodiment;

[0024] FIG. 2 is a diagram showing the horizontal imaging range of each fisheye camera according to one embodiment;

[0025] FIG. 3 is a diagram showing the vertical imaging range of the left-side fisheye camera according to one embodiment;

[0026] FIG. 4 is a diagram showing the vertical imaging range of the rear fisheye camera according to one embodiment;

[0027] FIG. 5 is a flowchart showing the procedure of processing performed by an image processing device according to one embodiment;

[0028] FIG. 6 is an explanatory diagram of processing of an image captured when the side mirrors are folded according to one embodiment;

[0029] FIG. 7 is an explanatory diagram of processing of an image captured when the side mirrors are not folded according to one embodiment;

[0030] FIG. 8 is a diagram showing an example of superimposing a virtual line according to one embodiment.

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0011] <Configuration> Fig. 1 is a block diagram of a vehicle 1 according to one embodiment. Fig. 1 shows an outline of the vehicle 1 in a plan view and a side view. As an example, the vehicle 1 is a four-wheeled sedan-type passenger car. The vehicle 1 may be such a four-wheeled vehicle, or may be a two-wheeled vehicle or another type of vehicle.

[0012] The vehicle 1 includes a vehicle control device 2 (hereinafter simply referred to as the control device 2) that controls the vehicle 1. The control device 2 includes multiple ECUs (Electronic Control Units) 20-29 that are communicatively connected via an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit), a memory such as a semiconductor memory, an interface with external devices, etc. The memory stores programs executed by the processor and data used by the processor for processing. Each ECU may include multiple processors, memories, interfaces, etc. For example, the ECU 20 includes one or more processors 20a and one or more memories 20b. The processor 20a executes instructions including a program stored in the memory 20b, thereby performing processing by the ECU 20. Alternatively, the ECU 20 may include a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for performing processing by the ECU 20. The same applies to the other ECUs.

[0013] The following describes the functions and the like that are handled by each of the ECUs 20 to 29. The number of ECUs and the functions that they are responsible for can be designed as appropriate, and they can be subdivided or integrated more than in this embodiment.

[0014] The ECU 20 executes control related to the automatic driving of the vehicle 1. In the automatic driving, the ECU 20 automatically controls at least one of the steering and the acceleration / deceleration of the vehicle 1. The automatic driving by the ECU 20 may include automatic driving that does not require driving operation by the driver (also called automatic driving) and automatic driving that assists driving operation by the driver (also called driving assistance).

[0015] The ECU 21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism for steering the front wheels in response to a driver's operation (steering operation) of the steering wheel 31. The electric power steering device 3 also includes a motor that generates driving force for assisting the steering operation and automatically steering the front wheels, a sensor that detects the steering angle, etc. When the vehicle 1 is in an autonomous driving state, the ECU 21 automatically controls the electric power steering device 3 in response to an instruction from the ECU 20, and controls the traveling direction of the vehicle 1.

[0016] The ECUs 22 and 23 control the detection units that detect the vehicle's surroundings and process information on the detection results. The vehicle 1 includes one standard camera 40 and four fisheye cameras 41 to 44 as detection units that detect the vehicle's surroundings. The standard camera 40 and the fisheye cameras 42 and 44 are connected to the ECU 22. The fisheye cameras 41 and 43 are connected to the ECU 23. The ECUs 22 and 23 can extract the contours of targets and lane markings (such as white lines) on the road by analyzing images captured by the standard camera 40 and the fisheye cameras 41 to 44. In other words, the ECUs 22 and 23 can function as image processing devices.

[0017] The fisheye cameras 41 to 44 are cameras equipped with fisheye lenses. The configuration of the fisheye camera 41 will be described below. The other fisheye cameras 42 to 44 may have a similar configuration. The angle of view of the fisheye camera 41 is wider than that of the standard camera 40. Therefore, the fisheye camera 41 can capture a wider range than the standard camera 40. Images captured by the fisheye camera 41 have greater distortion than images captured by the standard camera 40. Therefore, the ECU 23 may perform a conversion process (hereinafter referred to as "distortion correction process") to reduce distortion on the images captured by the fisheye camera 41 before analyzing the images. On the other hand, the ECU 22 may not perform distortion correction on the images captured by the standard camera 40 before analyzing them. In this way, the standard camera 40 is an imaging device that captures images that are not subject to distortion correction process, while the fisheye camera 41 is an imaging device that captures images that are subject to distortion correction process. Instead of the standard camera 40, another imaging device that captures images that are not subject to distortion correction processing, such as a camera equipped with a wide-angle lens or a telephoto lens, may be used.

[0018] The standard camera 40 is attached to the center of the front of the vehicle 1 and captures the surroundings in front of the vehicle 1. The fisheye camera 41 is attached to the center of the front of the vehicle 1 and captures the surroundings in front of the vehicle 1. In FIG. 1 , the standard camera 40 and the fisheye camera 41 are shown aligned horizontally. However, the arrangement of the standard camera 40 and the fisheye camera 41 is not limited to this; for example, they may be aligned vertically. Furthermore, at least one of the standard camera 40 and the fisheye camera 41 may be attached to the front of the roof of the vehicle 1 (e.g., on the passenger compartment inside of the windshield). The fisheye camera 42 is attached to the right side mirror of the vehicle 1 and captures the surroundings to the lower right of the vehicle 1. The fisheye camera 43 is attached to the center of the rear of the vehicle 1 and captures the surroundings behind the vehicle 1. The fisheye camera 44 is attached to the left side mirror of the vehicle 1 and captures the surroundings to the lower left of the vehicle 1.

[0019] The type, number, and mounting positions of the cameras of the vehicle 1 are not limited to the above example. The vehicle 1 may also include a light detection and ranging (lidar) or millimeter wave radar as a detection unit for detecting targets around the vehicle 1 and measuring the distance to the targets.

[0020] The ECU 22 controls the standard camera 40 and the fisheye cameras 42 and 44 and processes information on the detection results. The ECU 23 controls the fisheye cameras 41 and 43 and processes information on the detection results. By dividing the detection units that detect the vehicle's surroundings into two systems, the reliability of the detection results can be improved. The image processing device according to this embodiment mainly corresponds to the ECU 22, but the ECUs 22 and 23 may be integrated into a single ECU, which is configured as the image processing device.

[0021] The ECU 24 controls the gyro sensor 5, the GPS sensor 24b, and the communication device 24c, and processes information on the detection results or communication results. The gyro sensor 5 detects the rotational motion of the vehicle 1. The path of the vehicle 1 can be determined based on the detection results of the gyro sensor 5, the wheel speed, etc. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c wirelessly communicates with a server that provides map information and traffic information to acquire this information. The ECU 24 can access a database 24a of map information built in memory, and the ECU 24 performs tasks such as searching for a route from the current location to a destination. The ECU 24, the map database 24a, and the GPS sensor 24b constitute a so-called navigation device.

[0022] The ECU 25 includes a communication device 25a for vehicle-to-vehicle communication. The communication device 25a performs wireless communication with other vehicles in the vicinity, and exchanges information between the vehicles.

[0023] The ECU 26 controls the power plant 6. The power plant 6 is a mechanism that outputs driving force to rotate the drive wheels of the vehicle 1 and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine in response to the driver's driving operation (accelerator operation or acceleration operation) detected by an operation detection sensor 7a provided on the accelerator pedal 7A, for example, and switches the gear position of the transmission based on information such as the vehicle speed detected by a vehicle speed sensor 7c. When the vehicle 1 is in an autonomous driving state, the ECU 26 automatically controls the power plant 6 in response to instructions from the ECU 20, and controls the acceleration and deceleration of the vehicle 1.

[0024] The ECU 27 controls lighting devices (headlights, taillights, etc.) including turn signals 8. In the example of FIG. 1, the turn signals 8 are provided at the front, door mirrors, and rear of the vehicle 1.

[0025] The ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and receives information input from the driver. The audio output device 91 notifies the driver of information by audio. The display device 92 notifies the driver of information by displaying an image. The display device 92 is, for example, disposed on the surface of the driver's seat and constitutes an instrument panel or the like. Note that, while audio and display are exemplified here, information may be notified by vibration or light. Information may also be notified by a combination of audio, display, vibration, or light. Furthermore, the combination or the notification mode may be varied depending on the level of the information to be notified (e.g., urgency). The input device 93 is a group of switches disposed in a position operable by the driver and used to issue instructions to the vehicle 1, but may also include an audio input device.

[0026] The ECU 29 controls the brake device 10 and a parking brake (not shown). The brake device 10 is, for example, a disc brake device provided on each wheel of the vehicle 1. It applies resistance to the rotation of the wheel to decelerate or stop the vehicle 1. The ECU 29 controls the operation of the brake device 10 in response to a driving operation (brake operation) by the driver detected, for example, by an operation detection sensor 7b provided on the brake pedal 7B. When the vehicle 1 is in an autonomous driving state, the ECU 29 automatically controls the brake device 10 in response to instructions from the ECU 20, and controls the deceleration and stopping of the vehicle 1. The brake device 10 and the parking brake can also be activated to maintain the vehicle 1 in a stopped state. Furthermore, if the transmission of the power plant 6 is equipped with a parking lock mechanism, this can also be activated to maintain the vehicle 1 in a stopped state.

[0027] 2A to 2C, the shooting ranges of the standard camera 40 and the fisheye cameras 41 to 44 will be described. Fig. 2A shows the horizontal shooting range of each camera, Fig. 2B shows the vertical shooting range of the fisheye camera 44 attached to the left side mirror of the vehicle 1, and Fig. 2C shows the vertical shooting range of the fisheye camera 43 attached to the rear of the vehicle 1.

[0028] First, the imaging range in a plan view of the vehicle 1 (i.e., the horizontal direction of the vehicle 1) will be described with reference to Figure 2A. The standard camera 40 captures the scenery included in the imaging range 200. The imaging center 200C of the standard camera 40 faces directly ahead of the vehicle 1. The horizontal angle of view of the standard camera 40 may be less than 90°, and may be, for example, approximately 45° or 30°.

[0029] Fisheye camera 41 captures the scenery included in shooting range 201. The shooting center 201C of fisheye camera 41 faces directly ahead of vehicle 1. Fisheye camera 42 captures the scenery included in shooting range 202. The shooting center 202C of fisheye camera 42 faces to the lower right of vehicle 1. Fisheye camera 43 captures the scenery included in shooting range 203. The shooting center 203C of fisheye camera 43 faces directly behind vehicle 1. Fisheye camera 44 captures the scenery included in shooting range 204. The shooting center 204C of fisheye camera 44 faces to the lower left of vehicle 1. The horizontal angle of view of fisheye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, or greater than 180°, or may be, for example, approximately 180°. FIG. 2A shows an example in which the horizontal angle of view of the fisheye cameras 41 to 44 is 180°.

[0030] The shooting range 201 can be divided into an area 201L diagonally forward and to the left of the vehicle 1, an area 201F directly in front of the vehicle 1, and an area 201R diagonally forward and to the right of the vehicle 1. The shooting range 202 can be divided into an area 202L diagonally forward and to the right of the vehicle 1, an area 202F directly to the right of the vehicle 1, and an area 202R diagonally rear and to the right of the vehicle 1. The shooting range 203 can be divided into an area 203L diagonally rear and to the right of the vehicle 1, an area 203F directly behind the vehicle 1, and an area 203R diagonally rear and to the left of the vehicle 1. The shooting range 204 can be divided into an area 204L diagonally rear and to the right of the vehicle 1, an area 204F directly to the left of the vehicle 1, and an area 204R diagonally forward and to the left of the vehicle 1. The imaging range 201 may be divided equally into three areas 201L, 201F, and 201R (i.e., so that the angle of view of each area is equal). The other imaging ranges 202 to 204 may also be divided equally into three areas.

[0031] Since the standard camera 40 and the fisheye cameras 41 to 44 have the above-described shooting ranges 200 to 204, the area directly ahead of the vehicle 1 and the four oblique directions are included in the shooting ranges of the two separate cameras. Specifically, the area directly ahead of the vehicle 1 is included in both the shooting range 200 of the standard camera 40 and the area 201F of the shooting range 201 of the fisheye camera 41. The area diagonally forward to the right of the vehicle 1 is included in both the area 201R of the shooting range 201 of the fisheye camera 41 and the area 202L of the shooting range 202 of the fisheye camera 42. The same applies to the other three oblique directions of the vehicle 1.

[0032] Next, with reference to Figures 2B and 2C, the vertical imaging range of vehicle 1 will be described. Figure 2B describes the vertical imaging range of fisheye camera 44, and Figure 2C describes the vertical imaging range of fisheye camera 43. The vertical imaging ranges of the other fisheye cameras 41 and 42 may be similar.

[0033] The vertical angle of view of the fisheye cameras 41 to 44 may be, for example, greater than 90°, greater than 150°, or greater than 180°, and may be, for example, approximately 180°. FIGS. 2B and 2C show an example in which the vertical angle of view of the fisheye cameras 41 to 44 is 180°. In the example shown, the imaging center 203C of the fisheye camera 43 faces downward (toward the ground) from a direction parallel to the ground. Alternatively, the imaging center 203C of the fisheye camera 43 may face parallel to the ground or upward (away from the ground) from a direction parallel to the ground. Furthermore, the imaging center 204C of the fisheye camera 44 faces downward (toward the ground) from a direction parallel to the ground. In other words, the imaging center 204C of the fisheye camera 44 faces downward and to the left of the vehicle 1. The imaging center 204C of the fish-eye camera 44 may be oriented vertically downward.

[0034] Furthermore, the imaging centers 201C to 204C of the fisheye cameras 41 to 44 may face in different directions in the vertical direction. In addition, although the present embodiment shows an example in which the fisheye cameras 41 to 44 are arranged corresponding to the front, right side, rear, and left side of the vehicle 1, the present invention is not limited to this example. A configuration in which fisheye cameras are arranged on the right side and left side and standard cameras are arranged in the front and rear may also be used.

[0035] <Processing> Next, the procedure of processing performed by the ECU 22 functioning as an image processing device according to one embodiment will be described with reference to the flowchart of FIG. 3 and FIGS.

[0036] In step S301, the ECU 22 acquires a captured image captured by a fisheye camera mounted on the side mirror. FIG. 4 is an explanatory diagram illustrating processing of a captured image captured with the side mirror folded according to an embodiment. FIG. 5 is an explanatory diagram illustrating processing of a captured image captured with the side mirror unfolded according to an embodiment. In FIGS. 4 and 5 , the letters A, B, C, and D are displayed on the ground. These letters are provided to facilitate understanding and recognition of changes in the image due to distortion correction, cropping, and image conversion processes. Note that these marks do not exist on the actual road (ground). In this step, a captured image 401 or a captured image 501 is acquired. While the following description will be given using an image captured by the fisheye camera 44 mounted on the left side mirror as an example, similar processing is also performed on an image captured by the fisheye camera 42 mounted on the right side mirror. Situations in which the side mirrors need to be folded include, for example, when driving forward on a narrow road or reversing into a narrow parking lot, and driving at low speed with the side mirrors folded while checking the image of the surroundings.

[0037] In step S302, the ECU 22 performs distortion correction processing on the image captured by the fisheye camera (processing for converting the fisheye image into a planar image). After the distortion correction is performed, the corrected image 402 or the corrected image 502 is obtained.

[0038] In step S303, the ECU 22 determines whether the captured image is being acquired with the side mirrors folded. This can be determined from the control state of the side mirrors. If the answer to this step is Yes, the process proceeds to step S304. On the other hand, if the answer to this step is No, the process proceeds to step S307.

[0039] In step S304, the ECU 22 performs a trimming process to trim a predetermined range of the corrected image. Specifically, the trimming process is performed on the corrected image 402 to obtain a trimmed image 403. Here, the predetermined range may be a range extending forward and to the side from the end of the vehicle body and including the front wheels when the vehicle 1 is moving forward (or the shift lever is in the D range). On the other hand, the predetermined range may be a range extending backward and to the side from the end of the vehicle body and including the rear wheels when the vehicle 1 is moving backward (or the shift lever is in the R range). This allows the vehicle to move forward or backward while checking the surrounding environment, thereby improving user convenience. Furthermore, the trimming range in step S304 is controlled to be the same as the trimming range in step S307.

[0040] In step S305, the ECU 22 performs image conversion processing on the trimmed image. In this embodiment, folding the side mirrors results in a rotation of, for example, about 90 degrees. By converting the image (rotating the image) by the rotation angle of the side mirrors, the image can be converted to the same orientation as when the side mirrors are not folded. Specifically, the trimmed image 403 is converted to obtain a converted image 404.

[0041] In step S306, the ECU 22 performs display control to superimpose a virtual line indicating the area in which the vehicle 1 can travel when the side mirrors are folded on the converted image and display the superimposed virtual line on the screen of the display device 92. Details of the superimposed display of the virtual line will be described later with reference to FIG.

[0042] In step S307, the ECU 22 executes a trimming process to trim a predetermined range of the corrected image. Specifically, the ECU 22 executes the trimming process on the corrected image 502 to obtain a trimmed image 503. Here, the predetermined range is the same as the range described in step S304. The trimming range in step S307 is controlled to be the same as the trimming range in step S304.

[0043] In step S308, the ECU 22 performs display control to superimpose a virtual line indicating an area in which the vehicle 1 can travel when the side mirrors are not folded on the trimmed image and display the superimposed virtual line on the screen of the display device 92. Details of the superimposed display of the virtual line will be described later with reference to FIG.

[0044] Here, the superimposed display will be described with reference to Fig. 6. Fig. 6 is an explanatory diagram of a state in which a virtual line is superimposed on a converted image or a cropped image according to one embodiment. A first virtual line 601 is a virtual line superimposed on the converted image 404 in a state in which the side mirrors are folded. A second virtual line 602 is a virtual line superimposed on the converted image 503 in a state in which the side mirrors are folded. By viewing the image and checking the positional relationship between the virtual line and objects around the vehicle 1, the user can easily determine whether or not the vehicle can travel without a collision.

[0045] The first virtual line 601 is spaced a first distance from the vehicle 1. The second virtual line 602 is spaced a second distance, which is longer than the first distance, from the vehicle 1. This is because, when the side mirrors are folded, the vehicle 1 can travel even if the area width is relatively narrow, but when the side mirrors are not folded, a relatively wide area width must be ensured in order for the vehicle 1 to travel. This completes the series of processes in Figure 3.

[0046] In the above embodiment, the process of superimposing the virtual line is executed, but the converted image may be displayed without the virtual line being superimposed. Alternatively, instead of and / or in addition to the processes of S306 and S308, an object detection process may be executed on the converted image generated in S305 or the cropped image generated in S307 to detect an obstacle. Furthermore, a first warning may be issued when an obstacle is detected. Furthermore, a second warning that is more emphasized than the first warning may be issued when the virtual line intersects with an obstacle.

[0047] In the above embodiment, an example in which fisheye cameras are installed on the left and right side mirrors has been described, but the present invention is not limited to this example. The processing of this embodiment can also be applied to a case in which wide-angle standard cameras are installed on the left and right side mirrors. In this case, the distortion correction processing of S302 is skipped.

[0048] Although the above embodiment describes an example in which trimming processing is performed, trimming processing may not be performed, in which case the processing in S304 or S307 is skipped.

[0049] Furthermore, in the above embodiment, processing of an image captured by a fisheye camera provided on one of the left and right side mirrors has been described, but in practice, a left captured image and a right captured image acquired by each of the fisheye cameras 44, 42 provided on both the left and right side mirrors may be displayed side by side on the display device 92. Alternatively, a left converted image obtained by converting the left captured image and a right converted image obtained by converting the right captured image may be displayed side by side on the display device 92. The first virtual line 601 and the second virtual line 602 may also be displayed side by side, superimposed on the left and right images, respectively.

[0050] As described above, in this embodiment, when a captured image is acquired in a first state in which the side mirrors are folded, the captured image is converted (rotated) so that it has the same orientation as when the captured image is acquired in a second state in which the side mirrors are not folded.

[0051] This prevents the orientation of the captured image from changing due to changes in the orientation of the side mirrors, allowing the user to properly grasp the surrounding environment. Also, even when driving on narrow roads with the side mirrors folded, the user can properly grasp the surrounding environment, further improving driving safety.

[0052] Summary of the embodiment 1. The image processing device (22) according to the above embodiment is an image processing device that processes a captured image acquired by an image capturing unit (42, 44) provided in a side mirror of a vehicle (1), and includes an image conversion unit (22) that converts the captured image based on a state of the side mirror, and when the captured image is acquired in a first state in which the side mirror is folded, the image conversion unit converts the captured image so that the orientation of the captured image is the same as when the captured image is acquired in a second state in which the side mirror is not folded.

[0053] According to this embodiment, the user can properly grasp the surrounding environment (particularly the environment on the left and right sides of the vehicle).

[0054] 2. The image processing device (22) according to the above embodiment further includes a processing unit (22) that performs processing to trim a predetermined range of the captured image before the captured image is converted by the image conversion unit.

[0055] According to this embodiment, processing can be limited to a region of interest, so that an image of the region that the user wants to visually recognize can be obtained.

[0056] 3. In the image processing device (22) according to the above embodiment, when the vehicle is moving forward, the predetermined range is a range extending forward and to the side from the end of the vehicle body and including the front wheels.

[0057] According to this embodiment, information on the sides in front of the vehicle can be obtained when the vehicle is moving forward.

[0058] 4. In the image processing device (22) according to the above embodiment, when the vehicle is moving backward, the predetermined range is a range extending rearward and laterally from the end of the vehicle body and including the rear wheels.

[0059] According to this embodiment, information on the side area behind the vehicle can be obtained when the vehicle is moving backward.

[0060] 5. The image processing device (22) according to the above embodiment further includes a display control unit (22, 28) that causes a display device (92) to display a converted image of the image taken in the first state or the image taken in the second state, and the display control unit superimposes virtual lines (601, 602) indicating an area in which the vehicle can travel on the converted image or the taken image.

[0061] According to this embodiment, the boundaries of the area in which the vehicle can travel become easier to recognize, and by looking at the displayed image, it becomes possible to easily recognize whether the vehicle can travel without colliding with surrounding objects.

[0062] 6. In the image processing device (22) according to the above embodiment, the display control unit superimposes a first virtual line (601) spaced a first distance from the vehicle on the converted image in the first state, and superimposes a second virtual line (602) spaced a second distance from the vehicle that is longer than the first distance on the captured image in the second state.

[0063] This allows the boundaries of the area in which the vehicle can travel to be recognized in more detail, whether the side mirrors are folded or not, making it easier to recognize whether the vehicle can travel through narrow roads if the side mirrors are folded.

[0064] 7. In the image processing device (22) according to the above embodiment, the display control unit causes the display device to display a left captured image and a right captured image acquired by the image capturing units provided on the left and right side mirrors of the vehicle side by side, or causes the display device to display a left converted image obtained by converting the left captured image and a right converted image obtained by converting the right captured image side by side.

[0065] This allows the driver to easily recognize the conditions on both sides of the vehicle on a single screen, facilitating fine adjustments of the driver's driving operation (steering operation) of the steering wheel 31.

[0066] 8. In the image processing device (22) according to the above embodiment, the photographing unit is a fisheye camera (42, 44), and the image processing device (22) further includes a correction unit (22) that performs distortion correction on the photographed image, and the image conversion unit converts the corrected image that has been subjected to the distortion correction by the correction unit.

[0067] According to this embodiment, it is possible to prevent blind spots from occurring because a wide range of images can be captured, and it is also easy to trim an image of a range of interest from a wide range of captured images.

[0068] 9. In the image processing device (22) according to the above embodiment, the photographing unit photographs a downward view from the side mirror.

[0069] According to this embodiment, information on the side of the vehicle can be obtained.

[0070] 10. The vehicle (1) according to the above embodiment is a vehicle characterized by including the image processing device according to the above embodiment.

[0071] According to this embodiment, the processing of the image processing device can be realized in the vehicle.

[0072] 11. A control method for an image processing device (22) according to the above embodiment is a control method for an image processing device that processes a captured image acquired by an image capturing unit (42, 44) provided in a side mirror of a vehicle (1), and includes an image conversion step (S305) of converting the captured image based on a state of the side mirror, wherein in the image conversion step, when the captured image is acquired in a first state in which the side mirror is folded, the captured image is converted so that the orientation of the captured image is the same as when the captured image is acquired in a second state in which the side mirror is not folded.

[0073] According to this embodiment, the user can properly grasp the surrounding environment (particularly the environment on the left and right sides of the vehicle).

[0074] 12. The program according to the above embodiment is a program for causing a computer to function as the image processing device according to the above embodiment.

[0075] According to this embodiment, the functions of the image processing device can be realized as a program.

[0076] 13. The program according to the above embodiment is a storage medium storing a program for causing a computer to function as the image processing device according to the above embodiment.

[0077] According to this embodiment, the functions of the image processing device can be realized as a storage medium.

[0078] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0079] 22, 28: ECU, 42, 44: fisheye camera, 92: display device, 601: first virtual line, 602: second virtual line

Claims

1. An image processing device that processes captured images acquired by an image capturing unit provided on a vehicle's side mirror, comprising an image conversion unit that converts the captured image based on the state of the side mirror, wherein when the captured image is acquired with the side mirror in a first state (folded), the image conversion unit converts the captured image so that the orientation is the same as when the captured image is acquired with the side mirror in a second state (unfolded).

2. The image processing device according to claim 1, further comprising a processing section for carrying out a process of trimming a predetermined range of the captured image before the captured image is converted by the image conversion section.

3. The image processing device according to claim 2, wherein when the vehicle is moving forward, the predetermined range is a range extending forward and to the side from the end of the vehicle body and including the front wheels.

4. The image processing device according to claim 2, characterized in that, when the vehicle is moving backward, the predetermined range is a range extending rearward and sideways from the end of the vehicle body and including the rear wheels.

5. The image processing device described in claim 1, further comprising a display control unit that causes a converted image of the image taken in the first state or an image taken in the second state to be displayed on a display device, wherein the display control unit superimposes a virtual line indicating an area in which the vehicle can travel on the converted image or the taken image.

6. The image processing device described in claim 5, characterized in that the display control unit superimposes a first virtual line spaced a first distance from the vehicle on the converted image in the first state, and superimposes a second virtual line spaced a second distance from the vehicle longer than the first distance on the captured image in the second state.

7. The image processing device described in claim 5, characterized in that the display control unit causes the left captured image and the right captured image acquired by each capturing unit provided on the left and right side mirrors of the vehicle to be displayed side by side on the display device, or causes the left converted image converted from the left captured image and the right converted image converted from the right captured image to be displayed side by side on the display device.

8. The image processing device according to claim 1, further comprising a correction unit that performs distortion correction on the captured image, and wherein the image conversion unit converts the corrected image that has been subjected to the distortion correction by the correction unit.

9. The image processing device according to claim 1, wherein the image capturing unit captures an image of a downward direction from the side mirror.

10. A vehicle comprising an image processing device according to any one of claims 1 to 9.

11. A control method for an image processing device that processes a captured image acquired by an image capturing unit provided on a side mirror of a vehicle, comprising an image conversion process for converting the captured image based on the state of the side mirror, wherein in the image conversion process, when the captured image is acquired with the side mirror in a first state in which it is folded, the captured image is converted so that it has the same orientation as when the captured image is acquired with the side mirror in a second state in which it is not folded.

12. A program for causing a computer to function as the image processing device according to any one of claims 1 to 9.

13. A storage medium storing a program for causing a computer to function as the image processing device according to any one of claims 1 to 9.

Images