Image processing device, method for controlling image processing device, vehicle, program, and storage medium

Abstract

An image processing device for processing a captured image acquired by an imaging unit provided in a side mirror of a vehicle, the image processing device being provided with an image conversion unit for converting the captured image on the basis of the state of the side mirror, the image conversion unit converts the captured image such that when the captured image is acquired in a first state in which the side mirror is folded, the captured image is acquired in the same direction as when the captured image is acquired in a second state in which the side mirror is not folded.

Description

TECHNICAL FIELD

[0001] The present application relates to an image processing apparatus, a control method of an image processing apparatus, a vehicle, a program, and a storage medium. BACKGROUND

[0002] Patent Literature 1 discloses a technology of synthesizing and displaying image data acquired by a photographing camera provided to a side mirror of a vehicle to photograph a rear of the vehicle and image data acquired by another photographing camera.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2010-287163 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, in Patent Literature 1, a state in which the side mirror is folded is not considered. For example, when driving in front of a narrow road or parking in a parking lot having a narrow width, there is a demand to confirm an image of the surroundings while driving at a low speed in a state in which the side mirror is folded. In the technology described in Patent Literature 1, there is a problem that the direction of the image changes in the state in which the side mirror is folded, and it is difficult for a user to properly grasp the surrounding environment.

[0008] The present application has been made in view of the above problems, and aims to provide a technology for enabling a user to properly grasp a surrounding environment.

[0009] SOLUTION TO PROBLEM

[0010] An aspect of the present application for achieving the above object relates to an image processing apparatus that processes a photographed image acquired by a photographing section provided to a side mirror of a vehicle, the image processing apparatus characterized by comprising an image conversion section that converts the photographed image based on a state of the side mirror, the image conversion section converting the photographed image in such a manner that a case where the photographed image is acquired in a state in which the side mirror is in a first state of being folded and a case where the photographed image is acquired in a state in which the side mirror is in a second state of not being folded become the same direction.

[0011] EFFECT OF THE INVENTION

[0012] According to the present application, a user is able to properly grasp a surrounding environment.

[0013] The following description will be made with reference to the accompanying drawings, so that other features and advantages of the present application can be apparent. Moreover, in the drawings, the same or similar structures are attached with the same reference numerals. Attached Figure Description

[0014] The specification includes accompanying drawings, which form part of the specification, illustrating embodiments of the invention and explaining the principles of the invention together with the description of the embodiments.

[0015] Figure 1 This is a block diagram of a vehicle according to one embodiment.

[0016] Figure 2A This is a diagram illustrating the horizontal shooting range of each fisheye camera involved in one embodiment.

[0017] Figure 2B This is a diagram showing the vertical shooting range of a fisheye camera on the left side according to one embodiment.

[0018] Figure 2C This is a diagram showing the vertical shooting range of a rear fisheye camera according to one embodiment.

[0019] Figure 3 This is a flowchart illustrating the sequence of processes performed by an image processing apparatus according to one embodiment.

[0020] Figure 4 This is an explanatory diagram illustrating the processing of images captured when the side mirrors are folded, according to one embodiment.

[0021] Figure 5 This is an explanatory diagram illustrating the processing of images captured when the side mirrors are not folded, according to one embodiment.

[0022] Figure 6 This is a diagram illustrating an example of the overlap of imaginary lines in one embodiment. Detailed Implementation

[0023] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the following embodiments are not intended to limit the scope of the claims, and the present invention does not require a combination of all the features described in the embodiments. Alternatively, two or more features from the plurality of features described in the embodiments may be arbitrarily combined. Additionally, the same or identical structures will be given the same reference numerals, and repeated descriptions will be omitted.

[0024] <Structure>

[0025] Figure 1 This is a block diagram of vehicle 1 according to one embodiment. Figure 1 The diagram shows a general outline of vehicle 1 using top and side views. Vehicle 1 is an example of a four-wheeled passenger car. Vehicle 1 can be such a four-wheeled vehicle, or it can be a two-wheeled vehicle or other types of vehicles.

[0026] The vehicle 1 includes a vehicle control device 2 (hereinafter referred to as control device 2) that controls the vehicle 1. The control device 2 includes a plurality of ECUs (Electronic Control Units) 20 to 29 communicably connected through an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit), a memory such as a semiconductor memory, an interface to an external device, and the like. In the memory, a program executed by the processor, data used by the processor in processing, and the like are stored. Each ECU can also have a plurality of processors, memories, interfaces, and the like. For example, the ECU 20 has one or more processors 20a and one or more memories 20b. The processor 20a executes instructions including a program stored in the memory 20b, whereby the ECU 20 performs processing. Alternatively, the ECU 20 can have an ASIC (Application Specific Integrated Circuit) or the like dedicated to the processing performed by the ECU 20. The same applies to the other ECUs.

[0027] The functions and the like handled by each ECU 20 to 29 will be described below. The number of ECUs and the functions handled thereby can be appropriately designed, and can be more detailed or more integrated than in the present embodiment.

[0028] The ECU 20 performs control related to automatic travel of the vehicle 1. In automatic driving, at least one of the steering, acceleration, and deceleration of the vehicle 1 is automatically controlled. The automatic travel performed by the ECU 20 can include automatic travel in which the driver does not need to perform a travel operation (also referred to as automatic driving), and automatic travel for assisting the driver in performing a travel operation (also referred to as driving assistance).

[0029] The ECU 21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism that turns the front wheels in accordance with a steering operation (steering operation) of the driver on the steering wheel 31. In addition, the electric power steering device 3 includes a motor that generates a driving force for assisting the steering operation or automatically turning the front wheels, a sensor that detects a steering angle, and the like. In a case where the driving state of the vehicle 1 is automatic driving, the ECU 21 automatically controls the electric power steering device 3 in accordance with an instruction from the ECU 20, and controls the traveling direction of the vehicle 1.

[0030] The ECU 22 and the ECU 23 control the detection components that detect the surrounding situation of the vehicle and process the detection results. The vehicle 1 includes one standard camera 40 and four fisheye cameras 41 to 44 as the detection components that detect the surrounding situation of the vehicle. The standard camera 40, the fisheye camera 42, and the fisheye camera 44 are connected to the ECU 22. The fisheye camera 41 and the fisheye camera 43 are connected to the ECU 23. The ECU 22 and the ECU 23 analyze the images captured by the standard camera 40 and the fisheye cameras 41 to 44, thereby being able to extract the outlines of object marks and the lane boundary lines (white lines and the like) on the road. That is, the ECU 22 and the ECU 23 are able to function as image processing devices.

[0031] The fisheye cameras 41 to 44 refer to cameras to which fisheye lenses are attached. Hereinafter, the structure of the fisheye camera 41 will be described. The other fisheye cameras 42 to 44 can have the same structure. The fisheye camera 41 has a larger field of view than the standard camera 40. Therefore, the fisheye camera 41 is able to capture a larger range than the standard camera 40. The image captured by the fisheye camera 41 has a large distortion compared to the image captured by the standard camera 40. Therefore, it can be that the ECU 23 performs a conversion process for reducing the distortion (hereinafter, referred to as "distortion correction process") on the image captured by the fisheye camera 41 before analyzing the image. On the other hand, it can be that the ECU 22 does not perform the distortion correction process on the image captured by the standard camera 40 before analyzing the image. In this way, the standard camera 40 is an imaging device that captures an image that is not a target of the distortion correction process, and the fisheye camera 41 is an imaging device that captures an image that is a target of the distortion correction process. Instead of the standard camera 40, another imaging device that captures an image that is not a target of the distortion correction process, such as a camera to which a wide-angle lens or a telephoto lens is attached, can be used.

[0032] The standard camera 40 is attached to the center of the front portion of the vehicle 1 and captures the surrounding situation in front of the vehicle 1. The fisheye camera 41 is attached to the center of the front portion of the vehicle 1 and captures the surrounding situation in front of the vehicle 1. The fisheye camera 42 is attached to the right side of the front portion of the vehicle 1 and captures the surrounding situation on the right side of the vehicle 1. The fisheye camera 43 is attached to the left side of the front portion of the vehicle 1 and captures the surrounding situation on the left side of the vehicle 1. The fisheye camera 44 is attached to the rear portion of the vehicle 1 and captures the surrounding situation behind the vehicle 1. Figure 1In the present embodiment, the standard camera 40 and the fisheye camera 41 are arranged in the horizontal direction. However, the arrangement of the standard camera 40 and the fisheye camera 41 is not limited to this, and for example, these cameras can be arranged in the vertical direction. Also, at least one of the standard camera 40 and the fisheye camera 41 can be installed in the front portion of the roof of the vehicle 1 (for example, the inner side of the front window of the vehicle 1). The fisheye camera 42 is installed in the right side mirror of the vehicle 1, and captures the surrounding situation in the lower right of the vehicle 1. The fisheye camera 43 is installed in the center of the rear portion of the vehicle 1, and captures the surrounding situation in the rear of the vehicle 1. The fisheye camera 44 is installed in the left side mirror of the vehicle 1, and captures the surrounding situation in the lower left of the vehicle 1.

[0033] The kind, number, and installation position of the cameras provided in the vehicle 1 are not limited to the above-described examples. Also, the vehicle 1 can include a laser radar (Light Detection and Ranging), a millimeter wave radar, as a detection component for detecting an object marker in the surroundings of the vehicle 1 or measuring the distance to the object marker.

[0034] The ECU 22 controls the standard camera 40, the fisheye camera 42, and the fisheye camera 44, and information processes the detection results. The ECU 23 controls the fisheye camera 41 and the fisheye camera 43, and information processes the detection results. By dividing the detection component that detects the surrounding situation of the vehicle into two systems, it is possible to improve the reliability of the detection results. The image processing device according to the present embodiment mainly corresponds to the ECU 22, but the ECU 22 and the ECU 23 can be integrated into one ECU, and configured as an image processing device.

[0035] The ECU 24 controls the gyro sensor 5, the GPS sensor 24b, the communication device 24c, and information processes the detection results or the communication results. The gyro sensor 5 detects the rotational motion of the vehicle 1. From the detection results of the gyro sensor 5, the wheel speed, and the like, it is possible to determine the advancing path of the vehicle 1. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c wirelessly communicates with a server that provides map information and traffic information, and acquires these information. The ECU 24 can access the database 24a of the map information constructed in the memory, and the ECU 24 performs route search from the current location to the destination, and the like. The ECU 24, the map database 24a, and the GPS sensor 24b constitute a so-called navigation device.

[0036] The ECU 25 is provided with a communication device 25a for vehicle-to-vehicle communication. The communication device 25a wirelessly communicates with other vehicles in the surroundings, and exchanges information between the vehicles.

[0037] The ECU 26 controls the powertrain 6. The powertrain 6 is a mechanism that outputs driving force to rotate the drive wheels of the vehicle 1, and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine in correspondence with the driving operation (accelerator operation or acceleration operation) of the driver detected by the operation detection sensor 7a provided at the accelerator pedal 7A, for example, or switches the shift stage of the transmission on the basis of information such as the vehicle speed detected by the vehicle speed sensor 7c. In the case where the driving state of the vehicle 1 is automatic driving, the ECU 26 automatically controls the powertrain 6 in correspondence with the instruction from the ECU 20 to control the acceleration and deceleration of the vehicle 1.

[0038] The ECU 27 controls the light devices (headlights, taillights, and the like) including the direction indicator 8 (turn signal). In the case of the example shown in FIG. 1, the direction indicator 8 is provided at the front portion, the door mirror, and the rear portion of the vehicle 1. Figure 1

[0039] The ECU 28 controls the input and output device 9. The input and output device 9 outputs information to the driver and accepts input of information from the driver. The sound output device 91 notifies the driver of information by sound. The display device 92 notifies the driver of information by displaying an image. The display device 92 is provided, for example, on the front surface of the driver's seat, and constitutes an instrument panel or the like. Also, in this case, sound and display are exemplified, but information can also be notified by vibration or light. In addition, information can also be notified by a combination of a plurality of sound, display, vibration, or light. Also, it can be that the combination or the notification method is made different in accordance with the level (for example, the degree of urgency) of the information that should be notified. The input device 93 is a switch group that instructs the vehicle 1 and is provided at a position operable by the driver, but can also include a sound input device.

[0040] The ECU 29 controls the brake device 10 and the parking brake (not shown). The brake device 10 is, for example, a disc brake device provided at each wheel of the vehicle 1, and decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheel. The ECU 29 controls the operation of the brake device 10 in correspondence with the driving operation (brake operation) of the driver detected by the operation detection sensor 7b provided at the brake pedal 7B, for example. In the case where the driving state of the vehicle 1 is automatic driving, the ECU 29 automatically controls the brake device 10 in correspondence with the instruction from the ECU 20 to control the deceleration and stop of the vehicle 1. The brake device 10 and the parking brake can also operate to maintain the stopped state of the vehicle 1. In addition, in the case where the transmission of the powertrain 6 is provided with a parking lock mechanism, the parking lock mechanism can also operate to maintain the stopped state of the vehicle 1.

[0041] <Photographing Range>

[0042] ​Then, the shooting ranges of the standard camera 40 and the fisheye cameras 41 to 44 are described with reference to Figures 2A to 2C Figure 2A The shooting ranges of the respective cameras in the horizontal direction are shown in FIG. 2. Figure 2B The shooting range in the vertical direction of the fisheye camera 44 installed at the left side mirror of the vehicle 1 is shown in FIG. 3. Figure 2C The shooting range in the vertical direction of the fisheye camera 43 installed at the rear of the vehicle 1 is shown in FIG. 4.

[0043] First, the shooting ranges in the plan view of the vehicle 1 (i.e., the horizontal direction of the vehicle 1) are described with reference to Figure 2A The standard camera 40 shoots the scene included in the shooting range 200. The shooting center 200C of the standard camera 40 faces the front of the vehicle 1. The field of view in the horizontal direction of the standard camera 40 can be less than 90°, for example, it can be about 45° or about 30°.

[0044] The fisheye camera 41 shoots the scene included in the shooting range 201. The shooting center 201C of the fisheye camera 41 faces the front of the vehicle 1. The fisheye camera 42 shoots the scene included in the shooting range 202. The shooting center 202C of the fisheye camera 42 faces the lower right of the vehicle 1. The fisheye camera 43 shoots the scene included in the shooting range 203. The shooting center 203C of the fisheye camera 43 faces the rear of the vehicle 1. The fisheye camera 44 shoots the scene included in the shooting range 204. The shooting center 204C of the fisheye camera 44 faces the lower left of the vehicle 1. The field of view in the horizontal direction of the fisheye cameras 41 to 44 can be greater than 90°, greater than 150°, or greater than 180°, for example, it can be about 180°. Figure 2A An example in which the field of view in the horizontal direction of the fisheye cameras 41 to 44 is 180° is shown in FIG. 5.

[0045] ​The imaging range 201 can be divided into a region 201L in the left oblique front of the vehicle 1, a region 201F in the front of the vehicle 1, and a region 201R in the right oblique front of the vehicle 1. The imaging range 202 can be divided into a region 202L in the right oblique front of the vehicle 1, a region 202F in the right side surface of the vehicle 1, and a region 202R in the right oblique rear of the vehicle 1. The imaging range 203 can be divided into a region 203L in the right oblique rear of the vehicle 1, a region 203F in the rear of the vehicle 1, and a region 203R in the left oblique rear of the vehicle 1. The imaging range 204 can be divided into a region 204L in the right oblique rear of the vehicle 1, a region 204F in the left side surface of the vehicle 1, and a region 204R in the left oblique front of the vehicle 1. The imaging range 201 can also be divided into three regions 201L, 201F, 201R equally (i.e., the field angles of the regions are equal). The other imaging ranges 202 to 204 can also be divided into three equally.

[0046] The standard camera 40 and the fisheye cameras 41 to 44 have the above-described imaging ranges 200 to 204, whereby the imaging ranges of the two separate cameras include the front of the vehicle 1 and the four oblique directions. Specifically, both the imaging range 200 of the standard camera 40 and the region 201F of the imaging range 201 of the fisheye camera 41 include the front of the vehicle 1. Both the region 201R of the imaging range 201 of the fisheye camera 41 and the region 202L of the imaging range 202 of the fisheye camera 42 include the right oblique front of the vehicle 1. The same applies to the other three oblique directions of the vehicle 1.

[0047] Then, the imaging ranges in the vertical direction of the vehicle 1 are described with reference to Figure 2B and Figure 2C In Figure 2B , the imaging range in the vertical direction of the fisheye camera 44 is described, and in Figure 2C , the imaging range in the vertical direction of the fisheye camera 43 is described. The same applies to the imaging ranges in the vertical direction of the other fisheye cameras 41 and 42.

[0048] The field angle in the vertical direction of the fisheye cameras 41 to 44 can be greater than 90°, greater than 150°, or greater than 180°, for example, and can be, for example, 180° or so. Figure 2B and Figure 2CExamples are shown where the vertical field of view of fisheye cameras 41-44 is 180°. In the illustrated example, the shooting center 203C of fisheye camera 43 is oriented downwards (towards the ground) relative to a direction parallel to the ground. Alternatively, the shooting center 203C of fisheye camera 43 may be oriented parallel to the ground, or it may be oriented upwards (towards the opposite side of the ground) relative to a direction parallel to the ground. Furthermore, the shooting center 204C of fisheye camera 44 is oriented downwards (towards the ground) relative to a direction parallel to the ground. That is, the shooting center 204C of fisheye camera 44 is oriented towards the lower left of vehicle 1. The shooting center 204C of fisheye camera 44 may also be oriented vertically downwards.

[0049] Alternatively, the shooting centers 201C to 204C of the fisheye cameras 41 to 44 may face different directions vertically. Furthermore, this embodiment shows an example of fisheye cameras 41 to 44 positioned corresponding to the front, right, rear, and left sides of vehicle 1, but is not limited to this example. It is also possible to configure fisheye cameras on the right and left sides, and standard cameras on the front and rear sides.

[0050] <Processing>

[0051] Then, refer to Figure 3 Flowchart and Figures 4 to 6 This describes the sequence of processing performed by the ECU 22, which functions as an image processing device according to one embodiment.

[0052] In step S301, ECU 22 acquires an image captured by a fisheye camera mounted on the side mirror. Here, Figure 4 This is an explanatory diagram illustrating the processing of images captured when the side mirrors are folded, according to one embodiment. Additionally, Figure 5 This is an explanatory diagram illustrating the processing of images captured when the side mirrors are not folded, according to one embodiment. Figure 4 as well as Figure 5 The Roman letters A, B, C, and D are displayed on the ground, but this is for the purpose of understanding and recognizing the changes in the image caused by various processes such as distortion correction, cropping, and image transformation. Note that such a display does not exist on a real road (ground). In this step, either image 401 or image 501 is acquired. Furthermore, the image captured by the fisheye camera 44 installed in the left side mirror will be used as an example for explanation, but the same processing is performed on the image captured by the fisheye camera 42 installed in the right side mirror. Situations where the side mirrors need to be folded include, for example, when driving forward on a narrow road or backing up to park in a narrow parking lot, driving at low speed while checking the surrounding image with the side mirrors folded.

[0053] In step S302, the ECU 22 performs a distortion correction process (a conversion process from a fisheye image to a planar image) on the captured image captured by the fisheye camera. When the distortion correction is performed, the corrected image 402 or the corrected image 502 is acquired.

[0054] In step S303, the ECU 22 determines whether the captured image is acquired in a state in which the side mirror is folded. This can be determined from the control state of the side mirror. In a case in which this step is "Yes", the processing proceeds to step S304. On the other hand, in a case in which this step is "No", the processing proceeds to step S307.

[0055] In step S304, the ECU 22 performs a cropping process that crops a predetermined range on the corrected image. Specifically, the cropping process is performed on the corrected image 402, and the cropped image 403 is acquired. Here, the predetermined range can also be a range that, in a case in which the vehicle 1 is advancing (or in a case in which the shift lever is in the D range), faces forward and sideways from the end portion of the vehicle body and includes the front wheels. On the other hand, the predetermined range can also be a range that, in a case in which the vehicle 1 is retreating (or in a case in which the shift lever is in the R range), faces backward and sideways from the end portion of the vehicle body and includes the rear wheels. Thereby, it is possible to advance or retreat while confirming the surrounding environment, and thus the convenience of the user is improved. In addition, the cropping range in step S304 is controlled to be the same as the cropping range in step S307.

[0056] In step S305, the ECU 22 performs an image conversion process on the cropped image. In the present embodiment, for example, a rotation of 90° or more is generated due to the folding of the side mirror. By performing an image conversion (image rotation) using only the rotation angle of the side mirror, it is possible to convert to the same direction as the direction in a state in which the side mirror is not folded. Specifically, the cropped image 403 is image-converted, and the converted image 404 is acquired.

[0057] In step S306, the ECU 22 performs display control such that the virtual line that indicates the region in which the vehicle 1 can travel in a case in which the side mirror is folded is superimposed on the converted image, and causes the image to be displayed on the screen of the display device 92. The details of the superimposed display of the virtual line are described later. Figure 6 The details of the superimposed display of the virtual line are described later.

[0058] In step S307, the ECU 22 performs a cropping process that crops a predetermined range on the corrected image. Specifically, the cropping process is performed on the corrected image 502, and the cropped image 503 is acquired. Here, the predetermined range is the same as the range described in step S304. In addition, the cropping range in step S307 is controlled to be the same as the cropping range in step S304.

[0059] In step S308, the ECU 22 performs display control such that the virtual line indicating the region in which the vehicle 1 is able to travel without folding the side mirror is superimposed on the cropped image, and causes the image to be displayed on the screen of the display device 92. Referring to Figure 6 The detailed contents of the superimposed display of the virtual line are described later.

[0060] Here, referring to Figure 6 the superimposed display is described. Figure 6 is a diagram of the state in which the virtual line is superimposed on the transformed image or the cropped image according to the embodiment. The first virtual line 601 is a virtual line superimposed on the transformed image 404 in the state in which the side mirror is folded. The second virtual line 602 is a virtual line superimposed on the transformed image 503 in the state in which the side mirror is folded. The user observes the image to confirm the positional relationship of the virtual line and the object of the periphery of the vehicle 1, and thus is able to easily determine whether or not it is possible to travel without collision.

[0061] The first virtual line 601 is separated from the vehicle 1 by a first distance. The second virtual line 602 is separated from the vehicle 1 by a second distance longer than the first distance. This is because, in the state in which the side mirror is folded, even if the region width is relatively narrow, the vehicle 1 is able to travel, but in the state in which the side mirror is not folded, it is necessary to secure a relatively wide region width in order for the vehicle 1 to travel. Above, Figure 3 the series of processes of

[0062] Furthermore, in the above-described embodiment, the process of superimposing and displaying the virtual line is performed, but it can also be configured not to superimpose and display the virtual line on the transformed image. Or it can also be that, instead of the processes of S306 and S308 and / or in addition to these processes, an obstacle detection is performed by performing an object detection process on the transformed image generated in S305 or the cropped image generated in S307. It can also be that, in the case in which an obstacle is detected, a first warning is notified. It can also be that, in the case in which the virtual line intersects with the obstacle, a second warning more conspicuous than the first warning is notified.

[0063] In addition, in the above-described embodiment, the example in which the fisheye camera is disposed on the left and right side mirrors is described, but is not limited to this example. In the case in which a standard camera of a wide angle is disposed on the left and right side mirrors, the process of the present embodiment can also be applied. In this case, the distortion correction process of S302 is skipped.

[0064] In addition, in the above-described embodiment, the example in which the cropping process is performed is described, but the cropping process can also not be performed. In this case, the process of S304 or S307 is skipped.

[0065] In addition, in the above-described embodiment, the processing of the captured image of the fisheye camera provided on one of the left and right side mirrors was described, but in fact, the left captured image and the right captured image acquired by the respective fisheye cameras 44, 42 provided on both of the left and right side mirrors can be arranged and displayed on the display device 92. Alternatively, a left transformed image transformed from the left captured image and a right transformed image transformed from the right captured image can be arranged and displayed on the display device 92. The first imaginary line 601 and the second imaginary line 602 can be overlaid on each of the left and right images and arranged and displayed.

[0066] As described above, in the present embodiment, the captured image is transformed (rotated) in a manner such that the case where the state of the side mirror is in the first state in which it is folded and the case where the captured image is acquired in the second state in which the side mirror is not folded become the same direction.

[0067] Thus, it is possible to prevent the direction of the acquired image from changing due to a change in the direction of the side mirror, and therefore the user can appropriately grasp the surrounding environment. In addition, even in a situation such as driving on a narrow road in a state in which the side mirror is folded, the user can appropriately grasp the surrounding environment, and therefore it is possible to further improve the safety of driving.

[0068] <Summary of Embodiment>

[0069] 1. The image processing device 22 described in the above embodiment processes a captured image acquired by the capturing section 42, 44 provided on the side mirror of the vehicle 1, and includes an image transformation section 22 that transforms the captured image based on the state of the side mirror, the image transformation section transforming the captured image in a manner such that the case where the state of the side mirror is in a first state in which it is folded and the case where the captured image is acquired in a second state in which the side mirror is not folded become the same direction.

[0070] According to this embodiment, the user can appropriately grasp the surrounding environment (particularly, the environment on the left and right sides of the vehicle).

[0071] 2. The image processing device 22 described in the above embodiment further includes a processing section 22 that performs a processing of clipping a predetermined range on the captured image before the captured image is transformed by the image transformation section.

[0072] According to this embodiment, it is possible to perform the processing limited to the region of interest, and therefore the user can acquire an image of a region that the user wants to visually confirm.

[0073] 3. The image processing apparatus 22 according to any one of the above embodiments, wherein in a case where the vehicle is advancing, the predetermined range is a range from the end of the vehicle body toward the front and side and including the front wheel.

[0074] According to this embodiment, it is possible to acquire information of the side portion in front of the vehicle while advancing.

[0075] 4. The image processing apparatus 22 according to any one of the above embodiments, wherein in a case where the vehicle is retreating, the predetermined range is a range from the end of the vehicle body toward the rear and side and including the rear wheel.

[0076] According to this embodiment, it is possible to acquire information of the side portion in the rear of the vehicle while retreating.

[0077] 5. The image processing apparatus 22 according to any one of the above embodiments, further comprising a display control section 22, 28 that causes a transformed image of the captured image in the first state or the captured image in the second state to be displayed on a display device 92, the display control section superimposing an imaginary line 601, 602 indicating a region in which the vehicle is able to travel with respect to the transformed image or the captured image.

[0078] According to this embodiment, it is possible to easily recognize the boundary of the region in which the vehicle is able to travel, and the observer is able to easily recognize whether or not it is possible to travel without collision with the surrounding objects and the vehicle by observing the display image.

[0079] 6. The image processing apparatus 22 according to any one of the above embodiments, wherein the display control section superimposes a first imaginary line 601 separated by a first distance from the vehicle with respect to the transformed image in the first state, and superimposes a second imaginary line 602 separated by a second distance longer than the first distance from the vehicle with respect to the captured image in the second state.

[0080] Thus, in the case where the side mirror is folded and in the case where the side mirror is not folded, it is possible to more specifically recognize the boundary of the region in which the vehicle is able to travel. Therefore, it is easier to recognize whether or not it is possible to travel on a narrow road if the side mirror is folded.

[0081] 7. The image processing apparatus 22 according to any one of the above embodiments, wherein the display control section causes a left captured image and a right captured image acquired by each of the capturing sections provided at the left and right side mirrors of the vehicle to be arranged and displayed on the display device, or causes a left transformed image in which the left captured image is transformed and a right transformed image in which the right captured image is transformed to be arranged and displayed on the display device.

[0082] Thus, it is possible to easily recognize the situation of the left and right side portions of the vehicle in one screen. Therefore, it is easier for the driver to finely adjust the steering operation (steering wheel operation) of the steering wheel 31.

[0083] 8. The image processing apparatus 22 according to the above embodiment, wherein the imaging section is a fisheye camera 42, 44, and the image processing apparatus further includes a correction section 22 that performs distortion correction on the captured image, and the image conversion section converts a corrected image that has been subjected to the distortion correction by the correction section.

[0084] According to this embodiment, a wide range can be imaged, and thus it is possible to suppress the occurrence of dead angles. In addition, it becomes easy to crop an image of a range of interest from a captured image of a wide range.

[0085] 9. The image processing apparatus 22 according to the above embodiment, wherein the imaging section images the lower side from the side mirror.

[0086] According to this embodiment, it is possible to acquire information about the side of the vehicle.

[0087] 10. The vehicle 1 according to the above embodiment, characterized by including the image processing apparatus according to the above embodiment.

[0088] According to this embodiment, it is possible to implement the processing of the image processing apparatus in the vehicle.

[0089] 11. A control method of the image processing apparatus 22 according to the above embodiment, which processes a captured image acquired by the imaging section 42, 44 provided to the side mirror of the vehicle 1, the control method of the image processing apparatus including an image conversion step S305 in which the captured image is converted on the basis of the state of the side mirror, and in the image conversion step, the captured image is converted in such a manner that a case where the captured image is acquired in a state in which the side mirror is in a folded first state and a case where the captured image is acquired in a state in which the side mirror is not folded in a second state become the same direction.

[0090] According to this embodiment, the user can appropriately grasp the surrounding environment (particularly, the environment on the left and right sides of the vehicle).

[0091] 12. The program according to the above embodiment, which causes a computer to function as the image processing apparatus according to the above embodiment.

[0092] According to this embodiment, it is possible to implement the function of the image processing apparatus as a program.

[0093] 13. The program according to the above embodiment, which stores a program for causing a computer to function as the image processing apparatus according to the above embodiment.

[0094] According to this embodiment, it is possible to implement the function of the image processing apparatus as a storage medium.

[0095] The application is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the gist of the application.

[0096] Legend of reference numerals

[0097] 22, 28: ECU; 42, 44: fisheye camera; 92: display device; 601: first imaginary line; 602: second imaginary line.

Claims

1. An image processing apparatus for processing images captured by a camera mounted on a side mirror of a vehicle, characterized in that, The device includes an image transformation unit that transforms the captured image based on the state of the side mirror. The image transformation unit transforms the captured image in such a way that the captured image obtained in the first state where the side mirror is folded is in the same direction as the captured image obtained in the second state where the side mirror is not folded.

2. The image processing apparatus according to claim 1, characterized in that, It also includes a processing unit that performs cropping of the captured image by a predetermined range before the image transformation unit transforms the captured image.

3. The image processing apparatus according to claim 2, characterized in that, When the vehicle is moving forward, the defined range is the area extending from the end of the vehicle body towards the front and sides, including the front wheels.

4. The image processing apparatus according to claim 2, characterized in that, When the vehicle is reversing, the predetermined range is the area extending from the end of the vehicle body toward the rear and sides, including the rear wheels.

5. The image processing apparatus according to claim 1, characterized in that, It also includes a display control unit, which displays a transformed image of the captured image in the first state or a captured image in the second state on the display device. The display control unit uses an imaginary line to represent the area where the vehicle can drive, superimposed on the transformed image or the captured image.

6. The image processing apparatus according to claim 5, characterized in that, The display control unit overlays the transformed image of the first state with a first imaginary line that separates the vehicle by a first distance. And for the second state, the captured image is superimposed with a second imaginary line that is a second distance longer than the first distance between the vehicle and the captured image.

7. The image processing apparatus according to claim 5, characterized in that, The display control unit arranges and displays the left and right images captured by the cameras installed in the left and right side mirrors of the vehicle on the display device, or arranges and displays a left-transformed image (derived from the left image) and a right-transformed image (derived from the right image) on the display device.

8. The image processing apparatus according to claim 1, characterized in that, The camera unit is a fisheye camera. The image processing device also includes a correction unit for correcting distortion in the captured image. The image transformation unit transforms the corrected image obtained by the distortion correction performed by the correction unit.

9. The image processing apparatus according to claim 1, characterized in that, The camera unit takes a picture from below through the side mirror.

10. A vehicle, characterized in that, The device comprises an image processing apparatus according to any one of claims 1 to 9.

11. A control method for an image processing apparatus, comprising processing an image captured by a camera mounted on a side mirror of a vehicle, characterized in that... The image transformation step includes transforming the captured image based on the state of the side mirror. In the image transformation step, the captured image is transformed in such a way that the captured image obtained in the first state where the side mirror is folded is in the same direction as the captured image obtained in the second state where the side mirror is not folded.

12. A program for enabling a computer to function as an image processing apparatus according to any one of claims 1 to 9.

13. A storage medium storing a program for enabling a computer to function as an image processing apparatus according to any one of claims 1 to 9.

Images