Image Processing Apparatus and Image Processing Method

Abstract

Provided are an image processing apparatus and an image processing method capable of making the synthesis of images captured by a plurality of imaging components clear. The apparatus has: side cameras (110L, 110R) mounted on the sides of a vehicle, which provide images of the rear of the sides including the vehicle body area; a fisheye camera (120) mounted on the rear of the vehicle, which provides an image of the rear of the vehicle captured with a relatively wide field of view; a narrow-angle camera (130) mounted on the rear of the vehicle, which provides an image of the rear of the vehicle captured with a relatively narrow field of view; and a synthesis component capable of synthesizing, with respect to the vehicle body area of the image of the side camera (110), the images of the fisheye camera (120) and the narrow-angle camera (130) corresponding to the vehicle body area.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for processing images captured by in-vehicle cameras, and particularly to a method for synthesizing an image captured by a rear camera with an image captured by a side camera. Background Art

[0002] In order to achieve driver assistance and improved visual recognition, a CMS (Camera Monitor System) that displays images captured by in-vehicle cameras on a display is mounted on a vehicle. For example, as an alternative component to a side mirror, a side camera is installed on the side of the vehicle, and an image of the rear of the side of the vehicle captured by the side camera is displayed on the display. There is the following method: in the image captured by the side camera, a part of the vehicle body is included in the same manner as the image of an optical side mirror, but an area that becomes a blind spot due to the vehicle body is synthesized with an image of a rear camera so that the vehicle body appears transparent (hereinafter, referred to as a see-through view).

[0003] As shown in Patent Document 1 Figure 1 when synthesizing an image of a side camera and an image of a rear camera, a first field-of-view transformation process is performed on a right camera image captured by a right camera 3R and a reference vehicle body masking image of the right camera 3R to generate a field-of-view transformed image R and a field-of-view transformed vehicle body masking image, a second field-of-view transformation process is performed on a rear camera image captured by a rear camera 3B to generate a field-of-view transformed image BR, and based on the pixel values of the field-of-view transformed vehicle body masking image, the field-of-view transformed image R and the field-of-view transformed image BR are synthesized. In this way, a rear camera image captured by the rear camera 3B is synthesized with the vehicle body area of the field-of-view transformed image R to generate a see-through view that complements the blind spot caused by the vehicle body. In addition, in Patent Document 1, a distance sensor such as an ultrasonic wave or a laser is used to calculate the position of a projection plane behind the own vehicle, and the image is synthesized using the projection plane.

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2015-74436 Summary of the Invention

[0007] Problems to be Solved by the Invention

[0008] The schematic procedure of a conventional method for generating a see-through view is as Figure 2 shown. Figure 2 (A) of shows an image of the rear of the side of the vehicle captured by a side camera, Figure 2(B) represents an image of the rear of the vehicle taken by a rear camera with a fish-eye lens. Figure 2 (C) represents detecting a rear vehicle through recognition processing of an image taken by the rear camera and calculating the distance to the position of the rear vehicle. Figure 2 (D) represents setting a virtual plane (projection plane) according to the calculated distance and projecting the image taken by the rear camera onto the virtual plane. Figure 2 (E) represents the synthesis of the image of the side camera and the projected image.

[0009] The rear camera is usually installed at the rear of the vehicle, and the rear image taken by the rear camera is used for parking assistance or obstacle detection when the vehicle is reversing. Therefore, the rear camera uses a fish-eye lens with a large field of view (for example, the field of view is 180 degrees or more) so that it can capture the road surface near the rear end of the vehicle. On the other hand, the side camera is installed on the side of the vehicle near the door mirror, and its field of view is smaller than that of the rear camera (for example, 90 degrees or less) so that it can capture the distance behind the side of the vehicle.

[0010] Figure 3 Schematically shows the shooting range when a fish-eye camera with a large viewable angle S1 is used for the rear camera and the shooting range when a narrow-angle camera with a narrow viewable angle S2 is used. P is a projection plane set to generate a perspective image, and the image taken by the side camera and the image taken by the rear camera are coordinate-transformed so as to be projected onto the projection plane P. The projection plane P changes dynamically according to the position of the rear vehicle 10. If the rear vehicle 10 approaches the host vehicle M, the image quality improves in the fish-eye camera. On the other hand, there is a tendency for the blind spot to become larger in the narrow-angle camera.

[0011] When a fish-eye camera is used for the rear camera, due to the wide field of view, it can capture the road surface directly below near the rear end of the vehicle. That is, even if the rear vehicle 10 approaches the host vehicle M, the rear vehicle 10 can be captured without omission, and the blind spot (the range that cannot be captured) is small. However, due to the wide field of view, the image quality is low and the aberration is large. Therefore, if the distance to the rear vehicle 10 increases, its image becomes unclear and the distortion becomes larger. In contrast, when a narrow-angle camera is used for the rear camera, due to the narrow field of view, the image quality is high and the aberration is small. Therefore, even if the distance to the rear vehicle 10 is large, its image is clear. However, if the rear vehicle 10 approaches the host vehicle M, the blind spot that cannot be captured by the narrow-angle camera becomes larger. Thus, the fish-eye camera and the narrow-angle camera each have their advantages and disadvantages.

[0012] Figure 4 (A) is a display example when a perspective image is generated by using a fish-eye camera for the rear camera. In this case, the perspective part of the vehicle body area of the image of the side camera, that is, the part synthesized by the image of the fish-eye camera, has a reduced image quality.Figure 4 Figure (B) shows an example of display when a narrow-angle camera is used for the rear camera to generate a perspective image. In this case, the image quality of the perspective part is good, but if the rear vehicle 10 approaches, a blind spot area BL is generated. In Figure 4 Figure (B), the blind spot areas BL at three locations, i.e., the upper side, the side part, and the lower side of the rear vehicle, are shown. Thus, in the generation of a conventional perspective image, there is a problem that a clear image cannot be obtained when an image captured by a side camera is synthesized with an image captured by a fish-eye camera or a narrow-angle camera.

[0013] The present invention solves such a conventional problem, and an object thereof is to provide an image processing apparatus, an image processing method, and an image processing program capable of making the synthesis of images captured by a plurality of imaging components clear.

[0014] Means for Solving the Problem

[0015] The image processing apparatus according to the present invention includes: a first imaging component mounted on a side part of a vehicle, providing a first image that captures a rear part of the side including a vehicle body area; a second imaging component mounted on a rear part of the vehicle, providing a second image that includes a rear of the vehicle captured at a first field of view angle; a third imaging component mounted on the rear part of the vehicle, providing a third image that includes a rear of the vehicle captured at a second field of view angle narrower than the first field of view angle; and a synthesis component capable of synthesizing the second image and the third image corresponding to the vehicle body area of the first image with the vehicle body area.

[0016] In one embodiment, the synthesizing component supplements and completes the blind spot area of the third image with the second image. In one embodiment, when the blind spot area of the third image meets a predetermined condition, the synthesizing component supplements and completes the blind spot area with the second image. In one embodiment, when the blind spot area of the third image does not meet the predetermined condition, the synthesizing component does not supplement and complete the blind spot area with the second image but synthesizes only the second image for the vehicle body area of the first image. In one embodiment, the predetermined condition specifies the position where the blind spot area is generated, the number of blind spot areas generated, the size of the blind spot areas generated, or the distance from the vehicle to the rear object. In one embodiment, when the blind spot area of the third image is generated only below the rear object, the synthesizing component supplements and completes the blind spot area with the second image. In one embodiment, when the blind spot area of the third image is generated not only below the rear object, the synthesizing component does not supplement and complete the blind spot area with the second image but synthesizes only the second image for the vehicle body area of the first image. In one embodiment, the synthesizing component supplements and completes the blind spot area with the second image in response to a dynamic change in the blind spot area of the third image, or does not supplement and complete the blind spot area with the second image but synthesizes only the second image for the vehicle body area of the first image. In one embodiment, when migrating from the synthesis using the second image and the third image to the synthesis using only the second image, the synthesizing component controls the image synthesis so that the transmittance of the second image gradually increases in the overlapping area of the second image and the third image. In one embodiment, the image processing device further includes a calculating component that calculates the distance to the rear object. The synthesizing component sets a projection plane based on the calculated distance, transforms the coordinates of the first image, the second image, and the third image to the set projection plane, and synthesizes the transformed second image and third image for the vehicle body area of the transformed first image. In one embodiment, the calculating component calculates the distance to the rear object based on the parallax when performing stereoscopic vision on the second image and the third image. In one embodiment, the calculating component calculates the distance to the rear object through a distance sensor such as a radar or LiDAR (Light Detection and Ranging). In one embodiment, the second image can be used as an image for a rear view when the vehicle is reversing, and the third image can be used as an image for an interior mirror. In one embodiment, the image processing device further includes a display component that displays the image synthesized by the synthesizing component. In one embodiment, the second imaging component includes a fish-eye lens with a wide-angle shooting range, and the third imaging component includes a narrow-angle lens with a relatively narrow shooting range compared to the fish-eye lens.

[0017] The image processing method according to the present invention is used in an image processing apparatus, and the image processing apparatus includes: a first photographing component, which is installed on a side portion of a vehicle and provides a first image that photographs a rear side portion including a vehicle body area; a second photographing component, which is installed on a rear portion of the vehicle and provides a second image that includes the rear of the vehicle photographed at a first field of view angle; and a third photographing component, which is installed on the rear portion of the vehicle and provides a third image that includes the rear of the vehicle photographed at a second field of view angle narrower than the first field of view angle. The image processing method includes: a synthesizing step capable of synthesizing, with respect to the vehicle body area of the first image, the second image and the third image corresponding to the vehicle body area.

[0018] In one embodiment, when a blind spot area of the third image satisfies a predetermined condition, the synthesizing step supplements and perfects the blind spot area with the second image. In one embodiment, when the blind spot area of the third image does not satisfy the predetermined condition, the synthesizing step does not supplement and perfect the blind spot area with the second image and only synthesizes the second image with respect to the vehicle body area of the first image.

[0019] The image processing program according to the present invention is executed by an image processing apparatus, and the image processing apparatus includes: a first photographing component, which is installed on a side portion of a vehicle and provides a first image that photographs a rear side portion including a vehicle body area; a second photographing component, which is installed on a rear portion of the vehicle and provides a second image that includes the rear of the vehicle photographed at a first field of view angle; and a third photographing component, which is installed on the rear portion of the vehicle and provides a third image that includes the rear of the vehicle photographed at a second field of view angle narrower than the first field of view angle. The image processing program includes: a synthesizing step capable of synthesizing, with respect to the vehicle body area of the first image, the second image and the third image corresponding to the vehicle body area.

[0020] In one embodiment, when a blind spot area of the third image satisfies a predetermined condition, the synthesizing step supplements and perfects the blind spot area with the second image. In one embodiment, when the blind spot area of the third image does not satisfy the predetermined condition, the synthesizing step does not supplement and perfect the blind spot area with the second image and only synthesizes the second image with respect to the vehicle body area of the first image.

[0021] Advantages of the Invention

[0022] According to the present invention, by being able to synthesize the second image photographed at the first field of view angle and the third image photographed at the second field of view angle narrower than the first field of view angle into the vehicle body area of the first image, the respective advantages of the image photographed at the first field of view angle and the image photographed at the second field of view angle (that is, the wide shooting range of the image photographed at the first field of view angle and the high image quality of the image photographed at the second field of view angle) can be utilized for the synthesized image, and a clear synthesized image can be obtained. Description of the Drawings

[0023] Figure 1 It is a diagram showing an example of synthesizing an image captured by a rear camera with an image captured by a side camera in the past.

[0024] Figure 2 It is a diagram explaining a method for generating a perspective image in the past.

[0025] Figure 3 It is a schematic diagram of the field of view when a fish-eye camera and a narrow-angle camera are used for the rear camera.

[0026] Figure 4 (A) of is a display example of a perspective image when a fish-eye camera is used, Figure 4 and (B) of is a display example of a perspective image when a narrow-angle camera is used.

[0027] Figure 5 It is a block diagram showing the structure of an image processing apparatus according to an embodiment of the present invention.

[0028] Figure 6 It is a top view showing an installation example of a photographing camera according to an embodiment of the present invention.

[0029] Figure 7 It is a flowchart explaining the generation operation of a perspective image according to an embodiment of the present invention.

[0030] Figure 8 It is a diagram showing an example of generating a perspective image by supplementing and perfecting a blind spot area of an image of a narrow-angle camera with an image of a fish-eye camera and synthesizing it with an image of a side camera according to an embodiment of the present invention.

[0031] Figure 9 It is a diagram showing an example of generating a perspective image by synthesizing only an image of a fish-eye camera with an image of a side camera according to an embodiment of the present invention.

[0032] Figure 10 It is a diagram explaining a method for synthesizing an image of a narrow-angle camera and an image of a fish-eye camera according to a modified example of the present invention.

[0033] Reference Signs Explanation

[0034] 100: Image Processing Apparatus

[0035] 110L, 110R: Photographing Cameras (Side Cameras)

[0036] 120: Photographing Camera (Fish-Eye Camera)

[0037] 130: Photographing Camera (Narrow-Angle Camera)

[0038] 140: Control Unit

[0039] 150: Distance Calculation Unit

[0040] 160: Display unit Detailed implementation mode

[0041] The image processing apparatus according to the present invention has a function of synthesizing images captured by a camera mounted on a moving body or the like. The synthesized image is used for a CMS (Camera Monitor System) and is displayed on a display. In a preferred mode, the image processing apparatus synthesizes an image captured by a rear camera for an area that becomes a blind spot of the vehicle body in the image captured by a side camera, and generates a perspective image.

[0042] Embodiment

[0043] Figure 5 FIG. is a block diagram showing the structure of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus 100 of the present embodiment includes: a plurality of imaging cameras 110L, 110R, 120, 130, which are mounted on a vehicle; a control unit 140, which acquires images captured by these imaging cameras and performs various controls including image processing; a distance calculation unit 150, which calculates the distance from the host vehicle to an object behind (for example, a vehicle behind or the like); and a display unit 160, which displays an image obtained by image processing by the control unit 140. The image processing apparatus of the present embodiment can also perform operations in cooperation with in-vehicle devices (for example, electronic devices having audio-visual / navigation functions or the like) mounted on the vehicle.

[0044] Figure 6 FIG. is a top view of a vehicle showing an installation example of an imaging camera. The imaging cameras 110 to 130 include, for example, imaging elements such as CCD sensors or CMOS sensors and imaging lenses. The side cameras 110L and 110R are side cameras that replace side mirrors, and are respectively mounted inside or near the left and right side mirrors of the vehicle M, or replace the side mirrors (hereinafter, referred to as side cameras 110L and 110R). The field of view angle θ1 of the imaging lenses of the side cameras 110L and 110R, that is, the imaging range is, for example, 90 degrees or less from the optical axis. In the image projected onto an optical side mirror, the side portion of the vehicle body is projected in order to easily grasp the positional relationship between the vehicle M and surrounding objects. Therefore, the optical axes of the side cameras 110L and 110R are adjusted so that at least a part of the side portion of the vehicle body is imaged in the same manner as the optical side mirror.

[0045] At the rear of the vehicle M, two imaging cameras 120 and 130 are installed. The imaging camera 120 is installed approximately at the center of the left and right sides of the rear of the vehicle M, for example, near the height of the license plate or the bumper. The imaging camera 120 is a fish-eye lens with a relatively wide viewing angle, and its field of view angle θ2 is about 180 degrees or larger (hereinafter, referred to as a fish-eye camera). The fish-eye camera 120 captures an image of the rear of the vehicle M in a wide range, and the captured image includes the road surface at the rear end of the vehicle M. The image captured by the fish-eye lens 120 is displayed as a rear image on the display unit 160 when the vehicle M is reversing (that is, when the gear is in reverse). The driver visually recognizes the rear image while confirming obstacles behind or performing parking. In addition, as will be described later, the image captured by the fish-eye camera 120 is also used for synthesizing the images captured by the side cameras 110L and 110R.

[0046] Another imaging camera 130 is installed at approximately the same height as the fish-eye camera 120 at the rear of the vehicle M and at a position away from the fish-eye camera 120 by a certain distance. The imaging camera 130 includes an imaging lens with a smaller field of view angle than the fish-eye lens, and its field of view angle θ3 is about 90 degrees or less (hereinafter, referred to as a narrow-angle camera). The narrow-angle camera 130 can be used as an alternative to an interior mirror (room mirror). In this case, the rear of the vehicle M is captured in the same shooting range as the image reflected by the interior mirror. The image captured by the narrow-angle camera 130 is displayed as a rear image on the display unit 160 when the vehicle is moving forward or the like. In addition, the narrow-angle camera 130 is also used together with the fish-eye camera 120 for synthesizing the images captured by the side cameras 110L and 110R.

[0047] The control unit 140 acquires the images captured by the side cameras 110L and 110R, the fish-eye camera 120, and the narrow-angle camera 130, processes the acquired images, and displays them on the display unit 160. The control unit 140 sets the projection plane P based on the distance to the object behind calculated by the distance calculation unit 150 (refer to Figure 3) On the projection plane P, the images captured by the cameras 110 to 130 with different installation positions or field angles are respectively projected, thereby performing coordinate transformation of these images. The control unit 140 calculates the correspondence relationship between the pixels of the transformed images of the side cameras 110L, 110R, the fisheye camera 120, and the narrow-angle camera 130 in the projection plane P. When generating a perspective image, the control unit 140 identifies or extracts the images of the fisheye camera 120 and the narrow-angle camera 130 corresponding to the image of the vehicle body area in the images of the side cameras 110L, 110R on the projection plane P, and synthesizes the identified or extracted images into the vehicle body area. Furthermore, the control unit 140 calculates which area in the images of the side cameras 110L, 110R on the projection plane P corresponds to the vehicle body area according to the installation position of the side cameras, the direction of the optical axis, the shooting range, the vehicle body shape, etc.

[0048] The control unit 140 is, for example, a processor for performing image processing, including a microcomputer, ROM / RAM, etc. In a preferred example, the control unit 140 executes an image processing program stored in the ROM / RAM to control the hardware and perform the image processing required for generating images such as perspective images or rear images.

[0049] The distance calculation unit 150 calculates the distance from the host vehicle M to the object behind. The calculation is performed, for example, at a certain cycle. The object is, for example, a rear vehicle traveling behind the host vehicle, or an obstacle or road existing behind the host vehicle. The method for calculating the distance is, for example, to use a distance sensor such as a radar or LiDAR (laser radar) to calculate the distance to the object, or to perform stereo vision on the two images of the fisheye camera 120 and the narrow-angle camera 130 to calculate the distance to the object. In the latter case, the distance calculation unit 150 performs stereo vision on the two images transformed onto the projection plane P and calculates the distance to the object based on the result. Each time the distance calculation unit 150 calculates the distance, it provides it to the control unit 140. The control unit 140 sets the projection plane P for generating a perspective image based on the distance obtained from the distance calculation unit 150. The position of this projection plane P changes each time according to the distance to the object.

[0050] The display unit 160 acquires the image data processed by the control unit 140 and displays it. The display unit 160 includes one or more of display media such as a liquid crystal display, an organic EL display, and a projection device (HUD). The installation position of the display unit 160 is not particularly limited. For example, it is a liquid crystal display module in the instrument panel, a display for in-vehicle devices, or an image projected onto the front windshield or the like. In a preferred example, the display unit 160 displays an image of the rear side of the vehicle captured by the side mirrors 110L and 110R (perspective image display), or an image of the rear of the vehicle captured by the fisheye camera 120 (rear image display), or an image of the rear of the vehicle captured by the narrow-angle camera 130 (interior mirror image display).

[0051] Next, the generation of the perspective image in this embodiment will be described. Figure 7 This is a flowchart showing the generation operation of the perspective image in this embodiment. The generation of the perspective image is common to the side cameras 110L and 110R. Therefore, here, the side camera 110L will be described.

[0052] The control unit 140 acquires the image captured by the side camera 110L, the image captured by the fisheye camera 120, and the image captured by the narrow-angle camera 130 (S100). In addition, the distance calculation unit 150 calculates the distance from the rear of the vehicle to the object (S110) and provides the calculated distance to the control unit 140.

[0053] When the control unit 140 acquires the distance from the distance calculation unit 150, it sets the projection plane P based on the distance, and transforms the coordinates of the images captured by the side camera 110L, the fisheye camera 120, and the narrow-angle camera 130 on the projection plane P (S120).

[0054] Next, when the control unit 140 synthesizes the image of the narrow-angle camera 130 with the vehicle body area of the image of the side camera 110L, it determines whether a blind spot is generated in the image of the narrow-angle camera 130 (S130). For example, the field of view angle θ3 (or the shooting range) of the narrow-angle camera 130 is about 90 degrees, and it is installed near the bumper of the vehicle. If the distance from the vehicle to the rear object becomes, for example, about 1.4 m or so, a blind spot where the entire rear object cannot be photographed will be generated. In the projection plane P, the control unit 140 calculates the coordinate space of the vehicle body area of the side mirror 110L. If all the pixels corresponding to this coordinate space can be photographed by the narrow-angle camera 130, it is determined that no blind spot is generated. If not, it is determined that a blind spot is generated.

[0055] When there is no blind spot in the image of the narrow-angle camera 130 to be synthesized into the vehicle body area, the control unit 140 synthesizes the image of the narrow-angle camera 130 into the vehicle body area of the image of the side mirror 110L (S150). Since the vehicle body area is synthesized with the high-quality image of the narrow-angle camera 130, the perspective image can be displayed more clearly.

[0056] On the other hand, when a blind spot occurs in the image of the narrow-angle camera 130 to be synthesized, the control unit 140 determines whether the blind spot meets a pre-determined condition (S160). The pre-determined condition refers to the condition for determining whether to supplement the blind spot area of the image of the narrow-angle camera 130 with the image of the fisheye camera 120. For example, if the blind spot area is in a relatively inconspicuous position, size, or number, even if it is supplemented with the low-quality image of the fisheye camera 120, it will not have a great impact on the clarity of the image. On the other hand, when the blind spot area overlaps with a part of the vehicle behind, or is relatively large in size, or multiple blind spots occur, if it is supplemented with the image of the fisheye camera 120, the image will become unclear instead. In this case, rather than supplementing the blind spot area with the image of the fisheye camera 120, it is better not to use the image of the narrow-angle camera 130 and only synthesize the image of the fisheye camera 120 to make the synthesized image clear.

[0057] The pre-determined condition can also be specified as the position, size, and number of the blind spot area as described above. Or, since the generation of the blind spot area is a function of the distance calculated by the distance calculation unit 150, the distance can also be set as a condition. In this case, if the distance from the vehicle itself to the object behind is equal to or greater than the threshold, it is determined to meet the condition; if it is less than the threshold, it is determined not to meet the condition.

[0058] When the blind spot meets the pre-determined condition (S160), the control unit 140 supplements the blind spot area of the narrow-angle camera 130 with the corresponding image of the fisheye camera 120, and synthesizes the supplemented image of the narrow-angle camera 130 and the image of the fisheye camera 120 into the vehicle body area of the image of the side camera 110L (S170). In the projection plane P, the corresponding relationship between the pixels of the narrow-angle camera 130 and the fisheye camera 120 is known, so the image of the fisheye camera 130 corresponding to the blind spot area is extracted or cut out and synthesized into the blind spot area.

[0059] For example, supplementing the blind spot area with the image of the fisheye camera 130 Figure 8As shown. In this example, the blind spot area BL of the image of the narrow-angle camera is generated in an inconspicuous part on the lower side of the vehicle behind. Therefore, the control unit 140 determines that the blind spot area meets the conditions, extracts or cuts out the image corresponding to the blind spot area BL from the fisheye camera 130, and synthesizes it with the blind spot area BL to supplement and perfect the blind spot area BL. Then, the image after the blind spot area BL is supplemented and perfected is synthesized into the vehicle body area of the image of the side camera 110L, and a perspective image is generated.

[0060] "Q" in the perspective image represents the boundary of the vehicle body area of the side camera, and "T" represents the boundary between the image of the narrow-angle camera 130 and the image of the fisheye camera 120. This boundary T is near the vehicle behind and the road surface, so it is not very conspicuous. In addition, due to the synthesis of the image of the narrow-angle camera 130, an image quality similar to that of the image of the side camera 110L can be obtained, and the blind spot area is supplemented and perfected by the image of the fisheye camera 130, so a relatively clear and high-quality perspective image can be obtained.

[0061] On the other hand, when the blind spot does not meet the pre-determined conditions (S160), the control unit 140 does not supplement and perfect the blind spot area of the narrow-angle camera 130 with the image of the fisheye camera 120, and only synthesizes the image of the fisheye camera 120 into the vehicle body area of the image of the side camera 110L (S180). The situation of this synthesis is as Figure 9 shown. Here, in the image of the narrow-angle camera 130, blind spot areas BL are generated in multiple parts on the lower side, side, and upper side of the vehicle behind. In addition, the blind spot area on the lower side is so large that it overlaps with a part of the vehicle behind. Therefore, the control unit 140 determines that the blind spot area does not meet the conditions, does not supplement and perfect the blind spot area BL with the image of the fisheye camera 120, but synthesizes the image of the fisheye camera 120 into the vehicle body area of the image of the side camera 110L to generate a perspective image. It should be noted here that in the perspective image, the blind spot area is replaced with the image of the fisheye camera before it appears. In this way, although the use of the image of the fisheye camera 130 causes some reduction in image quality, it can prevent the display of blind spot areas in the perspective image when the vehicle behind approaches.

[0062] Figure 7 The steps shown are repeatedly executed during the driving (forward or backward) or parking of the vehicle, and a perspective image is continuously generated. In addition, if the distance to the object behind calculated by the distance calculation unit 150 changes, the projection plane P changes, and accordingly, the generation of the blind spot area changes dynamically, and the image to be synthesized into the vehicle body area of the side camera image is appropriately switched to the image of the fisheye camera 120 and / or the narrow-angle camera 130.

[0063] Thus, according to this embodiment, the image of the narrow-angle camera with the alternative function of the interior mirror and the image of the fish-eye camera used for the rear image can be synthesized into the image of the side camera, so that a clear perspective image that makes use of the advantages of the narrow-angle camera and the fish-eye camera can be generated.

[0064] Next, a modified example of the present invention will be described. As described above, the blind spot area generated in the image of the narrow-angle camera 130 changes dynamically according to the distance, that is, the projection plane, calculated by the distance calculation unit 150. Thus, the image for synthesis is transferred from the images of the narrow-angle camera 130 and the fish-eye camera 120 to the image of only the fish-eye camera 120. In order to make the switching of the images during this transfer smooth, in the modified example, the synthesis of the two images is controlled so that in the overlapping range of the fish-eye camera 120 and the narrow-angle camera 130, the transmittance of the image of the fish-eye camera 120 gradually increases, or the transmittance of the image of the narrow-angle camera 130 gradually decreases in this determination.

[0065] The situation of this image synthesis is as Figure 10 shown. It is assumed that at time T1, the blind spot area of the narrow-angle camera 130 is supplemented and completed by the image of the fish-eye camera 120, and at time T5, the blind spot area is replaced by the image of the fish-eye camera 120. The control unit 140 synthesizes the transmitted image of the narrow-angle camera and the image of the fish-eye camera at a ratio of 7:3, for example, at time T2, at a ratio of 5:5 at time T3, and at a ratio of 3:7 at time T4. By performing such synthesis, it is possible to prevent the image quality from changing drastically.

[0066] As described above, the preferred embodiments of the present invention have been described in detail, but the present invention is not limited to specific embodiments, and various modifications and changes can be made within the scope of the gist of the invention described in the claims.

Claims

1. An image processing device, wherein, it has: a first imaging component, which is installed on the side of a vehicle and provides a first image that captures the rear of the side including the vehicle body area; a second imaging component, which is installed at the rear of the vehicle and provides a second image that includes the rear of the vehicle captured at a first field of view angle; a third imaging component, which is installed at the rear of the vehicle and provides a third image that includes the rear of the vehicle captured at a second field of view angle narrower than the first field of view angle; and a synthesizing component, which is capable of synthesizing the second image and the third image corresponding to the vehicle body area of the first image for the vehicle body area of the first image, when a blind spot area of the third image meets a predetermined condition, the synthesizing component supplements and perfects the blind spot area with the second image, when the blind spot area of the third image does not meet the predetermined condition, the synthesizing component does not supplement and perfect the blind spot area with the second image, but only synthesizes the second image for the vehicle body area of the first image.

2. The image processing device according to claim 1, wherein, the predetermined condition specifies the position where the blind spot area is generated.

3. The image processing device according to claim 1, wherein, the predetermined condition specifies the distance from the vehicle to a rear object.

4. The image processing device according to claim 1, wherein, when the blind spot area of the third image is generated only on the lower side of a rear object, the synthesizing component supplements and perfects the blind spot area with the second image.

5. An image processing device, wherein, it has: a first imaging component, which is installed on the side of a vehicle and provides a first image that captures the rear of the side including the vehicle body area; a second imaging component, which is installed at the rear of the vehicle and provides a second image that includes the rear of the vehicle captured at a first field of view angle; a third imaging component, which is installed at the rear of the vehicle and provides a third image that includes the rear of the vehicle captured at a second field of view angle narrower than the first field of view angle; and a synthesizing component, which is capable of synthesizing the second image and the third image corresponding to the vehicle body area of the first image for the vehicle body area of the first image, when a blind spot area of the third image meets a predetermined condition, the synthesizing component supplements and perfects the blind spot area with the second image, the predetermined condition specifies the number of blind spot areas generated.

6. An image processing device, wherein, it has: a first imaging component, which is installed on the side of a vehicle and provides a first image that captures the rear of the side including the vehicle body area; a second imaging component, which is installed at the rear of the vehicle and provides a second image that includes the rear of the vehicle captured at a first field of view angle; a third imaging component, which is installed at the rear of the vehicle and provides a third image that includes the rear of the vehicle captured at a second field of view angle narrower than the first field of view angle; and a synthesizing component, which is capable of synthesizing the second image and the third image corresponding to the vehicle body area of the first image for the vehicle body area of the first image, when a blind spot area of the third image meets a predetermined condition, the synthesizing component supplements and perfects the blind spot area with the second image, the predetermined condition specifies the size of the blind spot area generated.

7. An image processing method for an image processing device, the image processing device includes: A first imaging component, which is installed on the side of a vehicle and provides a first image that captures the rear of the side including the vehicle body area; A second imaging component, which is installed at the rear of the vehicle and provides a second image that includes the rear of the vehicle captured at a first field of view angle; and A third imaging component, which is installed at the rear of the vehicle and provides a third image that includes the rear of the vehicle captured at a second field of view angle narrower than the first field of view angle, The image processing method comprises: A synthesizing step capable of synthesizing, for the vehicle body area of the first image, the second image and the third image corresponding to the vehicle body area, In the synthesizing step, when a blind spot area of the third image meets a predetermined condition, the blind spot area is supplemented and perfected with the second image, when a blind spot area of the third image does not meet the predetermined condition, the blind spot area is not supplemented and perfected with the second image, and only the second image is synthesized for the vehicle body area of the first image.

8. An image processing method for an image processing device, The image processing device comprises: A first imaging component, which is installed on the side of a vehicle and provides a first image that captures the rear of the side including the vehicle body area; A second imaging component, which is installed at the rear of the vehicle and provides a second image that includes the rear of the vehicle captured at a first field of view angle; and A third imaging component, which is installed at the rear of the vehicle and provides a third image that includes the rear of the vehicle captured at a second field of view angle narrower than the first field of view angle, The image processing method comprises: A synthesizing step capable of synthesizing, for the vehicle body area of the first image, the second image and the third image corresponding to the vehicle body area, In the synthesizing step, when a blind spot area of the third image meets a predetermined condition, the blind spot area is supplemented and perfected with the second image, The predetermined condition defines the number of blind spot areas generated.

9. An image processing method for an image processing device, The image processing device comprises: A first imaging component, which is installed on the side of a vehicle and provides a first image that captures the rear of the side including the vehicle body area; A second imaging component, which is installed at the rear of the vehicle and provides a second image that includes the rear of the vehicle captured at a first field of view angle; and A third imaging component, which is installed at the rear of the vehicle and provides a third image that includes the rear of the vehicle captured at a second field of view angle narrower than the first field of view angle, The image processing method comprises: A synthesizing step capable of synthesizing, for the vehicle body area of the first image, the second image and the third image corresponding to the vehicle body area, In the synthesizing step, when a blind spot area of the third image meets a predetermined condition, the blind spot area is supplemented and perfected with the second image, The predetermined condition defines the size of the blind spot areas generated.

Images