Image display apparatus and image display method

Abstract

To effectively display a captured image on a display medium while eliminating distortion in the captured image.SOLUTION: A vehicle display apparatus uses a projection screen 50 when displaying a display image corresponding to a captured image from a camera on a monitor. The projection screen 50 is composed of a central screen part 52, a side screen part 54, and a lower screen part 56. The side screen part 54 and the lower screen part 56 are tilted and made to face the monitor such that the central screen part 52 is positioned on a back side. This configuration enables an image obtained by vertically compressing the captured image to be displayed on the monitor, and thereby, enables the image to be effectively displayed on the monitor while eliminating distortion in the captured image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image display device and an image display method. [Background technology]

[0002] Patent Document 1 describes a method for converting a camera image of the surroundings of a vehicle captured on an image sensor through a lens such as a fisheye lens into a display image projected substantially parallel to a projection plane parallel to the image sensor plane, and displaying the image on a display device for viewing by the driver. In Patent Document 1, a virtual plane defined at a predetermined distance from the vehicle camera is divided into multiple virtual pixels, the projection of each virtual pixel through the lens onto a camera image including multiple camera pixels is calculated, and the display image is generated using display pixels generated based on camera pixels present in the vicinity of the projection of each virtual pixel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-504414 Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, in-vehicle cameras that capture a wide range of images around a vehicle use fisheye lenses, which produce distorted images. When this distortion is eliminated by projecting the image onto a flat surface or converting it into a pinhole model image, the wider the display angle, the more extreme the stretching of the image's peripheral edges, resulting in a smaller image displayed near the center. Furthermore, when an image captured by an in-vehicle camera is displayed on a display medium (display area) with a shorter vertical dimension than horizontal dimension, lines corrected to straight lines after distortion removal are distorted again. For this reason, there is room for improvement in the effective display of images captured using wide-angle lenses such as fisheye lenses.

[0005] The present invention has been made in consideration of the above-mentioned facts, and aims to provide an image display device and an image display method that can effectively display an image on a display medium by reducing the display size, particularly in the vertical direction, while eliminating distortion while maintaining a wide angle of view in the horizontal and vertical directions (vertical direction) of an image captured using a wide-angle lens such as a fisheye lens. [Means for solving the problem]

[0006] A first aspect of an image display device for achieving the above-mentioned object includes an imaging means provided in a vehicle for capturing an image of the outside of the vehicle from the vehicle body side through a wide-angle lens with a field of view of 180° or more; a display medium arranged with its vertical direction being the up-down direction and its horizontal direction being the horizontal direction, and having an area that is longer in the horizontal direction than the vertical direction as a display area for an image captured by the imaging means; and a display control unit that generates a compressed display image by aligning the horizontal direction with the horizontal direction of the display area of ​​the display medium and the vertical direction with the vertical direction of the display medium for an image area in which the vertical field of view along the up-down direction in the image captured by the imaging means is narrower than the horizontal field of view along the horizontal direction, and displays the generated display image in the display area.

[0007] In the second aspect of the image display device, in the first aspect, the display control unit is provided with left and right side screen sections that correspond to the left and right sides of the captured image from the horizontal center position and have a vertical dimension narrower toward the horizontal center position than at both horizontal ends, and a lower screen section whose upper side is in contact with the lower sides of the left and right side screen sections and that spans the left and right side screen sections across the entire horizontal area of ​​the captured image, and uses a screen in which each of the side screen sections and the lower screen section is inclined so that the central positions of the side screen sections are away from the captured image, and generates a projected image in which the captured image is projected radially onto the screen from a required center point as the display image.

[0008] The image display device of the third aspect includes an imaging means provided in a vehicle and configured to capture an image of the outside of the vehicle body from the vehicle body side through a wide-angle lens with a field angle of 180° or more; a display medium arranged with a vertical direction as the up-down direction and a horizontal direction as the horizontal direction, in which an area longer in the horizontal direction than the vertical direction is used as a display area for an image captured by the imaging means; left and right side screen portions each corresponding to a left and right side portion of the image area in which the vertical field angle along the up-down direction of the image captured by the imaging means is narrower than the horizontal field angle along the horizontal direction, and in which the vertical dimension on the horizontal center position side is narrower than both ends in the horizontal direction; and a display control unit that uses a screen having a lower screen section that is in contact with the underside of the left and right side screen sections and that spans the left and right side screen sections across the entire horizontal area of ​​the captured image, with the side screen sections and the lower screen section each inclined so that the center position of the side screen section is away from the captured image, that generates a compressed display image by aligning the horizontal direction of a projected image, formed by projecting the captured image radially from a required center point onto the screen, with the horizontal direction aligned with the horizontal direction of the display area of ​​the display medium and the vertical direction aligned with the vertical direction of the display medium, and that displays the generated display image in the display area.

[0009] The image display device of the fourth aspect is the image display device of the second or third aspect, wherein the screen is provided with a central screen section between the left and right side screen sections, connecting the horizontal center positions of the two side screen sections.

[0010] The image display device of a fifth aspect is the fourth aspect, wherein the central screen portion is curved toward the back when viewed from the captured image side.

[0011] An image display device of a sixth aspect is any one of the second to fifth aspects, wherein the display control unit generates a display image in which the horizontal angle of view is 150° or more as the display image.

[0012] A seventh aspect of the image display device is any one of the second to sixth aspects, wherein the imaging means is installed at the rear of the vehicle and captures an image of the area behind the vehicle at a predetermined dip angle, and the display control unit controls the up and down direction of the screen so that an image tilted at an angle corresponding to the dip angle relative to the vertical direction is displayed on the display medium.

[0013] An eighth aspect of the image display device is any one of the first to seventh aspects, wherein the imaging means captures an image including a part of the vehicle body, and the display control unit generates the display image including the captured part of the vehicle body.

[0014] In a ninth aspect of the image display method, when an image captured by an imaging means provided in a vehicle and capturing an image of the outside of the vehicle body from the vehicle body side through a wide-angle lens is displayed on a display medium that is arranged with the vertical direction being the up-down direction and the horizontal direction being the horizontal direction, and an area that is longer in the horizontal direction than the vertical direction is the display area for the image captured by the imaging means, a display image is generated in which the horizontal direction is aligned with the horizontal direction of the display area of ​​the display medium and the vertical direction is aligned with the vertical direction of the display medium for an image area in which the vertical angle of view along the up-down direction in the image captured by the imaging means is narrower than the horizontal angle of view along the horizontal direction, and the generated display image is displayed in the display area of ​​the display medium.

[0015] In a tenth aspect of the image display method, when an image captured by an imaging means provided in a vehicle and capturing an image of the outside of the vehicle body through a wide-angle lens from the vehicle body side is displayed on a display medium that is arranged with a vertical direction as the up-down direction and a horizontal direction as the horizontal direction, and an area that is longer in the horizontal direction than the vertical direction is used as a display area for the image captured by the imaging means, left and right side screens that correspond to each of left and right sides of a horizontal center position and have a vertical dimension narrower on the horizontal center position side than on both lateral ends in the horizontal direction in an image area in which a vertical angle of view along the up-down direction in the image captured by the imaging means is narrower than a horizontal angle of view along the horizontal direction, are displayed. The screen includes a screen section and a lower screen section whose upper side is in contact with the lower sides of the left and right side screen sections and which spans the left and right side screen sections across the entire horizontal area of ​​the captured image, and the side screen sections and lower screen sections are each inclined so that the central positions of the side screen sections are away from the captured image.The captured image is projected radially onto the screen from a required center point, and a compressed display image is generated by aligning the horizontal direction of the image with the horizontal direction of the display area of ​​the display medium and the vertical direction with the vertical direction of the display medium, and the generated display image is displayed in the display area. [Effects of the Invention]

[0016] In the image display device of the first aspect, an imaging means is provided on a vehicle, and the imaging means captures an image of the outside of the vehicle from the vehicle body side through a wide-angle lens. The vertical direction of the display medium corresponds to the up-down direction of the captured image, and the horizontal direction of the captured image corresponds to the horizontal direction of the captured image, and the display area for the captured image of the imaging means is an area that is longer in the horizontal direction than in the vertical direction.

[0017] The display control unit generates a compressed display image by aligning the horizontal direction with the horizontal direction of the display area of ​​the display medium and the vertical direction with the vertical direction of the display medium for an image area in which the vertical angle of view along the up-down direction in the image captured by the imaging means is narrower than the horizontal angle of view along the horizontal direction, and displays the generated display image in the display area.

[0018] This allows the photographed image to be displayed on the display medium after being compressed in the vertical direction compared to the horizontal direction, so that the image can be effectively displayed on the display medium by suppressing the vertical size while removing distortion of the photographed image photographed at a wide angle using a wide-angle lens.

[0019] In the image display device of the second aspect and the third aspect, a screen is provided. The screen includes left and right side screen sections that correspond to the left and right sides of the horizontal center position of the captured image and have a vertical dimension narrower toward the horizontal center position than toward both horizontal ends, and a lower screen section whose upper side is in contact with the lower sides of the left and right side screen sections and that spans the left and right side screen sections across the entire horizontal area of ​​the captured image. Furthermore, the side screen sections and the lower screen section are each inclined so that the center positions of the side screen sections are away from the captured image.

[0020] The display control unit projects a desired area of ​​the captured image onto a screen using light that can be considered to radiate from the center point of the camera in a pinhole camera model.

[0021] This allows a captured image to be projected (drawn) onto the screen using a single-point projection method, and the captured image can be displayed on the display medium while effectively removing distortion and reducing the vertical size.

[0022] In the image display device of the fourth aspect, a central screen section is provided between the left and right side screen sections on the screen, connecting the horizontally central sides of the screen sections, thereby preventing the left and right images from bending at the lateral (horizontal) central position of the displayed image, and allowing a high-quality displayed image to be displayed on the display medium.

[0023] In the image display device of the fifth aspect, the central screen portion is curved toward the back when viewed from the captured image side, which allows the center position of the displayed image to have depth in the lateral (horizontal) direction, thereby enabling a higher quality displayed image to be displayed on the display medium.

[0024] In the image display device of the sixth aspect, the display control unit generates a display image having a horizontal angle of view of 150° or more as the display image, thereby making it possible to display the surroundings of the vehicle in a wide range with reduced distortion.

[0025] In the image display device of the seventh aspect, an imaging means installed at the rear of the vehicle captures an image of the area behind the vehicle at a predetermined dip angle. The display control unit generates a display image in which the up-down direction of the screen is tilted relative to the vertical direction at an angle corresponding to the dip angle of the imaging means. This allows the image of the area below the rear of the vehicle captured by the imaging means to be displayed, for example, like an image displayed on an interior mirror.

[0026] In the image display device of the eighth aspect, the imaging means captures an image including a part of the vehicle body, and the display control unit generates a display image including the captured part of the vehicle body. This makes it easy to recognize the area captured by the imaging means, and easily assists the occupant in viewing the area around the vehicle. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a block diagram showing a schematic configuration of a vehicle display device according to an embodiment of the present invention; [Figure 2] FIG. 1 is a plan view showing a schematic configuration of a vehicle. [Figure 3] (A) is a front view of the projection screen, (B) is a top view of the projection screen, (C) is a side view of the projection screen, and (D) and (E) are each top views showing other examples of the projection screen. [Figure 4] 1A and 1B are schematic diagrams showing the projection of a captured image onto a projection screen, where 1A shows the projection screen as seen from the side, and 1B shows the projection screen as seen from above. [Figure 5] (A) and (B) are schematic diagrams showing a side view of the lens, where (A) shows a dip angle β and (B) shows a dip angle of zero. [Figure 6] 10 is a flowchart illustrating an example of a correspondence information generation process. [Figure 7]10 is a flowchart showing an example of a display process using correspondence information. [Figure 8] (A) and (B) are schematic diagrams showing examples of display images, where (A) shows an example in which a captured image is displayed as is, and (B) shows a display image processed by an image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0028] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0029] This embodiment includes a display image generation method in which, when an image captured by an imaging means provided on a vehicle and capturing an image of the outside of the vehicle body from the vehicle body side through a wide-angle lens is displayed on a display medium that is arranged with the vertical direction being the up-down direction and the horizontal direction being the horizontal direction, and an area that is longer in the horizontal direction than the vertical direction is used as a display area for the image captured by the imaging means, the display image is generated by compressing the image area in which the vertical angle of view along the up-down direction in the image captured by the imaging means is narrower than the horizontal angle of view along the horizontal direction, by aligning the horizontal direction with the horizontal direction of the display area of ​​the display medium and the vertical direction with the vertical direction of the display medium.

[0030] In addition, in this embodiment, when an image captured by an imaging means provided in a vehicle and capturing an image of the outside of the vehicle body through a wide-angle lens from the vehicle body side is displayed on a display medium that is arranged with the vertical direction as the up-down direction and the horizontal direction as the horizontal direction, and an area that is longer in the horizontal direction than the vertical direction is used as a display area for the image captured by the imaging means, the image is displayed on left and right side portions that correspond to the left and right sides of the horizontal center position and are narrower in the vertical dimension on the horizontal center position side than on both lateral ends in the horizontal direction in an image area in which the vertical angle of view along the up-down direction in the image captured by the imaging means is narrower than the horizontal angle of view along the horizontal direction. The present invention also includes a display image generation method that uses a screen that includes a screen unit and a lower screen unit whose upper side is in contact with the lower sides of the left and right side screen units and that straddles the left and right side screen units across the entire horizontal area of ​​a captured image, and in which the side screen units and the lower screen unit are each inclined so that the central positions of the side screen units are away from the captured image, and that projects the captured image radially from a required center point onto the screen, aligning the horizontal direction of the image with the horizontal direction of the display area of ​​the display medium and compressing the vertical direction of the image to match the vertical direction of the display medium to generate the display image.

[0031] Fig. 1 shows a schematic configuration diagram of a vehicle display device 10 as an image display device according to this embodiment, and Fig. 2 shows a schematic plan view of a vehicle 12 equipped with the vehicle display device 10, as viewed from above. In the drawing, the front side of the vehicle is indicated by an arrow FR, the vehicle width direction is indicated by an arrow W, and the upward direction is indicated by an arrow UP. In the drawing, the horizontal direction (horizon line or horizontal plane) is indicated by the symbol HP.

[0032] 1, the vehicle display device 10 includes a camera 14 as an imaging means, a monitor 16 as a display medium, and an image processing device 18 configured with a display control unit for displaying a display image corresponding to the image captured by the camera 14 on the monitor 16. The vehicle display device 10 uses the camera 14 to capture an image of the outside of the vehicle 12 (outside the vehicle body), and displays an image (display image) corresponding to the captured image on the monitor 16, thereby assisting the occupant (driver) of the vehicle 12 in visually recognizing the outside of the vehicle. In the following description, the horizontal and vertical directions of the captured image are defined as directions along the horizontal and vertical directions relative to the vehicle, respectively.

[0033] The camera 14 includes a lens 20 and an image sensor 22 serving as an imaging medium, which uses a CCD sensor, a CMOS sensor, or the like as an imaging element. The imaging area of ​​the image sensor 22 is a substantially rectangular area (a substantially square area). In the camera 14, a captured image passes through the lens 20 and is focused on the imaging area of ​​the image sensor 22.

[0034] As shown in Fig. 2, camera 14 is disposed at the rear of vehicle 12 and in the center in the vehicle width direction. A fisheye lens serving as a wide-angle lens is used for lens 20 of camera 14. The angle of view α0 of lens (fisheye lens) 20 may be 170° or greater (for example, 170°≦α0≦220°), and in this embodiment, α0 = 195°. This allows camera 14 to capture an image over a range of angle of view α0 = 195° in the vehicle width direction, and camera 14 captures an image of the rear of vehicle 12 at a wide angle. Note that fisheye lenses are available in a variety of projection methods, such as equidistant projection and stereoscopic projection, and a stereoscopic projection fisheye lens is used for lens 20 of camera 14 according to this embodiment.

[0035] The horizontal angle of view (horizontal angle of view) of camera 14 may be different from the vertical angle of view (vertical angle of view), and when the horizontal angle of view of camera 14 is angle of view α0, the vertical angle of view can be in the range of 3 / 4 to 9 / 16 of the horizontal angle of view (angle of view α0) (for example, approximately 120°).

[0036] 1, the camera 14 is installed on the vehicle 12 so that the optical axis Oa is tilted downward and the dip angle β between the horizontal direction and the optical axis Oa is a required angle. As a result, the image captured by the camera 14 is centered below the horizontal direction, and the captured image includes a bumper (rear bumper) 12A as part of the vehicle body, the road surface behind the vehicle, the space behind the vehicle, etc.

[0037] The monitor 16 is installed on an instrument panel (not shown) of the vehicle 12, and is easily visible to a passenger (driver) seated in the driver's seat of the vehicle 12. The monitor 16 uses, for example, full HD, and has a resolution of 1920 dpi x 1080 dpi and an aspect ratio of 16:9. A display area 16A of the monitor 16 is divided into a camera image area 16B for displaying a display image (video) corresponding to an image (video) captured by the camera 14, and an area other than the camera image area 16B (non-camera image area 16C), and images are displayed in these areas. The width and length of the display surface (display image) of the monitor 16 are horizontal and up-down directions, respectively.

[0038] The camera image area 16B of the monitor 16 is the upper part of the display area 16A, and the monitor 16 can display a rear image captured by the camera 14 while displaying various information, images, videos, etc. in the non-camera image area 16C.

[0039] The camera image area 16B has a required aspect ratio in which the width is longer than the height (for example, a width:height ratio of approximately 16:4.5), which enables the monitor 16 (camera image area 16B) to display an image that is long in the vehicle width direction (for example, a wide image of approximately 1920 dpi x 500 dpi).

[0040] In this embodiment, the camera image area 16B is provided within the display area 16A of the monitor 16, but the display medium may be one that displays the image captured by the camera 14 (camera image) across the entire display area. A display medium that can display the camera image across the entire area has a required aspect ratio (e.g., 16:4.5) and can be placed near the top of the front windshield glass (a position corresponding to the inner mirror) within the vehicle cabin. Note that the camera image area 16B of the monitor 16 only needs to be longer in the horizontal direction than in the vertical direction, and may have an aspect ratio of, for example, 16:9 (full HD).

[0041] Image processing device 18 reads the image captured by camera 14, generates a display image corresponding to the captured image, and displays the display image (camera video) in camera video area 16B of monitor 16. Furthermore, image processing device 18 sets the horizontal angle of view αh of the display image displayed on monitor 16 to be equal to or narrower than the horizontal angle of view α0 of the image captured by camera 14 (αh≦α0). Furthermore, the horizontal angle of view αh of the display image displayed on monitor 16 is set to at least 150° or more (150°≦αh≦α0), preferably 180° or more (180°≦α≦α0), and in this embodiment, the horizontal angle of view αh is set to 180° (αh=180°).

[0042] In addition, in image processing device 18, the vertical angle of view αv of the display image displayed on monitor 16 is set to be narrower than the vertical angle of view of camera 14, and in image processing device 18, camera image area 16B on monitor 16 is set to be horizontal, and in image processing device 18, the vertical angle of view αv of the display image displayed on monitor 16 is set to be 110° (αv = 110°). Note that, in the following, for simplicity of explanation, image display in camera image area 16B of monitor 16 will be described as image display on monitor 16, with the description of camera image area 16B omitted.

[0043] The image processing device 18 is provided with a camera control unit 30, a display image generation unit 32, a display processing unit 34, a correspondence information generation unit 36, and a storage unit 38. In the image processing device 18, the camera 14 (image sensor 22) is electrically connected to the camera control unit 30, and the monitor 16 is electrically connected to the display processing unit 34.

[0044] The image processing device 18 includes a microcomputer (not shown) in which a CPU, a ROM, a RAM, a storage as a non-volatile storage medium, an input / output interface, and the like are interconnected via a bus. The image processing device 18 stores a camera control program, a display image generation program, a display control program, and a correspondence information generation program in the ROM and storage. In the image processing device 18, the CPU reads and executes each of the camera control program, the display image generation program, the display control program, and the correspondence information program from the ROM and storage, thereby realizing the functions of the camera control unit 30, the display image generation unit 32, the display processing unit 34, and the correspondence information generation unit 36. The storage unit 38 is also realized by the storage. Note that a GPU (Graphic Processing Unit), an FPGA (Field Programmable Gate Array), or the like may also be used as the image processing device 18.

[0045] In the image processing device 18, information (map) indicating the correspondence between each pixel on the image sensor 22 and each pixel on the monitor 16 (pixels in the camera image area) is stored in the storage unit .

[0046] The camera control unit 30 controls the camera 14 to acquire a captured image. The display image generation unit 32 sets the pixel value of each pixel on the monitor 16 from the correspondence information stored in the storage unit 38 and the pixel value of each pixel of the image sensor 22. The display image generation unit 32 also sets each pixel on the monitor 16 using information (map) indicating the correspondence stored in the storage unit 38, and sets a pixel value for each pixel, thereby generating a display image to be displayed on the monitor 16.

[0047] The display processing unit 34 controls the display of the monitor 16 in accordance with the display image (video) generated by the display image generating unit 32, thereby displaying the display image on the monitor 16 (camera video area 16B).

[0048] The correspondence information generation unit 36 ​​generates information indicating the correspondence between each pixel (pixel coordinates) on the monitor 16 (camera image area 16B) and the pixel (pixel coordinates) on the image sensor 22. This generates correspondence information (map) indicating the correspondence between each pixel (pixel coordinates) on the monitor 16 (camera image area 16B) and each corresponding pixel (pixel coordinates) on the image sensor 22. The storage unit 38 stores the correspondence information generated by the correspondence information generation unit 36.

[0049] The operation of this embodiment will be described below, including the processing of the image processing device 18 in the vehicle display device 10 and the display of the image captured by the camera 14 on the monitor 16 (camera image area 16B).

[0050] In the image processing device 18, a pinhole camera model or a perspective projection camera model is applied to derive an image captured by the image sensor 22. In general, a pinhole camera model geometrically models the coordinate position (pixel coordinate) onto which a ray of light emitted from an arbitrary point in three-dimensional space is projected after passing through a pinhole (pinhole point) Ph on a two-dimensional image projection surface. Onto the image projection surface of the pinhole camera model, an image is projected that is flipped vertically and horizontally with the pinhole Ph, which is the optical center, as the symmetrical point. For actual calculations, a perspective projection camera model is generally used in which the image projected onto the image projection surface (the imaging area of ​​the image sensor 22) is flipped vertically and horizontally with respect to the pinhole Ph.

[0051] In the perspective projection camera model, an image plane is assumed to be placed at a position that is a focal point f away from the camera center point as a projection plane corresponding to the image sensor 22 between the three-dimensional space to be imaged and the camera center point that corresponds to the pinhole Ph, and the pixel coordinate is the point where the ray from the image plane toward the camera center point intersects with the projection plane.

[0052] In the perspective projection camera model, a coordinate system (camera-centered coordinate system) with the camera center point (pinhole Ph) as the origin is used, which makes it easy to convert coordinates in three-dimensional space into two-dimensional coordinates on the projection surface, and by setting the focus f = 1, the projected image projected onto the projection surface can be normalized.

[0053] Furthermore, for example, a point expressed in a world coordinate system whose coordinate axes are tilted relative to the coordinate axes of the camera coordinate system can be expressed in the coordinates of the camera coordinate system by a rigid transformation using a rotation matrix and a translation vector determined between the camera coordinate system and the world coordinate system (world coordinate transformation). In other words, the world coordinate transformation is applied to align one of two different coordinate systems with the other.

[0054] In the image processing device 18, a known camera model such as a perspective projection camera model (pinhole camera model) and various processes related to the camera model are applied in basic processing. Since known techniques (processing) can be applied to these processes, detailed explanations are omitted in this embodiment.

[0055] Meanwhile, in the vehicle display device 10, a projection screen 50 is set as a virtual screen in the image processing device 18. In the image processing device 18, the projection screen 50 is used to display an image corresponding to the image captured by the image sensor 22 on the monitor 16 (camera image area 16B), and the correspondence information generating unit 36 ​​generates correspondence information using (intervening) the projection screen 50.

[0056] The correspondence information indicates the correspondence between pixel positions (pixel coordinates) on the monitor 16 (camera image area 16B) based on the pinhole Ph and pixel positions (pixel coordinates) on the image sensor 22. An xy coordinate system (two-dimensional Cartesian coordinate system) is set for the monitor 16 and the image sensor 22, with the pinhole Ph serving as the reference point set as the origin (0, 0). As a result, the correspondence information indicates pixel coordinates (xp, yp) on the monitor 16 that correspond to each of the pixel coordinates (pxf, pyf) on the image sensor 22. Note that by including pixel value information for each pixel coordinate (pxf, pyf), pixel coordinates (xp, yp) including pixel value information can be obtained, and therefore, hereinafter, it is assumed that pixel coordinates (pxf, pyf) and pixel coordinates (xp, yp) include pixel value information.

[0057] Fig. 3 shows a schematic three-view diagram of a projection screen 50 facing the monitor 16. Fig. 3(A) shows the projection screen 50 as seen from the front, Fig. 3(B) shows the projection screen 50 as seen from above, and Fig. 3(C) shows the projection screen 50 as seen from the side, with the front view of the projection screen 50 corresponding to the front view of the monitor 16 as seen from the driver's side. Fig. 3(D) shows a top view of a projection screen 50A that can be used in place of the projection screen 50, and Fig. 3(E) shows a top view of a projection screen 50B that can be used in place of the projection screen 50.

[0058] 3(A) to 3(C), the projection screen 50 includes a central screen portion (front screen portion) 52 at a predetermined width above the pinhole Ph in the center in the width direction, side screen portions (side screen portions) 54 on either side of the central screen portion 52, and a lower screen portion (bottom screen portion) 56 below the central screen portion 52. The side screen portion 54 is made up of a side screen portion 54L on the left side of the central screen portion 52 and a side screen portion 54R on the right side.

[0059] The central screen portion 52 has a substantially rectangular shape when viewed from the front from the monitor 16 side. When viewed from above, the central screen portion 52 is curved in a convex shape facing away from the monitor 16 (and the image sensor 22). The radius R of curvature of the central screen portion 52 can be, for example, a radius corresponding to a hemispherical screen set in three-dimensional space in a perspective projection camera model.

[0060] The side screen sections 54 each have an upper base in the area that is in contact with the central screen section 52 in the vertical direction, and a lower base on the side opposite the central screen section 52 that covers the entire width of both ends of the projection screen 50, giving them a generally trapezoidal shape when viewed from the front. The lower screen section 56 has an upper base in the area that is in contact with the lower part of the central screen section 52, and a lower base that covers the entire lower edge of the projection screen 50, giving them a generally trapezoidal shape when viewed from the front.

[0061] The projection screen 50 faces the monitor 16 with the up-down direction of the central screen portion 52 approximately parallel to the up-down direction of the monitor 16. The projection screen 50 is also inclined at a predetermined angle γs so that the lower base side of each of the side screen portions 54 approaches the monitor 16, and the lower base side of the lower screen portion 56 is inclined at a predetermined angle γu so that it approaches the monitor 16.

[0062] In this case, when viewed from the front, the projection screen 50 has a vertical dimension H1 that corresponds to the vertical dimension of the monitor 16 (camera image area 16B), and a width dimension W1 that corresponds to the width dimension of the monitor 16. In other words, the ratio of the vertical dimension H1 to the width dimension W1 of the projection screen 50 is the same as the aspect ratio of the camera image area 16B on the monitor 16.

[0063] In the image processing device 18, instead of the projection screen 50, a projection screen 50A shown in FIG. 3(D) or a projection screen 50B shown in FIG. 3(E) can also be used.

[0064] 3(D), in the projection screen 50A, the central screen portion 52 is omitted, and the side screen portions 54A, which are extensions of the side screen portions 54, are formed so as to contact each other at the center position in the width direction. In addition, in the projection screen 50A, by omitting the central screen portion 52, the lower screen portions 56A, which contact each of the side screen portions 54A, are triangular in shape.

[0065] Furthermore, as shown in FIG. 3(E), in the projection screen 50B, a planar central screen portion 52A is formed in place of the central screen portion 52, and in the projection screen 50A, by forming the central screen portion 52A, the lower screen portions 56A that contact each of the central screen portion 52A and the side screen portions 54A are made to have an approximately trapezoidal shape.

[0066] Among these projection screens 50, 50A, and 50B, the projection screen 50A is the basic form, and the projection screen 50B is configured such that a substantially flat central screen section 52A is bridged between the side screen sections 54A of the projection screen 50A. Also, the projection screen 50 is configured such that a curved central screen section 52 is disposed in place of the central screen section 52A of the projection screen 50B.

[0067] On the other hand, the projection screen 50 can be used as a screen that applies to one-point perspective projection, with the central screen portion 52 located at the rear and the vanishing point set on the optical axis passing through the pinhole Ph. Figures 4(A) and 4(B) are schematic diagrams showing the projection of a captured image onto the projection screen 50. Note that Figure 4(A) shows the projection screen 50 as seen from the side, and Figure 4(B) shows the projection screen 50 as seen from above.

[0068] When projecting a captured image onto the projection screen 50, the points (intersections) on the projection screen 50 of the light rays emitted from the vanishing point (pinhole Ph) for the captured image are set as pixel coordinates on the projection screen 50 for each pixel (pixel coordinates) of the captured image. As shown in Figures 4(A) and 4(B), when projecting a captured image onto the projection screen 50, the vanishing point is set on an axis passing through the pinhole Ph. Note that in Figures 4(A) and 4(B), the pinhole Ph is set as the vanishing point.

[0069] In such projection onto the projection screen 50, in a perspective projection camera model, the projection screen 50 is placed as a projection surface between the three-dimensional space to be imaged and the camera center point (pinhole Ph). In this case, the point where a light ray heading toward the camera center point (and toward the image sensor 22) intersects with the projection screen 50 can be set as the pixel coordinates (xn, yn, zn) on the projection screen 50 corresponding to the pixel coordinates (pxf, pyf) on the image sensor 22 (the pixel coordinates of each pixel in the captured image).

[0070] On the other hand, when projecting an image on the projection screen 50 onto a two-dimensional plane, pixel coordinates (xn, yn, zn) on the projection screen 50 can be expressed as pixel coordinates (r, θ) in a polar coordinate system on the two-dimensional plane by providing a required relationship between the coordinate axes. Similarly, when projecting an image on a two-dimensional plane onto the projection screen 50, pixel coordinates (r, θ) in a polar coordinate system on the two-dimensional plane can be expressed as pixel coordinates (xn, yn, zn) on the projection screen 50. Note that the pixel coordinates (r, θ) differ depending on whether the two-dimensional plane faces the monitor 16 or the image sensor 22.

[0071] Furthermore, pixel coordinates (r, θ) in a polar coordinate system on a two-dimensional plane can be expressed as pixel coordinates (xp, yp) in a Cartesian coordinate system on the two-dimensional plane (when the two-dimensional plane faces the monitor 16) or pixel coordinates (pxf, pyf) (when the two-dimensional plane faces the image sensor 22). By using such a polar coordinate system on a two-dimensional plane, distortion correction for wide-angle lenses such as fisheye lenses becomes possible.

[0072] That is, each lens 20 (fisheye lens) has its own inherent distortion, which differs from lens to lens, and image display requires processing that takes distortion into account. This distortion of lens 20 can be taken into account using a general method. For example, if the projection method of the fisheye lens used as lens 20 is stereoscopic projection, where y is the image height, f is the focal length, and θ is the half angle of view for the lens (1 / 2 the angle of view), then the image height y is y = 2f tan(θ / 2).

[0073] The inherent distortion θd of the lens 20 is approximated by, for example, the following general polynomial: θd=θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 ) (However, k1, k2, k3, and k4 are coefficients determined for each lens.)

[0074] In the image processing device 18, when each pixel coordinate (r, θ) on the projection screen 50 is made to correspond to a pixel coordinate (pxf, pyf) on the image sensor 22 or to a pixel coordinate (xp, yp) on the monitor 16, a normalization process is performed in which the focus of the lens 20 is set to f=1, and a process that takes into account the distortion of the lens 20 (a process to remove the distortion), for example.

[0075] 5(A) and 5(B) are schematic side views (side views) of the lens 20 as seen from one side along the vehicle width direction. Note that Fig. 5(A) shows the lens 20 with a dip angle β, and Fig. 5(B) shows the lens 20 with a dip angle β of zero.

[0076] As shown in Figure 5(A), on the projection screen 50, by setting one of the axes (for example, the optical axis Oa) passing through the origin (pinhole Ph) as the z-axis, each coordinate position on the central screen portion 52, the side screen portion 54, and the lower screen portion 56 can be expressed in terms of coordinates in an xyz coordinate system (a three-dimensional Cartesian coordinate system).

[0077] Also, as shown in FIG. 5B, the camera 14 may not have a dip angle β (β=0°). In this case, the optical axis of the lens 20 relative to the projection screen 50 is different. In this case, by performing world coordinate transformation, a projection screen 50 corresponding to the lens 20 with a dip angle β can be applied. From this, by performing world coordinate transformation on the coordinate system on the image sensor 22 or the coordinate system of the projection screen 50, the coordinate system of the image sensor 22 and the coordinate system of the projection screen 50 can be aligned to a predetermined relationship. As a result, for each pixel coordinate (xn, yn, zn) on the projection screen 50, by corresponding it to the coordinate system of the image sensor 22, pixel coordinates (pxf, pyf) in the two-dimensional Cartesian coordinate system can be calculated.

[0078] On the other hand, in the vehicle display device 10 (image processing device 18), the projection screen 50 is set to overlap the captured image formed on the image sensor 22. In the projection screen 50, the height (height dimension) which is the size in the up-down direction corresponds to the height of the captured image on the image sensor 22, and the width dimension which is the size in the width direction (vehicle width direction) corresponds to the width of the captured image on the image sensor 22.

[0079] In addition, in the vehicle display device 10, the width dimension W2 of the central screen portion 52 of the projection screen 50 is set to a field angle αs that is slightly wider than the width dimension (vehicle width) of the vehicle 12 on the image captured by the camera 14.

[0080] In the vehicle display device 10, the angles γs, γu and width dimension W2 of the projection screen 50 can be determined by determining the width dimensions W1, W2 of the projection screen 50 and the height of the projection screen 50A.

[0081] This makes it possible to calculate the corresponding pixel coordinates (pxf, pyf) on the image sensor 22 for each pixel coordinate (xn, yn, zn) on the projection screen 50, and obtain correspondence information indicating the pixel coordinates (pxf, pyf) on the image sensor 22 that correspond to each pixel coordinate (xp, yp) on the monitor 16. Furthermore, by using this correspondence information, it is possible to obtain each pixel coordinate (xp, yp) on the monitor 16 that corresponds to each pixel coordinate (pxf, pyf) on the image sensor 22.

[0082] Using this correspondence information, the display image generation unit 32 calculates pixel values ​​at each pixel coordinate (xp, yp) from the captured image formed on the image sensor 22. At this time, the display image generation unit 32 cuts out the captured image in accordance with the horizontal angle of view αh (αh=150°) and vertical angle of view αv (αv=110°) displayed on the monitor 16. Note that the captured image may be cut out on the projection screen 50.

[0083] Fig. 6 shows a flow chart of an example of a process for generating correspondence information, and Fig. 7 shows a flow chart of an example of a process for displaying an image on monitor 16. Fig. 8(A) shows an example of an image captured by camera 14, and Fig. 8(B) shows an example of an image displayed on monitor 16 for the image captured by Fig. 8(A). Fig. 8(A) shows the image captured by camera 14 as an image displayed on monitor 16 without undergoing the processing according to this embodiment.

[0084] 6, in the first step 100, the correspondence information generation unit 36 ​​of the image processing device 18 calculates the pixel coordinates (xn, yn, zn) on the projection screen 50 for each pixel coordinate (xp, yp) on the monitor 16 (camera image area 16B). In the next step 102, the correspondence information generation unit 36 ​​calculates the pixel coordinates (polar coordinates) (r, θ) corresponding to the pixel coordinates (xn, yn, zn) on the projection screen 50 for each of the pixel coordinates (xn, yn, zn) on the projection screen 50.

[0085] Next, in step 104, the correspondence information generation unit 36 ​​performs normalization and distortion correction processing corresponding to the lens 20 for each pixel coordinate (r, θ). Note that the correspondence information generation unit 36 ​​performs world coordinate conversion if it is necessary to perform world coordinate conversion between the coordinate system of the image sensor 22 and the coordinate system of the pixel coordinates (xn, yn, zn). The world coordinate conversion can be performed before or after step 104 is performed.

[0086] Thereafter, in step 106, the correspondence information generation unit 36 ​​calculates the pixel coordinates (pxf, Pry) on the image sensor 22 that correspond to each of the pixel coordinates (r, θ), and in step 108 generates correspondence information that indicates the pixel coordinates (xp, yp) on the monitor 16 that correspond to each of the pixel coordinates (pxf, pyf) on the image sensor 22. The correspondence information generated in the correspondence information generation unit 36 ​​is stored in the storage unit 38 and is used by the display image generation unit 32 to generate a display image.

[0087] Note that such correspondence information is determined for each vehicle display device 10 (combination of camera 14 and monitor 16), and therefore may be generated in advance and stored in the storage unit 38 of the image processing device 18.

[0088] 7, in the image processing device 18, in the first step 120, the camera control unit 30 controls the imaging of the camera 14 to capture an image of the area behind the vehicle, thereby obtaining a captured image. Next, in the image processing device 18, in step 122, the display image generation unit 32 uses the correspondence information to generate an image to be displayed on the monitor 16 from the captured image. At the same time, in the image processing device 18, in step 124, the display image generation unit 32 cuts out an area corresponding to the horizontal angle of view αh and the vertical angle of view αv, and generates a display image to be displayed on the monitor 16.

[0089] In the image processing device 18, when the display image is generated, in step 126, the display processing unit 34 controls the display image to be displayed on the monitor 16, and the rear of the vehicle imaged by the camera 14 is displayed on the monitor 16.

[0090] As a result, an area corresponding to the horizontal angle of view αh and the vertical angle of view αv is cut out from the image captured by camera 14 (showing the image corresponding to Figure 8(A)), and the cut-out image is displayed on monitor 16 (see Figure 8(B)).

[0091] In this way, in the vehicle display device 10, for an image area in which the vertical angle of view αv along the up-down direction in the captured image is narrower than the horizontal angle of view αh, a display image is generated in which the horizontal direction is aligned with the horizontal direction of the camera image area 16B of the monitor 16 and the up-down direction (vertical direction) is aligned with the vertical direction of the camera image area 16B, and the generated display image is displayed on the monitor 16. As a result, the vehicle display device 10 can display a captured image by compressing it vertically compared to the horizontal direction, and therefore the captured image captured at a wide angle using a wide-angle lens can be effectively displayed on the monitor 16 while removing distortion.

[0092] Furthermore, the vehicle display device 10 uses the projection screen 50 to generate a display image to be displayed in the camera image area 16B of the monitor 16. As a result, the vehicle display device 10 can compress the captured image in the vertical direction to match the height dimension of the camera image area 16B of the monitor 16, and can effectively display an image with reduced distortion in the captured image in the camera image area 16B of the monitor 16, thereby assisting the occupant in viewing the area behind the vehicle.

[0093] Furthermore, the vehicular display device 10 generates a display image by cutting out a photographed image captured at a wide angle according to the required horizontal angle of view αh and vertical angle of view αv. In this case, the vehicular display device 10 sets the horizontal angle of view αh behind the vehicle to αh=150°, so that a wide range of the area behind the vehicle can be displayed on the monitor 16.

[0094] Furthermore, the vehicle display device 10 uses a projection screen 50 provided with a curved central screen portion 52 that protrudes rearward of the vehicle in the captured image. As a result, the image displayed on the monitor 16 has depth in the central portion in the vehicle width direction, so that a high-quality image can be displayed that gives the impression of actually looking rearward of the vehicle.

[0095] Furthermore, the vehicle display device 10 uses the camera 14 to capture an image of a part of the vehicle body (bumper 12A) and displays it on the monitor 16. This allows the occupant to recognize the relative positions of the vehicle 12 and the objects around the vehicle 12, thereby effectively assisting the occupant's visibility.

[0096] Furthermore, the vehicle display device 10 can generate a display image similar to that obtained by applying a one-point projection method by using the projection screen 50. This allows the display image that is easy for the occupant to view to be displayed on the monitor 16 while more effectively removing distortion of the image captured using a wide-angle lens, thereby more effectively assisting the occupant's visibility.

[0097] In the above-described embodiment, the projection screen 50 is provided with a curved central screen portion 52 on the rear side. However, the projection screen 50A shown in Fig. 3(D) or the projection screen 50B shown in Fig. 3(E) may also be used.

[0098] The projection screen 50A has a simple three-sided structure using a side screen portion 54A and a lower screen portion 56A, so compared to when the projection screen 50 or the projection screen 50B is used, the processing load for image processing is reduced, and an image can be displayed that is easy for occupants to recognize while suppressing distortion.

[0099] Furthermore, since the projection screen 50B uses the flat central screen portion 52A, the image change in the central portion of the display image in the vehicle width direction can be made smoother than that of the projection screen 50A. This allows for the display of a display image that can assist the occupants' visibility while effectively suppressing distortion.

[0100] In this embodiment, the rear of the vehicle is imaged by the camera 14. However, the image capturing means may capture images of the sides of the vehicle (the right side or left side of the vehicle) or the front of the vehicle. This allows the surroundings of the vehicle to be displayed, effectively assisting the occupant's visibility. [Explanation of symbols]

[0101] 10···Vehicle display device (image display device), 12A···Bumper (part of vehicle body), 14···Camera, 16···Monitor (display medium), 16B··Camera image area, 18···Image processing device (display control unit), 20···Lens, 22···Image sensor, 32···Display image generation unit, 36···Corresponding information generation unit, 50, 50A, 50B···Projection screen, 52, 52A···Central screen unit, 54 (54L, 54R), 54A, 54B···Side screen unit, 56, 56A···Lower screen unit.

Claims

1. an imaging means provided in the vehicle for imaging an image of the outside of the vehicle body through a wide-angle lens having an angle of view of 180° or more from the vehicle body side; a display medium arranged with a vertical direction being the up-down direction and a horizontal direction being the horizontal direction, and an area longer in the horizontal direction than in the vertical direction is used as a display area for an image captured by the imaging means; a display control unit that generates a display image by compressing an image area in which a vertical angle of view along a top-bottom direction in an image captured by the imaging means is narrower than a horizontal angle of view along a horizontal direction by matching the horizontal direction to a width direction of the display area of ​​the display medium and the vertical direction to a height direction of the display medium, and displays the generated display image in the display area; An image display device comprising:

2. 2. The image display device according to claim 1, wherein the display control unit uses a screen having left and right side screen sections which correspond to left and right sides of the horizontal center position of the captured image and have a vertical dimension narrower toward the horizontal center position than both horizontal ends, and a lower screen section whose upper side contacts the lower sides of the left and right side screen sections and spans the left and right side screen sections over the entire horizontal area of ​​the captured image, and in which the side screen sections and the lower screen section are each inclined so that the central positions of the side screen sections are away from the captured image, and generates as the displayed image a projected image in which the captured image is projected radially from a required central point onto the screen.

3. an imaging means provided in the vehicle for imaging an image of the outside of the vehicle body through a wide-angle lens having an angle of view of 180° or more from the vehicle body side; a display medium arranged with a vertical direction being the up-down direction and a horizontal direction being the horizontal direction, and an area longer in the horizontal direction than in the vertical direction is used as a display area for an image captured by the imaging means; a display control unit that uses a screen having left and right side screen sections each having a vertical dimension narrower on the horizontal center side than both ends corresponding to a left and right side portion of a horizontal center position in an image area in which a vertical angle of view in a top-down direction in an image captured by the imaging means is narrower than a horizontal angle of view in a horizontal direction, and a lower screen section whose upper side contacts the lower sides of the left and right side screen sections and spans the left and right side screen sections over the entire horizontal area of ​​the captured image, and in which the side screen sections and the lower screen section are each inclined so that a center position of the side screen sections is away from the captured image, and that generates a display image by compressing a projected image obtained by projecting the captured image radially from a required center point onto the screen so that the horizontal direction of the projected image matches the horizontal direction of the display area of ​​the display medium and the vertical direction of the projected image matches the vertical direction of the display medium, and displays the generated display image on the display area; An image display device comprising:

4. 3. The image display device according to claim 2, wherein the screen is provided with a central screen portion between the left and right side screen portions, the central portion connecting the left and right side screen portions at their horizontal center positions.

5. 5. The image display device according to claim 4, wherein the central screen portion is curved toward the rear when viewed from the photographed image side.

6. The image display device according to claim 2 , wherein the display control unit generates, as the display image, a display image in which the horizontal angle of view is 150° or more.

7. the imaging means is installed at a rear portion of the vehicle and images an area behind the vehicle at a predetermined dip angle; The image display device according to claim 2 , wherein the display control unit controls the display medium to display an image inclined at an angle corresponding to the dip angle with respect to the vertical direction of the screen.

8. The imaging means captures an image including a part of the vehicle body, The image display device according to claim 1 , wherein the display control unit generates the display image including a part of the captured vehicle body.

9. When an image captured by an imaging means provided in a vehicle for capturing an image of the outside of the vehicle body through a wide-angle lens from the vehicle body side is displayed on a display medium arranged such that the vertical direction is the up-down direction and the horizontal direction is the horizontal direction, and an area longer in the horizontal direction than in the vertical direction is used as a display area for the captured image of the imaging means, generating a display image in which an image area in which a vertical angle of view along a top-bottom direction in the captured image of the imaging means is narrower than a horizontal angle of view along a horizontal direction is compressed by aligning the horizontal direction with a horizontal direction of a display area of ​​the display medium and by aligning the vertical direction with a vertical direction of the display medium; The image display method displays the generated display image in a display area of ​​the display medium.

10. When an image captured by an imaging means provided in a vehicle for capturing an image of the outside of the vehicle body through a wide-angle lens from the vehicle body side is displayed on a display medium arranged such that the vertical direction is the up-down direction and the horizontal direction is the horizontal direction, and an area longer in the horizontal direction than in the vertical direction is used as a display area for the captured image of the imaging means, a screen including left and right side screen sections each having a vertical dimension narrower on the lateral center side than on both lateral ends corresponding to the left and right side portions of an image area in which a vertical angle of view in the up-down direction in an image captured by the imaging means is narrower than a horizontal angle of view in the horizontal direction, and a lower screen section whose upper side is in contact with the lower sides of the left and right side screen sections and which straddles the left and right side screen sections over the entire horizontal area of ​​the captured image, and each of the side screen sections and the lower screen section are inclined so that the center positions of the side screen sections are away from the captured image, A captured image is projected radially from a desired center point onto the screen, and a display image is generated in which the horizontal direction of the image is aligned with the horizontal direction of the display area of ​​the display medium and the vertical direction is aligned with the vertical direction of the display medium, and compressed; The image display method includes displaying the generated display image in the display area.

Images