Image processing device, camera system, and image processing method

Abstract

This image processing device includes a first signal processing unit that generates a first image with a first angle of view from an image captured by a camera attached to the vehicle body, and an image superimposition unit that superimposes an image representing a part of the vehicle body onto the first image.

Description

[Technical field]

[0001] The present invention relates to an image processing device, a camera system, a monitor device, and an image processing method for processing images captured by a camera installed on a vehicle body. [Background technology]

[0002] Recently, an increasing number of vehicles are being equipped with rearview cameras that assist drivers in parking by displaying a rear image captured by the camera on the screen of a car navigation system when the vehicle is backed up for parking, etc. Many rearview cameras include a part of the rear bumper of the vehicle in the imaging field of view, thereby displaying an image of the rear of the vehicle, including a part of the rear bumper, on the rearview screen. This allows the driver to get a sense of distance in the rearview image space based on the rear bumper (see Patent Document 1).

[0003] On the other hand, in order to realize automobiles that do not have rearview mirrors, the number of cars equipped with camera monitoring systems (CMS: Camera Monitor Systems) that replace rearview mirrors is increasing. It is expected that CMS will realize safer automobiles by providing rear images with fewer blind spots and a wider field of view. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-150938 A (FIG. 3, paragraphs

[0016] and

[0029] ) Summary of the Invention [Problem to be solved by the invention]

[0005] Both the rearview camera and CMS provide the driver with images of the rear of the vehicle, but the rearview camera is intended to monitor the situation directly behind the vehicle, whereas the CMS is intended to monitor the vehicles following behind the vehicle, so the heights of the imaging fields of view are different between the rearview camera and the CMS camera. For this reason, separate cameras with different imaging fields are usually used for each. However, in order to reduce the total cost, it is desirable to realize the rearview and CMS with one camera, or to realize the rearview using a camera for the CMS. However, when trying to realize a rear view using images from a CMS camera, various problems arise because the CMS camera's imaging field of view does not include the vehicle's bumper.

[0006] In view of the above circumstances, an object of the present technology is to provide an image processing device, a camera system, and an image processing method that can provide a high-quality rearview image. [Means for solving the problem]

[0007] In order to solve the above problems, an image processing device according to the present technology includes a first signal processing unit that generates a first image with a first angle of view from an image captured by a camera attached to a vehicle body, and an image superimposition unit that superimposes an image representing a part of the vehicle body onto the first image.

[0008] According to the image processing device of the present technology, by superimposing an image depicting a part of the vehicle body on a first image generated from an image captured by a camera, it is possible to obtain a captured image that makes it easy for the user to grasp the sense of distance based on that part of the vehicle body.

[0009] The image of the portion of the vehicle body may be a pre-stored image.

[0010] More specifically, the pre-stored image may be an image that is stored in the first signal processing unit before the image of the camera is acquired.

[0011] The image of the portion of the vehicle body may be an image of an area outside the field of view of the camera.

[0012] The image of the portion of the vehicle body may be a virtual image of at least a portion of a bumper of the vehicle body.

[0013] The camera is a camera mounted so as to capture an image of the front, rear or side of the vehicle body.

[0014] The image superimposing unit may be configured to adjust the virtual image to be superimposed on the first image based on speed information of the vehicle body.

[0015] The image overlay unit may be configured to overlay an embedding image in a gap area between the first video and the virtual image where no image exists.

[0016] The embedded image may be an image that approximates a color tone of the first image adjacent to the gap region.

[0017] The image superimposing section may be configured to generate the embedded image by cutting out an image of a region corresponding to the gap region from the past first image.

[0018] The image superimposing section may be configured to output an image in which at least the image is superimposed on the first image to a first display section.

[0019] The image processing device of the present technology may further include a second signal processing unit that generates a second image having a second angle of view different from the first image from the first image and outputs the second image to a second display unit.

[0020] The centre of the second angle of view may be at a higher position in the first image space than the centre of the first angle of view.

[0021] A camera system according to another aspect of the present technology includes a camera attached to a vehicle body, a first signal processing unit that generates a first image having a first angle of view from an image captured by the camera, and an image overlay unit that overlays an image representing a part of the vehicle body onto the first image.

[0022] An image processing device according to yet another aspect of the present technology includes a first signal processing unit that generates a first image with a first angle of view from an image captured by a camera attached to a vehicle body, an image superimposition unit that superimposes an image representing a part of the vehicle body on the first image, and a speed detection unit that detects the speed of the vehicle body, and is configured such that the image superimposition unit adjusts the image to be superimposed on the image based on the detected speed.

[0023] In a signal processing method according to yet another aspect of the present technology, a first signal processing unit generates a first image having a first angle of view from an image captured by a camera attached to a vehicle body, and an image superimposition unit superimposes an image representing a part of the vehicle body onto the first image. [Brief description of the drawings]

[0024] [Figure 1] 1 is a block diagram showing an overall configuration of a camera system 100 according to the present technology. [Diagram 2] 2 is a side view showing the mounting position and orientation of a camera 10 in the camera system 100 of FIG. [Diagram 3] 3 is a diagram showing an example of a visual field image of the camera 10 in FIG. 2. [Figure 4] 1 is a diagram showing an example of the positional relationship between the image space of the camera 10 and the CMS image space. [Diagram 5] FIG. 2 is a diagram showing an example of the configuration of a rearview image 6. [Figure 6] FIG. 13 is a diagram showing another example of the configuration of the rearview image 6. [Figure 7] 11A to 11C are diagrams illustrating a method of adjusting a virtual image 4 in accordance with a vehicle speed. [Figure 8]11 is an explanatory diagram of a method for superimposing a virtual image 4 on a rearview original video image 3 in a static positional relationship in a second embodiment according to the present technology. FIG. [Figure 9] FIG. 11 is a diagram showing an example of a rear view image in the second embodiment. [Figure 10] FIG. 11 is a diagram showing another example of a rearview image in the second embodiment. [Figure 11] FIG. 11 is an explanatory diagram of an embedding process using past video. [Figure 12] 1 is a diagram for explaining a method for actually measuring the distance from the position of camera 10 to a position corresponding to the lower end of the angle of view of the image from camera 10 using a chart. [Figure 13] FIG. 13 is a diagram illustrating a method for installing a chart. [Figure 14A] FIG. 11 is a diagram showing an example of a rear view image in normal view mode. [Figure 14B] FIG. 11 is a diagram showing an example of a rear view image in a wide view mode. [Figure 14C] FIG. 11 is a diagram showing an example of a rear view image in a top-down view mode. [Figure 15] FIG. 13 illustrates a camera viewpoint after transformation in top-down view mode. [Figure 16] 10 is a flowchart showing the overall operation of a rearview signal processor 23 and an image superimposing unit 24 in the camera system 100 of the second embodiment. [Figure 17] 1 is a block diagram showing a first modified example of the configuration of an image processing device according to the present technology. [Figure 18] FIG. 11 is a block diagram showing a second modified example of the configuration of the image processing device according to the present technology. [Figure 19] FIG. 13 is a block diagram showing a configuration of a monitor device which is a third modified example of the image processing device according to the present technology. [Figure 20] FIG. 13 is a block diagram showing a configuration of a camera system that is a fourth modified example of an image processing device according to the present technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Hereinafter, embodiments of the present technology will be described in the following order with reference to the drawings. 1. Overall system configuration 2. Configuration of the image processing device 20 3. Image overlay processing 4.Adjusting the virtual image according to the vehicle speed 5. Embedding of past footage 6. Switching rear view modes 7. Overall Control Flow 8. Variations

[0026] <First embodiment> (1. Overall system configuration) FIG. 1 is a block diagram showing an overall configuration of a camera system 100 according to the present technology. The camera system 100 includes a camera 10, an image processing device 20, a CMS display unit 30, and a rearview display unit 40.

[0027] The CMS display unit 30 and the rearview display unit 40 are separate monitors that can be viewed by the driver of the vehicle body 60. The CMS display unit 30 may be a monitor that replaces a room mirror such as a rearview mirror or a door mirror. The rearview display unit 40 may be, for example, the screen of a car navigation system.

[0028] Fig. 2 is a side view showing the mounting position and orientation of camera 10 in camera system 100 in Fig. 1, and Fig. 3 is a diagram showing an example of a visual field image of camera 10 in Fig. 2. Note that in this embodiment, it is assumed that camera 10 is disposed at the rear of the vehicle body 60 so as to be able to capture an image of the rear of the vehicle body 60. However, the present invention is not limited to this, and camera 10 may be disposed so as to be able to capture an image of the front of the vehicle body 60.

[0029] The camera 10 is a digital video camera equipped with a CCD sensor or a CMOS sensor, and is preferably a high definition (HD) digital video camera. As shown in Figs. 2 and 3, the camera 10 of the camera system 100 is installed at a relatively high position, such as the top part of the rear of the vehicle body 60, with the optical axis almost horizontal or slightly tilted downward so that it can capture distant objects. In the case of such a camera arrangement, bumpers such as the rear bumper 61 of the vehicle body 60 are outside the field of view of the camera 10. For this reason, when the camera system 100 is used for monitoring the rear of the vehicle body in place of a rearview mirror, for example, there is a problem that the camera image 1 (see Fig. 3) makes it difficult to grasp the sense of distance to the rear of the vehicle body due to the absence of an image of the rear bumper 61. This is one of the problems that the present technology aims to solve.

[0030] (2. Configuration of image processing device 20) The image processing device 20 includes at least a first signal processing unit (rearview signal processing unit 23) that generates a first image with a first angle of view from an image captured by a camera 10 attached to the vehicle body 60, and an image superimposition unit 25 that superimposes an image representing a part of the vehicle body 60 on the first image.

[0031] More specifically, the image processing device 20 generates an image for CMS from the image 1 captured by the camera 10 and supplies it to the CMS display unit 30, while also generating an original image for rear view from the image captured by the camera 10 and generating a rear view image by superimposing at least an image of a virtual rear bumper (hereinafter referred to as a "virtual image") on this original image for rear view, and supplies the generated rear view image to the rear view display unit 40. There are two methods for superimposing a virtual image on the original rearview image: a method for adjusting the position of the virtual image based on the vehicle speed, and a method for superimposing the virtual image at a fixed position regardless of the vehicle speed. The former method is described in the first embodiment, and the latter method is described in the second embodiment.

[0032] 1, the image processing device 20 is composed of, for example, a splitter 21, a CMS signal processing unit 22 (second signal processing unit), a rearview signal processing unit 23, and an image superimposition unit 24. The image processing device 20 is composed of, for example, an integrated circuit such as an IC (Integrated Circuit) or an LSI (large-scale integrated circuit), a memory for buffering, and the like.

[0033] The splitter 21 supplies the video 1 (see FIG. 3) captured by the camera 10 to a CMS signal processor 22 and a rearview signal processor 23.

[0034] The CMS signal processing unit 22 generates a CMS image to be supplied to the CMS display unit 30 by cutting out the image of the target area for CMS signal processing from the image 1 supplied through the splitter 21, performing image quality adjustments such as distortion correction according to the view mode, and white balance adjustment.

[0035] 4 is a diagram showing an example of the positional relationship between the image space of the camera 10 and the CMS image space. As shown in the figure, the CMS signal processor 22 generates a CMS image by cutting out, from the camera image 1, an image area 2 that has a distant subject as the main subject and corresponds to an angle of view part where the horizon S is slightly below the midpoint, as a target area for CMS signal processing.

[0036] The rearview signal processor 23 generates an original image for rearview from the camera image 1 by performing image quality adjustments such as cropping an image of a target area for rearview signal processing, distortion correction according to the view mode, and white balance adjustment. The rearview signal processor 23 may process all pixels of the camera image 1. FIG. 4 shows a case in which all pixels of the camera image 1 are used as the original image 3 for rearview. The original image 3 for rearview is a camera image consisting of only the image of the camera 10 before a virtual image of a bumper and an embedded image, which will be described later, are superimposed, and will be referred to as the "original rearview image 3" hereinafter. The image in which the virtual image and embedded image of the bumper, which will be described later, are superimposed on this original rearview image 3 is the "rearview image 6".

[0037] The image processing device 20 has a communication interface (not shown) for communicating with an ECU (Engine Control Unit) 200 mounted on the vehicle body 60. The communication protocol with the ECU 200 is, for example, CAN (Controller Area Network). The image superimposing unit 24 receives status information such as vehicle speed information from the ECU 200 using the above communication interface, and adjusts the position of the virtual image based on the received vehicle speed information. The details of the virtual image superimposing process by the image superimposing unit 24 will be described later.

[0038] (3. Image overlay processing) FIG. 5 is a diagram showing an example of the configuration of the rearview image 6. The image superimposing unit 24 superimposes an image expressing a part of the vehicle body, such as a rear bumper, that does not exist within the field of view of the camera 10 as a virtual image 4 on the rearview original image 3 generated by the rearview signal processing unit 23, to generate the rearview image 6. The virtual image 4 is, for example, an image created in advance by computer graphics or the like, and is information stored in a non-volatile memory (not shown) in the image processing device 20 before the image from the camera 10 is supplied to the image processing device 20 or before the system is started. Note that the image superimposed on the rearview original image 3 is not necessarily an image created by computer graphics or the like, and may be, for example, an image processed from a photograph. In this specification, images created by computer graphics or the like and photo-processed images are collectively referred to as "virtual images 4."

[0039] 5 shows a case where all pixels of the image 1 of the camera 10 are used as the rearview original image 3 to generate the rearview image 6, but the present technology is not limited to this. For example, as shown in FIG. 6, a partial rectangular area including the bottommost area of ​​the image 1 of the camera 10 may be used as the rearview original image 3a, and a virtual image 4 or the like may be superimposed on this to generate the rearview image 6a.

[0040] (4. Adjustment of virtual image according to vehicle speed) The image superimposing unit 24 can adjust the position of the virtual image 4 to be superimposed on the rearview video 6, based on vehicle speed information obtained from the ECU 200.

[0041] FIG. 7 is a diagram for explaining a method of adjusting the virtual image 4 in accordance with the vehicle speed. In the same figure, h is the installation height of the camera 10, θ is the angle of the light ray corresponding to the lower end of the angle of view of the camera 10 relative to the camera optical axis, φ is the angle of the light ray corresponding to the lower end of the angle of view of the camera 10 relative to the vertical, and t is a timing parameter preset in the image superimposition unit 24.

[0042] Here, φ is the inclination (downward inclination) of the optical axis of the camera 10 with respect to the horizontal, and β is the following: φ=90−θ−β (1) It is calculated using the following formula.

[0043] The timing parameter t is a parameter for displaying a virtual image 4 having an end at a point where the rear bumper 61 of the vehicle body 60 will reach in t seconds at the current speed while the vehicle is reversing. The setting of this timing parameter t may be arbitrarily changed by the user. The virtual image here shows an area where the vehicle will reach in t seconds / within t seconds when reversing at the current speed. Furthermore, the virtual image may be superimposed and displayed as a semi-transparent image so that the driver can confirm the state of the area of ​​the rearview original image 3 on which the virtual image is superimposed.

[0044] First, the image superimposing unit 24 calculates the display position of the virtual image 4 from h, t, and the vehicle speed v using the following formula. If the angle of view of the original rearview video 3 is the overall rearview display angle of view, the display position of the virtual image 4 is calculated based on how far away from the bottom end of the overall rearview display angle the top end of the virtual image 4 should be. The top end position of the virtual image 4 is a point moved by vt+e from the position of the camera 10. Therefore, the image superimposing unit 24 first calculates vt+e, where e is the distance from the position of the camera 10 to the edge of the rear bumper 61.

[0045] Next, α is calculated from the calculation result of vt+e. α is the angle with respect to the vertical in the direction in which the point vt+e on the road surface is viewed from the camera 10. α is calculated by the following formula. α=tan -1 ((vt+e) / h) (2) Therefore, the higher the vehicle speed, the further to the left the top edge position of the virtual image 4 is in FIG.

[0046] Next, the image superimposing unit 24 superimposes the virtual image 4 on the area from the bottom end of the rearview original video 3 to a position (the top end position of the virtual image 4) on the rearview original video 3 that corresponds to the angle α calculated by the above formula (2). As a result, the virtual image 4 is superimposed at an appropriate position on the rearview original video 3, and a rearview video 6 that makes it easy to grasp the sense of distance from the vehicle body 60 is obtained.

[0047] However, when the vehicle speed v is excessively high, most of the area of ​​the rearview image 6 is occupied by the virtual image 4, and the space behind the vehicle body seen by the user through the rearview display unit 40 becomes too narrow. Therefore, the image superimposing unit 24 superimposes the virtual image 4 only when, for example, the following condition is satisfied. θ / 2<θ+φ-α<θ (3) If the above formula (3) is not satisfied, only the rearview original image 3 is displayed on the rearview display unit 40 as the rearview image 6. This makes it possible to prevent the image space behind the vehicle body seen by the user through the rearview display unit 40 from becoming too narrow.

[0048] <Second embodiment> Next, a second embodiment according to the present technology will be described. In this embodiment, the image superimposing unit 24 superimposes the virtual image 4 on the rearview original video 3 in a static positional relationship, regardless of the vehicle speed.

[0049] FIG. 8 is an explanatory diagram of a method for superimposing a virtual image 4 on a rearview original image 3 in a static positional relationship in the second embodiment, and FIG. 9 is a diagram showing an example of a rearview image in this embodiment. As shown in these figures, the virtual image 4 is disposed in a position corresponding to the region from the position of the camera 10 to the end of the rear bumper 61 of the vehicle body 60 in the display space of the rearview display unit 40. At this time, a gap region where no image exists may be formed between the rearview original video 3 and the virtual image 4 in the display space of the rearview display unit 40. The image superimposing unit 24 embeds an embedded image 5 in this gap region to facilitate discrimination from the virtual image 4. This embedded image 5 is preferably an image having characteristics different from the virtual image 4, for example, by diagonal lines, shading, color, etc. It is more preferable that the embedded image 5 uses a color close to the color of the rearview original video 3 near the gap region, in order to facilitate discrimination from the virtual image 4. Furthermore, as shown in FIG. 10, character information 7 having a value indicating the distance in real space of the gap between the rearview original video 3 and the virtual image 4 may be superimposed on the embedded image 5. Here, the virtual image area shown in Fig. 8 corresponds to a part of the projection area in which the rear part of the vehicle body (including the rear bumper) is projected onto the road surface. The edge of the virtual image area corresponds to the edge of this projection area (projection area of ​​the rear part of the vehicle body).

[0050] (5. Embedding of past footage) A past image, for example an image cut out from an image one frame before, may be used as the embedded image 5. This process is performed using the past rear view original image 3 stored in the buffer while the vehicle body 60 is moving backward.

[0051] FIG. 11 is an explanatory diagram of the embedding process using past video. 60a is the vehicle body at its current position, and 60b is the vehicle body at its position one frame before. vf is the distance the vehicle body has moved at a speed v during one frame, and f is the frame period. The gap area between the original rearview image 3 captured at the current position of the vehicle body 60a and the virtual image 4 is filled with an image in the angle range from θ1 to θ2 with respect to the camera optical axis in the original rearview image 3 captured one frame before. θ1 and θ2 are angles calculated as follows. θ1=θ+φ-tan -1 ((htanφ+vf / h) (4) θ2=θ+φ-tan -1 ((vf+e) / h) (5)

[0052] In this way, by filling the gap area between the lower end of the angle of view of the current camera 10 and the virtual image 4 with the image of the same area from one frame before, a more natural rear view image can be obtained.

[0053] Note that, although the image to be embedded is cut out from the image of the previous frame and embedded here, it is also possible to cut out an image of an area corresponding to the current gap area from an even earlier frame and embed it.

[0054] In FIG. 11 and the above equation (4), htanφ, which is the distance from the position of camera 10 to the position corresponding to the lower end of the angle of view of the image of camera 10, can be obtained by calculation. However, since htanφ is a fixed value for each vehicle body, it can also be obtained by actual measurement.

[0055] 12 and 13 are explanatory diagrams showing the case where a chart is used for measuring htanφ. As shown in FIG. 12, the chart 70 is a sheet or the like on which a scale is marked in increments of, for example, 2 cm and 10 cm. As shown in FIG. 13, during actual measurement, the chart 70 is placed on the road surface so that the position of the base point (0 point) of the chart 70 is aligned with the edge of the rear bumper 61 of the vehicle body 60. The value of the scale visible at the bottom edge of the angle of view of the image 71 of the camera 10 displayed on the screen from the rear view image captured by the camera 10 can be visually obtained as the distance from the edge of the rear bumper 61 to the bottom edge of the angle of view of the image 71 of the camera 10. In the example of FIG. 12, it can be seen that the distance from the edge of the rear bumper 61 to the bottom edge of the angle of view of the camera 10 is 44 cm. The obtained distance is set in the image processing device 20 by manual input into the system, etc., and is used to generate the virtual image superimposed rear view image 6.

[0056] (6. Switching rear view modes) The rearview signal processor 23 has three types of rearview modes, and can switch the display angle of the rearview image based on rearview mode selection information appropriately set by the user.

[0057] The rear view mode includes a normal view mode, a wide view mode, a top-down view mode, etc. The user can arbitrarily select the rear view mode at any time by using an operation button or the like.

[0058] 14A, 14B, and 14C are diagrams showing examples of rear view images in normal view mode, wide view mode, and top-down view mode, respectively. The wide view is a mode in which a rear view image with a wider angle range than the normal view is displayed by capturing an image using a wide-angle lens. As shown in FIG. 15, the top-down view is a mode in which a rear-view image of the road surface directly behind the vehicle is displayed almost from directly above by performing signal processing such as viewpoint conversion on a rear-view image in normal view mode, etc.

[0059] The image superimposing unit 24 performs distortion correction on the virtual image 4 and the embedded image 5 according to the type of rearview mode.

[0060] (7. Overall Control Flow) Next, the flow of operations of the rearview signal processor 23 and the image superimposing unit 24 in the camera system 100 of the second embodiment will be described. FIG. 16 is a flowchart showing the flow of the overall operation of the rearview signal processor 23 and the image superimposing unit 24 in the camera system 100 of the second embodiment.

[0061] The rearview signal processor 23 and the image superimposing unit 24 are activated, for example, by acquiring from the ECU 200 a status indicating that the vehicle body is in reverse gear, and execute the following control.

[0062] After being started, the rearview signal processing unit 23 checks the rearview display mode and holds the result (step S101). Then, the rearview signal processing unit 23 inputs the rearview original video captured by the camera 10 through the splitter 21 (step S102). The input rearview original video is held in a buffer in the image processing device 20 (step S103).

[0063] Based on the confirmation result of the rearview display mode, the image superimposition unit 24 generates a virtual image in a form corresponding to the currently selected rearview display mode, and creates an infill image for the gap between the lower end of the field of view of the original rearview image and the virtual image, and generates a virtual image superimposed rearview image by superimposing these on the original rearview image (steps S104, S105, S106, S107).

[0064] Next, image superimposition unit 24 determines whether or not the vehicle body has started to move backward based on the status from ECU 200 (step S108). If the vehicle body has not started to move backward, rearview signal processing unit 23 outputs the virtual image superimposed rearview video generated in any one of steps S105, S106, and S107 to rearview display unit 40 (step S109). At this time, the embedded image in the virtual image superimposed rearview video is a uniform image such as a diagonal line image, a shaded image, or a color image, which is merely distinguishable from the virtual image.

[0065] Until one frame has elapsed since the vehicle body started to move backward (N in step S110), the image superimposition unit 24 continues to output the virtual image superimposed rearview image generated in any one of steps S105, S106, or S107 to the rearview display unit 40 (step S109).

[0066] When one frame's worth of time has elapsed since the vehicle started to move backward (Y in step S110), the image superimposing unit 24 acquires vehicle speed information from the ECU 200 and checks the value (step S111). The image superimposing unit 24 reads the rearview original video of the past frame from the buffer, and calculates a corresponding area in the rearview original video of the past frame that corresponds to the area of ​​the embedded image in the virtual image superimposed rearview video based on the checked vehicle speed (step S112). Next, the image superimposing unit 24 cuts out the past video of the corresponding area from the past frame video, and performs distortion correction on the cut out past video according to the rearview display mode, thereby generating an embedded image from the past video (step S113).

[0067] Next, the image overlay unit 24 overlays (overwrites) the generated embedded image from the past video onto the area of ​​the embedded image of the current virtual image overlay rearview video (step S114), thereby completing a virtual image overlay rearview video in which the virtual image and the embedded image from the past video are overlayed, and outputs this virtual image overlay rearview video to the rearview display unit 40 (step S115).

[0068] (8. Modifications) In the camera system 100 of the above embodiment, the image processing device 20 includes the splitter 21, the CMS signal processing unit 22, the rearview signal processing unit 23, and the image superimposition unit 24, but the present technology is not limited to this.

[0069] (First Modification) For example, as shown in FIG. 17, instead of configuring the image processing device 20A to acquire vehicle speed information from the ECU 200, a speed detection unit 80 that detects the speed of the vehicle body may be separately provided, and the image superimposition unit 24 may acquire the vehicle speed information from the speed detection unit 80 and adjust the virtual information.

[0070] (Second Modification) As shown in FIG. 18, an image processing device 20B according to the present technology may be configured by excluding at least the CMS signal processing unit 22 from the configuration of the image processing device 20 in FIG. 1 and including a rearview signal processing unit 23 and an image superimposition unit 24.

[0071] (Third Modification) As shown in FIG. 19, the present technology can be provided as a monitor device 20C incorporating a rearview signal processor 23, an image superimposing unit 24, and a display 25 as one system. Alternatively, the rearview signal processing unit 23 and the image superimposing unit 24 may be provided as a single system in the form of a camera device incorporating the camera 10.

[0072] (Fourth Modification) As shown in FIG. 20, the present technology can also be provided as a camera system 100B in which the camera 10, the rearview signal processing unit 23, the image superimposing unit 24, and the display unit 25 are individual devices. Furthermore, the combination of functional blocks built into one device may be changed for each of FIGS. 1, the system may include a camera device incorporating camera 10, splitter 21, rearview signal processing unit 23, and image superimposition unit 24, a display device incorporating CMS signal processing unit 22 and CMS display unit 30, a rearview display unit 40, and ECU 200. Also, the system may include a camera device incorporating camera 10, splitter 21, and CMS signal processing unit 22, a rearview display device incorporating rearview signal processing unit 23, image superimposition unit 24, and rearview display unit 40, and CMS display unit 30 and ECU 200.

[0073] In each of the above embodiments, images from a camera mounted so as to be able to capture an image of the rear of the vehicle body in particular are used, but the present technology can also be applied in the same way when images from a camera capable of capturing an image of the front or sides of the vehicle body are used.

[0074] In each of the above embodiments, the virtual image 4 of the bumper is superimposed on the camera image as the virtual image 4 of a part of the vehicle body 60, but the present technology does not limit where the part of the vehicle body 60 is superimposed on the camera image. Any part that is not within the field of view of the camera 10 installed on the vehicle body 60 and that can easily allow the user to grasp the sense of distance in the camera image may be used. For example, if the side of the vehicle body is imaged by the camera, a virtual image of the side of the vehicle body may be superimposed. Furthermore, even when monitoring the rear and front of the vehicle body, the virtual image of the bumper does not necessarily have to be superimposed. For example, a virtual image of a license plate or a part of the silhouette of the vehicle body may be superimposed.

[0075] Furthermore, the present technology is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention.

[0076] Furthermore, the present technology can also be configured as follows. (1) a first signal processing unit that generates a first image having a first angle of view from an image captured by a camera attached to a vehicle body; an image superimposing unit that superimposes an image representing a part of the vehicle body on the first image; An image processing device comprising:

[0077] (2) The image processing device according to (1), The image of the part of the vehicle body is an image stored in advance in a storage unit. Image processing device.

[0078] (3) The image processing device according to (2), The pre-stored image is an image stored in the first signal processing unit before the image of the camera is acquired. Image processing device.

[0079] (4) Any one of the image processing devices according to (1) to (3) above, The image of the part of the vehicle body is an image of an area outside the field of view of the camera. Image processing device.

[0080] (5) Any one of the image processing devices according to (2) to (4) above, The image of the part of the vehicle body is a virtual image of at least a part of a bumper of the vehicle body. Image processing device.

[0081] (6) Any one of the image processing devices according to (1) to (5) above, The camera is a camera mounted so as to capture an image of the front, rear or side of the vehicle body. Image processing device.

[0082] (7) Any one of the image processing devices according to (1) to (6) above, The image superimposing unit adjusts the image to be superimposed on the first image based on speed information of the vehicle body. Image processing device.

[0083] (8) Any one of the image processing devices according to (1) to (6) above, The image superimposing unit superimposes an embedding image into a gap area between the first video and the virtual image where no image exists. Image processing device.

[0084] (9) The image processing device according to (8), The embedded image is an image that is close to the color of the first image adjacent to the gap region. Image processing device.

[0085] (10) The image processing device according to (8), The image superimposing unit generates the embedding image by cutting out an image of a region corresponding to the gap region from the past first image. Image processing device.

[0086] (11) Any one of the image processing devices according to (1) to (10) above, The image superimposing unit outputs an image in which at least the image is superimposed on the first image to a first display unit. Image processing device.

[0087] (12) Any one of the image processing devices according to (1) to (11) above, a second signal processing unit that generates a second image having a second angle of view different from the first image from the first image and outputs the second image to a second display unit; The image processing device further comprises:

[0088] (13) The image processing device according to (12), The center of the second angle of view is at a higher position in the first image space than the center of the first angle of view. Image processing device.

[0089] (14) The image processing device according to (1), The first image is an image for rear view. Image processing device.

[0090] (15) The image processing device according to (12), The second image is an image equivalent to a rear-view mirror image. Image processing device.

[0091] (16) A camera mounted on the vehicle body; a first signal processing unit that generates a first image having a first angle of view from an image captured by the camera; an image superimposing unit that superimposes an image representing a part of the vehicle body on the first image; A camera system comprising:

[0092] (17) The camera system according to (16), The image of the part of the vehicle body is an image stored in advance in a storage unit. Camera system.

[0093] (18) The camera system according to (17), The pre-stored image is an image stored in the first signal processing unit before the image of the camera is acquired. Camera system.

[0094] (19) Any of the camera systems according to (16) to (18), The image of the part of the vehicle body is an image of an area outside the field of view of the camera. Camera system.

[0095] (20) Any of the camera systems according to (17) to (19) above, The image of the part of the vehicle body is a virtual image of at least a part of a bumper of the vehicle body. Camera system.

[0096] (21) Any of the camera systems according to (16) to (20) above, The camera is a camera mounted so as to capture an image of the front, rear or side of the vehicle body. Camera system.

[0097] (22) Any of the camera systems according to (16) to (21), The image superimposing unit adjusts the image to be superimposed on the first image based on speed information of the vehicle body. Camera system.

[0098] (23) Any of the camera systems according to (16) to (21) above, The image superimposing unit superimposes an embedding image into a gap area between the first video and the virtual image where no image exists. Camera system.

[0099] (24) The camera system according to (23), The embedded image is an image that is close to the color of the first image adjacent to the gap region. Camera system.

[0100] (25) The camera system according to (23), The image superimposing unit generates the embedding image by cutting out an image of a region corresponding to the gap region from the past first image. Camera system.

[0101] (26) Any of the camera systems according to (23) to (25) above, The image superimposing unit outputs an image in which at least the image is superimposed on the first image to a first display unit. Camera system.

[0102] (27) Any of the camera systems according to (16) to (26) above, a second signal processing unit that generates a second image having a second angle of view different from the first image from the first image and outputs the second image to a second display unit; The camera system further comprises:

[0103] (28) The camera system according to (27), The center of the second angle of view is at a higher position in the first image space than the center of the first angle of view. Camera system.

[0104] (29) The camera system according to (16), The first image is an image for rear view. Camera system.

[0105] (30) The camera system according to (27), The second image is an image equivalent to a rear-view mirror image. Camera system.

[0106] (31) A first signal processing unit that generates a first image having a first angle of view from an image captured by a camera attached to the vehicle body; an image superimposing unit that superimposes an image representing a part of the vehicle body on the first image; a display unit that displays a video on which the image is superimposed; A monitor device comprising:

[0107] (32) The monitor device according to (31), The image of the part of the vehicle body is an image stored in advance in a storage unit. Monitoring equipment.

[0108] (33) The monitor device according to (32), The pre-stored image is an image stored in the first signal processing unit before the image of the camera is acquired. Monitoring equipment.

[0109] (34) Any of the monitor devices according to (31) to (33) above, The image of the part of the vehicle body is an image of an area outside the field of view of the camera. Monitoring equipment.

[0110] (35) Any of the monitor devices according to (31) to (34) above, The image of the part of the vehicle body is a virtual image of at least a part of a bumper of the vehicle body. Monitoring equipment.

[0111] (36) Any of the monitor devices according to (31) to (35) above, The camera is a camera mounted so as to capture an image of the front, rear or side of the vehicle body. Monitoring equipment.

[0112] (37) Any of the monitor devices according to (31) to (36) above, The image superimposing unit adjusts the image to be superimposed on the first image based on speed information of the vehicle body. Monitoring equipment.

[0113] (38) Any of the monitor devices according to (31) to (36) above, The image superimposing unit superimposes an embedding image into a gap area between the first video and the virtual image where no image exists. Monitoring equipment.

[0114] (39) The monitor device according to (38), The embedded image is an image that is close to the color of the first image adjacent to the gap region. Monitoring equipment.

[0115] (40) The monitor device according to (37), The image superimposing unit generates the embedding image by cutting out an image of a region corresponding to the gap region from the past first image. Monitoring equipment.

[0116] (41) The monitor device according to (31), The first image is an image for rear view. Monitoring equipment.

[0117] (42) A first signal processing unit generates a first image having a first angle of view from an image captured by a camera attached to the vehicle body; An image superimposing unit superimposes an image showing a part of the vehicle body on the first image. Image processing methods.

[0118] (43) The image processing method according to (42), The image of the part of the vehicle body is an image stored in advance in a storage unit. Image processing methods.

[0119] (44) The image processing method according to (43), The pre-stored image is an image stored in the first signal processing unit before the image of the camera is acquired. Image processing methods.

[0120] (45) Any of the image processing methods according to (42) to (44), The image of the part of the vehicle body is an image of an area outside the field of view of the camera. Image processing methods.

[0121] (46) Any one of the image processing methods according to (42) to (44), The image of the part of the vehicle body is a virtual image of at least a part of a bumper of the vehicle body. Image processing methods.

[0122] (47) Any one of the image processing methods according to (42) to (46), The camera is a camera mounted so as to capture an image of the front, rear or side of the vehicle body. Image processing methods.

[0123] (48) Any of the image processing methods according to (42) to (47), The image superimposing unit adjusts the image to be superimposed on the first image based on speed information of the vehicle body. Image processing methods.

[0124] (49) Any of the image processing methods according to (42) to (47), The image superimposing unit superimposes an embedding image into a gap area between the first video and the virtual image where no image exists. Image processing methods.

[0125] (50) The image processing method according to (49), The embedded image is an image that is close to the color of the first image adjacent to the gap region. Image processing methods.

[0126] (51) The image processing method according to (49), The image superimposing unit generates the embedding image by cutting out an image of a region corresponding to the gap region from the past first image. Image processing methods.

[0127] (52) Any of the image processing methods according to (42) to (51), The image superimposing unit outputs an image in which at least the image is superimposed on the first image to a first display unit. Image processing methods.

[0128] (53) Any of the image processing methods according to (42) to (52), A second signal processing unit generates a second image having a second angle of view different from the first image from the first image, and outputs the second image to a second display unit. Image processing methods.

[0129] (54) The image processing method according to (53), The center of the second angle of view is at a higher position in the first image space than the center of the first angle of view. Image processing methods.

[0130] (56) The image processing method according to (42), The first image is an image for rear view. Image processing methods.

[0131] (57) The image processing method according to (53), The second image is an image equivalent to a rear-view mirror image. Image processing methods. [Explanation of symbols]

[0132] 1...Camera footage 3…Rear view original image 4. Virtual image 5. Embedded images 6…Rear view image 10. Camera 20, 20A, 20B...Image processing device 20C…Monitoring device 22…CMS signal processing section 23...Rearview signal processing unit 24...Image superimposition section 30...CMS display section 40…Rear view display section 60…Body 61…Rear bumper 80...Speed ​​detector 100…Camera system 100B…Camera monitoring system 200…ECU

Claims

1. a first signal processing unit that generates a rearview image from a first video captured by a camera that is attached to a vehicle body and has a first angle of view that does not include the vehicle body in an imaging field of view; a second signal processing unit that generates a second image having a second angle of view different from the first image from the first image and outputs the second image as an image for room monitoring; an image superimposing unit that synthesizes a virtual image representing a part of the vehicle body with respect to the rear view image in a static positional relationship, cuts out an image corresponding to a region corresponding to a gap region where no image exists between the rear view image and the virtual image from a previously captured image, generates an embedded image, and embeds the embedded image in the gap region; Equipped with Image processing device.

2. 2. The image processing device according to claim 1, The virtual image is an image stored in advance in a storage unit. Image processing device.

3. 3. The image processing device according to claim 2, The pre-stored image is an image stored in the first signal processing unit before the image of the camera is acquired. Image processing device.

4. 2. The image processing device according to claim 1, The virtual image is an image including at least a part of a front or rear bumper of the vehicle body. Image processing device.

5. 2. The image processing device according to claim 1, The embedded image is an image having a color similar to that of the first image adjacent to the gap region. Image processing device.

6. 2. The image processing device according to claim 1, The image superimposing unit is configured to output an image in which at least the virtual image and the embedded image are combined with the first image to a first display unit.

7. An image processing device according to claim 1, The center of the second angle of view is at a higher position in the first image space than the center of the first angle of view. Image processing device.

8. a camera attached to a vehicle body and having a first angle of view that does not include the vehicle body in an imaging field of view; a first signal processing unit that generates a rearview image from a first image captured by the camera; a second signal processing unit that generates a second image having a second angle of view different from the first image from the first image and outputs the second image as an image for room monitoring; an image superimposing unit that synthesizes a virtual image representing a part of the vehicle body with respect to the rear view image in a static positional relationship, cuts out an image corresponding to a region corresponding to a gap region where no image exists between the rear view image and the virtual image from a previously captured image, generates an embedded image, and embeds the embedded image in the gap region; A camera system comprising:

9. a first signal processing unit generates a rearview image from a first video captured by a camera attached to the vehicle body and having a first angle of view in which the vehicle body is not included in an imaging field of view; a second signal processing unit generates a second image having a second angle of view different from the first angle of view from the first image, and outputs the second image as an image for room monitoring; An image superimposing unit synthesizes a virtual image expressing a part of the vehicle body with respect to the rear view image in a static positional relationship, cuts out an image corresponding to a region corresponding to a gap region where no image exists between the rear view image and the virtual image from a previously captured image, generates an embedded image, and embeds the embedded image in the gap region. Image processing methods.

Images