Image processing device, camera system and image processing method

By superimposing a virtual body bumper in the image processing device, the problem of not including bumpers in the field of view is solved, high-quality rearview image processing is realized, and the driver's sense of distance to the rearview image space is improved Reasoning to reduce costs and improve vehicle safety.

CN113039781BActive Publication Date: 2024-11-08SONY SEMICON SOLUTIONS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980075070.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-11-22
Filing Date
2019-11-21
Publication Date
2024-11-08
Estimated Expiration
2039-11-21

AI Technical Summary

Technical Problem

Existing rear-view cameras and CMS cameras have different captured field heights, making it difficult to achieve high-quality rear-view and CMS images through one camera, especially because the capture field of view of the CMS camera does not include the bumper of the body, making it difficult to grasp. The sense of distance in the rearview image.

Method used

Through the image processing device, an image of a first viewing angle is generated by the first signal processing unit, and an image representing a part of the vehicle body is superimposed on the first image by the image superposition unit, including a bumper of the virtual vehicle body, adjusting the position of the virtual image based on the Vehicle speed, generating rearview images that are easy for driver reference.

Benefits of technology

It realizes the high-quality rearview image provided through a camera, helping the driver better grasp the sense of distance behind the vehicle body, reducing the total cost and improving the safety of the vehicle.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113039781B_ABST
    Figure CN113039781B_ABST
Patent Text Reader

Abstract

An image processing device includes: a first signal processing unit that generates a first image at a first viewing angle based on an image captured by a camera attached to a vehicle body; and an image superimposing unit that superimposes an image representing a portion of the vehicle body on the first image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image processing device, a camera system, a monitoring device, and an image processing method for processing an image of a camera installed in a vehicle body. Background Art

[0002] Recently, when the vehicle body is moved backward during parking, etc., the number of vehicles equipped with rear-view cameras for supporting parking operations is increasing, by presenting the driver with a rear image captured by the camera through the screen of the car navigation system. Many rear-view cameras display an image of the rear of the vehicle body including the rear bumper on the rear-view screen by including the rear bumper in the captured field of view. Therefore, the driver will grasp the sense of distance in the rear-view image space with reference to the rear bumper (see Patent Document 1).

[0003] On the other hand, in order to realize vehicles that do not require rearview mirrors, etc., there is also an increase in vehicle bodies equipped with camera monitoring systems (CMS) instead of rearview mirrors, etc. CMS is expected to realize vehicles with higher safety such as providing rear images with smaller blind spots and wider fields of view.

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2005-150938 ( Figure 3 , paragraphs

[0016] and

[0029] ) Summary of the invention

[0007] Problems to be solved by the present invention

[0008] Although both the above-mentioned rear-view camera and CMS provide the driver with an image of the rear of the vehicle body, the rear-view camera is intended to monitor the conditions behind the vehicle body. In contrast, the CMS is intended to monitor the following vehicles of the vehicle body. Therefore, the rear-view camera and the CMS camera have different capture field heights. For this reason, separate cameras with different capture fields of view are generally used respectively. However, in order to suppress the total cost, it is expected to realize rear-view and CMS by one camera or realize rear-view by using a camera for CMS.

[0009] However, when attempts are made to use the images of a CMS camera to achieve rearward vision, various problems have occurred due to the fact that the bumper of the vehicle body is not included in the capture field of view of the CMS camera.

[0010] In view of the above circumstances, an object of the present technology is to provide an image processing device, a camera system, and an image processing method capable of providing a high-quality rear view image.

[0011] Solution to the problem

[0012] In order to solve the above problems, the image processing device according to the present technology includes: a first signal processing unit for generating a first image of a first perspective based on an image captured by a camera attached to a vehicle body; and an image superimposition unit for superimposing an image representing a part of the vehicle body on the first image.

[0013] According to the image processing device according to the present technology, by superimposing an image representing a portion of the vehicle body on a first image generated from an image captured by a camera, a captured image in which a user can easily grasp a sense of distance by referring to the portion of the vehicle body can be obtained.

[0014] The image of the portion of the vehicle body may be a pre-stored image.

[0015] More specifically, the pre-stored image may be an image stored in the first signal processing unit before acquiring the image of the camera.

[0016] The image of the portion of the vehicle body may be an image of an area outside the field of view of the camera.

[0017] The image of the portion of the vehicle body may be a virtual image obtained by virtualizing at least a portion of a bumper of the vehicle body.

[0018] The camera is a camera that is attached to capture the front, rear or sides of the vehicle body.

[0019] The image superimposing unit may be configured to adjust the virtual image superimposed on the first image based on speed information of the vehicle body.

[0020] The image superimposing unit may be configured to superimpose the embedded image on a gap region in which an image between the first image and the virtual image does not exist.

[0021] The embedded image may be an image having a hue close to that of the first image adjacent to the gap region.

[0022] The image superposition unit may be configured to crop an image of an area corresponding to the gap area from the past first image to generate the embedded image.

[0023] The image superimposing unit may be configured to output an image in which at least the image is superimposed on the first image to the first display unit.

[0024] The image processing device according to the present technology may further include a second signal processing unit that generates a second image having a second viewing angle different from the first viewing angle from the first image and outputs the second image to the second display unit.

[0025] In the first image space, the center of the second perspective may be higher than the center of the first perspective.

[0026] A camera system according to another aspect of the present technology includes: a camera attached to a vehicle body; a first signal processing unit for generating a first image of a first perspective based on an image captured by the camera; and an image superimposition unit for superimposing an image representing a portion of the vehicle body on the first image.

[0027] According to another aspect of the present technology, an image processing device includes: a first signal processing unit for generating a first image of a first perspective based on an image captured by a camera attached to a vehicle body; an image superposition unit for superimposing an image representing a portion of the vehicle body on the first image; and a speed detection unit for detecting the speed of the vehicle body, wherein the image superposition unit is configured to adjust the image superimposed on the image based on the detected speed.

[0028] A signal processing method according to another aspect of the present technology includes: generating, by a first signal processing unit, a first image of a first perspective based on an image captured by a camera attached to a vehicle body; and superimposing, by an image superimposing unit, an image representing a portion of the vehicle body on the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] [ Figure 1 ] is a block diagram showing the overall configuration of a camera system 100 according to the present technology.

[0030] [ Figure 2 ] is shown Figure 1 1 is a side view of the attachment position and orientation of camera 10 in camera system 100 .

[0031] [ Figure 3 ] is shown Figure 2 Schematic diagram of an example of a view image of camera 10.

[0032] [ Figure 4 ] is a diagram showing an example of the positional relationship between the image space of camera 10 and the CMS image space.

[0033] [ Figure 5 ] is a diagram showing a configuration example of the rear view image 6.

[0034] [ Figure 6 ] is a diagram showing another configuration example of the rear view image 6.

[0035] [ Figure 7 ] is a diagram illustrating a method of adjusting the virtual image 4 according to the vehicle speed.

[0036] [ Figure 8 ] is an illustrative view of a method of superimposing a virtual image 4 relative to a rear-view original image 3 in a static positional relationship according to a second embodiment of the present technology.

[0037] [ Fig. 9 ] is a diagram showing an example of a rear view image in the second embodiment.

[0038] [ Fig.10 ] is a diagram showing another example of a rear view image in the second embodiment.

[0039] [ Fig.11 ] is an illustrative view of the embedding process through past images.

[0040] [ Fig.12 ] is a diagram illustrating a method of actually measuring the distance from the position of the camera 10 to the position corresponding to the lower end of the viewing angle of the camera 10 using a graph.

[0041] [ Fig.13 ] is a diagram for explaining the installation method of the chart.

[0042] [ Fig.14A ] is a diagram showing an example of a rear view image in a normal view mode.

[0043] [ Fig. 14B ] is a diagram showing an example of a rear view image in a wide-angle view mode.

[0044] [ Fig. 14C ] is a diagram showing an example of a rear view image in a top-down view mode.

[0045] [ Fig.15 ] is a diagram showing the camera viewpoint of the top-down view mode after transformation.

[0046] [ Fig.16 ] is a flowchart showing the overall operation flow of the rear view signal processing unit 23 and the image superposition unit 24 in the camera system 100 of the second embodiment.

[0047] [ Fig.17 ] is a block diagram showing a first variation of the configuration of the image processing device according to the present technology.

[0048] [ Fig.18 ] is a block diagram showing a second variation of the configuration of the image processing device according to the present technology.

[0049] [ Fig.19 ] is a block diagram showing the configuration of a monitoring device which is a third variation of the image processing device according to the present technology.

[0050] [ Fig. 20 ] is a block diagram showing the configuration of a camera system as a fourth variation of the image processing device according to the present technology. DETAILED DESCRIPTION

[0051] An embodiment of the present technology will be described in the following order with reference to the drawings.

[0052] 1. Overall system configuration

[0053] 2. Configuration of the Image Processing Device 20

[0054] 3. Image overlay processing

[0055] 4. Adjust the virtual image according to the vehicle speed

[0056] 5. Embedding of Past Images

[0057] 6. Switch to rear view mode

[0058] 7. Overall control process

[0059] 8. Variant

[0060] <First Embodiment>

[0061] (1. Overall system configuration)

[0062] Figure 1 is a block diagram showing the overall configuration of a camera system 100 according to the present technology.

[0063] The camera system 100 includes a camera 10 , an image processing device 20 , a CMS display unit 30 , and a rear view display unit 40 .

[0064] The CMS display unit 30 and the rear view display unit 40 are separate monitors that can be viewed by the driver of the vehicle body 60. The CMS display unit 30 may be a monitor or the like that replaces an interior mirror such as a rearview mirror, a reverse mirror, etc. The rear view display unit 40 may be, for example, a screen of a car navigation system.

[0065] Figure 2 It shows Figure 1 A side view of the attachment position and orientation of the camera 10 in the camera system 100, and Figure 3 It shows Figure 2 1 is a diagram showing an example of a view image of the camera 10. Note that the present embodiment assumes that the camera 10 is arranged behind the vehicle body so as to be able to capture the rear of the vehicle body 60. Without being limited thereto, the camera 10 may be arranged so as to be able to capture the front of the vehicle body 60.

[0066] The camera 10 is a digital video camera equipped with a CCD sensor or a CMOS sensor, preferably a high-resolution (HD: high definition) digital video camera. Figure 2 and Figure 3As shown in , the camera 10 of the camera system 100 is installed so that the optical axis is approximately horizontal or slightly tilted downward at a relatively high position such as the top portion of the rear of the vehicle body 60 so as to be able to capture a distant location. With such a camera arrangement, a bumper such as the rear bumper 61 of the vehicle body 60 is outside the field of view of the camera 10. Therefore, for example, when an attempt is made to use the camera system 100 instead of a rearview mirror to monitor the rear of the vehicle body, there is no image of the rear bumper 61, resulting in the presence of an image from the camera 10 (see Figure 3 ) It is difficult to grasp the distance sense behind the vehicle body. This is one of the problems to be solved by this technology.

[0067] (2. Configuration of Image Processing Device 20)

[0068] The image processing device 20 includes: at least a first signal processing unit (rearview signal processing unit 23) for generating a first image of a first perspective based on an image captured by a camera 10 attached to the vehicle body 60; and an image overlay unit 25 for overlaying an image representing a portion of the vehicle body 60 on the first image.

[0069] More specifically, the image processing device 20 generates an image for the CMS based on the image captured by the camera 10 and supplies it to the CMS display unit 30, and at the same time generates an original image for rear vision based on the image captured by the camera 10, generates a rear vision image in which at least an image such as a virtual rear bumper (hereinafter, referred to as a "virtual image") is superimposed on the original image for rear vision and supplies it to the rear vision display unit 40.

[0070] Note that methods of superimposing a virtual image on an original image for rear view include a method of adjusting the position of the virtual image based on the vehicle speed and a method of superimposing the virtual image at a fixed position regardless of the vehicle speed, and the former method will be described in the first embodiment and the latter method will be described in the second embodiment.

[0071] like Figure 1 As shown in , the image processing device 20 includes, for example, a separator 21, a CMS signal processing unit 22 (second signal processing unit), a rear view signal processing unit 23, an image superposition unit 24, etc. For example, the image processing device 20 includes an integrated circuit such as an IC (Integrated Circuit) and an LSI (Large Scale Integrated Circuit) and a memory for buffering.

[0072] The separator 21 converts the image 1 captured by the camera 10 (see Figure 3 ) is supplied to the CMS signal processing unit 22 and the rearview signal processing unit 23.

[0073] The CMS signal processing unit 22 generates a CMS image to be supplied to the CMS display unit 30 by cropping an image of a target area for CMS signal processing from the image 1 supplied via the separator 21 and performing image quality adjustment such as distortion correction and white balance adjustment depending on a viewing mode.

[0074] Figure 4 is a diagram showing an example of the positional relationship between the image space of the camera 10 and the CMS image space. As shown in the same figure, the CMS signal processing unit 22 generates a CMS image by cutting out an image area 2 corresponding to the image angle portion from the image 1 of the camera as a target area for CMS signal processing, so that a distant subject is treated as a main subject and the horizontal line S is slightly lower than the height of approximately the middle.

[0075] The rear view signal processing unit 23 generates a raw image for rear view by cutting out the image of the target area of ​​rear view signal processing from the image 1 of the camera and performing image quality adjustment such as distortion correction and white balance adjustment depending on the viewing mode. Note that the rear view signal processing unit 23 can process all pixels of the image 1 of the camera 10. Figure 4 A case where all pixels of the image 1 of the camera 10 are used as the original image 3 for rear view in this manner is shown. The original image 3 for rear view refers to an image of the camera including only the image of the camera 10 before a virtual image or an embedded image of a bumper to be described later is superimposed, and is hereinafter referred to as the "rear view original image 3". An image obtained by superimposing a virtual image or an embedded image of a bumper to be described later on the rear view original image 3 is the "rear view image 6".

[0076] The image processing device 20 has a communication interface (not shown) for communicating with an ECU (engine control unit) 200 mounted on the vehicle body 60. The communication protocol with the ECU 200 includes CAN (controller area network) and the like. The image superimposing unit 24 receives state information such as vehicle speed information from the ECU 200 using the above-mentioned communication interface, and adjusts the position of the virtual image based on the received vehicle speed information. The details of the superimposing process of the virtual image by the image superimposing unit 24 will be described later.

[0077] (3. Image overlay processing)

[0078] Figure 5is a diagram showing a configuration example of a rear view image 6. The image superimposition unit 24 superimposes an image representing a portion of a vehicle body such as a rear bumper that does not exist in the field of view of the camera 10 as a virtual image 4 on the rear view original image 3 generated by the rear view signal processing unit 23, and generates a rear view image 6. The virtual image 4 is an image created in advance, for example, by computer graphics, etc., and is information stored in a non-volatile memory (not shown) in the image processing device 20 before the image is supplied from the camera 10 to the image processing device 20 or before the system is started. Note that the image to be superimposed on the rear view original image 3 is not necessarily an image created by computer graphics, etc., and may be an image processed from a photograph, for example. In this specification, images created by computer graphics, etc. and processed image photography are collectively referred to as "virtual image 4".

[0079] Note that in Figure 5 , a case where a rear view image 6 is generated by using all pixels of an image 1 of a camera 10 as a rear view original image 3 is shown, but the present technology is not limited thereto. Figure 6 As shown in , a partial rectangular area including the lowest area of ​​the image 1 of the camera 10 may be used as the rearward-view original image 3a, and a rearward-view image 6a may be generated by superimposing the virtual image 4 and the like on the partial rectangular area.

[0080] (4. Adjust the virtual image according to the vehicle speed)

[0081] The image superimposing unit 24 can adjust the position of the virtual image 4 to be superimposed on the rear view image 6 and the like based on the vehicle speed information obtained from the ECU 200 .

[0082] Figure 7 2 is a diagram for explaining a method of adjusting the virtual image 4 according to the vehicle speed.

[0083] In the same figure, h is the installation height of the camera 10, θ is the angle between the lower end of the viewing angle of the camera 10 and the camera optical axis of the light beam, is the angle between the lower end of the viewing angle of the camera 10 and the vertical line of the light beam, and t is a timing parameter preset in the image superposition unit 24 .

[0084] Here, it is assumed that the slope of the optical axis of the camera 10 relative to the horizontal line is It is calculated by the following formula:

[0085] The timing parameter t is a parameter for displaying the virtual image 4 taking the point where the rear bumper of the vehicle body 60 reaches after t seconds at the current speed as the end when the vehicle body moves backward. The timing parameter t can be arbitrarily changed by the user. Here, the virtual image shows the area where the vehicle reaches after t seconds / within t seconds when the vehicle moves backward at the current speed.

[0086] In addition, the virtual image may be superimposed and displayed as a semi-transparent image so that the driver can confirm the state of the area of ​​the rear-view original image 3 on which the virtual image is superimposed.

[0087] First, the image superposition unit 24 determines the display position of the virtual image 4 according to h, t and the vehicle speed v by the following formula. If the viewing angle of the rear view original image 3 is set to the display angle of the entire rear view, the calculation target of the display position of the virtual image 4 is how far the position from the lower end of the display angle of the entire rear view is set as the upper end of the virtual image 4. The upper end position of the virtual image 4 corresponds to a point moved by vt+e from the position of the camera 10. Therefore, the image superposition unit 24 first calculates vt+e. By the way, e is the distance from the position of the camera 10 to the end of the rear bumper 61.

[0088] Next, α is calculated based on the calculation result of vt+e. α is the angle of the direction from the camera 10 to the point vt+e on the road surface relative to the vertical line. α is calculated by the following formula: α=tan -1 ((vt+e) / h)(2).

[0089] Therefore, the greater the vehicle speed, Figure 7 The upper end position of the virtual image 4 in the image is closer to the left.

[0090] Next, the image superimposing unit 24 superimposes the virtual image 4 in the area from the lower end of the rear view original image 3 to the position corresponding to the angle α calculated by the above formula (2) on the rear view original image 3 (the upper end position of the virtual image 4). Therefore, the virtual image 4 is superimposed on the appropriate position of the rear view original image 3, thereby obtaining a rear view image 6 with a sense of distance relative to the vehicle body 60 that can be easily grasped.

[0091] Incidentally, if the vehicle speed v is too large, most of the area of ​​the rear view image 6 is occupied by the virtual image 4, and the space behind the vehicle body visible to the user through the rear view display unit 40 becomes too narrow. Therefore, only when the following conditions are met: The image superimposition unit 24 superimposes the virtual image 4 only when .

[0092] If the above formula (3) is not satisfied, only the rear view original image 3 is displayed as the rear view image 6 on the rear view display unit 40. As a result, it is possible to prevent the image space behind the vehicle body that the user can see through the rear view display unit 40 from becoming too narrow.

[0093] <Second Embodiment>

[0094] Next, a second embodiment according to the present technology will be described.

[0095] In the present embodiment, the image superimposing unit 24 superimposes the virtual image 4 on the rear-view original image 3 in a static positional relationship without depending on the vehicle speed.

[0096] Figure 8 is an explanatory view of a method of superimposing a virtual image 4 in a static positional relationship with respect to a rear-view original image 3 in the second embodiment, and Fig. 9 : is a diagram showing an example of a rear view image in the present embodiment.

[0097] As shown in these figures, the virtual image 4 is arranged in the display space of the rearview display unit 40 at a position corresponding to the area from the position of the camera 10 to the end of the rear bumper 61 of the vehicle body 60. At this time, in the display space of the rearview display unit 40, there may be a gap area in which there is no image between the rearview original image 3 and the virtual image 4. The image superposition unit 24 embeds the embedded image 5 in the gap area to help distinguish it from the virtual image 4. For example, the embedded image 5 is preferably an image having features that are different from those of the virtual image 4 by hatching, shadows, colors, etc. For the embedded image 5, it is more preferable to use a color that is close to the hue of the rearview original image 3 near the gap area because it is easy to distinguish it from the virtual image 4. In addition, as Fig.10 As shown in FIG. 1 , the embedded image 5 may be superimposed with character information 7 having a value indicating a distance of a gap between the rear-view original image 3 and the virtual image 4 in the real space.

[0098] (5. Embedding of past images)

[0099] For the embedded image 5, a past image, for example, an image cut out from an image one frame ago, can be used. This process is performed by using the past rearward-view original image 3 held in the buffer when the vehicle body 60 moves backward.

[0100] Fig.11 is an illustrative view of the embedding process through past images.

[0101] 60a is the vehicle body at the current position, and 60b is the vehicle body at the position one frame ago. Vf is the distance the vehicle body moves at speed v between frames, and f is the frame period.

[0102] The gap area between the rear view original image 3 captured at the current position of the vehicle body 60a and the virtual image 4 is filled with an image having an angle range of θ1 to θ2 relative to the camera optical axis in the rear view original image 3 captured one frame ago. θ1 and θ2 are angles calculated by:

[0103] Therefore, by filling the gap area between the lower end of the viewing angle obtained by the current camera 10 and the virtual image 4 with the image of the same area one frame ago, a more natural rear view image can be obtained.

[0104] It should be noted that although the image for embedding is cropped from the image one frame before and embedded here, the image of the area corresponding to the current gap area may be cropped from the past frame and embedded.

[0105] By the way, in Fig.11 In the above formula (4), is the distance from the position of the camera 10 to the position corresponding to the lower end of the viewing angle of the camera 10 and can be determined by calculation, but since for each vehicle body is a fixed value and can therefore also be determined through actual measurement.

[0106] Fig.12 and Fig.13 Use the chart for An illustration of an actual measurement situation.

[0107] like Fig.12 As shown in FIG. 7 , the graph 70 is, for example, a table on which numerical scales are written in increments of 2 cm and 10 cm. Fig.13 As shown in , in actual measurement, the chart 70 is arranged on the road surface so as to align the position of the base point (0 point) of the chart 70 with the end of the rear bumper 61 of the vehicle body 60. The value of the numerical scale visible at the lower end of the viewing angle of the image 71 of the camera 10 displayed on the screen in the rear view image captured by the camera 10 can be visually obtained as the distance between the end of the rear bumper 61 and the lower end of the viewing angle of the image 71 of the camera 10. Fig.12 In the example of , it can be seen that the distance from the end of the rear bumper 61 to the lower end of the viewing angle of the camera 10 is 44 cm. The obtained distance is set in the image processing device 20 by manual input to the system, etc., and is used to generate the rear view image 6 superimposed with the virtual image.

[0108] (6. Switch to rear view mode)

[0109] The rear view signal processing unit 23 has three types of rear view modes, and can switch the display angle of the rear view image based on rear view mode selection information appropriately set by the user.

[0110] The rear view mode includes a normal view mode, a wide angle view mode and a top-down view mode. The user can arbitrarily select the rear view mode at any time by operating a button or the like.

[0111] Fig.14A , Fig. 14B and Fig. 14C 2 are diagrams showing examples of respective rear view images of the normal view mode, the wide-angle view mode, and the top-down view mode.

[0112] The wide-angle view is in a mode of displaying a rear view image of a wide angle range compared with a normal view by capturing using a wide-angle lens.

[0113] like Fig.15 As shown in , the top-down view is a mode in which a rear view image viewed from approximately directly above the road surface directly behind the vehicle body is displayed through signal processing such as viewpoint conversion of the rear view image from the normal view mode.

[0114] The image superposition unit 24 performs distortion correction of the virtual image 4 and the embedded image 5 corresponding to the type of the rearward viewing mode.

[0115] (7. Overall control process)

[0116] Next, the operation flow of the rear view signal processing unit 23 and the image superposition unit 24 in the camera system 100 in the second embodiment will be described.

[0117] Fig.16 : is a flowchart showing the overall operation flow of the rear view signal processing unit 23 and the image superposition unit 24 in the camera system 100 in the second embodiment.

[0118] The rear view signal processing unit 23 and the image superposition unit 24 are activated by obtaining a state indicating that the vehicle body is shifted to the reverse gear from the ECU 200, for example, and perform the following control.

[0119] After starting, the rear view signal processing unit 23 confirms the rear view display mode and holds the result (step S101). Subsequently, the rear view signal processing unit 23 inputs the rear view original image captured by the camera 10 through the separator 21 (step S102). The input rear view original image is held in a buffer in the image processing device 20 (step S103).

[0120] The image overlay unit 24 generates a virtual image in a form corresponding to the currently selected rearview display mode based on the confirmation result of the rearview display mode, generates an embedded image of the gap between the lower end of the viewing angle of the rearview original image and the virtual image, and overlays the embedded image on the rearview original image to generate a rearview image overlaid with the virtual image (steps S104, S105, S106, S107).

[0121] Next, the image superimposing unit 24 determines whether the vehicle body is started to move backward based on the status from the ECU 200 (step S108). If the vehicle body has not been started to move backward, the rear view signal processing unit 23 outputs the rear view image superimposed with the virtual image generated by any of steps S105, S106 or S107 to the rear view display unit 40 (step S109). At this time, the embedded image in the rear view image superimposed with the virtual image is a unified image that can be easily distinguished from a virtual image such as a hatched image, a shadow image or a color image.

[0122] During the period from when the vehicle body starts to move backward until one frame of time has passed (No in step S110), the image superimposition unit 24 continues to output the rear view image superimposed with the virtual image generated by any one of steps S105, S106 or S107 to the rear view display unit 40 (step S109).

[0123] When one frame of time has passed since the vehicle body started to move backward (Yes in step S110), the image superimposing unit 24 acquires the vehicle speed information from the ECU 200 and confirms the value (step S111). The image superimposing unit 24 reads the rear view original image of the past frame from the buffer, and calculates the corresponding area corresponding to the area of ​​the embedded image in the rear view image of the superimposed virtual image in the rear view original image of the past frame based on the confirmed vehicle speed (step S112). Subsequently, the image superimposing unit 24 crops the past image of the corresponding area from the past frame image, and performs distortion correction corresponding to the rear view display mode on the cropped past image to generate an embedded image based on the past image (step S113).

[0124] Next, the image overlay unit 24 superimposes (overwrites) the generated embedded image of the past image onto the area of ​​the embedded image of the rearview image of the current superimposed virtual image (step S114), thereby completing the rearview image of the superimposed virtual image on which the virtual image and the embedded image of the past image are superimposed, and outputs the rearview image of the superimposed virtual image to the rearview display unit 40 (step S115).

[0125] (8. Variant)

[0126] In the camera system 100 of the above-described embodiment, the image processing device 20 includes the separator 21 , the CMS signal processing unit 22 , the rear view signal processing unit 23 , and the image superposition unit 24 , but the present technology is not limited thereto.

[0127] (First variant)

[0128] For example, Fig.17 As shown in , instead of being configured to acquire vehicle speed information from the ECU 200, the image processing device 20A may also include a speed detection unit 80 for detecting the speed of the vehicle body, and the image superposition unit 24 may acquire the vehicle speed information from the speed detection unit 80 and adjust the virtual information.

[0129] (Second variant)

[0130] like Fig.18 As shown in Figure 1 In the configuration of the image processing device 20 , at least the CMS signal processing unit 22 is omitted, and the image processing device 20B according to the present technology can be configured by the rearview signal processing unit 23 and the image superposition unit 24 .

[0131] (Third variant)

[0132] like Fig.19 As shown in , the present technology can be provided as a monitoring device 20C that internally includes a rear view signal processing unit 23, an image superposition unit 24, and a display 25 as one system.

[0133] Alternatively, it may be provided as a camera device which internally includes the camera 10 including the rear view signal processing unit 23 and the image superposition unit 24 as a system.

[0134] (Fourth variant)

[0135] like Fig. 20 As shown in , the present technology can also be provided as a camera system 100B having the camera 10 , the rear view signal processing unit 23 , the image superposition unit 24 , and the display 25 as separate devices.

[0136] In addition, the combination of functional blocks included in a device can be relative to Figure 1 , Fig.17 and Fig. 20 The functional blocks shown in FIG.

[0137] For example, in Figure 1In the embodiment, the system may include: a camera device, which internally includes a camera 10, a separator 21, a rear view signal processing unit 23, and an image superposition unit 24; a display device, which internally includes a CMS signal processing unit 22, a CMS display unit 30, a rear view display unit 40, and an ECU 200. In addition, the system may include: a camera device, which internally includes a camera 10, a separator 21, and a CMS signal processing unit 22; a rear view display device, which internally includes a rear view signal processing unit 23, an image superposition unit 24, and a rear view display unit 40; a CMS display unit 30; and an ECU 200.

[0138] In the above embodiments, images of cameras attached particularly to be able to capture the rear of the vehicle body are used, but the present technology is also applicable to the case of using images of cameras capable of capturing the front and sides of the vehicle body.

[0139] In the above embodiments, the virtual image 4 of the bumper is superimposed on the camera image as the virtual image 4 of a portion of the vehicle body 60, but the position of the portion of the vehicle body 60 superimposed on the camera image is not limited in the present technology. Any portion is possible as long as the portion does not enter the field of view of the camera 10 installed in the vehicle body 60 and the portion allows the user to easily grasp the sense of distance in the space of the camera image. For example, if the side of the vehicle body is captured by the camera, a virtual image of the side of the vehicle body can be superimposed. In addition, if the rear and front of the vehicle body are monitored, the virtual image of the bumper should not necessarily be superimposed. For example, a virtual image of a license plate or a portion of the outline of the vehicle body can be superimposed.

[0140] Furthermore, the present technology is not limited to the above-described embodiments, and various modifications may be made without departing from the gist of the present invention.

[0141] Furthermore, the present technology can also be configured as follows.

[0142] (1) An image processing device comprising:

[0143] a first signal processing unit for generating a first image of a first perspective based on an image captured by a camera attached to the vehicle body; and

[0144] An image superimposing unit is used to superimpose an image representing a portion of the vehicle body on the first image.

[0145] (2) The image processing device according to (1), wherein

[0146] The image of the portion of the vehicle body is a pre-stored image.

[0147] (3) The image processing device according to (2), wherein

[0148] The pre-stored image is an image stored in the first signal processing unit before acquiring the image of the camera.

[0149] (4) The image processing device according to any one of (1) to (3), wherein

[0150] The image of the portion of the vehicle body is an image of an area outside the field of view of the camera.

[0151] (5) The image processing device according to any one of (2) to (4), wherein

[0152] The image of the portion of the vehicle body is a virtual image obtained by virtualizing at least a portion of a bumper of the vehicle body.

[0153] (6) The image processing device according to any one of (1) to (5), wherein

[0154] The camera is a camera attached to capture the front, rear or side of the vehicle body.

[0155] (7) The image processing device according to any one of (1) to (6), wherein

[0156] The image superimposing unit adjusts an image superimposed on the first image based on speed information of the vehicle body.

[0157] (8) The image processing device according to any one of (1) to (6), wherein

[0158] The image superimposing unit superimposes the embedded image on a gap area where an image between the first image and the virtual image does not exist.

[0159] (9) The image processing device according to (8), wherein

[0160] The embedded image is an image having a hue close to a hue of the first image adjacent to the gap area.

[0161] (10) The image processing device according to (8), wherein

[0162] The image superposition unit cuts out an image of a region corresponding to the gap region from the past first image to generate the embedded image.

[0163] (11) The image processing device according to any one of (1) to (10), wherein

[0164] The image superimposing unit outputs an image in which at least the image is superimposed on the first image to the first display unit.

[0165] (12) The image processing device according to any one of (1) to (11), further comprising:

[0166] A second signal processing unit is provided, wherein the second signal processing unit generates a second image having a second viewing angle different from the first viewing angle according to the first image, and outputs the second image to a second display unit.

[0167] (13) The image processing device according to (12), wherein

[0168] In the first image space, the center of the second viewing angle is higher than the center of the first viewing angle.

[0169] (14) The image processing device according to (1), wherein

[0170] The first image is an image for rearward viewing.

[0171] (15) The image processing device according to (12), wherein

[0172] The second image is an image equivalent to the room mirror image.

[0173] (16) A camera system comprising:

[0174] a camera, attached to the vehicle body;

[0175] a first signal processing unit, configured to generate a first image of a first viewing angle according to the image captured by the camera; and

[0176] An image superimposing unit is used to superimpose an image representing the portion of the vehicle body onto the first image.

[0177] (17) The camera system according to (16), wherein

[0178] The image of the portion of the vehicle body is a pre-stored image.

[0179] (18) The camera system according to (17), wherein

[0180] The pre-stored image is an image stored in the first signal processing unit before acquiring the image of the camera.

[0181] (19) The camera system according to any one of (16) to (18), wherein

[0182] The image of the portion of the vehicle body is an image of an area outside the field of view of the camera.

[0183] (20) The camera system according to any one of (17) to (19), wherein

[0184] The image of the portion of the vehicle body is a virtual image obtained by virtualizing at least a portion of a bumper of the vehicle body.

[0185] (21) The camera system according to any one of (16) to (20), wherein

[0186] The camera is a camera attached to capture the front, rear or side of the vehicle body.

[0187] (22) The camera system according to any one of (16) to (21), wherein

[0188] The image superimposing unit adjusts an image superimposed on the first image based on speed information of the vehicle body.

[0189] (23) The camera system according to any one of (16) to (21), wherein

[0190] The image superimposing unit superimposes the embedded image on a gap area where an image between the first image and the virtual image does not exist.

[0191] (24) The camera system according to (23), wherein

[0192] The embedded image is an image having a hue close to a hue of the first image adjacent to the gap area.

[0193] (25) The camera system according to (23), wherein

[0194] The image superposition unit cuts out an image of a region corresponding to the gap region from the past first image to generate the embedded image.

[0195] (26) The camera system according to any one of (23) to (25), wherein

[0196] The image superimposing unit outputs an image in which at least the image is superimposed on the first image to the first display unit.

[0197] (27) The camera system according to any one of (16) to (26), further comprising:

[0198] A second signal processing unit is provided, wherein the second signal processing unit generates a second image having a second viewing angle different from the first viewing angle according to the first image, and outputs the second image to a second display unit.

[0199] (28) The camera system according to (27), wherein

[0200] In the first image space, the center of the second viewing angle is higher than the center of the first viewing angle.

[0201] (29) The camera system according to (16), wherein

[0202] The first image is an image for rearward viewing.

[0203] (30) The camera system according to (27), wherein

[0204] The second image is an image equivalent to the room mirror image.

[0205] (31) A monitoring device comprising:

[0206] a first signal processing unit for generating a first image of a first perspective based on an image captured by a camera attached to the vehicle body;

[0207] an image superimposing unit for superimposing an image representing the portion of the vehicle body onto the first image; and

[0208] The display unit is used to display the superimposed image.

[0209] (32) The monitoring device according to (31), wherein

[0210] The image of the portion of the vehicle body is a pre-stored image.

[0211] (33) The monitoring device according to (32), wherein

[0212] The pre-stored image is an image stored in the first signal processing unit before acquiring the image of the camera.

[0213] (34) The monitoring device according to any one of (31) to (33), wherein

[0214] The image of the portion of the vehicle body is an image of an area outside the field of view of the camera.

[0215] (35) The monitoring device according to any one of (31) to (34), wherein

[0216] The image of the portion of the vehicle body is a virtual image obtained by virtualizing at least a portion of a bumper of the vehicle body.

[0217] (36) The monitoring device according to any one of (31) to (35), wherein

[0218] The camera is a camera attached to capture the front, rear or side of the vehicle body.

[0219] (37) The monitoring device according to any one of (31) to (36), wherein

[0220] The image superimposing unit adjusts an image superimposed on the first image based on speed information of the vehicle body.

[0221] (38) The monitoring device according to any one of (31) to (36), wherein

[0222] The image superimposing unit superimposes the embedded image on a gap area where an image between the first image and the virtual image does not exist.

[0223] (39) The monitoring device according to (38), wherein

[0224] The embedded image is an image having a hue close to a hue of the first image adjacent to the gap area.

[0225] (40) The monitoring device according to (37), wherein

[0226] The image superposition unit cuts out an image of a region corresponding to the gap region from the past first image to generate the embedded image.

[0227] (41) The monitoring device according to (31), wherein

[0228] The first image is an image for rearward viewing.

[0229] (42) An image processing method, comprising:

[0230] generating, by a first signal processing unit, a first image of a first perspective based on an image captured by a camera attached to the vehicle body; and

[0231] An image representing the portion of the vehicle body is superimposed on the first image by an image superimposition unit.

[0232] (43) The image processing method according to (42), wherein

[0233] The image of the portion of the vehicle body is a pre-stored image.

[0234] (44) The image processing method according to (43), wherein

[0235] The pre-stored image is an image stored in the first signal processing unit before acquiring the image of the camera.

[0236] (45) The image processing method according to any one of (42) to (44), wherein

[0237] The image of the portion of the vehicle body is an image of an area outside the field of view of the camera.

[0238] (46) The image processing method according to any one of (42) to (44), wherein

[0239] The image of the portion of the vehicle body is a virtual image obtained by virtualizing at least a portion of a bumper of the vehicle body.

[0240] (47) The image processing method according to any one of (42) to (46), wherein

[0241] The camera is a camera attached to capture the front, rear or side of the vehicle body.

[0242] (48) The image processing method according to any one of (42) to (47), wherein

[0243] The image superimposing unit adjusts an image superimposed on the first image based on speed information of the vehicle body.

[0244] (49) The image processing method according to any one of (42) to (47), wherein

[0245] The image superimposing unit superimposes the embedded image on a gap area where an image between the first image and the virtual image does not exist.

[0246] (50) The image processing method according to (49), wherein

[0247] The embedded image is an image having a hue close to a hue of the first image adjacent to the gap area.

[0248] (51) The image processing method according to (49), wherein

[0249] The image superposition unit cuts out an image of a region corresponding to the gap region from the past first image to generate the embedded image.

[0250] (52) The image processing method according to any one of (42) to (51), wherein

[0251] The image superimposing unit outputs an image in which at least the image is superimposed on the first image to the first display unit.

[0252] (53) The image processing method according to any one of (42) to (52), further comprising:

[0253] A second signal processing unit is provided, wherein the second signal processing unit generates a second image having a second viewing angle different from the first viewing angle according to the first image, and outputs the second image to a second display unit.

[0254] (54) The image processing method according to (53), wherein

[0255] In the first image space, the center of the second viewing angle is higher than the center of the first viewing angle.

[0256] (56) The image processing method according to (42), wherein

[0257] The first image is an image for rearward viewing.

[0258] (57) The image processing method according to (53), wherein

[0259] The second image is an image equivalent to the room mirror image.

[0260] Reference numerals list

[0261] 1 Camera Image

[0262] 3. Rear view original image

[0263] 4 Virtual Image

[0264] 5 Embedded Images

[0265] 6 Rear view image

[0266] 10. Camera

[0267] 20, 20A, 20B Image processing device

[0268] 20C Monitoring Device

[0269] 22 CMS signal processing unit

[0270] 23 Rear view signal processing unit

[0271] 24 Image overlay unit

[0272] 30 CMS display units

[0273] 40 Rear view display unit

[0274] 60 Body

[0275] 61 Rear bumper

[0276] 80 Speed ​​detection unit

[0277] 100 Camera System

[0278] 100B Camera Surveillance System

[0279] 200 ECU

Claims

1. An image processing device, comprising: a first signal processing unit for generating a first image of a first perspective based on an image captured by a camera attached to the vehicle body; as well as an image superimposing unit for superimposing an image representing a portion of the vehicle body on the first image, the image of the portion of the vehicle body being an image of an area outside the field of view of the camera, The image superposition unit adjusts the image superimposed on the first image based on the speed information of the vehicle body, and the image superimposed on the first image shows the area that the vehicle reaches after t seconds when the vehicle moves backward at the current speed, and t is a timing parameter preset in the image superposition unit.

2. The image processing apparatus according to claim 1, wherein The image of the portion of the vehicle body is a pre-stored image.

3. The image processing apparatus according to claim 2, wherein The pre-stored image is an image stored in the first signal processing unit before acquiring the image of the camera.

4. The image processing apparatus according to claim 1, wherein The image of the portion of the vehicle body is a virtual image obtained by virtualizing at least a portion of a bumper of the vehicle body.

5. The image processing apparatus according to claim 4, wherein The camera is a camera attached to capture the front, rear or side of the vehicle body.

6. The image processing apparatus according to claim 4, wherein The image superimposing unit superimposes the embedded image on a gap area where an image between the first image and the virtual image does not exist.

7. The image processing apparatus according to claim 6, wherein The embedded image is an image having a hue close to a hue of the first image adjacent to the gap area.

8. The image processing apparatus according to claim 7, wherein The image superposition unit cuts out an image of a region corresponding to the gap region from the past first image to generate the embedded image.

9. The image processing apparatus according to claim 1, wherein The image superimposing unit outputs an image in which at least the image is superimposed on the first image to the first display unit.

10. The image processing apparatus according to claim 1, further comprising: A second signal processing unit is provided, wherein the second signal processing unit generates a second image having a second viewing angle different from the first viewing angle according to the first image, and outputs the second image to a second display unit.

11. The image processing apparatus according to claim 10, wherein In the first image space, the center of the second viewing angle is higher than the center of the first viewing angle.

12. The image processing apparatus according to claim 1, wherein The first image is an image for rearward viewing.

13. The image processing apparatus according to claim 10, wherein The second image is an image equivalent to the room mirror image.

14. A camera system comprising: a camera, attached to the vehicle body; a first signal processing unit, configured to generate a first image of a first viewing angle according to the image captured by the camera; as well as an image superimposing unit for superimposing an image representing a portion of the vehicle body onto the first image, the image of the portion of the vehicle body being an image of an area outside the field of view of the camera, The image superposition unit adjusts the image superimposed on the first image based on the speed information of the vehicle body, and the image superimposed on the first image shows the area that the vehicle reaches after t seconds when the vehicle moves backward at the current speed, and t is a timing parameter preset in the image superposition unit.

15. An image processing method, comprising: generating, by a first signal processing unit, a first image of a first perspective based on an image captured by a camera attached to the vehicle body; as well as superimposing an image representing the portion of the vehicle body on the first image by an image superimposition unit, the image of the portion of the vehicle body being an image of an area outside the field of view of the camera, The image superposition unit adjusts the image superimposed on the first image based on the speed information of the vehicle body, and the image superimposed on the first image shows the area that the vehicle reaches after t seconds when the vehicle moves backward at the current speed, and t is a timing parameter preset in the image superposition unit.