Video image composing method and video image composing device

Abstract

To provide a video image composing method which performs projective transformation processing for substituting a video image region of a blocking object with a complementary video image acquired from the other camera even if the blocking object blocking a part of a field of view is present.SOLUTION: A first camera acquires a first video image, a second camera acquires a second video image, a projective transformation section generates a complementary video image by performing projective transformation on the first video image, and a composition section superposes the complementary video image on a part of the second video image. In the projective transformation, a position for each pixel on a first virtual plane is substituted with a corresponding position for each pixel on a third virtual plane, the corresponding position for each pixel on the third virtual plane is then transformed into a position for each pixel on a second virtual plane and a camera viewpoint of the first camera is selected on the first virtual plane, thereby determining a camera viewpoint of the second camera on the third virtual plane on which the projective transformation is performed.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The embodiments relate to an image synthesis method and an image synthesis device. [Background technology]

[0002] For example, there is a so-called electronic mirror, which is equipped with a plurality of cameras in a vehicle and can display images from the cameras on a display device instead of images from a conventional rearview mirror or side mirror. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4883977 [Patent Document 2] Patent No. 5500877 [Patent Document 3] JP 2011-213186 A [Patent Document 4] JP 2014-197818 A Summary of the Invention [Problem to be solved by the invention]

[0004] In an electronic mirror system, multiple cameras are installed at different positions on a vehicle. Therefore, when multiple cameras capture the scenery behind the vehicle, even if one camera's field of view captures a valid area, the other camera's field of view may capture a blind spot (invalid area).

[0005] As described above, in an electronic mirror, when a plurality of cameras each capture an image of the scenery behind the vehicle, an image of a valid area and an image of an invalid area are generated depending on the camera.

[0006] Therefore, the object of this embodiment is to provide an image synthesis method that can replace the image corresponding to the obstruction with an effective image (supplementary image) to obtain a natural synthesized image, even if an obstruction is present that blocks part of the field of view. [Means for solving the problem]

[0007] According to one embodiment, an image synthesis device includes a first camera, a second camera, a projective transformation unit, and a synthesis unit; The first camera captures a first image, and the second camera captures a second image; The projective transformation unit performs projective transformation on the first image to generate a supplementary image, and the synthesis unit superimposes the supplementary image on a part of the second image. replacing a position of each pixel on a first virtual plane of the first image with a corresponding position of each pixel on a third virtual plane, and then converting the corresponding position of each pixel on the third virtual plane into a position of each pixel on a second virtual plane; a camera viewpoint of the second camera on the third virtual plane that has been subjected to the projective transformation is determined by selecting a camera viewpoint of the first camera on the first virtual plane; In the image synthesis method, when the camera viewpoint of the second camera is changed, the camera viewpoint of the first camera in the projective transformation unit can be adjusted. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a plan view showing a vehicle. [Diagram 2] FIG. 2 is a side view of the vehicle of FIG. [Diagram 3] FIG. 3 is a block diagram of a digital mirroring system to which an embodiment is applied. [Figure 4] FIG. 4 is an image diagram shown for explaining a basic operation example of the digital mirror system of FIG. [Diagram 5]FIG. 5 is an image diagram shown for explaining an example of an operation involving projective transformation of the digital mirror system of FIG. [Figure 6] FIG. 6 is an explanatory diagram shown for explaining the basic principle of projective transformation. [Figure 7] FIG. 7 is an explanatory diagram showing an example of a determinant for projective transformation. [Figure 8] FIG. 8 is an explanatory diagram shown for explaining the brightness averaging process of the image of the vehicle boundary surface. [Figure 9] FIG. 9 is an explanatory diagram showing another example of generating an image to be synthesized in a complete blind spot area. [Figure 10] FIG. 10 is an explanatory diagram showing an example in which a pseudo image is synthesized in a complete blind spot area. [Figure 11] FIG. 11 is an explanatory diagram showing another example in which a pseudo image is synthesized in a complete blind spot area. [Figure 12] FIG. 12 is an explanatory diagram showing an example in which the distance from a vehicle traveling on a road to a reference plane position (distant plane position and ground position) changes depending on the speed of the vehicle. [Figure 13] FIG. 13 is a diagram showing an example of an image on a digital mirror when the distance from the vehicle to the reference (virtual) plane position is extended to a distant plane position. [Figure 14] FIG. 14 is a diagram showing an example of an image on the electronic mirror when the distance from the vehicle to the reference (virtual) plane position is shortened to a short-distance plane position. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment will be described with reference to the drawings. Fig. 1 shows a plan view of a vehicle 100. Fig. 2 shows the vehicle of Fig. 1 from the side.

[0010] The first camera 101 is placed on the trunk or near the license plate of the car 100, and can capture the scenery behind the car. The second camera 102R and the third camera 102L are placed in the side mirrors of the car 100, and can also capture the scenery behind the car.

[0011] As described above, the multiple cameras 101, 102R, and 102L are attached at different positions relative to one vehicle 100. Therefore, when the multiple cameras each capture the scenery behind the vehicle, even if the effective area is imageable according to the field of view W1 of one camera 101, there is a blind spot area (invalid area = field of view W3) that cannot be captured according to the field of view (for example, W2) of the other camera 102R or 102L.

[0012] That is, according to one embodiment of the image synthesis device for electronic mirrors, the first camera 101 captures a first image (rear image) of a first scene from a first position (for example, trunk position) using a first field of view W1. The second camera 102R captures a second image from a second position (a side position in the lateral direction of the driver's seat, a side door position) different from the first position (trunk position) using a second field of view W2 in which a scenery obstruction (the side of the vehicle) is partially positioned in the direction of the first scene. For this reason, part of the second image includes an image of the side of the vehicle.

[0013] Here, the image processing device 200 described later connects a part of the first image to a part of the second image as a supplementary image. In this case, an image of a scenery obstruction (side of the vehicle) included in the second image is processed into a semi-transparent image, and the supplementary image is displayed by overlaying it on the semi-transparent image. In addition, a boundary between the second image and the supplementary image is provided at the outline of the semi-transparent image (i.e., the outline of the side of the vehicle), and the supplementary image is connected to the second image to obtain a third image.

[0014] The supplementary image that is overlaid on the above-mentioned semi-transparent image area is an image that has been projectively transformed based on the far plane position and then projectively transformed based on the ground position that is closer to the shooting position than the far plane, and then the two images are connected (or composited, or spliced ​​together).

[0015] Furthermore, when the horizontal line direction of the first and second cameras is left-right and the vertical line direction is up-down, the upper area of ​​the supplementary image is an image projected and transformed based on the far plane position, and the lower area of ​​the supplementary image is an image projected and transformed based on the ground position.

[0016] Furthermore, the image of the scenery obstruction is made semi-transparent by averaging the brightness of multiple frames, which prevents the light of cars running alongside or passing by, or headlights from being reflected and turned into images on the car door (which has a mirror effect).

[0017] In addition, the lower area of ​​the supplementary image has a fill-in image with a color that is approximated from the color of the surrounding image. This allows the lower part of the car (the road) to be displayed by estimation, giving a sense of stability to the entire screen.

[0018] Fig. 3 shows an electrical processing block for performing the above-mentioned processing. In Fig. 3, 101, 102R, and 102L are a first camera (which may be called a rear camera), a second camera 102R (which may be called a right side camera), and a third camera 10L (which may be called a left side camera), respectively. The image signals of the cameras 101, 102R, and 102L are input to a video processing device 200.

[0019] That is, the image capturing signal of the first camera 101 is input to an image cropping section 221a of the first image processing section 221. The image capturing signal of the second camera 102R (which may be referred to as the right side camera) is input to an image cropping section 222a of the second image processing section 222, and the image capturing signal of the third camera 102L (which may be referred to as the left side camera) is input to an image cropping section 223a of the third image processing section 223.

[0020] The first camera 101, the second camera 102R, and the third camera 102L obtain wide-angle area images including an area wider than the image area to be used. Each of the cropping units 221a, 222a, and 222b crops the image area to be used from the wide-angle area images. The cropping position and cropping area may be changed depending on the speed of the vehicle, the vibration of the vehicle, and the steering direction. In addition, the cropping position and / or cropping area can be changed from the normal setting position by the user's operation. This is also effective when conducting video monitoring experiments and tests.

[0021] In first image processing unit 221, the image signal cut out by image cut-out unit 221a is projectively transformed in projection transformation units 221b and 221c, and then input to synthesis unit 230. Projective transformation units 221b and 221c have different reference positions for projection, and convert the viewpoint position of first camera 101 into the viewpoint position of second camera 102.

[0022] That is, the projective transformation units 221b and 221c can obtain an image that has been projectively transformed based on the far plane position and an image that has been projectively transformed based on the ground position that is closer to the shooting position than the far plane, and output this image to the synthesis unit 230 as a supplementary image.

[0023] Meanwhile, in the second video processing unit 222, the video signal cut out by the video cutout unit 222a is projectively transformed in the projective transformation unit 222b. Note that the processing by this projective transformation unit 222b may be omitted. Similarly, in the third video processing unit 223, the video signal cut out by the video cutout unit 223a is projectively transformed in the projective transformation unit 223b. The processing by this projective transformation unit 223b may also be omitted. It is clear that the degree of transformation performed by the projective transformation units 222b and 223b is smaller than the degree of transformation performed by the previous projective transformation units 221b and 221c.

[0024] Projective transformation, also known as planar projective transformation technology or homography transformation technology, is a technology that converts a planar figure on a virtual plane seen from a certain viewpoint (first viewpoint) into a planar figure on a virtual plane seen from another viewpoint (second viewpoint).

[0025] The image signal acquired by the second image processor 222 includes an image of the side of the vehicle (an image of the side of the right door taken toward the rear, i.e., an image of a scenery obstruction). Similarly, the image signal acquired by the third image processor 223 includes an image of the side of the vehicle (an image of the side of the left door taken toward the rear, i.e., an image of a scenery obstruction).

[0026] The mounting position of the second camera (side camera) 102R on the vehicle is fixed, so it is known in advance which area of ​​the angle of view the area of ​​the scenery obstruction image in the second image will be in. Similarly, the mounting position of the third camera (side camera) 102L on the vehicle is fixed, so it is known in advance which area of ​​the angle of view the area of ​​the scenery obstruction image in the third image will be in.

[0027] Therefore, the scenery obstruction image (image of the right side of the vehicle) included in part of the second video undergoes luminance averaging processing in the luminance averaging processing unit 222c. The reason for this averaging is to prevent the light of a car running parallel to the vehicle, a passing car, or a lamp from being reflected and imaged on the car door (which has a mirror effect). Similarly, the scenery obstruction image (image of the left side of the vehicle) included in part of the third video undergoes luminance averaging processing in the luminance averaging processing unit 223c.

[0028] Next, the scenery obstruction image (image of the right side of the vehicle) included in the brightness-averaged second image is subjected to a semi-transparent process in the semi-transparent process unit 222d. Also, the scenery obstruction image (image of the left side of the vehicle) included in the brightness-averaged third image is subjected to a semi-transparent process in the semi-transparent process unit 223d. The degree of semi-transparency may be arbitrarily controllable by information from the electronic controller 250.

[0029] Furthermore, the second image output from the semi-transparency processing unit 222d is input to the edge processing unit 222e, where it can be subjected to edge (contour) processing of the scenery obstruction image. Similarly, the third image output from the semi-transparency processing unit 223e is also input to the edge processing unit 223e, where it can be subjected to edge (contour) processing of the scenery obstruction image. The edge processing is, for example, processing such as adding a line or a dotted line to the contour (edge ​​line addition processing).

[0030] The color, thickness, darkness, and other elements of the edge line may be changed in various ways depending on the driving conditions of the vehicle and / or the external environment, and these elements may be adjusted by user operation.

[0031] For example, when a vehicle is reversed, the edges can be made clearer, making it easier to see the distance between an obstacle and the side of the vehicle or in the direction of vehicle travel.

[0032] The first video signal that has undergone the above processing in the first video processing unit 221, the second video signal that has undergone the above processing in the second video processing unit 222, and the third video signal that has undergone the above processing in the third video processing unit 223 are combined in the combining unit 230.

[0033] The electronic controller 250 keeps track of vehicle speed information J1, various manual operation information J2, direction information J3 indicating whether the vehicle is turning right, turning left, going straight, or reversing, time of day information J4, light illumination information, outside temperature information, interior temperature information, and other information.

[0034] The speed information J1 can be used, for example, in the projective transformation units 221b, 221c, 222b, and 223b as adjustment information for automatically adjusting the distance between the camera and a reference plane (reference image) that serves as a reference for changing the viewpoint position of the camera.

[0035] The manual operation information J2 includes, for example, information for adjusting the reference position of the cropping area of ​​the image, and information for initially setting or adjusting the reference plane for performing projective transformation.

[0036] The direction information J3 can also be used as control information for, for example, expanding the cut-out area of ​​the image, enlarging the image, correcting the position of the cut-out area, making the image semi-transparent, changing the thickness, color, density, etc. of the edges, etc. For example, when the car is moving forward, the transparency of the semi-transparency is made high (light). Also, when the car is turning left or right, the edges become thicker and darker, making it easier for the driver to recognize the distance to an obstacle next to the car.

[0037] The time zone information J4 can also be used to adjust the sensitivity of the camera, for example by increasing the detection sensitivity of the camera's image signal when the surroundings are dark and decreasing the detection sensitivity when the surroundings are bright.

[0038] Furthermore, there are various types of sensor information, such as information on outside humidity and outside temperature. By utilizing this sensor information, more appropriate images can be obtained from the camera. For example, if it is raining outside, a compressed air cleaner installed next to the camera can be used to remove water and dust around the lens. For this purpose, an outlet for blowing compressed air can be provided in front of the camera lens. Furthermore, since snow and other particles can freeze if they adhere to the front of the camera, a heater can also be provided in front of the camera lens.

[0039] In this system, the display 226 is disposed inside the vehicle at a position that is easily visible to the driver. Additional displays may be provided in front of the rear seat or in front of the passenger seat so that people other than the driver can see them.

[0040] Also, the inside or a part of the image processing device 200 in Fig. 3 can be realized by software. The inside or a part of the image processing device 200 may be a processing device operated by software recorded in a storage medium. Furthermore, the software may be transmitted from the outside by a communication means, and the inside or a part of the image processing device 200 may be a device that receives and executes the software. Also, a recording and reproducing device may be connected to the image processing device 200, and an image signal may be recorded in the recording and reproducing device in response to, for example, a specially set running speed, vibration, sound, or the like.

[0041] 4 is an explanatory diagram for explaining the basic process of image synthesis. A rear image 410 is a rear image cut out from an image captured by the rear camera 101. A side image 420 is a side image cut out from an image captured by the right side camera 102R.

[0042] 4 shows a rear image 410 and a side image 420 from a right side camera 102R, but an actual device also uses a side image from a left side camera 102L on the left side. Here, for the sake of simplicity, the embodiment will be described in relation to the rear image 410 and the side image 420.

[0043] An image A0 of the side of the vehicle (which may be called an image of the side of the vehicle or an image of a scenery obstruction) is reflected in a part (left side) of the side image 420. This reflected area (area of ​​image A0) is a blind spot area for the scenery behind the vehicle for the right side camera 102R.

[0044] Therefore, the area of ​​this image A0 is subjected to watermark processing (semi-transparency processing by the semi-transparency processing unit 222d), and a part of the image of the rear image 410 (the image of the area corresponding to the image A0) is overlaid and synthesized by the synthesis unit 230.

[0045] Then, as shown in the composite image 430, a part of the rear image is displayed superimposed on the semi-transparent scenery obstruction image. In the composite image 430, an edge line 435 indicates the boundary (which may also be called a joining position, a connection position, a contour, etc.) between the semi-transparent image A0 and the image captured by the right side camera 102R. This edge line 435 is added by the edge processing unit 222e.

[0046] The edge line 435 is useful for the driver to recognize the vehicle width when driving forward or backward. The edge line 435 may be variable in line width, line color, brightness (depth), etc., depending on the surrounding conditions, for example, when it is dark or bright outside. This makes it possible to call the driver's attention.

[0047] Furthermore, a non-display line 432 is shown between the upper area A1 and the lower area A2 of the semi-transparent image A0 in the composite image 430. This line 432 is not displayed as an image, but is shown for convenience of explanation.

[0048] Although many diamond shapes are shown in Fig. 4 and in other drawings to be described later, these shapes are not displayed in the actual image. These diamond shapes are shown to help the viewer understand that a rectangle is transformed into a diamond shape as a result of viewpoint conversion. These many diamond shapes are not displayed in the actual image.

[0049] The images of the upper area A1 and the lower area A2 are images obtained by converting a rear image 410 from the viewpoint of the rear camera 101 into an image from the viewpoint of the right side camera 102R. In this case, the conversion reference position of the upper area A1 is a far plane position 412, and the conversion reference position of the lower area A2 is a ground position 413 that is closer to the camera position than the far plane position 412. The far plane position 412 and the ground position 413 correspond to 422 and 423 shown in the figure in the side image 420 of the right side camera 102R.

[0050] 5 shows a first projectively transformed image 410H obtained by projectively transforming a rear image 410 from the viewpoint of the rear camera 101 into an image from the viewpoint of the right side camera 102R based on the previous far plane position (transformed by the projective transformation unit 221b in FIG. 4). Also, a second projectively transformed image 420H obtained by projectively transforming a rear image 410 from the viewpoint of the rear camera 101 into an image from the viewpoint of the right side camera 102R based on the previous far plane position (transformed by the projective transformation unit 221c in FIG. 3).

[0051] The above-mentioned far plane position and ground plane position may be manually adjusted by the driver before driving. As will be described later, a method of automatically changing the positions to preset positions according to the vehicle speed may be adopted.

[0052] A part of the first projective transformation image 410H (around P1) and a part of the second projective transformation image 420H (around P2) are cut out and are composited with the upper region A1 and lower region A2 of the semi-transparent image A0, respectively. In this case, when the upper region A1 and lower region A2 are cut out from the projective transformation image 410H and the projective transformation image 420H, the upper region A1 and lower region A2 are cut out so that the images of each region and these images are naturally connected to the side image of the second camera.

[0053] Furthermore, in the region of the image of the scenery obstruction in the side image 420 of the right side camera 102R, there is also a blind spot region of the rear camera 101. That is, there are blind spots of both cameras.

[0054] This blind spot area is a complete blind spot area 433. This area corresponds to the area under the rear of the vehicle (usually the road surface). In Fig. 5, it is shown at the bottom left of the composite image 420. This complete blind spot area 433 has no image, so if no processing is performed, it may result in a completely black or completely white image.

[0055] Therefore, in this system, for example, this region is filled with image data of the color of the image around the complete blind spot region 433 or a similar color to be interpolated.

[0056] In other words, this system makes the most efficient use of the images captured by the rear camera 101, the right side camera 102R, and the left side camera 102L.

[0057] Figure 6 shows the concept of converting the viewpoint. For example, assume that viewpoint C1 is the viewpoint of the first camera, and viewpoint C2 is the viewpoint of the second camera. First, pixel A1-D1 on the reference plane P1 of viewpoint C1 is temporarily replaced with pixel AD on virtual plane P. Next, pixel AD on virtual plane P is converted to pixel A2-D2 on the virtual plane P2 of the second camera at viewpoint C2.

[0058] Here, if the virtual planes P1 and P2 are at the same position (the previous ground position), the image of the ground captured by the first camera can be converted to look like an image captured by the second camera.

[0059] For example, if there are four points AD on a plane P, each of these points corresponds to A1-D1 on a virtual plane P1 in an image captured from viewpoint C1, and to A2-D2 on a virtual plane C2 in an image captured from viewpoint C2.

[0060] Here, if the correspondence between A1-D1 and A2-D2 is determined, the image captured by the first camera can be transformed to look like the image captured by the second camera. This correspondence is called a projective transformation (or homography transformation) and can be expressed as a matrix.

[0061] Figure 7 shows the transformation matrix H. Now, if the coordinates of A1-D1 on the virtual plane P1 are (xi, yi, 1), this shows that they are transformed into coordinates (ui, vi, 1) on the virtual plane P2 by this transformation matrix H. Here, i means 0, 1, 2, 3, and the coordinates are expressed in a three-dimensional homogeneous coordinate system in which one component is added to the two-dimensional plane.

[0062] That is, the coordinates of A1-D1 and A2-D2 correspond as follows:

[0063] A1: (x0, y0, 1), B1: (x1, y1, 1), C1: (x2, y2, 1), D1: (x3, y3, 1)A2: (u0, v0, 1), B2: (u1, v1, 1), C2: (u2, v2, 1)D2: (u3, v3, 1), In FIG. 7, if (xi, yi, 1) and (ui, vi, 1) are known, the matrix elements h0-h7 can be found.

[0064] Since the transformation matrix H has 8 components, if we know the correspondence between the four points, we can find h0-h7. The equation is expanded for i=0,1,2,3, and the results of factoring out h0-h7 are shown in Figure 7 (the determinant at the bottom of Figure 7). Using this equation, we can find h0-h7.

[0065] This transformation matrix H makes it possible to transform any point on the virtual plane P1 seen from the viewpoint c1 into a point on the virtual plane P2 seen from the viewpoint c2.

[0066] Fig. 8 shows an example of an image that has been luminance-averaged by the luminance averaging units 222c and 223c shown in Fig. 3. For example, as shown in image 510, a bus passes on the right side of a vehicle. In this case, the side of the vehicle functions as a mirror and reflects the bus. If this state is captured by right side camera 102R as is, an image B1 of the bus will also be present in the image captured by right side camera 102R (see images 510 and 511).

[0067] As the car is moving, various images are reflected (buses, scenery, houses, etc.). In this system, the brightness of image B1 of the side of the car is averaged to create image B12 (see image 512). When brightness-averaged image B12 is made semi-transparent, the reflected image does not have a significant effect on the semi-transparent image, as shown in image 513. Image 511 is an image that is not subjected to brightness averaging processing, and an obstructive image is reflected in the semi-transparent image. However, with this system, it is possible to make the image of the semi-transparent portion easier to see.

[0068] 9 is an explanatory diagram showing another embodiment in which the complete blind spot area 433 is filled (or painted over). If the vehicle has a fourth camera that captures the front, a partial image 466 of the captured image can be embedded in a part of the composite image 430. In this case, a cutout unit that cuts out the image captured by the front camera, a conversion unit that converts the front-rear direction of the cutout image, and a time adjustment unit are used. The time-adjusted partial image 466 is then sent to the synthesis unit 230 as a pseudo image and used as the image of the complete blind spot area 433.

[0069] Fig. 10 is an example in which a composite image 430 is displayed on display unit 226 in a digital mirror using a standard camera. Fig. 10 shows the right half of the image, but the left half of the image is also displayed on display unit 226. In the left half of the image, a supplementary image is also displayed superimposed on a semi-transparent image, and a line indicating the boundary between the image from the rear camera and the supplementary image is displayed at the outline of the semi-transparent image.

[0070] 11 shows an example of a composite image 430A obtained when using the first, second, and third cameras capable of panoramic photography with a wider viewing angle than the standard. A completely blind spot area 431A also occurs in this composite image 430A, so this part is filled with a color or pattern similar to the color or pattern of the surroundings.

[0071] Fig. 12 shows cars 502, 502 traveling on a road in a plan view. The distance from the car 502 to a reference plane position (for example, the far plane position and the ground position shown in Fig. 4) may be automatically adjusted depending on the speed. When the speed is slow (for example, 0Km / H-35Km / H), attention should be paid to a relatively close following car (for example, car 503), when the speed is medium (for example, 35Km / H-65Km / H or higher), attention should be paid to a further distant following car, and when the speed is higher than that (for example, 65Km / H or higher), attention should be paid to a further distant following car.

[0072] In such a case, the distance from the vehicle to the reference plane position (for example, the far plane position and the ground position shown in FIG. 4) may be automatically adjusted according to the user's setting.

[0073] Fig. 13 is an example of a composite image 430 when the reference plane position (virtual plane position) is far away, and Fig. 14 is an example of a composite image 430 when the reference plane position is close. In the example of Fig. 13, when the reference plane position is far away, the vertical lines of a building in the distant rear match in the first image and the second image. However, for the image of a nearby car, a step occurs at the connection between the first image and the second image.

[0074] In contrast, as shown in Fig. 14, when the reference plane position (virtual plane position) is close, the vertical lines of the building in the distant rear will be misaligned (a step) between the first and second images. However, the image of the nearby car will be smoothly connected at the connection point between the first and second images.

[0075] Therefore, by adjusting the virtual plane position in accordance with the vehicle speed, it is possible to clearly identify rear objects at a distance that the driver wishes to focus on.

[0076] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the present invention also includes the separation or combination of each element described in the claims. [Explanation of symbols]

[0077] 100...vehicle, 101...first camera, 102...second camera...102R, third camera...102L, 200...image processing device, 221a, 222a, 223a...image cropping unit, 221b, 221c, 222b, 223b...projection transformation unit, 222c, 223c...luminance averaging unit, 222d, 223d...semi-transparency processing unit, 222e, 223e...edge processing unit, 230...compositing unit, 226...display unit, 250...electronic controller.

Claims

1. The image synthesis device includes a first camera, a second camera, a projection transformation unit, and a synthesis unit, The first camera captures a first image; the second camera captures a second image; the projective transformation unit performs projective transformation on the first image to generate a supplemental image; The synthesizing unit overlays the supplemental image on a portion of the second image; In the projective transformation, replacing a position of each pixel on a first virtual plane of the first image with a corresponding position of each pixel on a third virtual plane, and then converting the corresponding position of each pixel on the third virtual plane into a position of each pixel on a second virtual plane; a camera viewpoint of the second camera on the third virtual plane obtained by the projective transformation is determined by selecting a camera viewpoint of the first camera on the first virtual plane; When changing the camera viewpoint of the second camera, the camera viewpoint of the first camera in the projective transformation unit is adjustable.

13. A video compositing method comprising:

2. the camera viewpoints of the first camera include, on the first virtual plane, a first camera viewpoint corresponding to a distant image, and a first camera viewpoint and a second camera viewpoint corresponding to a near image closer to the first camera than the distant image; 2. The image synthesis method according to claim 1, wherein a portion of a first image whose pixels have been projectively transformed based on the first camera viewpoint and a portion of a second image whose pixels have been projectively transformed based on the second camera viewpoint are arranged side by side on the third virtual plane.

3. An image synthesis device including a first camera, a second camera, a projection transformation unit, and a synthesis unit, the first camera having means for acquiring a first image; the second camera having means for acquiring a second image; the projection transformation unit has means for projectively transforming the first image to generate a supplementary image, the synthesizing unit has means for superimposing the supplemental image on a portion of the second image; The projective transformation unit is replacing a position of each pixel on a first virtual plane of the first image with a corresponding position of each pixel on a third virtual plane, and then converting the corresponding position of each pixel on the third virtual plane into a position of each pixel on a second virtual plane; a camera viewpoint of the second camera on the third virtual plane obtained by the projective transformation is determined by selecting a camera viewpoint of the first camera on the first virtual plane; a means for adjusting a camera viewpoint of the first camera in the projection transformation unit when changing the camera viewpoint of the second camera; 1. A video synthesizing device comprising:

4. 4. The image synthesizing device according to claim 3, wherein the first camera is a rear camera at a rear of the vehicle, and the second camera is a side camera at a side of the vehicle.

Images