Image synthesis apparatus and image synthesis method

Abstract

This embodiment provides an image synthesis device that performs projection transformation processing to replace the image area of ​​an obstruction that blocks part of the field of view with supplemental image acquired from another camera. [Solution] In one embodiment, a first camera and a second camera are mounted on a moving vehicle at different locations. The second camera acquires a second image that includes a landscape obstruction image caused by part of the vehicle being captured in the image. Therefore, the image synthesis device is configured to average the brightness of the landscape obstruction image included in the second image and superimpose a part of the first image onto the brightness-averaged landscape obstruction image.

Description

Technical Field

[0001] Embodiments relate to a video synthesizing apparatus and a video synthesizing method.

Background Art

[0002] For example, there is a so-called electronic mirror in which a plurality of cameras are mounted on a vehicle, and video from the cameras can be displayed on a display instead of images from conventional rearview mirrors and side mirrors.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of Invention

Problems to be Solved by the Invention

[0004] In an electronic mirror system, each of the plurality of cameras is mounted at a different position on a single vehicle. Therefore, when each of the plurality of cameras captures a scene behind the vehicle, even if a region is an effective region that can be captured within the field of view of one camera, there is a blind spot region (ineffective region) that cannot be captured within the field of view of the other camera.

[0005] As described above, in an electronic mirror, when each of the plurality of cameras captures a scene behind the vehicle, a video of an effective region and a video of an ineffective region are generated depending on the camera.

[0006] Therefore, the objective of this embodiment is to provide an image synthesis device that can obtain a natural composite image by replacing the image corresponding to the obstruction that blocks part of the field of view with an effective image (supplementary image). [Means for solving the problem]

[0007] According to one embodiment, a first camera and a second camera are mounted on a moving vehicle at different locations. The second camera acquires a second image that includes images of obstructions in the landscape caused by part of the vehicle being captured in the image. Therefore, an image synthesis device is provided that averages the brightness of the images of obstructions in the landscape included in the second image and superimposes a part of the first image onto the brightness-averaged images of obstructions in the landscape. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a plan view of a car. [Figure 2] Figure 2 is a side view of the vehicle shown in Figure 1. [Figure 3] Figure 3 is a block diagram of an electronic mirror system to which one embodiment is applied. [Figure 4] Figure 4 is an illustrative diagram illustrating a basic example of the operation of the electronic mirror system shown in Figure 3. [Figure 5] Figure 5 is an illustrative diagram illustrating an example of one operation involving projection transformation of the electronic mirror system shown in Figure 3. [Figure 6] Figure 6 is an explanatory diagram shown to illustrate the basic principles of projective transformation. [Figure 7] Figure 7 is an explanatory diagram illustrating an example of a determinant for a projection transformation. [Figure 8] Figure 8 is an explanatory diagram shown to illustrate the brightness averaging process of the image on the vehicle's surface. [Figure 9] Figure 9 is an explanatory diagram illustrating another example of generating an image that is composited in a completely blind spot area. [Figure 10]Figure 10 is an explanatory diagram showing an example in which a pseudo-image is superimposed in a completely blind spot area. [Figure 11] Figure 11 is an explanatory diagram showing another example in which a pseudo-image is superimposed on a completely blind spot area. [Figure 12] Figure 12 is an explanatory diagram illustrating an example where the distance from a vehicle to a reference plane position (far plane position and ground position) changes according to the speed of a vehicle traveling on a road. [Figure 13] Figure 13 shows an example of the image from an electronic mirror when the distance from the car to the reference (virtual) plane position is extended to a distant plane position. [Figure 14] Figure 14 shows an example of the image from an electronic mirror when the distance from the car to the reference (virtual) plane position is shortened to a short-range plane position. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. Figure 1 shows the car 100 in plan view. Figure 2 shows the car of Figure 1 from the side.

[0010] Vehicle 100 has a first camera 101 positioned above the rear trunk or near the license plate, which can capture the scenery behind it. Vehicle 100 also has a second camera 102R and a third camera 102L positioned where the side mirrors are located, which can also capture the scenery behind it.

[0011] As described above, the multiple cameras 101, 102R, and 102L are each mounted in different positions on a single vehicle 100. Therefore, when multiple cameras each photograph the scenery behind the vehicle, there are blind spots (ineffective areas = field of view W3) that can be photographed according to the field of view W1 of one camera 101, but cannot be photographed according to the field of view (e.g., W2) of the other cameras 102R or 102L.

[0012] That is, according to one embodiment of the image synthesizing apparatus for an electronic mirror of the present invention, the first camera 101 acquires a first image (rear image) of a first scene through a first field of view W1 from a first position (for example, a trunk room position). The second camera 102R acquires a second image through a second field of view W2 in which a scene blocking object (the side surface of the own vehicle) is partially arranged in the direction of the first scene, from a second position (a side position in the lateral direction of the driver's seat, a side door position) different from the first position (trunk room position). For this reason, a part of the second image includes an image of the side surface of the own vehicle.

[0013] The image processing apparatus 200 described later herein connects a part of the first image to a part of the second image as a supplementary image. In this case, the image of the scene blocking object (the side portion of the own vehicle) included in the second image is processed into a translucent image, and the supplementary image is superimposed and displayed on this translucent image. Further, a boundary between the second image and the supplementary image is provided at the contour portion of the translucent image (that is, the contour of the side of the vehicle), and the supplementary image is connected to the second image to obtain a third image.

[0014] The supplementary image superimposed and displayed on the region of the above-described translucent image is an image in which an image that has been subjected to projective transformation based on a distant plane position and an image that has been subjected to projective transformation based on a ground position closer to the imaging position than the distant plane are connected (or synthesized, or joined together).

[0015] Further, when the horizontal line direction of the first and second cameras is the left-right direction and the vertical line direction is the up-down direction, the upper region of the supplementary image is an image that has been subjected to projective transformation based on the distant plane position, and the lower region of the supplementary image is an image that has been subjected to projective transformation based on the ground position.

[0016] Furthermore, the image of the scene blocking object is made translucent by performing an averaging process on the luminance of a plurality of frames. This can prevent vehicles running side by side, vehicles passing by, lamp light, and the like from being reflected on the vehicle door (which has a mirror effect) and imaged.

[0017] Furthermore, the lower area of ​​the supplementary image contains a colored image with a color approximation based on the surrounding image. This allows the lower part of the car (the road) to be displayed through estimation, resulting in a sense of overall stability to the screen.

[0018] Figure 3 shows the electrical processing block for performing the above-described processing. In Figure 3, 101, 102R, and 102L are the first camera (may also be called the rear camera), the second camera 102R (may also be called the right side camera), and the third camera 10L (may also be called the left side camera), respectively, and the capture signals from each of the cameras 101, 102R, and 102L are input to the video processing device 200.

[0019] Specifically, the shooting signal from the first camera 101 is input to the image extraction unit 221a of the first video processing unit 221. The shooting signal from the second camera 102R (which may also be called the right side camera) is input to the image extraction unit 222a of the second video processing unit 222, and the shooting signal from the third camera 102L (which may also be called the left side camera) is input to the image extraction unit 223a of the third video processing unit 223.

[0020] The first camera 101, second camera 102R, and third camera 102L described above acquire wide-angle images that include an area wider than the image area to be used. The respective cropping units 221a, 222a, and 222b crop the image area to be used from the wide-angle image. The cropping position and cropping area may be varied depending on the vehicle's speed, vibration, and steering direction. Furthermore, the cropping position and / or cropping area can be changed from their normal settings by user operation. This is also effective when conducting video surveillance experiments and tests.

[0021] In the first video processing unit 221, the video signal extracted by the video extraction unit 221a is subjected to projection transformation by the projection transformation units 221b and 221c, and then input to the synthesis unit 230. Although the reference positions for projection differ, the projection transformation units 221b and 221c convert the viewpoint position of the first camera 101 to the viewpoint position of the second camera 102.

[0022] In other words, the projection transformation units 221b and 221c can obtain an image projected based on the position of the far plane and an image projected based on a ground position closer to the shooting position than the far plane, and output this image as a supplementary image to the synthesis unit 230.

[0023] Meanwhile, in the second video processing unit 222, the video signal extracted by the video extraction unit 222a is subjected to projection transformation in the projection transformation unit 222b. Note that the processing in this projection transformation unit 222b may be omitted. Similarly, in the third video processing unit 223, the video signal extracted by the video extraction unit 223a is subjected to projection transformation in the projection transformation unit 223b. The processing in this projection transformation unit 223b may also be omitted. It is clear that the degree of transformation performed by the projection transformation units 222b and 223b is smaller than the degree of transformation performed by the aforementioned projection transformation units 221b and 221c.

[0024] Projective transformation, also known as planar projection transformation or homography transformation, is a technique that transforms a planar figure on a virtual plane viewed from one viewpoint (first viewpoint) into a planar figure on a virtual plane viewed from another viewpoint (second viewpoint).

[0025] The video signal acquired by the second video processing unit 222 mentioned above includes video of the side of the vehicle (video of the side of the right door taken facing backward, i.e., video of obstructing scenery). Similarly, the video signal acquired by the third video processing unit 223 includes video of the side of the vehicle (video of the side of the left door taken facing backward, i.e., video of obstructing scenery).

[0026] Since the mounting position of the second camera (side camera) 102R on the car is fixed, the area of ​​the field of view where the obstructing scenery is located within the second image is known in advance. Similarly, since the mounting position of the third camera (side camera) 102L on the car is fixed, the area of ​​the field of view where the obstructing scenery is located within the third image is also known in advance.

[0027] Therefore, the image of obstructing scenery (image from the right side of the vehicle) included in part of the second image undergoes brightness averaging processing in the brightness averaging processing unit 222c. The reason for this averaging is to prevent reflections of cars driving alongside, passing cars, and lamp lights from being captured in the car door (which has a mirror effect). Similarly, the image of obstructing scenery (image from the left side of the vehicle) included in part of the third image undergoes brightness averaging processing in the brightness averaging processing unit 223c.

[0028] Next, the image of the obstructing scenery (image on the right side of the vehicle) included in a portion of the second image that has been brightness-averaged is made semi-transparent by the semi-transparency processing unit 222d. Similarly, the image of the obstructing scenery (image on the left side of the vehicle) included in a portion of the third image that has been brightness-averaged is also made semi-transparent by the semi-transparency processing unit 223d. The degree of semi-transparency may be arbitrarily controlled by information from the electronic controller 250.

[0029] Furthermore, the second image output from the translucency processing unit 222d is input to the edge processing unit 222e and can undergo edge (contour) processing of the image of the obstructing scenery. Similarly, the third image output from the translucency processing unit 223e is also input to the edge processing unit 223e and can undergo edge (contour) processing of the image of the obstructing scenery. Edge processing is a process such as adding lines or dotted lines to the contours (edge ​​line addition processing).

[0030] The edge lines may change in various ways, such as color, thickness, and density, depending on the driving conditions and / or external environment, and these elements may also be adjusted by the user.

[0031] For example, when a car is reversing, the edges can be made clearer, making it easier to see the distance between obstacles and the side of the car or in the direction of travel.

[0032] The first video signal processed in the first video processing unit 221, the second video signal processed in the second video processing unit 222, and the third video signal processed in the third video processing unit 223 are combined in the combining unit 230.

[0033] The electronic controller 250 is aware of the vehicle's speed information J1, various manual operation information J2, direction information J3 indicating the status of right turn, left turn, straight, and reverse, time of day information J4, light status information, outside temperature information, interior temperature information, and other information.

[0034] The velocity information J1 can be used, for example, in the projection transformation units 221b, 221c, 222b, and 223b as adjustment information for automatically adjusting the distance between the camera and a reference plane (reference image) that serves as a reference for changing the camera's viewpoint position.

[0035] Manual operation information J2 includes, for example, information for adjusting the reference position of the image cropping area, and information for initializing or adjusting the reference plane for performing projection transformations.

[0036] Furthermore, the directional information J3 can be used as control information when performing processes such as expanding the image cutout area, enlarging it, correcting the position of the cutout area, making it semi-transparent, and changing the thickness, color, and density of the edges. For example, when the car is moving forward, the transparency of the semi-transparent area is increased (made lighter). Also, when the car is turning left or right, the edges become thicker and darker, making it easier for the driver to recognize the distance to obstacles located to the side of the car.

[0037] Furthermore, the time-of-day information J4 can be used to adjust the camera's sensitivity. For example, the camera's detection sensitivity for the shooting signal can be increased when the surroundings are dark, and decreased when the surroundings are bright.

[0038] Furthermore, various types of sensor information are available, including outside humidity and outside temperature. By utilizing this sensor information, more appropriate images can be obtained from the camera. For example, if it is raining outside, a compressed air cleaner installed next to the camera can be used to remove water and dust around the lens. For this purpose, there may be an outlet in front of the camera lens that blows compressed air. In addition, since snow and other debris can freeze if they adhere to the front of the camera, a heater may be installed in front of the camera lens.

[0039] In this system, the display unit 226 is located inside the car, in a position easily visible to the driver. Additional displays may be installed in front of the rear seats or passenger seats so that people other than the driver can see them.

[0040] Furthermore, the internal workings or a part thereof of the video processing device 200 in Figure 3 can be implemented by software. The internal workings or a part thereof may also be a processing device that operates using software recorded on a storage medium. Moreover, the software may be transmitted externally via communication means, and the internal workings or a part thereof may be a device that receives and executes the software. A recording and playback device may also be connected to the video processing device 200, and for example, video signals may be recorded in the recording and playback device in response to specially set driving speed, vibration, sound, etc.

[0041] Figure 4 is an explanatory diagram illustrating the basic processing of image synthesis. Rear image 410 is a rear image extracted from the image captured by the rear camera 101. Side image 420 is a side image extracted from the image captured by the right side camera 102R.

[0042] Figure 4 shows the rear image 410 and the side image 420 from the right side camera 102R, but the actual device also uses the side image from the left side camera 102L. For the sake of simplicity, the embodiment will be described here in terms of the relationship between the rear image 410 and the side image 420.

[0043] A portion of the side image 420 shown above (on the left) includes image A0 of the side of the car (which may also be called the image of the side of the car or the image of obstructions in the scenery). This area of ​​reflection (the area of ​​image A0) is a blind spot for the right side camera 102R in relation to the scenery behind the car.

[0044] Therefore, the area of ​​video A0 is given a watermark (transparency processing by the semi-transparency processing unit 222d), a portion of the rear video 410 (the video in the area corresponding to video A0) is superimposed, and the two are combined by the compositing unit 230.

[0045] As shown in the composite image 430, a portion of the rear image is superimposed on the semi-transparent image of the obstructed scenery. In the composite image 430, the edge line 435 represents the boundary (which may also be called the joining position, connection position, contour, etc.) between the semi-transparent image A0 and the image captured by the right side camera 102R. This edge line 435 is added in the edge processing unit 222e.

[0046] This edge line 435 is useful for the driver to recognize the width of their vehicle when driving forward or backward. The width, color, and brightness (intensity) of this edge line 435 may be varied depending on the surrounding conditions, such as when it is dark or bright outdoors. This can also help to draw the driver's attention.

[0047] Furthermore, in the composite image 430, a non-visible line 432 is shown between the upper region A1 and the lower region A2 of the semi-transparent image A0. This line 432 is not displayed as part of the image, but is shown for illustrative purposes.

[0048] Note that, as is the case with Figure 4 and the diagrams explained later, numerous rhombus shapes are shown, but these shapes are not displayed in the actual video. These rhombus shapes are shown to illustrate that the quadrilateral has been transformed into a rhombus as a result of the viewpoint change. These numerous rhombus shapes are not displayed in the actual video.

[0049] The images in the upper region A1 and lower region A2 are rear images 410 from the viewpoint of the rear camera 101, which have been converted to images from the viewpoint of the right side camera 102R. In this case, the conversion reference position for the upper region A1 is the far-field plane position 412, and the conversion reference position for the lower region A2 is the ground position 413, which is closer to the camera position than the far-field plane position 412. These far-field plane position 412 and ground position 413 correspond to 422 and 423 in the side image 420 of the right side camera 102R.

[0050] Figure 5 shows the first projected image 410H, which is obtained by projecting the rear image 410 from the viewpoint of the rear camera 101 onto the image from the viewpoint of the right side camera 102R based on the aforementioned distant plane position (this transformation is performed in the projection transformation unit 221b in Figure 4). It also shows the second projected image 420H, which is obtained by projecting the rear image 410 from the viewpoint of the rear camera 101 onto the image from the viewpoint of the right side camera 102R based on the aforementioned distant plane position (this transformation is performed in the projection transformation unit 221c in Figure 3).

[0051] The above-mentioned distant plane position and ground plane position may be manually adjusted by the driver before driving. Alternatively, as described later, a system may be adopted in which the position automatically changes according to the vehicle's speed to a preset position.

[0052] A portion of the first projection-converted image 410H (including the periphery of P1) and a portion of the second projection-converted image 420H (including the periphery of P2) are cropped and composited into the upper region A1 and lower region A2 of the semi-transparent image A0, respectively. In this case, when the upper region A1 and lower region A2 are cropped from the projection-converted image 410H and projection-converted image 420H, the upper region A1 and lower region A2 are cropped in such a way that the images of each region and these images connect naturally with the side image of the second camera.

[0053] Furthermore, in the side image 420 from the right side camera 102R, the area of ​​the image showing obstructions in the scenery also includes the blind spot area of ​​the rear camera 101. In other words, blind spots exist for both cameras.

[0054] This blind spot area is a complete blind spot area 433. This area corresponds to the area below the rear of the car (usually the road surface). In Figure 5, it is shown in the lower left of the composite image 420. Because there is no image in this complete blind spot area 433, if no processing is performed, the image may appear completely dark or completely white.

[0055] Therefore, in this system, for example, this area is interpolated by filling it with video data of the color of the surrounding video of the complete blind spot area 433 or a similar color.

[0056] In other words, this system makes the most effective use of the images captured by the rear camera 101, the right side camera 102R, and the left side camera 102L.

[0057] Figure 6 illustrates the concept of changing viewpoints. For example, let's assume that viewpoint C1 is the viewpoint of the first camera and viewpoint C2 is the viewpoint of the second camera. First, pixels A1-D1 of the reference plane P1 of viewpoint C1 are temporarily replaced with pixels AD of the virtual plane P. Next, pixels AD of the virtual plane P are converted to pixels A2-D2 of the virtual plane P2 of the second camera at viewpoint C2.

[0058] Here, if virtual planes P1 and P2 are at the same position (the same ground position), the ground image captured by the first camera can be transformed to look like the image captured by the second camera.

[0059] For example, if there are four points A and D on a plane P, each of these points corresponds to A1-D1 on the virtual plane P1 in the video captured from viewpoint C1, and to A2-D2 on the virtual plane C2 in the video captured from viewpoint C2.

[0060] Once the correspondence between A1-D1 and A2-D2 is determined, the image captured by the first camera can be transformed to appear as if it were captured by the second camera. This correspondence is called a projective transformation (or homography transformation) and can be represented by a matrix.

[0061] Figure 7 shows the transformation matrix H. Now, if the coordinates of A1-D1 on the virtual plane P1 are (xi, yi, 1), this transformation matrix H shows that they are transformed to coordinates (ui, vi, 1) on the virtual plane P2. Here, i represents 0, 1, 2, or 3, and the coordinates are represented in a three-dimensional homogeneous coordinate system, which is a two-dimensional plane with one additional component.

[0062] In other words, the coordinates A1-D1 and A2-D2 correspond as follows:

[0063] A1: (x0, y0, 1), B1: (x1, y1, 1), C1: (x2, y2, 1), D1: (x3, y3, 1)A2: (u0, v0, 1), B2: (u1, v1, 1), C2: (u2, v2, 1)D2: (u3, v3, 1), In Figure 7, if (xi, yi, 1) and (ui, vi, 1) are known, then the matrix components h0-h7 can be determined.

[0064] Since the transformation matrix H has 8 components, if we can find the correspondence between the four points, we can find h0-h7. Expanding the equation for i=0,1,2,3 and factoring out h0-h7 is shown in Figure 7 (the determinant at the bottom of Figure 7). We can find h0-h7 using this equation.

[0065] This transformation matrix H makes it possible to transform any point on the virtual plane P1 as seen from viewpoint c1 into a point on the virtual plane P2 as seen from viewpoint c2.

[0066] Figure 8 shows an example of a video with averaged brightness achieved by the brightness averaging units 222c and 223c shown in Figure 3. For example, as shown in video 510, suppose a bus passes on the right side of the car. In this case, the side of the car acts like a mirror and reflects the bus. If this state is captured by the right side camera 102R, the video B1 of the bus will also be present in the video captured by the right side camera 102R (see videos 510 and 511).

[0067] Because the car is in motion, various images are reflected in it (buses, scenery, houses, etc.). Therefore, in this system, the brightness of the image B1 on the side of the car is averaged to create image B12 (see image 512). When the brightness-averaged image B12 is made semi-transparent, as shown in image 513, the reflected images have little impact on the semi-transparent image. In the case of image 511, the image is not processed with brightness averaging, and distracting images are reflected in the semi-transparent part. However, with this system, the image in the semi-transparent part can be made easier to see.

[0068] Figure 9 is an explanatory diagram showing another embodiment for filling (or filling in) the complete blind spot area 433. If the vehicle has a fourth camera that takes pictures of the front, a portion of the image 466 captured by that camera can be embedded in a portion of the composite image 430. In this case, an extraction unit that extracts the image captured by the front camera, a conversion unit that changes the front-to-back direction of the extracted image, and a time adjustment unit are used. The time-adjusted portion of the image 466 is then sent to the composite unit 230 as a pseudo-image and used as the image of the complete blind spot area 433.

[0069] Figure 10 shows an example of a composite image 430 being displayed on the display unit 226 in an electronic mirror using a standard camera. Although Figure 10 shows the image on the right half, the display unit 226 also displays the image on the left half. In the image on the left half, the supplementary image is superimposed on the semi-transparent image, and a line indicating the boundary between the image from the rear camera and the supplementary image is displayed at the outline of the semi-transparent image.

[0070] Figure 11 shows an example of a composite image 430A obtained when using the first, second, and third cameras, which are capable of panoramic shooting with a wider field of view than standard. In this composite image 430A, a complete blind spot area 431A also occurs, so this area is filled with a color or pattern similar to the surrounding color or pattern.

[0071] Figure 12 shows two cars 502 traveling on a road in a plan view. Depending on the speed, the distance from the car to the reference plane position (for example, the distant plane position and the ground position shown in Figure 4) may be automatically adjusted. When the speed is slow (for example, 0 km / h to 35 km / h), attention should be paid to relatively close following cars (for example, car 503); when the speed is medium (for example, 35 km / h to 65 km / h and above), attention should be paid to even more distant following cars; and when the speed is even higher (for example, 65 km / h and above), attention should be paid to even more distant following cars.

[0072] In such cases, the distance from the vehicle to the reference plane position (for example, the distant plane position and the ground position shown in Figure 4) may be automatically adjusted according to the user's settings.

[0073] Figure 13 shows an example of a composite image 430 when the reference plane position (virtual plane position) is far away, and Figure 14 shows an example of a composite image 430 when the reference plane position is close. In the example in Figure 13, when the reference plane position is far away, the vertical lines of the distant background building match in the first and second images. However, for the image of the nearby car, a step occurs at the connection point between the first and second images.

[0074] In contrast, as shown in Figure 14, when the reference plane position (virtual plane position) is close, the vertical lines of the distant, rearward building show a discrepancy (step) between the first and second images. However, for the image of a nearby car, the connection between the first and second images is smooth.

[0075] Therefore, by adjusting the virtual plane position according to the vehicle's speed, it is possible to clearly define objects behind the vehicle at a distance that the driver wishes to focus on.

[0076] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. Separating or combining the elements described in the claims is also included in the present invention. [Explanation of symbols]

[0077] 100...Car, 101...First camera, 102...Second camera...102R, Third camera...102L, 200...Image processing unit, 221a,222a,223a...Image extraction unit, 221b,221c,222b,223b...Projection conversion unit, 222c,223c...Brightness averaging unit, 222d,223d...Transparency processing unit, 222e,223e...Edge processing unit, 230...Composition unit, 226...Display unit, 250...Electronic controller.

Claims

1. It is an image synthesis device, It is mounted on a moving vehicle and consists of a first camera that acquires the first image, A second camera is mounted on the vehicle at a different location from the first camera and acquires a second image including a landscape obstruction image caused by part of the vehicle being captured in the image, A luminance averaging unit that averages the luminance of the image of the obstructing scenery included in the second image, A composite unit overlays the luminance-averaged landscape obstruction image with an image obtained by projecting a portion of the first image using a projection transformation unit, A video synthesis device equipped with the following features.

2. The aforementioned projection transformation unit is, The image synthesis apparatus according to claim 1, which creates the image by homography transformation, in which the positions of pixels in the first image at the viewpoint of the first camera are replaced with the positions of pixels on a virtual plane, and the positions of pixels on the virtual plane are replaced with the positions of pixels in the second image at the viewpoint of the second camera.

3. The part of the first image that the composite unit overlays is In the first video, the upper region video is a distant first viewpoint, This is a lower region image that represents a second viewpoint, approaching the second camera position in the first image. The aforementioned projection transformation unit is, A first projection transformation unit (homography transformation) replaces the pixel positions of the upper region image with the pixel positions on a first virtual plane, and replaces the pixel positions on the first virtual plane with the pixel positions of the second image at the viewpoint of the second camera, A second projection transformation (homography transformation) unit replaces the pixel positions of the lower region image with the pixel positions on a second virtual plane, and replaces the pixel positions on the second virtual plane with the pixel positions of the second image at the viewpoint of the second camera, The image synthesis apparatus according to claim 1, comprising:

4. A method of video synthesis, A first video acquisition process involves acquiring a first video using a first camera mounted on a moving vehicle, A second video acquisition step involves acquiring a second video, which includes a video of a landscape obstruction caused by a part of the vehicle being captured, using a second camera mounted on the vehicle at a different location from the first camera; A luminance averaging step, which involves averaging the luminance of the images of the obstructing scenery included in the second image, A synthesis process is performed by overlaying the luminance-averaged landscape obstruction image with an image obtained by projecting a portion of the first image using a projection transformation unit, A video synthesis method comprising the following features.

5. In the synthesis process, the portion of the first image that is superimposed is The image synthesis method according to claim 4, comprising the steps of replacing the pixel positions of the first image at the viewpoint of the first camera with the pixel positions on a virtual plane, and replacing the pixel positions on the virtual plane with the pixel positions of the second image at the viewpoint of the second camera.

Images