Image synthesis device and control method
The image synthesis device addresses the challenge of depicting relative vehicle positions by projecting camera images onto a virtual road surface and adjusting viewpoints, resulting in clearer spatial understanding.
Patent Information
- Application Number
- DE102020101539
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-08-22
- Filing Date
- 2020-01-23
- Publication Date
- 2024-11-07
- Estimated Expiration
- 2040-01-23
AI Technical Summary
Existing image synthesis apparatuses struggle to accurately depict the relative position between a viewer's vehicle and other vehicles in the synthetic image, making it difficult for users to understand the spatial relationship.
An image synthesis device that projects vehicle-mounted camera images onto a virtual projection plane corresponding to the road surface, adjusts the virtual viewpoint based on vehicle speed and position, and synthesizes component images to enhance spatial understanding.
Generates a synthetic image that clearly depicts the relative position between the viewer's vehicle and other vehicles, reducing image inconsistencies and enhancing user comprehension.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Area
[0001] The present invention relates to an image synthesis apparatus and a control method. background
[0002] Image display devices have been developed to capture a rearview image of a vehicle through a plurality of vehicle-mounted cameras and display the captured image. Examples of such devices include a so-called electronic vehicle mirror, which is an image display device that allows a driver to visually view the image showing the rearview image of the vehicle.
[0003] This image display device receives images from both the left and right rearview mirrors captured by cameras mounted near the left and right doors of the vehicle. The image display device also receives a center rearview mirror image captured by a camera mounted near the rear bumper or trunk lid. These images are combined to produce a single image comprising the left, right, and center rearview mirror images. The resulting image is displayed on a rearview mirror-type display device.
[0004] A known image synthesis device performs image synthesis by combining images (see, for example, Patent Document 1). The image synthesis device disclosed in Patent Document 1 maps points at infinity in images captured by cameras mounted near the right and left doors of the vehicle to a point at infinity in an image captured by a camera mounted at a rear of the vehicle. This creates a synthesized image as if viewed from a single virtual viewpoint.
[0005] Patent Document 2 describes an on-board display device comprising: a display having a display region on a passenger seat side with respect to a central position of a vehicle cabin in a vehicle width direction and capable of displaying an image in the display region;and a display control device that converts a second image captured by a second image pickup section into a vision-origin converted image as seen from a hypothetical vision origin at an installation position of the first image pickup section, combines a first image captured by the first image pickup section with an image of a region in the vision-origin converted image that does not overlap the first image to generate a composite image, and uses the display to display the composite image in the display region.
[0006] Patent Document 3 describes a multi-camera vision system for a vehicle, including first, second, and third image pickup devices arranged on respective vehicle parts. The field of view of the first image pickup device overlaps with the field of view of the third image pickup device, thereby defining a first overlap zone, and the field of view of the second image pickup device overlaps with the field of view of the third image pickup device, thereby defining a second overlap zone. In response to processing of the received image data by an image processor, a synthesized image is generated without duplicating the objects present in the first overlap zone and the second overlap zone.The synthesized image approximates a view as would be seen by a virtual camera at a single location outside the vehicle and is displayed on a single display screen of a reconfigurable display device that can be seen by a driver of the vehicle during normal vehicle operation.
[0007] Patent Document 4 describes a method and apparatus for expanding a rearview display. The method includes detecting a rearview display expansion condition; receiving a first image from a first camera facing a first area behind a vehicle in response to detecting the rearview display expansion condition and displaying the first received image on the rearview display. The method can be used to expand rearview displays to provide an unobstructed view when a view displayed on the rearview display is obstructed. List of citationsPatent literature Patent document 1: Japanese Unexamined Patent Disclosure No. JP H10-257482 A Patent document 2: Laid-open specification DE 10 2018 101 542 A1 Patent Document 3: US Unexamined Publication No. US 2013 / 0194426 A1 Patent specification 4: Laid-open specification DE 10 2018 101 592 A1 SummaryTechnical task
[0008] Unfortunately, the synthetic image generated by the image synthesis device disclosed in Patent Document 1 may cause difficulty for the viewer to visually understand a relative position between the viewer's vehicle and another vehicle seen in the synthetic image.
[0009] In response to the above-mentioned problem, an object of the present invention is to provide an image synthesis apparatus that generates a synthetic image allowing a viewer to easily understand a relative position between a vehicle of the viewer and another vehicle seen in the synthetic image. Solution to the task
[0010] According to one aspect of the present invention, an image synthesis apparatus according to claim 1 is provided.
[0011] It should be noted that general and specific aspects of the present invention may be embodied in a system, a method, an integrated circuit, a computer program, a computer-readable recording medium such as a CD-ROM, or any combination thereof. Beneficial effects
[0012] The image synthesis device according to the present invention is capable of generating a synthetic image that allows a user to easily understand a relative position between a viewer's vehicle and another vehicle seen in the synthetic image. Short description of the drawing [ Fig. 1] Fig. 1 is a schematic view of an example of a vehicle including cameras that acquire images to be synthesized by an image synthesis device according to the embodiment. [ Fig. 2] Fig. 2 shows images before and after synthesis performed by the image synthesis apparatus according to the embodiment. [ Fig. 3] Fig. 3 is a block diagram of the functional structure of the image synthesis apparatus according to the embodiment. [ Fig. 4] Fig. 4 illustrates images obtained by an Erlanger according to the embodiment. [ Fig. 5] Fig. 5 illustrates an example of a virtual projection plane according to the embodiment. [ Fig. 6] Fig. 6 illustrates an image converted by an image converter according to the embodiment. [ Fig. 7] Fig. 7 illustrates a synthetic image generated by a synthesizer according to the embodiment. [ Fig. 8] Fig. 8 shows an example of a virtual projection plane according to Comparative Example 1. [ Fig. 9] Fig. 9 illustrates an image converted by an image converter using the virtual projection plane according to Comparative Example 1. [ Fig. 10] Fig. 10 is a flowchart of the operations performed by the image synthesis apparatus according to the embodiment. [ Fig. 11] Fig. 11 illustrates virtual viewpoints according to Modification 1 of the embodiment. [ Fig. 12] Fig. 12 illustrates an example of a virtual projection plane according to Modification 2 of the embodiment. [ Fig. 13] Fig. 13 shows an example of a virtual projection plane according to Comparative Example 2. [ Fig. 14] Fig. 14 illustrates an example of a virtual projection plane according to Modification 3 of the embodiment. [ Fig. 15] Fig. 15 illustrates an example of a virtual projection plane according to Comparative Example 3. Description of the embodiment
[0013] According to one aspect of the present invention, there is provided an image synthesis apparatus comprising, inter alia: an acquirer that acquires a first image and a second image, both of which capture at least one same object, from two cameras, both mounted on a vehicle, and captures a rear image showing at least one road surface;an image converter that (i) projects, as a projection image, the first image and the second image in a predetermined relative position in which the first image and the second image overlap each other onto a virtual projection plane virtually placed near the vehicle and including a surface corresponding to the road surface on which the vehicle is located, and (ii) acquires a third image that is a view of the projection image from a virtual viewpoint, converting the first image and the second image projected onto the virtual projection plane into the third image; a generator that generates a component image showing a component included in the vehicle; and a synthesizer that synthesizes the component image generated by the generator into the third image.
[0014] In this aspect, the image synthesis device performs image conversion using the virtual projection plane containing the area corresponding to the road surface on which the vehicle is located. This image conversion allows the road surface and an object on the road surface, such as a vehicle (the same applies below), seen in the first image, and the road surface and the object on the road surface seen in the second image to be virtually projected onto the area corresponding to the road surface in the virtual projection plane. As a result, the road surface and the object on the road surface are shown at the appropriate positions in the generated third image. In addition, the component image showing a component of the vehicle generated by the generator is synthesized into the third image.The synthetic image created in this way is easy for the user to understand visually.
[0015] For example, the creator can create the component image that indicates a position where the component included in the vehicle must be located.
[0016] In this aspect, the image synthesis device synthesizes the component image showing the component of the vehicle generated by the generator into the third image. By viewing this synthetic image, the user can appropriately understand the distance and relative position between the user's vehicle and the road surface or the object on the road surface. In this way, the image synthesis device is capable of generating a synthetic image that allows the user to more easily understand the relative position between the user's vehicle and another vehicle seen in the synthetic image. Furthermore, the generator can be changed depending on vehicle types. This allows easy development of the image synthesis device for different vehicle types.
[0017] For example, the virtual viewing point can be in front of and above the vehicle.
[0018] In this aspect, the image synthesis device performs image conversion using the image seen from the virtual viewpoint located in front of and above the vehicle. Such a virtual viewpoint enables the image synthesis device to obtain the third image seen from above to show the road surface behind the vehicle and the object on the road surface. Furthermore, such virtual viewpoints enlarge the area of the road surface seen in the third image. By viewing the third image showing the enlarged area of the road surface, the user can more easily understand the relative position between the user's vehicle and another vehicle seen in the synthetic image.
[0019] For example, it is possible that the virtual projection plane includes a first surface and a second surface, the first surface being a surface corresponding to the road surface, the second surface being a remainder of the virtual projection surface excluding the first surface, and the image converter performing the correction processing on a part of the third image near a boundary between the first image and the second image, the part being obtained only from the second surface, instead of both the first surface and the second surface.
[0020] In this aspect, the image synthesis device performs correction processing on the image obtained from the second surface instead of the image obtained from the first surface. In the image obtained from the first surface for image conversion, the position of the surface corresponding to the road surface as a virtual projection plane coincides with the position of the road surface. Thus, inconsistencies or discontinuities are hardly caused in the image of the road surface or a low-height object at the boundary between the first and second images. However, inconsistencies or discontinuities may be caused in the image obtained from the second image for image conversion because the position of the virtual projection plane does not necessarily coincide with the position of an actual object.Consequently, the correction processing performed on the image obtained from the second surface reduces such unnaturalness at the boundary. Here, no correction processing is performed on the image obtained from the first surface. This eliminates the need to perform unnecessary processing, thus reducing the processing load and power consumption.
[0021] For example, it is possible for the synthesizer to (i) synthesize the component image generated by the generator and showing at least an exterior of a body of the vehicle into a suitable position in the third image at which an image of the exterior of the body of the vehicle is projected onto an image of the road surface, or to (ii) synthesize the component image generated by the generator and showing at least a rear wheel of the vehicle into a suitable position in the third image at which an image of the rear wheel of the vehicle is to be arranged.
[0022] In this aspect, the image synthesis device generates a synthetic image in which the image of the vehicle's body or rear wheel is synthesized to show its position. By viewing this synthetic image, the user can better understand the distance and relative position between the user's vehicle's body or rear wheel and another vehicle.
[0023] For example, it is possible for the image converter to obtain a speed of the vehicle, set the virtual viewpoint to a position in response to the obtained speed, and perform the image conversion using the set virtual viewpoint.
[0024] In this aspect, the image synthesis device performs image conversion using virtual viewpoints located at different positions depending on the vehicle's speed. This allows image synthesis appropriate for the vehicle's speed. By viewing the synthetic image appropriately generated in response to the vehicle's speed, the user can more easily understand the relative position between the user's vehicle and another vehicle visible in the synthetic image.
[0025] For example, it is possible for the image converter to set the virtual viewpoint to a lower position for a higher speed of the vehicle and to perform the image conversion using the set virtual viewpoint.
[0026] In this aspect, for a higher vehicle speed, the image synthesis device performs image conversion using the virtual viewpoint located at a lower position. As the vehicle speed is higher, a distance between the vehicle and another vehicle is typically greater. Thus, information about a relative position between the user's vehicle and another vehicle located at a greater distance from the user's vehicle on the road surface is useful to the user driving the vehicle. Thus, when the vehicle is traveling at a higher speed, the user can understand the relative position between the user's vehicle and another vehicle located at a greater distance from the user's vehicle on the road surface.
[0027] For example, it is possible for the image converter to obtain a displacement position of the vehicle, set the virtual viewpoint to a position in response to the obtained displacement position, and perform the image conversion using the set virtual viewpoint.
[0028] In this aspect, the image synthesis device performs image conversion using virtual viewpoints located at different positions depending on the vehicle's displacement position. This allows image synthesis appropriate for the vehicle's displacement position. By viewing the synthetic image appropriately generated in response to the vehicle's displacement position, the user can more easily understand the relative position between the user's vehicle and another vehicle visible in the synthetic image.
[0029] For example, it is possible for the image converter to set the virtual viewpoint (i) to a higher position for a smaller displacement position of the vehicle, or (ii) to a higher position for a reverse displacement position of the vehicle than a position of a virtual viewpoint used when the displacement position is not the reverse displacement position, and perform the image conversion using the set virtual viewpoint.
[0030] In this aspect, for a smaller displacement position of the vehicle, the image synthesis device performs image conversion using the virtual viewpoint located at a higher position. When a vehicle travels in a low gear, the speed of the vehicle is low, and a distance between the vehicle and another vehicle is typically smaller. Thus, information about a relative position between the user's vehicle and another vehicle located a shorter distance from the user's vehicle on the road surface is useful to the user driving the vehicle. Thus, when the vehicle travels in a low gear, the user can understand the relative position between the user's vehicle and another vehicle located a shorter distance from the user's vehicle on the road surface.When the reverse gear is engaged, the image synthesis device performs image conversion using the virtual viewpoint located at a higher position than the virtual viewpoint used when the vehicle is traveling in any other gear. When a vehicle is traveling in reverse, the vehicle speed is low, and the distance between the vehicle and another vehicle or object is typically smaller. Thus, information about the relative position between the user's vehicle and another vehicle or object located at a shorter distance from the user's vehicle on the road surface is useful to the user driving the vehicle.Thus, when the vehicle is traveling in reverse, the user can more easily understand the relative position between the user's vehicle and another vehicle located at a smaller distance from the user's vehicle on the road surface.
[0031] For example, it is possible for the image synthesis device to include three cameras, including the two cameras, the three cameras comprising: a left camera that captures a left rear image of the vehicle; a right camera that captures a right rear image of the vehicle; and a middle camera that captures a middle rear image of the vehicle, wherein the acquirer obtains: (a) a left rear image captured by the left camera; (b) a right rear image captured by the right camera;and (c) a middle reverse image taken by the middle camera, wherein the acquirer obtains as first and second images (i) one of the left and right reverse images and (ii) the middle reverse image and generates as the third image a fourth image in which the component image is synthesized by the synthesizer, and the acquirer obtains as first and second images (i) the fourth image and (ii) the other of the left and right reverse images and generates as the third image a fifth image in which the component image is synthesized by the synthesizer;
[0032] In this aspect, the image synthesis device synthesizes the images captured by the left, right, and center cameras mounted on the vehicle. Thus, while reducing the unnaturalness caused by the image synthesis, the image synthesis device is capable of generating a synthetic image showing a wide area behind the vehicle, covering the area from the rear left to the rear center, then to the rear right of the vehicle, or from the rear right to the rear center, then to the rear left of the vehicle.
[0033] According to another aspect of the present invention, there is provided a control method for an image synthesis device, the control method comprising: acquiring a first image and a second image, both of which capture at least one same object, from two cameras both mounted on a vehicle, and capturing a reverse image showing at least one road surface;Projecting, as a projection image, the first image and the second image in a predetermined relative position in which the first image and the second image overlap each other onto a virtual projection plane virtually placed near the vehicle and including a surface corresponding to the road surface on which the vehicle is located, and obtaining a third image that is a view of the projection image from a virtual viewpoint, thereby converting the first image and the second image projected onto the virtual projection plane into the third image; generating a component image showing a component included in the vehicle; and synthesizing the component image generated by the generator into the third image.
[0034] This produces the same effects as those produced by the image synthesis device described above.
[0035] It should be noted that general and specific aspects of the present invention may be embodied in a system, a method, an integrated circuit, a computer program, a computer-readable recording medium such as a CD-ROM, or any combination thereof.
[0036] A specific exemplary embodiment will now be described in detail with reference to the accompanying drawings.
[0037] The following embodiment is a general or specific example of the present invention. Numerical values, shapes, materials, components, arrangement positions, and connection arrangement of components, steps, sequence of steps, etc., described in the following embodiment are only examples and therefore are not intended to limit the present invention. Embodiment
[0038] The present embodiment describes an image synthesis apparatus that generates a synthetic image that allows a viewer to easily understand a relative position between a vehicle of the viewer and another vehicle seen in the synthetic image.
[0039] Fig. 1 is a schematic view of an example of vehicle 1 according to the present embodiment. Fig. 2 shows images before and after synthesis performed by the image synthesis apparatus 10 according to the present embodiment.
[0040] As in Fig. As shown in Figure 1, the vehicle 1 includes the image synthesis device 10, cameras 12, 13, and 14, and an electronic mirror 40. Unless otherwise specified, the direction in which the vehicle 1 moves forward is defined as the forward direction. Based on this, the forward, backward, right, and left directions of the vehicle 1 are defined. These four directions are also defined as the forward, backward, right, and left directions of a driver of the vehicle 1. The cameras 12, 13, and 14 are mounted to point in the rearward direction of the vehicle 1, or more precisely, mounted to receive light coming from behind the vehicle 1. Each of these cameras captures an image showing at least one road surface behind the vehicle 1.
[0041] The camera 12 is mounted near a left door of the vehicle 1 and records a left rear image of the vehicle 1. The camera 12 generates an image from the recorded left rear image of the vehicle 1. The image generated by the camera 12 is also referred to herein as the "left rear image."
[0042] The camera 13 is mounted near a right door of the vehicle 1 and records a right rear image of the vehicle 1. The camera 13 generates an image from the recorded right rear image of the vehicle 1. The image generated by the camera 13 is also referred to herein as the "right rear image."
[0043] The camera 14 is mounted near a rear bumper or trunk lid of the vehicle 1 and captures a rear center image of the vehicle 1. The camera 14 generates an image from the captured rear center image of the vehicle 1. The image generated by the camera 14 is also referred to herein as the "rear center image."
[0044] The image capture areas of cameras 12 and 14 overlap. The image capture areas of cameras 13 and 14 overlap. Thus, the same object is seen in parts of both the left reverse image and the center reverse image. Likewise, the same object is seen in parts of both the right reverse image and the center reverse image.
[0045] Each of cameras 12, 13, and 14 captures an image under different recording conditions. Specifically, each of cameras 12, 13, and 14 is mounted in different positions facing different directions. For example, these cameras capture images at 60 frames per second. Furthermore, cameras 12, 13, and 14 may contain optical systems that may have different optical properties.
[0046] The electronic mirror 40 is an image display device that displays a rearview image 50 showing an image behind the vehicle 1. The electronic mirror 40 displays an image based on a video signal output by the image synthesis device 10. The electronic mirror 40 can be used instead of a conventional rearview mirror that shows a rear view from the vehicle 1 using light reflection.
[0047] The image synthesis device 10 synthesizes the images generated by the cameras 12, 13 and 14 into an image to be displayed by the electronic mirror 40. Examples of the images generated by the cameras 12, 13 and 14 are shown as images 51, 52 and 53 (also referred to as “image 51, etc.”) in Fig. 2. Images 51, 52, and 53 are captured by cameras 12, 13, and 14, respectively, at substantially the same time. Here, "substantially the same time" can refer not only to precisely the same time, but also to times that have a small difference that is barely perceptible to humans (such as a period of time almost equal to one frame interval).
[0048] As described above, the cameras 12, 13, and 14 capture the image of the same object under different shooting conditions. Thus, the image synthesis device 10 cannot obtain the natural rearview image 50 by simply combining the images generated by the cameras 12, 13, and 14. Here, a natural rearview image 50 refers to an image that is the same as the rear view from the vehicle 1 that a driver of the vehicle 1 can see through a conventional rearview mirror (called a physical mirror). More specifically, an image generated by simply combining the images may have double images or image dropouts near a boundary between the combined images. A double image shows a single object twice in the image. Image dropouts cause an object in the image to be unseen even though the object is present in the actual view.
[0049] To avoid such a problem, the image synthesis device 10 combines the images generated by the cameras 12, 13, and 14 after performing appropriate processing on the images. The image synthesis device 10 thus generates a natural reverse image 50 in which double images are reduced. Then, the image synthesis device 10 causes the electronic mirror 40 to display this image. Here, two adjacent images from the three images generated by the cameras 12, 13, and 14 may be synthesized first. Then, this synthetic image and the remaining one image may be synthesized to generate the reverse image 50. Alternatively, the three images generated by the cameras 12, 13, and 14 may be synthesized in parallel to generate the reverse image 50.
[0050] The functions and processes of the image synthesis device 10 are described below.
[0051] Fig. 3 is a block diagram of a functional configuration of the image synthesis device 10 according to the present embodiment. In this example, the image synthesis device 10 obtains a first image and a second image, both of which capture the same subject, from the cameras 12 and 14. The image synthesis device 10 then synthesizes these images. These two images can be the left reverse image and the center reverse image, or the right reverse image and the center reverse image.
[0052] In the following detailed description, the first image is the left reverse image, and the second image is the center reverse image. In this case, the image synthesis device 10 receives the first image and the second image, captured at 60 frames per second by the camera 12 and the camera 14. Each time the images are received, the image synthesis device 10 generates a synthetic image from the first image and the second image and causes the electronic mirror 40 to display the synthetic image.
[0053] As in Fig. 3, the image synthesis device 10 includes the erlanger 21, the image converter 22, the synthesizer 23, the display controller 24, and the generator 25. Each of these components may be implemented by a processor executing a predetermined program using, for example, a memory, or may be implemented as dedicated hardware.
[0054] The acquisition device 21 is an image acquisition device that acquires the first image and the second image, both of which capture at least the same object under different image acquisition conditions (or, more precisely, the camera 12 and the camera 14 are mounted at different locations). For example, the acquisition device 21 acquires image data generated by image acquisition with two cameras 12 and 14 mounted on the vehicle 1.
[0055] The image converter 22 is a processor that obtains a third image by converting the first image and the second image obtained by the acquirer 21. Specifically, the image converter 22 projects the first image and the second image, at a predetermined relative position where the first image and the second image overlap with each other, onto a virtual projection plane virtually located near the vehicle 1 and including a surface corresponding to the road surface on which the vehicle 1 is located. The image converter 22 obtains an image (corresponding to the third image) viewed from a virtual viewpoint through image conversion performed on the image projected onto the virtual projection plane (also referred to as a projection image). In this way, the image converter 22 converts the first image and the second image into the third image.
[0056] The virtual viewpoint differs from the positions of cameras 12 and 14, which receive the first and second images, respectively. Each of the positions of cameras 12 and 14 is referred to as a "real viewpoint" to distinguish it from the virtual viewpoint.
[0057] The road surface refers to the surface on which the vehicle 1 is located and which is located beneath the vehicle 1 when the vehicle 1 is traveling or parked. An intended purpose or use of this surface is not limited. Concepts of the road surface include, in addition to a surface of a road, the floor of a parking lot and a floor area of a business. Any coating may be applied to the surface, and the surface may be made of any material. The process performed by the image converter 22 for projecting the first image and the second image onto the virtual projection plane and the process for converting the projection image projected onto the virtual projection plane into the third image seen from the virtual viewpoint can be achieved by a computer through arithmetic processing.This arithmetic processing performed to convert an image viewpoint can be performed using a well-known technology.
[0058] The virtual viewpoint may be located inside vehicle 1, such as in a position above and between a driver's seat and a passenger seat, or more specifically in a position where the rearview mirror is located. Alternatively, the virtual viewpoint may be located in front of and above vehicle 1. In the following example, the virtual viewpoint is located in front of and above vehicle 1.
[0059] To be more specific, the virtual viewpoint is located in front of the rearview mirror of vehicle 1 at a predetermined distance (for example, more than 0 m, but not more than approximately 3 m) from the rearview mirror, and above the rearview mirror of vehicle 1 at a predetermined distance (for example, more than 0 m, but not more than approximately 3 m) from the rearview mirror. The position of the rearview mirror of vehicle 1 is set according to the specifications of vehicle 1. Such a virtual viewpoint enables the image converter 22 to obtain the third image seen from above, for example, to show the road surface and another vehicle behind the vehicle 1.
[0060] Furthermore, the image converter 22 performs correction processing on an area near the boundary between the first image and the second image. Here, the boundary refers to a boundary line where the first image and the second image overlap on the virtual projection plane. The correction processing is performed to correct inconsistencies and reduce discontinuities of an object on the boundary between the first image and the second image. The correction processing includes averaging pixel values and approximating pixels that form an image of the same object. The image converter 22 may perform correction processing on a partial image of the third image in the area near the boundary between the first image and the second image. The partial image is obtained only from a second surface, instead of both a first surface and the second surface.Here, the first surface refers to a planar surface, and the second surface refers to a remnant of the virtual projection plane without the first surface. The first surface and the second surface are connected.
[0061] The generator 25 is a processor that generates a component image indicating a position where a component included in the vehicle 1 must be located. The synthesizer 23 is a processor that synthesizes the component image generated by the generator 25 into the third image. In the third image, the component image is synthesized to a position indicating a suitable location for this component.
[0062] The generator 25 and the synthesizer 23 can be integrated into a single device. However, the separate construction of the generator 25 and the synthesizer 23 allows the generator 25 to be modified depending on the vehicle types, which have different component images. This allows for easy development of the image synthesis device 10 for different vehicle types.
[0063] The position of any object located in the image areas of the first and second images can typically be determined for the third image by the image converter 22 through arithmetic processing. For the third image, the synthesizer 23 calculates the position of a component from the components of the vehicle 1 present in the image areas of the first and second images. The synthesizer 23 then synthesizes the component image generated by the generator 25 to the calculated position. The component image generated by the generator 25 can be realistic (such as a photograph or a faithfully reproduced image) or schematic (such as a graphic representation using rectangles or cylinders).
[0064] The component may, for example, be a part of a body of the vehicle 1, or more precisely, a part of the rear body of the vehicle 1. In this case, an example of the component image shows at least an outer side of the body of the vehicle 1. The synthesizer 23 synthesizes the component image of at least the outer side of the body of the vehicle to a suitable position in the third image at which the image of the body of the vehicle 1 is projected onto the image of the road surface.
[0065] For example, the component may be a rear wheel of the vehicle 1. In this case, an example of the component image shows at least an outer side of the rear wheel of the vehicle 1. The synthesizer 23 synthesizes the component image of at least the rear wheel of the vehicle 1 to a suitable position in the third image where the rear wheel of the vehicle 1 is to be arranged.
[0066] The display controller 24 is a processor that controls the display of the synthetic image generated by the synthesizer 23. The display controller 24 outputs pixel values contained in the synthetic image to the electronic mirror 40, which is a display device. As a result, the electronic mirror 40 displays this synthetic image.
[0067] The following describes in detail the image conversion processing and the synthesis processing performed by the image synthesis device 10 with reference to examples of images and virtual projection planes.
[0068] This example describes how the synthesis of the left reverse image and the center reverse image is performed in parallel with the synthesis of the right reverse image and the center reverse image. In other words, the left reverse image, the right reverse image, and the center reverse image are synthesized together at the same time.
[0069] Fig. Fig. 4 illustrates images obtained by the Erlanger 21 according to the present embodiment. To be more specific, Fig. 4 shows image 61 as an example of the left reverse image, image 62 as an example of the right reverse image, and image 63 as an example of the middle reverse image. Images 61, 62, and 63 are examples of images taken by cameras 12, 13, and 14, respectively.
[0070] Here, the shooting conditions under which images 61, 62, and 63 are captured are different. These shooting conditions include the positions of the cameras and the optical properties of the optical systems. To be more specific, at least the positions of the cameras (i.e., cameras 12, 13, and 14) capturing the above-mentioned images are different.
[0071] For this reason, the size, shape, and angle of the same object seen in images 61, 62, and 63 are different for each of these images. If no conversion process is performed, it is difficult to obtain a natural reverse image simply by combining images 61 and 63 and by combining images 63 and 62.
[0072] Fig. Fig. 5 illustrates the projection plane 70, which is a first example of the virtual projection plane according to the present embodiment. The image conversion processing performed by the image converter 22 will be described with reference to Fig. 5 described.
[0073] As in Fig. 5, the projection plane 70 contains the planar surface 71 and the surface 72. The planar surface 71 corresponds to the first surface, and the surface 72 corresponds to the second surface.
[0074] The flat surface 71 is located at a position corresponding to the road surface on which the vehicle 1 is located. In other words, the flat surface 71 is located at a position in a virtual space corresponding to the road surface on which the vehicle 1 is located. The flat surface 71 extends from the position of the vehicle 1 to the intersection line with the surface 72. The flat surface 71 corresponds to the road surface on which the vehicle 1 is located, and thus it may not be exactly flat if the road surface is inclined or undulating. Strictly speaking, the virtual projection plane has a surface corresponding to the road surface on which the vehicle 1 is located. Nevertheless, the flat surface 71 will be described as the road surface below.
[0075] Surface 72 is located virtually behind vehicle 1 at a distance from vehicle 1. Surface 72 is a cylindrical surface or a portion of a cylindrical surface. Surface 72 extends upward from the position where surface 72 intersects flat surface 71. A distance from vehicle 1 to surface 72 is, for example, 15 m to 20 m.
[0076] A section of a cylinder having the cylindrical surface of surface 72 is not limited to an exact circle. The cross-section of the cylinder may be an ellipse or an irregular circle. Surface 72 may be flat or spherical.
[0077] The image converter 22 generates the image 61A on the projection plane 70 by converting the image 61 from the position of the camera 12 into Fig. 5 in the same direction as an image pickup direction of camera 12. In the same way, image converter 22 generates image 62A and image 63A on projection plane 70 by projecting image 62 and image 63 from the positions of camera 13 and camera 14 in the same direction as the image pickup directions of camera 13 and camera 14. Images 61A, 62A, and 63A are also referred to as projection images.
[0078] In Fig. In Figure 5, a gap is drawn between image 61A and image 63A for illustrative purposes, but this gap does not actually exist. Thus, image 61A and image 63A are arranged without any gap. A line connecting image 61A and image 63A corresponds to the boundary. The same applies to a gap between image 62A and image 63A.
[0079] Subsequently, the image converter 22 receives the third image as the projection, seen from the virtual viewing point 73 in the direction 74.
[0080] In this way, the image converter 22 performs image conversion of the images captured by the cameras 12, 13, and 14 to obtain the single third image. The third image thus obtained is described below.
[0081] Fig. 6 illustrates image 65, which is the third image obtained by the image converter 22 according to the present embodiment. Image 65 shows objects including: another vehicle C1 behind vehicle 1; another vehicle C2 on the right side behind vehicle 1; another vehicle C3 on the right side of vehicle 1.
[0082] The image 65 is generated from the images 61, 62 and 63 by image conversion performed by the image converter 22.
[0083] In particular, image 65 shows a correct relative position with respect to the road surface. Furthermore, no inconsistencies or discontinuities are visible in the boundary section B1 between the original images 61 and 63 and in the boundary section B2 between the original images 62 and 63. This causes no displacement between the original images because parts showing the road surface in images 61, 62, and 63 are projected onto the flat surface 71, which corresponds to the road surface in the virtual projection plane.
[0084] Next, the image synthesis processing performed by Synthesizer 23 is described.
[0085] Fig. Fig. 7 shows the image 69, that is, a synthetic image generated by the synthesizer 23 according to the present embodiment. Here, Fig. 7 shows that the image 69 is displayed on the electronic mirror 40 by the display control 24.
[0086] Image 69 is generated by synthesizing the component image into image 65. The component image includes: image 67, which shows the exterior of the body of vehicle 1; image 68L, which shows the left rear wheel of vehicle 1; and image 68R, which shows the right rear wheel of vehicle 1.
[0087] Image 67 is synthesized to the position in image 69 where the image of the body of vehicle 1 is projected to the appropriate position in the road surface image. Images 68L and 68R are synthesized to the positions in image 69 where the left and right rear wheels of vehicle 1 must be located.
[0088] By viewing the image 69 showing the component image including the images 67, 68L and 68R on the electronic mirror 40, the user can appropriately understand actual distances and relative positions between the vehicle 1 and other vehicles near the vehicle 1.
[0089] For example, by viewing image 69, which shows image 67 of the exterior of the body of vehicle 1, the user can see the distance L1 and a relative position between image 67 and the image of vehicle C1. This allows the user to roughly understand the actual distance and relative position between vehicle 1 and vehicle C1. In the same way, by viewing image 69, the user can see the distance L2 and a relative position between image 67 and the image of vehicle C2. This allows the user to roughly understand the actual distance and relative position between vehicle 1 and vehicle C2.
[0090] Further, by viewing image 69 showing image 68R indicating the position of the rear wheel of vehicle 1, the user can see the distance L3 and a relative position between image 68R and the image of vehicle C3 located near vehicle 1. This allows the user to roughly understand the actual distance and relative position between vehicle 1 and vehicle C3 located near vehicle 1.
[0091] Suppose that image 69 shows vehicles C1, C2, and C3, but does not display the component image. In this case, the user can see that vehicles C1, C2, and C3 are located behind, diagonally behind, and on the right side. However, the user cannot understand the respective distances and relative positions between vehicle 1 and vehicles C1, C2, and C3. For example, although the user visually determines that vehicle C3 is on the right side of vehicle 1, they cannot tell whether vehicle C3 is immediately to the right of vehicle 1 or to the rear right of vehicle 1.
[0092] However, image 69, in which the component image is synthesized, allows the user to understand the distances and relative positions between vehicle 1 and vehicles C1, C2, and C3. By viewing image 69, which shows the image of vehicle C3 on the right side of image 68R, the user can say, for example, that vehicle C3 is directly to the right of vehicle C1.
[0093] Next, advantageous effects achieved by using the projection plane 70 will be described in comparison with a case (hereinafter referred to as Comparative Example 1) in which a projection plane having a shape different from the shape of the projection plane 70 is used. In this comparative example, a virtual viewpoint from which a projection image projected onto the projection plane is viewed is located in the rearview mirror of the vehicle 1.
[0094] Fig. 8 shows an example of the virtual projection plane according to Comparative Example 1.
[0095] The projection plane 70B, which is the virtual projection plane according to Comparative Example 1, does not include a flat surface at the position of the road surface on which the vehicle is located. Specifically, the projection plane 70B does not include a surface corresponding to the flat surface 71 of the projection plane 70. The projection plane 70B corresponds to the surface 72 and is arranged vertically at a position of the road surface.
[0096] Fig. 9 illustrates the image 65B obtained by the image converter 22 through image conversion using the virtual projection plane according to Comparative Example 1. The image 65B is generated by the image converter 22 through image conversion using the projection plane 70B according to Comparative Example 1 as the virtual projection plane instead of the projection plane 70.
[0097] In image 65B, objects are displayed discontinuously in boundary section B3 between original image 61 and image 63 and in boundary section B4 between original image 62 and image 63. The discontinuity is partially prominent in the image showing the road surface. To be more specific, discontinuities are visible in an image of a curb located between the road surface and a sidewalk in boundary section B3 and in an image of pavement marking lines (white lines) painted on the road surface in boundary section B4.
[0098] Images 61, 62, and 63, each containing the image of the road surface, are projected onto the projection plane 70B. Here, because a difference between the position in which the image of the road surface is projected and the actual position of the road surface is greater than differences in other parts, this leads to the above-mentioned discontinuities. Thus, the image conversion using the Fig. 8, the image of the road surface is shifted at two positions.
[0099] Furthermore, image 65B shows the images of vehicles C1, C2, and C3 without the component image. Although the user currently determines that vehicle C3 is located to the right of vehicle 1, he cannot tell whether vehicle C3 is immediately to the right of vehicle 1 or to the rear right of vehicle 1.
[0100] Compared to Figure 65B in Fig. 9 shows Figure 65 in Fig. 6, a larger image proportion of the road surface. This is because the virtual viewpoint 73 used to generate the image 65 is located further forward and higher than the virtual viewpoint 73A used to generate the image 65B. Thus, the image 65 is generated from such a virtual viewpoint 73 looking down on the view. This increases the distance between the vehicle 1 and another vehicle in the image 65. Furthermore, such an image allows the user to easily understand the distance and relative position between the vehicle 1 and the other vehicle.
[0101] From all this, the following beneficial effects can be seen by comparing Fig. 6 according to the present embodiment with Fig. 9 according to Comparative Example 1.
[0102] Seen as in Fig. 9, vehicle C1 and vehicle C2 are the same size and thus appear to be driving side by side with the user. However, as can be seen in Figure 63 (see Fig. 4) before the image conversion, the vehicle C1 is actually located diagonally behind the vehicle C2. Thus, the synthesis of the boundaries can be done as in Fig. 9 cause the user to incorrectly detect the relative positions between the vehicle (Vehicle 1) of the user (for example, the driver) and the vehicles C1 and C2 located behind the Vehicle 1.
[0103] As described above, Figure 65 shows Fig. 6 the continuous lane marking lines (the white lines) painted on the road surface in the boundary sections B1 and B2. In particular, the relative positions between vehicles C1 and C2 in Figure 65 are the same as those in Figure 63 (see Fig. 4). Thus, when the user clicks on image 65 in Fig. 6, the probability of such a false detection as in Fig. 9 reduced.
[0104] In Figure 65B in Fig. 9, several posts are shown on the sidewalk parallel to the vertical direction of Figure 65A. In Figure 65 in Fig. 6, however, several posts on the sidewalk are inclined relative to the vertical direction of Figure 65. An angle of inclination relative to the vertical direction is greater toward the left side in Figure 65 (i.e., the front in the view of Figure 65). An outline of the vehicle C3 in Figure 65B in Fig. 9 is almost parallel to the vertical direction of Figure 65B. An outline of the vehicle C3 in Figure 65 in Fig. However, the image shown in Figure 65 is tilted toward the upper right. Thus, the present embodiment has a structure that prioritizes the display of the respective distances between the user's vehicle (vehicle 1) and vehicles C1, C2, and C3 without causing false recognition, even when the actual directions of the objects on the left and right sides in Figure 65 are different from the vertical direction in Figure 65.
[0105] Fig. 10 is a flowchart of the operations performed by the image synthesis device 10 according to the present embodiment. This processing corresponds to the control method of the image synthesis device 10.
[0106] In step S101 (Erlangen), the first image and the second image, in both of which at least the same object was captured, are obtained from two cameras mounted on a vehicle and having different image capturing conditions.
[0107] In step S102 (conversion), image conversion of the first image and the second image is performed using virtual projection. More specifically, the first image and the second image are projected onto a virtual projection plane, which is virtually located near vehicle 1 and includes a surface corresponding to the road surface on which vehicle 1 is traveling, at a predetermined relative position where the first image and the second image overlap each other. Then, a projection image including the first image and the second image projected onto the virtual projection plane is converted into a third image viewed from a virtual viewpoint.
[0108] In step S103 (generation), a component image showing a component of the vehicle is generated.
[0109] In step S104 (synthesizing), the component image generated in step S103 is synthesized into the third image.
[0110] Thus, the image synthesis device 10 generates a synthetic image that allows a viewer to easily understand a relative position between a vehicle of the viewer and another vehicle seen in the synthetic image. Modification 1 of the embodiment
[0111] This modification describes another example of the virtual viewpoint.
[0112] In the embodiment described above, the virtual viewpoint 73 is located in front of and above the position of the rearview mirror of the vehicle 1. However, the virtual viewpoint may be located at a different position.
[0113] Fig. 11 shows positions of virtual viewpoints according to the present modification.
[0114] In Fig. 11 (a) shows examples of positions of virtual viewpoints and directions in which a projection image is viewed from the virtual viewpoints. Specifically, (a) shows the virtual viewpoints 73, 73A, and 73B and the viewing directions 74, 74A, and 74B corresponding to these virtual viewpoints. The virtual viewpoint 73B is located further forward and higher than the virtual viewpoint 73A. The virtual viewpoint 73 is located further forward and higher than the virtual viewpoint 73B. An inclination angle in the viewing direction increases as the position of the virtual viewpoint is higher. Thus, the directions shown in (a) are Fig. 11 are expressed as θA < θB < θ.
[0115] The projection image viewed from the virtual viewpoint 73B or 73B shows a gap behind vehicle 1, viewed from a higher position, compared to the projection image viewed from the virtual viewpoint 73A. Thus, a proportion of the road surface in the generated third image increases. This usefully allows the user to easily understand the distance and relative position between vehicle 1 and another vehicle.
[0116] In Fig. 11 (b) shows various examples of positions of virtual viewpoints and directions in which a projection image is viewed from the virtual viewpoints. In particular, (b) shows the virtual viewpoints 73A, 73C, and 73D and the viewing directions 74A, 74C, and 74D corresponding to these virtual viewpoints. The virtual viewpoint 73C is located higher than the virtual viewpoint 73A. The virtual viewpoint 73D is located higher than the virtual viewpoint 73C. As in the case described above, the positions shown in (b) in Fig. 11 are expressed as θA < θC < θD.
[0117] As in (a) in Fig. 11, the projection image viewed from the virtual viewpoint 73C or 73D shows a gap behind vehicle 1 viewed from a higher position compared to the projection image viewed from the virtual viewpoint 73A. Thus, a proportion of the road surface in the generated third image increases. This usefully allows the user to easily understand the distance and relative position between vehicle 1 and another vehicle.
[0118] The image converter 22 can obtain information about the vehicle 1 and change the position of the virtual viewpoint based on the obtained information.
[0119] For example, the image converter 22 may receive a speed of the vehicle 1 and adjust the virtual viewpoint in response to the received speed. The image converter 22 then performs the image conversion using the adjusted virtual viewpoint. Specifically, the image converter 22 adjusts the virtual viewpoint to a lower position in response to a higher speed of the vehicle 1.
[0120] To be more specific, when the speed of the vehicle 1 increases, the virtual viewpoint is shifted from the virtual viewpoint 73 to the virtual viewpoint 73B, and then from the virtual viewpoint 73B to the virtual viewpoint 73A, as shown in (a) of Fig. 11. In contrast, when the speed of the vehicle 1 decreases, the virtual viewpoint is shifted from the virtual viewpoint 73A to the virtual viewpoint 73B, and then from the virtual viewpoint 73B to the virtual viewpoint 73. The same description applies to the virtual viewpoints 73D, 73C and 73A shown in (b) in Fig. 11 are shown.
[0121] Furthermore, the image converter 22 may receive a displacement position of the vehicle 1 and adjust the virtual viewpoint in response to the received displacement position. Then, the image converter 22 performs image conversion using the adjusted virtual viewpoint. Specifically, the image converter 22 adjusts the virtual viewpoint to a higher position in response to a lower displacement position. Alternatively, when the reverse gear is engaged, the image converter 22 adjusts the virtual viewpoint to a higher position than the virtual viewpoint used when the vehicle 1 is traveling in any other gear.
[0122] Special shifts among the virtual viewpoints are the same as those described above.
[0123] As described above, the virtual viewpoint is changed in response to the speed or displacement position of the vehicle 1. Thus, information useful to the user for driving the vehicle 1 can be presented to the user. By viewing the image seen from the virtual viewpoint corresponding to the speed or displacement position of the vehicle 1, the user can understand the relative position between the user's vehicle 1 and another vehicle. Variation 2 of the embodiment
[0124] The present modification describes an example of a virtual projection plane having a shape different from the shape of the virtual projection plane according to the above-described embodiment.
[0125] Fig. 12 illustrates an example of a virtual projection plane according to the present modification.
[0126] As in Fig. As shown in Figure 12, the projection plane 70C includes the planar surface 71C and the surface 72C. The planar surface 71C corresponds to the first surface, and the surface 72C corresponds to the second surface.
[0127] The flat surface 71C is located at a position corresponding to the road surface on which the vehicle 1 is located. The flat surface 71C extends from the position of the vehicle 1 to the intersection line with the surface 72C.
[0128] Surface 72C is located virtually behind vehicle 1 at a distance from vehicle 1. Surface 72C is a flat surface or a portion of a flat surface. Surface 72C extends upward from the position where surface 72C intersects with flat surface 71C. A distance from vehicle 1 to surface 72C is, for example, 15 m to 20 m.
[0129] The image converter 22 performs the image conversion as in the above-described embodiment, using the projection plane 70C instead of the projection plane 70.
[0130] Fig. 13 shows an example of a virtual projection plane according to Comparative Example 2.
[0131] As in Fig. As shown in Figure 13, the projection plane 70D, which is the virtual projection plane according to Comparative Example 2, does not include a flat surface at the position of the road surface on which the vehicle is located. Specifically, the projection plane 70D does not include a surface corresponding to the flat surface 71 of the projection plane 70. The projection plane 70D corresponds to the surface 72 and is arranged vertically at a position of the road surface.
[0132] The image converter 22 obtains an image by image conversion performed on the images 61, 62, and 63 using the projection plane according to Comparative Example 2. In this obtained image, objects in the boundary portions B1 and B2 are displayed discontinuously, as in the case shown in Fig. Figure 65B shown in Figure 9.
[0133] An image obtained by the image converter 22 through image conversion performed with the images 61, 62 and 63 using the Fig. 12, but shows a correct relative position with respect to the part of the road surface, as in Figure 65 in Fig. 6. Thus, the image synthesis device 10 generates a synthetic image that allows a viewer to easily understand a relative position between a vehicle of the viewer and another vehicle seen in the synthetic image. Variation 3 of the embodiment
[0134] The present modification describes another example of a virtual projection plane having a shape different from the shape of the virtual projection plane according to the above-described embodiment.
[0135] Fig. 14 illustrates an example of a virtual projection plane according to the present modification.
[0136] As in Fig. As shown in Figure 14, the projection plane 70E includes the planar surface 71E and the surface 72E. The planar surface 71E corresponds to the first surface, and the surface 72E corresponds to the second surface.
[0137] The flat surface 71E is virtually located at a position corresponding to the road surface on which the vehicle 1 is located. The flat surface 71E extends from the position of the vehicle 1 to the intersection line with the surface 72E.
[0138] Surface 72E is located virtually behind vehicle 1 at a distance from vehicle 1. Surface 72E is a spherical surface or a portion of a spherical surface. Surface 72E extends upward from the position where surface 72E intersects planar surface 71E. A distance from vehicle 1 to surface 72E is, for example, 15 m to 20 m.
[0139] The image converter 22 performs the image conversion as in the above-described embodiment, using the projection plane 70E instead of the projection plane 70.
[0140] Fig. 15 illustrates an example of a virtual projection plane according to Comparative Example 3.
[0141] As in Fig. As shown in Figure 15, the projection plane 70F, which is the virtual projection plane according to Comparative Example 3, does not include a flat surface at the position of the road surface on which the vehicle is located. Specifically, the projection plane 70F does not include a surface corresponding to the flat surface 71E of the projection plane 70E. The projection plane 70F corresponds to the surface 72E and is arranged vertically at a position of the road surface.
[0142] The image converter 22 obtains an image by image conversion performed on the images 61, 62, and 63 using the projection plane according to Comparative Example 3. In this obtained image, objects in the boundary portions B1 and B2 are displayed discontinuously, as in the case shown in Fig. Figure 65B shown in Figure 9.
[0143] An image obtained by the image converter 22 through image conversion performed with the images 61, 62 and 63 using the Fig. 15, but shows a correct relative position with respect to the part of the road surface, as in Figure 65 in Fig. 6. Thus, the image synthesis device 10 generates a synthetic image that allows a viewer to easily understand a relative position between a vehicle of the viewer and another vehicle seen in the synthetic image.
[0144] It should be noted that a method for synthesizing two images, such as the first and second images, is also capable of synthesizing three images captured by three cameras. For example, the image synthesis device 10 (1) acquires an image by synthesizing a left reverse image and a center reverse image as the first image and the second image, and (2) then synthesizes this acquired image and a right reverse image again as the first image and the second image. This finally results in a reverse image in which the left reverse image, the center reverse image, and the right reverse image are combined. Here, the image obtained by synthesizing the first and second images is also referred to as a fourth image, and the image obtained by synthesizing the fourth image and the right reverse image is referred to as a fifth image.
[0145] As described above, the image synthesis device 10 achieves the synthesis of the three images by (1) obtaining an image by synthesizing the left reverse image and the center reverse image as the first image and the second image, and then (2) resynthesizing this obtained image and the right reverse image as the first image and the second image. However, this synthesis is not intended to be limiting. For example, the image synthesis device 10 may (1) obtain an image by synthesizing a right reverse image and a center reverse image as the first image and the second image, and then (2) resynthesize this obtained image and a left reverse image as the first image and the second image. Here, an image obtained by synthesizing the fourth image obtained by synthesizing the first and second images and the left reverse image corresponds to the fifth image.
[0146] Thus, the acquirer 21 of the image synthesis device 10 can acquire one of the left and right reverse images and the center reverse image as the first and second images. Then, the synthesizer 23 of the image synthesis device 10 can generate the fourth image in which the component image is synthesized as the third image. After that, the acquirer 21 of the image synthesis device 10 can acquire the fourth image as the first and second images and the other of the left and right reverse images. Then, the synthesizer 23 of the image synthesis device 10 can generate the fifth image in which the component image is synthesized as the third image. In this case, the display controller 24 causes the fifth image to be displayed on the electronic mirror 40.
[0147] As described so far, the image synthesis apparatus according to the present embodiment and modifications performs image conversion using the virtual projection plane including the surface corresponding to the road surface on which the vehicle is located. This image conversion allows the road surface and an object on the road surface, such as a vehicle (the same applies below), seen in the first image, and the road surface and the object on the road surface seen in the second image to be virtually projected onto the area corresponding to the road surface in the virtual projection plane. As a result, the road surface and the object on the road surface are shown at the appropriate positions in the generated third image.In addition, the component image, which shows a component of the vehicle generated by the generator, is synthesized into the third image. The resulting synthetic image is easy for the user to visually understand.
[0148] Furthermore, the image synthesis device synthesizes the component image showing the component of the vehicle generated by the generator into the third image. By viewing this synthesized image, the user can appropriately understand the distance and relative position between the user's vehicle and the road surface or the object on the road surface. In this way, the image synthesis device is capable of generating a synthetic image that allows the user to more easily understand the relative position between the user's vehicle and another vehicle seen in the synthetic image. Furthermore, the generator can be changed depending on the vehicle type. This allows for easy development of the image synthesis device for different vehicle types.
[0149] Furthermore, the image synthesis device performs image conversion using the image seen from the virtual viewpoint located in front of and above the vehicle. Such a virtual viewpoint enables the image synthesis device to obtain the third image seen from above to show the road surface behind the vehicle and the object on the road surface. Furthermore, such virtual viewpoints enlarge the area of the road surface seen in the third image. By viewing the third image showing the enlarged area of the road surface, the user can more easily understand the relative position between the user's vehicle and another vehicle seen in the synthetic image.
[0150] Furthermore, the image synthesis device performs correction processing on the image obtained from the second surface instead of the image obtained from the first surface. In the image obtained from the first surface for image conversion, the position of the surface corresponding to the road surface as a virtual projection plane coincides with the position of the road surface. Thus, inconsistencies or discontinuities are hardly caused in the image of the road surface or a low-height object at the boundary between the first and second images. However, inconsistencies or discontinuities may be caused in the image obtained from the second image for image conversion because the position of the virtual projection plane does not necessarily coincide with the position of an actual object.Thus, the correction processing performed on the image obtained from the second surface reduces such unnaturalness at the boundary. Here, no correction processing is performed on the image obtained from the first surface. This eliminates the need to perform unnecessary processing, thus reducing the processing load and power consumption.
[0151] In addition, the image synthesis device generates a synthetic image in which the image of the vehicle's body or rear wheel is synthesized to show its position. By viewing this synthetic image, the user can better understand the distance and relative position between the user's vehicle's body or rear wheel and another vehicle.
[0152] In addition, the image synthesis device performs image conversion using virtual viewpoints located at different positions depending on the vehicle's speed. This allows image synthesis appropriate for the vehicle's speed. By viewing the synthetic image appropriately generated in response to the vehicle's speed, the user can more easily understand the relative position between the user's vehicle and another vehicle visible in the synthetic image.
[0153] For a higher vehicle speed, the image synthesis device performs image conversion using the virtual viewpoint located at a lower position. When the vehicle speed is higher, a distance between the vehicle and another vehicle is typically greater. Thus, information about a relative position between the user's vehicle and another vehicle located at a greater distance from the user's vehicle on the road surface is useful to the user driving the vehicle. Thus, when the vehicle is traveling at a higher speed, the user can understand the relative position between the user's vehicle and another vehicle located at a greater distance from the user's vehicle on the road surface.
[0154] In addition, the image synthesis device performs image conversion using virtual viewpoints located at different positions depending on the vehicle's displacement position. This allows image synthesis appropriate for the vehicle's displacement position. By viewing the synthetic image appropriately generated in response to the vehicle's displacement position, the user can more easily understand the relative position between the user's vehicle and another vehicle visible in the synthetic image.
[0155] For a smaller displacement position of the vehicle, the image synthesis device performs image conversion using the virtual viewpoint located at a higher position. When a vehicle travels in a low gear, the speed of the vehicle is low, and a distance between the vehicle and another vehicle is typically smaller. Thus, information about a relative position between the user's vehicle and another vehicle located a shorter distance from the user's vehicle on the road surface is useful to the user driving the vehicle. Thus, when the vehicle travels in a low gear, the user can understand the relative position between the user's vehicle and another vehicle located a shorter distance from the user's vehicle on the road surface.When the reverse gear is engaged, the image synthesis device performs image conversion using the virtual viewpoint located at a higher position than the virtual viewpoint used when the vehicle is traveling in any other gear. When a vehicle is traveling in reverse, the vehicle speed is low, and the distance between the vehicle and another vehicle or object is typically smaller. Thus, information about the relative position between the user's vehicle and another vehicle or object located at a shorter distance from the user's vehicle on the road surface is useful to the user driving the vehicle.Thus, when the vehicle is traveling in reverse, the user can more easily understand the relative position between the user's vehicle and another vehicle located at a smaller distance from the user's vehicle on the road surface.
[0156] In addition, the image synthesis device synthesizes the images captured by the left, right, and center cameras mounted on the vehicle. Thus, while reducing the unnaturalness caused by image synthesis, the image synthesis device is capable of generating a synthetic image showing a large area behind the vehicle, covering the area from the rear left to the rear center, then to the rear right of the vehicle, or from the rear right to the rear center, then to the rear left of the vehicle.
[0157] Each of the elements of the above-described embodiment may be constructed as a dedicated hardware product, or may be implemented by executing a software program suitable for the element. Each of the elements may be implemented by a program execution unit, such as a CPU or a processor, that reads and executes the software program stored on a recording medium, such as a hard disk or a semiconductor memory. An example of such a software program implementing the image synthesis device according to the above embodiment or the like is the following program.
[0158] More specifically, the above program is a control method for an image synthesis device, comprising: acquiring a first image and a second image, both of which capture at least one identical object, from two cameras mounted on a vehicle, and capturing a rearward image showing at least one road surface; projecting, as a projection image, the first image and the second image, in a predetermined relative position in which the first image and the second image overlap with each other, onto a virtual projection plane virtually placed near the vehicle and including a surface corresponding to the road surface on which the vehicle is located; and acquiring a third image, which is a view of the projection image from a virtual viewpoint, thereby converting the first image and the second image projected onto the virtual projection plane into the third image;Generating a component image showing a component included in the vehicle; and synthesizing the component image generated during the generation into the third image.
[0159] Although image synthesis devices according to one or more aspects of the present invention have been described based on the embodiment, the present invention is not limited to this embodiment. Those skilled in the art will readily understand that embodiments achieved by making various modifications to the exemplary embodiment or embodiments by selectively combining elements disclosed in the above embodiment may be included in one or more aspects of the present invention without departing from the scope of the present invention. Industrial applicability
[0160] The present invention is applicable to an image synthesis apparatus that synthesizes images taken by cameras mounted on a vehicle and displays the synthesized images on an electronic mirror. List of reference symbols 1, C1, C2, C3 vehicle 10 Image synthesis device 12, 13, 14 Camera 21 Erlangen residents 22 image converters 23 synthesizers 24 Display control 25 producers 40 electronic mirrors 50 reverse image 51, 52, 53, 61, 61A, 61B, 61D, 61E, 61F, 62, 62A, 62B, 62C, 62D, 62E, 62F, 63, 63A, 63B, 63C, 63D, 63E, 63F, 65, 65B, 67, 68L, 68R, 69 Image 70, 70B, 70C, 70D, 70E, 70F projection plane 71, 71C, 71E flat surface 72, 72C, 72E surface 73, 73A, 73B, 73C, 73D virtual viewpoint 74, 74A, 74B, 74C, 74D direction B1, B2, B3, B4 border section L1, L2, L3 distance
Claims
[1] Image synthesis device (10) comprising: an acquirer (21) which acquires a first image and a second image, in both of which at least one identical object is captured, from two cameras, both mounted on a vehicle (1, C1, C2, C3), and captures a reverse image (50) showing at least one road surface; an image converter (22) which (i) projects, as a projection image, the first image and the second image in a predetermined relative position in which the first image and the second image overlap each other onto a virtual projection plane (70, 70B, 70C, 70D, 70E, 70F) which is virtually placed in the surroundings of the vehicle (1, C1, C2, C3) and includes a surface corresponding to the road surface on which the vehicle (1, C1, C2, C3) is located, and (ii) acquires a third image which is a view of the projection image from a virtual viewpoint, converting the first image and the second image projected onto the virtual projection plane (70, 70B, 70C, 70D, 70E, 70F) into the third image; a generator (25) that generates a component image showing a component included in the vehicle (1, C1, C2, C3); and a synthesizer (23) which synthesizes the component image generated by the generator (25) into the third image, wherein the virtual projection plane (70, 70B, 70C, 70D, 70E, 70F) includes a first surface and a second surface, wherein the first surface is a surface corresponding to the road surface, the second surface is a remainder of the virtual projection surface without the first surface, and the image converter (22) prohibits correction processing on a partial image obtained from the first surface of the third image and permits a deviation in the vertical direction (74, 74A, 74B, 74C, 74D), and the image converter (22) performs the correction processing on a part of the third image up to a boundary between the first image and the second image, the part being obtained only from the second surface, instead of both the first surface and the second surface. [2] The image synthesis device (10) according to claim 1, wherein the generator (25) generates the component image indicating a position at which the component included in the vehicle (1, C1, C2, C3) must be located. [3] Image synthesis device (10) according to one of claim 1 and claim 2, wherein the virtual viewpoint is located in front of and above the vehicle (1, C1, C2, C3). [4] Image synthesis device (10) according to any one of claims 1 to 3, wherein the synthesizer (23) (i) synthesizing the component image generated by the generator (25) and showing at least an exterior of a body of the vehicle (1, C1, C2, C3) into a suitable position in the third image at which an image of the exterior of the body of the vehicle (1, C1, C2, C3) is projected onto an image of the road surface, or (ii) synthesizing the component image generated by the generator (25) and showing at least one rear wheel of the vehicle (1, C1, C2, C3) into a suitable position in the third image at which an image of the rear wheel of the vehicle (1, C1, C2, C3) is to be placed. [5] The image synthesis device (10) according to any one of claims 1 to 4, wherein the image converter (22) obtains a speed of the vehicle (1, C1, C2, C3), sets the virtual viewpoint to a position in response to the obtained speed, and performs the image conversion using the set virtual viewpoint. [6] The image synthesis device (10) according to claim 5, wherein the image converter (22) sets the virtual viewpoint to a lower position for a higher speed of the vehicle (1, C1, C2, C3) and performs the image conversion using the set virtual viewpoint. [7] The image synthesis device (10) according to any one of claims 1 to 6, wherein the image converter (22) obtains a displacement position of the vehicle (1, C1, C2, C3), sets the virtual viewpoint to a position in response to the obtained displacement position, and performs image conversion using the set virtual viewpoint. [8] Image synthesis device (10) according to claim 7, wherein the image converter (22) converts the virtual viewpoint (i) for a smaller displacement position of the vehicle (1, C1, C2, C3) to a higher position, or (ii) for a reverse displacement position of the vehicle (1, C1, C2, C3) to a higher position than a position of a virtual viewpoint used when the displacement position is not the reverse displacement position, and performs image conversion using the set virtual viewpoint. [9] Image synthesis device (10) according to any one of claims 1 to 8, comprising: three cameras (12, 13, 14), including the two cameras, wherein the three cameras (12, 13, 14) comprise: a left camera which records a left rearward image (50) of the vehicle (1, C1, C2, C3); a right camera which records a right rearward image (50) of the vehicle (1, C1, C2, C3); and a middle camera which records a middle rearward image (50) of the vehicle (1, C1, C2, C3), where the Erlanger (21) receives: (a) a left reverse image (50) taken by the left camera; (b) a right reverse image (50) taken by the right camera; and (c) a middle reverse image (50) taken by the middle camera, the Erlanger (21) receives as first and second images (i) one of the left and right reverse images (50) and (ii) the middle reverse image (50) and generates as the third image a fourth image in which the component image is synthesized by the synthesizer (23), and the Erlanger (21) receives as first and second images (i) the fourth image and (ii) another image of the left and right reverse images (50), and generates as third image a fifth image in which the component image is synthesized by the synthesizer (23). [10] A control method for an image synthesis device (10), the control method comprising: Obtaining a first image and a second image, both of which capture at least one identical object, from two cameras, both mounted on a vehicle (1, C1, C2, C3), and capturing a reverse image (50) showing at least one road surface; Projecting as a projection image the first image and the second image in a predetermined relative position in which the first image and the second image overlap each other onto a virtual projection plane (70, 70B, 70C, 70D, 70E, 70F) that is virtually placed in the surroundings of the vehicle (1, C1, C2, C3) and includes a surface corresponding to the road surface on which the vehicle (1, C1, C2, C3) is located, and obtaining a third image that is a view of the projection image from a virtual viewpoint, thereby converting the first image and the second image projected onto the virtual projection plane (70, 70B, 70C, 70D, 70E, 70F) into the third image; Generating a component image showing a component included in the vehicle (1, C1, C2, C3); and Synthesize the component image generated during creation into the third image wherein the virtual projection plane (70, 70B, 70C, 70D, 70E, 70F) includes a first surface and a second surface, wherein the first surface is a surface corresponding to the road surface, the second surface is a remainder of the virtual projection surface without the first surface, and the step of projecting prohibits correction processing on a partial image obtained from the first surface of the third image and permits a deviation in the vertical direction (74, 74A, 74B, 74C, 74D), and performs the correction processing on a part of the third image up to a boundary between the first image and the second image, the part being obtained only from the second surface, instead of both the first surface and the second surface.
Citation Information
Patent Citations
Vehicle surrounding condition display device
JP1998257482A
onboard display device, onboard display device control method and onboard display device control program
DE102018101542A1
METHOD AND DEVICE FOR EXTENSION OF A REARVIEW DISPLAY
DE102018101592A1
Multi-camera vision system for a vehicle
US20130194426A1
