Display control device, vehicle and display control method

Abstract

The display control device of the present invention includes a vehicle detection unit and an image processing unit, and the image processing unit performs: projection transformation of projecting multiple captured images onto a projection surface; and viewpoint transformation processing of transforming the captured images into images observed from a virtual viewpoint, and making the height above the ground of the virtual viewpoint when no surrounding vehicles are detected in a specified area lower than the height above the ground of the virtual viewpoint when surrounding vehicles are detected in the specified area.

Description

Technical Field

[0001] The invention relates to a display control device, a vehicle and a display control method. Background Art

[0002] Patent document 1 discloses a technology that provides a driver with an image that looks like a virtual bird's-eye view of the area surrounding the vehicle by photographing the area surrounding the vehicle with multiple cameras. In the technology of patent document 1, the distortion of the image projected onto the projection surface can be reduced by setting the projection surface to a bowl shape and changing the size of the projection surface according to the distance from the three-dimensional object.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2012 / 017560 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] However, when the bird's-eye view display of Patent Document 1 is applied to the projection surface of a moving vehicle, the moving direction of the image of the object on the road (e.g., a speed limit mark painted on the road surface, etc.) is different from the moving direction of the vehicle, so its movement is conspicuous and easily guides the driver's line of sight. Therefore, in the prior art, there is room for improvement in synthesizing the images behind the vehicle captured by multiple cameras.

[0008] Non-limiting embodiments of the present invention contribute to providing a display control device, a vehicle, and a display control method that can reduce distortion of an image projected onto a projection surface while making it difficult to guide a driver's line of sight.

[0009] Solutions to the problem

[0010] A display control device according to an embodiment of the present invention comprises: a vehicle detection unit for detecting surrounding vehicles existing around the vehicle based on a plurality of captured images of the surroundings of the vehicle captured by a plurality of camera devices mounted on the vehicle; and an image processing unit for generating a display image displayed on a display unit mounted on the vehicle by synthesizing the plurality of captured images, the image processing unit performing: a projection transformation for projecting the plurality of captured images onto a projection plane set to be located behind the vehicle; and a viewpoint transformation processing for transforming the captured images projected on the projection plane into images observed from a virtual viewpoint located at a position different from that of the camera devices, and the image processing unit causes the height above the ground of the virtual viewpoint when the surrounding vehicles are not detected within a specified area to be lower than the height above the ground of the virtual viewpoint when the surrounding vehicles are detected within the specified area.

[0011] A vehicle according to an embodiment of the present invention includes the above-mentioned display control device.

[0012] A display control method according to an embodiment of the present invention includes the following steps: a step of detecting surrounding vehicles existing around the vehicle based on a plurality of captured images of the surroundings of the vehicle captured by a plurality of camera devices mounted on the vehicle; a step of projecting the plurality of captured images onto a projection surface set to be located behind the vehicle; and a step of transforming the captured images projected on the projection surface into images observed from a virtual viewpoint located at a position different from that of the camera devices, wherein the step of moving the virtual viewpoint includes the step of making the height above the ground of the virtual viewpoint when the surrounding vehicles are not detected within a specified area lower than the height above the ground of the virtual viewpoint when the surrounding vehicles are detected within the specified area.

[0013] Effects of the Invention

[0014] According to an embodiment of the present invention, a display control device, a vehicle, and a display control method can be constructed that can reduce distortion of an image projected onto a projection surface while making it difficult to guide a driver's line of sight.

[0015] Further advantages and effects in one embodiment of the present invention will be explained through the description and drawings. These advantages and / or effects are provided by several embodiments and features described in the description and drawings, but not necessarily all of them are provided in order to obtain one or more of the same features. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a diagram showing a configuration example of a vehicle 100 according to an embodiment of the present invention.

[0017] Figure 2 It is a diagram showing a configuration example of a display control device 200 according to an embodiment of the present invention.

[0018] Figure 3 2 is a diagram showing an example of a synthesized image displayed on the display unit 7 .

[0019] Figure 4 This is a diagram for explaining the relationship between projection transformation and viewpoint transformation.

[0020] Figure 5 This is a diagram showing an example of a viewpoint-converted image when viewpoint conversion is performed.

[0021] Figure 6 This is a diagram for explaining the process of creating a display image by synthesizing images captured by three cameras.

[0022] Figure 7is used to illustrate the generation of Figure 1 A diagram of the process when a virtual viewpoint image is provided that looks like an overhead view of the vehicle.

[0023] Figure 8 It is a diagram for explaining a state in which the following vehicle 11 disappears from the display image.

[0024] Fig. 9 A diagram for explaining a state in which one object is displayed doubly on a display image.

[0025] Fig.10 This is a diagram for explaining a method of setting the screen projection surface 13A by the display control device 200 according to the present embodiment.

[0026] Fig.11 It is a diagram for explaining the application area (screen projection application area 16) of the screen projection surface 13A.

[0027] Fig.12 It is a diagram for explaining the height of the screen projection surface 13A.

[0028] Fig.13 This is a diagram for explaining blending processing at the boundary portion between the screen projection surface 13A and the projection surface 10 .

[0029] Fig.14 It is a diagram for explaining a method of dynamically controlling the boundary areas 17a and 17b.

[0030] Fig.15 This is a diagram for explaining control of the width of the boundary region according to the position of the following vehicle 11 .

[0031] Fig.16 This is a diagram for explaining the control of the mixing ratio in the boundary area.

[0032] Fig.17 This is a diagram for explaining a method of controlling the position of the virtual viewpoint 20 according to the presence or absence of the following vehicle 11 .

[0033] Fig.18 This is a diagram for explaining a method of controlling the position of the virtual viewpoint 20 according to the vehicle speed of the host vehicle.

[0034] Fig.19 It is a diagram for explaining a method of controlling the inclination of the projection surface 10 .

[0035] Fig. 20 It is a diagram for explaining a method of controlling the inclination of the projection surface 10 .

[0036] Fig.21 This is a diagram for explaining a control method for including a road surface in the projection plane 10 .

[0037] Fig. 22 This is a diagram for explaining a control method for adding a gentle slope surface 18 to the lower portion of the projection surface 10 .

[0038] Fig.23 This is a diagram showing an example of a bowl-shaped projection surface 10 .

[0039] Fig.24 It is a diagram showing an example of the curved cylindrical projection plane 10 and the screen projection plane 13A. DETAILED DESCRIPTION

[0040] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that in this specification and the drawings, components having substantially the same functions are denoted by the same reference numerals and repeated descriptions are omitted.

[0041] (Implementation Method)

[0042] Figure 1 1 is a diagram showing a configuration example of a vehicle 100 according to an embodiment of the present invention. The vehicle 100 includes a right camera 1 , a left camera 2 , a rear camera 3 , and a display control device 200 .

[0043] Each of the right camera 1 , the left camera 2 , and the rear camera 3 is an imaging unit that includes an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) and captures an image of the outside of the vehicle 100 .

[0044] The right camera 1 is, for example, installed at a position close to the right side with respect to the traveling direction of the vehicle 100, usually, at a position where a right rearview mirror is installed, facing obliquely right and slightly downward, and inputs image information representing the content of an image (right image) obtained by capturing a visual field 1a to the display control device 200. The visual field 1a is a fixed area behind the right side of the vehicle 100 (obliquely right).

[0045] The left camera 2 is, for example, installed at a position close to the left side with respect to the traveling direction of the vehicle 100, usually, at a position where a left rearview mirror is provided, facing obliquely left and slightly downward, and inputs image information representing the content of an image (left image) obtained by capturing a visual field 2a to the display control device 200. The visual field 2a is a fixed area behind the left side of the vehicle 100 (obliquely left).

[0046] The rear camera 3 is installed slightly downwardly, for example, at the rear side with respect to the traveling direction of the vehicle 100, and inputs image information showing the contents of an image (rear image) obtained by photographing a visual field 3a to the display control device 200. The visual field 3a is a fixed area behind the vehicle 100.

[0047] The display control device 200 is a device that displays images cut out from the right image, the left image, and the rear image side by side on the display unit 7 .

[0048] Next, refer to Figure 2 A configuration example of the display control device 200 will be described. Figure 2 It is a diagram showing a configuration example of a display control device 200 according to an embodiment of the present invention.

[0049] The display control device 200 is a control unit that is composed of, for example, one or a plurality of ECUs (Electronic Control Units) and performs various display control processes in the display unit 7 .

[0050] The display control device 200 is connected to the right camera 1 , the left camera 2 , the rear camera 3 , the operation unit 4 , the navigation device 5 , and the display unit 7 , and is connected to various sensors 8 via a CAN (Controller Area Network) 9 as an in-vehicle network.

[0051] The operation unit 4 is a user interface (turn signal lever, switches, etc.) that receives input operations from the occupant of the vehicle 100. The display unit 7 is a liquid crystal display that displays a composite image. The various sensors 8 are, for example, a vehicle speed sensor, a steering angle sensor, a shift position sensor, and the like.

[0052] The display control device 200 includes an image processing unit 61 , a vehicle detection unit 62 , a situation determination unit 63 , an image control unit 64 , and a CAN communication unit 65 .

[0053] The image processing unit 61 combines the images captured by the right camera 1 , the left camera 2 , and the rear camera 3 , and displays the combined image on the display unit 7 .

[0054] The vehicle detection unit 62 detects subsequent vehicles, etc., behind the vehicle 100 based on the images captured by the right camera 1, the left camera 2 and the rear camera 3 respectively, detects the positions of the subsequent vehicles, and outputs subsequent vehicle information indicating whether the subsequent vehicle is detected and the position of the subsequent vehicle.

[0055] The situation determination unit 63 inputs the subsequent vehicle information output from the vehicle detection unit 62 and the operation information indicating that an input operation has been performed on the operation unit 4 , and determines the display mode on the display unit 7 based on the information.

[0056] The image control unit 64 controls parameters for image conversion and synthesis according to the display mode determined by the situation determination unit 63. In the image processing unit 61, the images captured by the right camera 1, the left camera 2 and the rear camera 3 are synthesized according to the parameters controlled by the image control unit 64.

[0057] Next, refer to Figure 3 to Figure 9 , explaining the background of creating an embodiment of the present invention and previous problems in synthesizing captured images, etc.

[0058] <Mixed>

[0059] Figure 3 FIG. 2 is a diagram showing an example of a composite image displayed on the display unit 7. Figure 3 The left side of the figure schematically shows how to synthesize the images captured by the right camera 1, the left camera 2 and the rear camera 3 respectively. Figure 3 The right side diagram in shows the synthesized image displayed on the display unit 7.

[0060] If three cameras are used to capture the scenery behind the vehicle 100, the images captured by each camera are images captured from different viewpoints. Therefore, even if the object captured by these cameras is the same object, the shape of the image of the object differs depending on the viewpoint. Therefore, when these images are synthesized, the images may be discontinuous or distorted at the seams.

[0061] For example, the fence 6 disposed at a position far from the rear of the vehicle 100 extends linearly in a direction parallel to the lateral width direction of the vehicle 100 in real space. However, when the captured right image and the rear image are synthesized, if the heights of the image of the fence 6 at the joint position in the synthesized image are different, the image of the fence will not be aligned, resulting in distortion.

[0062] As a countermeasure, for example, when a display image is produced in a manner that the rear image is sandwiched between the right image and the left image, a boundary area is set at the boundary portion (joint portion) between the right image and the rear image, and a boundary area is also set at the joint portion between the left image and the rear image. And, by mixing the pixel values ​​of the two images (the right image and the rear image, or the left image and the rear image) in these boundary areas, a display image is formed. This is generally called "mixing".

[0063] In this mixing, the closer one image is to the other image (for example, from Figure 3 The closer the right image is to the rear image), the lower the blend ratio of the pixel value of one image relative to the pixel value of the other image is, so that the unnaturalness of the boundary area can be made less noticeable.

[0064] <Projection Transformation and Viewpoint Transformation>

[0065] Figure 4 This is a diagram used to explain the relationship between projection transformation and viewpoint transformation. When shooting with a camera, the optical information in the three-dimensional space observed from the actual viewpoint (camera position) is transformed into the information of a two-dimensional image (photographed image). In addition, projecting a two-dimensional image onto a surface (projection surface) in three-dimensional space is called projection. If the projection surface is tilted relative to the actual viewpoint, the image on the projection surface will be stretched.

[0066] The process of establishing a correspondence between the pixels on the captured image and the pixels on the projection surface is called mapping. By mapping, the captured image can be transformed into an image observed from another position. This transformation process is called viewpoint transformation or viewpoint movement, and the image obtained by this transformation process is called a viewpoint transformation image. The process of establishing a correspondence between the pixels on the projection surface and the pixels on the viewpoint transformation image can also be called mapping.

[0067] The portion of the image on the projection surface that is closer to the viewpoint due to the viewpoint transformation is magnified on the viewpoint transformation image, and the portion that is farther from the viewpoint due to the viewpoint transformation is reduced on the viewpoint transformation image. In other words, the image is also deformed by the viewpoint transformation.

[0068] It should be noted that when performing projection and viewpoint transformation, it is also possible to perform mapping to directly establish a correspondence between pixels on the captured image and pixels on the viewpoint transformation image, rather than performing projection mapping and then viewpoint transformation mapping. The table used to establish the correspondence in this case is called a mapping table. In other words, projection and viewpoint transformation are reduced to the process of deforming an image using a mapping table.

[0069] <The Reality and Virtuality of Virtual Viewpoint>

[0070] Figure 5 This is a diagram showing an example of a viewpoint conversion image when the viewpoint conversion is performed. Figure 5The front side 51 of the cube 50 shown in the figure is obtained in which the front side 51 is visible but the side side 52 is invisible. When the image observed from the actual viewpoint is transformed into the image observed from the virtual viewpoint on the right as shown in the figure, that is, when the image is transformed into a viewpoint-transformed image as if the cube 50 is virtually photographed from an oblique direction, the front side 51 is deformed into the same image as the image observed when observed from the virtual viewpoint on the right as shown in the figure, that is, it is deformed into an image in which the left side that becomes farther away due to the viewpoint movement is reduced. However, the side side 52 of the cube 50 that is not reflected when observed from the actual viewpoint is not reflected in the viewpoint-transformed image.

[0071] That is, viewpoint conversion is a virtual viewpoint movement, and therefore, in a viewpoint-converted image, it is not possible to obtain an image that is the same as an image captured by actually moving the viewpoint. After all, projection and viewpoint conversion are just deformations of a captured image.

[0072] <Generation of Virtual Viewpoint Image and Generation of Synthesized Image by Viewpoint Conversion>

[0073] Figure 6 This is a diagram for explaining the process of creating a display image by synthesizing images captured by three cameras.

[0074] In the process of creating the display image, first, each of the right image, the rear image, and the left image is converted into a virtual viewpoint image with the position of the display unit 7 as a virtual viewpoint by viewpoint conversion.

[0075] Next, these virtual viewpoint images are synthesized to create a display image. Through this process, the images captured by the three cameras are synthesized into an image that looks like an image observed from a virtual viewpoint, but as mentioned above, there are still differences in the images due to the different positions of the actual viewpoints, so discontinuities may sometimes appear at the seams of the images.

[0076] <Upward Movement of Virtual Viewpoint and Overlay Synthesis of Host Vehicle Model Image on Virtual Viewpoint Image>

[0077] Figure 7 is used to illustrate the generation of Figure 1 In the generation process, first, as in Figure 7 As shown on the left side of , the virtual viewpoint 20 is moved to the front and top of the host vehicle. Thus, three virtual viewpoint images that look like an image of looking down on the host vehicle are obtained.

[0078] Then, these virtual viewpoint images are synthesized to generate Figure 7As shown in the figure on the right side of FIG. 1 , a model image of the vehicle (for example, located at Figure 7 The trapezoid at the bottom of the right side of the figure is an image obtained by simulating the roof of the vehicle 100).

[0079] In an actual driving scenario of a common vehicle equipped with a rearview mirror, the driver observes the rear of the vehicle 100 and the rear of the vehicle 100 reflected in the rearview mirror, and grasps the position of the following vehicle based on the rear of the vehicle (e.g., the upper surface of the trunk, etc.) reflected in the rearview mirror. Figure 7 As shown, in a vehicle that does not have a mirror but uses a camera, the rear image of the vehicle is virtually superimposed when synthesizing three virtual viewpoint images ( Figure 7 The image in the figure is a simulated image of the roof), which can be used to construct a benchmark for grasping the positional relationship with the following vehicle.

[0080] <Situation where the following vehicle 11 disappears from the displayed image>

[0081] Figure 8 This is a diagram for explaining a state where the following vehicle 11 disappears from the display image. When the projection plane 10 is set to be far from the vehicle 100 and three virtual viewpoint images are projected onto the projection plane 10 in a horizontally continuous manner, a portion of each of the three camera images is cut out.

[0082] It should be noted that the projection surface 10 is not limited to a plane, and for example, it can be a bowl-shaped surface described later, or a curved surface. In addition, the projection surface 10 can be a cylindrical surface surrounding the entire circumference of the vehicle 100, or a conical surface surrounding the entire circumference of the vehicle 100.

[0083] like Figure 1 As shown in the visual field 1a, the visual field 2a, and the visual field 3a of the three camera images, the visual fields of the three camera images have overlapping parts. If the three camera images are synthesized as they are to produce a display image, the objects reflected in the overlapping visual fields are displayed multiple times on the display image, which may cause the driver to be confused when grasping the position of the object. Therefore, in order to avoid duplication on the display image, a cut-out process is performed to produce an image obtained by cutting out a part of the visual field of each camera image. Figure 8 The cut-out range 1a1 is from Figure 1 The right image of the field of view 1a is cropped out. Figure 8 The cut-out range 2a1 is from Figure 1 The cropped portion of the left image of the field of view 2a. Figure 8 The cut-out range 3a1 is from Figure 1 The rear image of the field of view 3a is cut out.

[0084] Here, when the following vehicle 11 that is located in front of the projection plane 10 is located in a portion that is not cut out from the camera image (blind spot area 12 that is not displayed in the background image), the blind spot area 12 becomes a blind spot for the driver who is looking at the displayed image, causing the following vehicle 11 to disappear from the displayed image. In other words, if the projection plane 10 is set to be located farther than the position of the following vehicle 11, there is a possibility that the following vehicle 11 will disappear from the synthesized image.

[0085] <When an object is displayed twice>

[0086] Fig. 9 The following is a diagram for explaining a state where one object is double-displayed on a display image. A problem when the projection plane 10 is set to be located near the vehicle 100 will be explained.

[0087] exist Fig. 9 In the example of FIG. 1 , a right image, a left image, and a rear image are cut out on a projection plane 10 set near a vehicle 100 so that the three rear images are continuous in the horizontal direction. The width ratio of each of the cut-out range 1a1, the cut-out range 3a1, and the cut-out range 2a1 is Figure 8 In this case, the clipping range 1a1, the clipping range 3a1, and the clipping range 2a1 do not overlap at the position of the projection plane 10, but have overlapping portions at a position farther than the projection plane 10. Therefore, if an object 13 that exists farther than the projection plane 10 is located in the overlapping portion, the object 13 is displayed in each of the two clipping ranges.

[0088] exist Fig. 9 In the example of FIG. 1 , a portion of the cutout range 1a1 overlaps a portion of the cutout range 3a1 at a position farther than the projection plane 10, so that a common object 13 is displayed in the right image and the rear image. That is, one object 13 is displayed as two objects 13a and 13b on the synthesized display image. When the object 13 is a two-wheeled vehicle, two two-wheeled vehicles are displayed on the projection plane 10 set to be near the vehicle 100. Therefore, it is difficult for the driver to correctly grasp the actual position of the two-wheeled vehicle.

[0089] In addition, for example, when the object 13 is a two-wheeled vehicle, if the two-wheeled vehicle existing in the overlapping portion of the cut-out ranges 3a1 and 1a1 moves to a position only included in the cut-out range 1a1, only the two-wheeled vehicle existing in the cut-out range 1a1 is displayed on the display unit 7. At this time, since the driver has already recognized that there are two two-wheeled vehicles, in this case, the driver will have the illusion that one of the motorcycles has disappeared from the rear area of ​​the vehicle 100.

[0090] Next, refer to Fig.10etc. describe embodiments for solving the above-mentioned problems.

[0091] (Example 1)

[0092] <Example 1-1: Setting of Screen Projection Surface 13A>

[0093] Fig.10 This is a diagram for explaining a method of setting the screen projection surface 13A by the display control device 200 according to the present embodiment.

[0094] When the following vehicle 11 is detected, the display control device 200 sets a screen-shaped projection surface (screen projection surface 13A) of a size that can surround the following vehicle 11 to be located at the front part of the following vehicle 11. In addition, the display control device 200 projects the area of ​​the image of the following vehicle 11 in the captured image onto the screen projection surface 13A to generate a screen image. At this time, if there is no captured image containing the entire image of the following vehicle 11, two captured images may be projected onto the screen projection surface 13A, thereby synthesizing the entire image of the following vehicle 11 on the screen projection surface 13A. If there is a captured image containing the entire image of the following vehicle 11, only one captured image may be projected onto the screen projection surface 13A. If only one captured image is projected onto the screen projection surface 13A, the discontinuity caused by the synthesis can be avoided, and therefore, a more preferred screen image can be generated. The setting area of ​​the screen projection surface 13A will be described later.

[0095] Furthermore, the display control device 200 sets the projection plane 10 to be located farther from the screen projection plane 13A, and projects a plurality of captured images onto the projection plane 10 to generate a background image (eg, an image including the object 13 ).

[0096] Furthermore, the display control device 200 generates a display image by embedding the screen image on the background image.

[0097] It should be noted that the screen projection surface 13A and the projection surface 10 may be a flat rectangular surface or a curved surface such as a spherical surface.

[0098] As a result, the image of the following vehicle 11 is pasted onto the continuous background image, thereby solving the problem of disappearance of the following vehicle 11 and double display of the image of the following vehicle 11. That is, the background image is cut and synthesized in such a way that the image is continuous on the distant projection surface 10 to avoid double display of the image, and the image of the following vehicle 11 is captured in the screen image and overlapped on the background image, so that the disappearance of the following vehicle 11 does not occur.

[0099] It should be noted that, when the following vehicle 11 is not detected, the display control device 200 does not set the screen projection surface 13A or perform projection onto the screen projection surface 13A, but sets the background image as the display image.

[0100] <Example 1-2: Setting Area of ​​Screen Projection Surface 13A>

[0101] Fig.11 It is a diagram for explaining the application area (screen projection application area 16) of the screen projection surface 13A.

[0102] Fig.11 The figure shows the center end a of the cropped range 1a1 of the right camera's image, the center end b of the cropped range 2a1 of the left camera's image, and the right end c1 and the left end c2 of the cropped range 3a1 of the rear camera's image when multiple captured images of the vehicle's periphery are projected onto the projection surface to generate a background image. The cropping performed when multiple captured images of the vehicle's periphery are projected onto the projection surface is a cropping that limits the range of projection of the multiple captured images of the vehicle's periphery so that the background image becomes a continuous image. Specifically, the cropping is performed in a manner that excludes the portion that overlaps with the adjacent image from the projection range. It should be noted that in Fig.11 In the example, the images are trimmed on the premise of blending to leave a narrow overlapping portion as room for blending. The screen projection application area 16 is set to include an area that is included in the field of view 3a of the rear camera but will not be displayed in the background image when trimming is performed in such a way that the background image becomes a continuous image.

[0103] Please refer to the example of cutting the background image so that it becomes a continuous image. Figure 8 .like Figure 8 As shown in FIG. 1 , when the range of projection of a plurality of captured images from the periphery of the vehicle is limited to cropping so that the background image becomes a continuous image, the blind spot area 12 not shown in the background image is expanded, so that if the two-wheeled vehicle 11 enters the blind spot area 12, it disappears from the background image. Fig.11 Similarly, in the example of FIG. 1 , the area between the end a of the clipping range 1a1 and the end c1 of the clipping range 3a1 becomes a blind spot area 12 that is not displayed in the background image. However, Figure 8 Compared with the blind spot area 12, its range is greatly reduced and is limited to the position close to the vehicle 100. The image of the following vehicle 11 entering the blind spot area 12 may disappear from the background image, but this is a blind spot that is inevitably generated due to the limitation of the camera configuration. It can be said that it is an area that the following vehicle will not enter when the normal vehicle-to-vehicle distance is maintained. Therefore, Fig.11The blind spot area 12 is not included in the screen projection application area 16 .

[0104] As described above, the display control device 200 sets the screen projection application area 16 to include the area that is included in the field of view range 3a of the rear camera but is not displayed in the background image when the background image is cut out so as to be a continuous image, and the display control device 200 sets the screen projection surface 13A to be located at the front end portion of the set screen projection application area 16. The reason for setting the width of the screen projection application area 16 to be larger than the area that is included in the field of view range 3a of the rear camera but is not displayed in the background image when the background image is cut out so as to be a continuous image is that, even if a part of the body of the following vehicle enters the area that is not displayed in the background image, the screen projection is applied to avoid that a part of the body of the following vehicle becomes invisible and hinders the driver's understanding of the situation.

[0105] It should be noted that the area in the background image that reflects only the image captured by one camera (for example Fig.11 In the cut-out range 1a1), even if the background image is cut out in such a way that it becomes a continuous image, the following vehicle 11 will not disappear, and the image will not be displayed twice. Therefore, it is not necessary to set the screen projection application area 16 for this area. Fig.11 In the embodiment, the area directly behind the vehicle 100 sandwiched between the two screen projection application areas 16 is also excluded from the screen projection application area 16, because the area is always displayed in the background image and no double image display occurs.

[0106] In addition, even if the screen projection application area 16 is applied to a position closer to the projection surface 10 or farther than the projection surface 10, there is no significant difference compared to the case where the screen projection application area 16 is not applied, so there is no need to set the screen projection application area 16 for these positions.

[0107] <Example 1-3: Setting the Height of Screen Projection Surface 13A>

[0108] Fig.12 It is a diagram for explaining the height of the screen projection surface 13A.

[0109] The height of the screen projection surface 13A is set so as to include a region from the upper end to the lower end of the following vehicle 11 when the following vehicle 11 is viewed from the virtual viewpoint 20 .

[0110] In the area above the upper end of the screen projection surface 13A, it is not necessary to display the image of the following vehicle 11, so the background image of the projection surface 10 is selected in this area on the display image. As a result, the background image is displayed without being blocked by the screen projection surface 13A in the upper part of the screen projection surface 13A. The background image is also selected in the lower part of the screen projection surface 13A, and the part where the road surface is photographed is displayed without being blocked by the screen projection surface 13A.

[0111] According to the first embodiment, by setting the screen projection surface 13A, the image of the following vehicle 11 is projected onto the screen projection surface 13A, and therefore, the disappearance of the following vehicle 11 can be prevented.

[0112] Even when the screen projection surface 13A is set, at least in the area of ​​the entire projection surface 10 that is not blocked by the screen projection surface 13A, the background image is not displayed in duplicate but appears continuous, so the driver does not feel a sense of discomfort with the background image.

[0113] Furthermore, even if the image of a distant object is displayed in duplicate near the image of the following vehicle 11 , the area where the image is displayed in duplicate is a relatively narrow range, and therefore does not become a visual obstacle.

[0114] <Example 1-4: Mixing of the Boundary Part between Screen Projection Surface 13A and Projection Surface 10>

[0115] Fig.13 This is a diagram for explaining blending processing at the boundary portion between the screen projection surface 13A and the projection surface 10 .

[0116] When the screen projection surface 13A is set, the screen projection image of the screen projection surface 13A and the background image of the projection surface 10 are discontinuous. Therefore, the display control device 200 can set a boundary area 17c between the screen projection image and the background image at the edge of the screen projection image of the screen projection surface 13A, and mix the screen projection image in the boundary area 17c in such a way that the transmittance of the screen projection image continuously increases as it approaches the outside, and the screen projection image is gradually integrated into the background image.

[0117] In the generation of the background image, the boundary parts of different images may also become discontinuous, so a boundary area may be set and mixed. Fig.14The boundary area 17a in the upper figure is the boundary area between the right image and the rear image, and the boundary area 17b is the boundary area between the left image and the rear image. Boundary areas 17a and 17b are set, and the pixel values ​​of the composite image are calculated in the boundary areas by weighted arithmetic in which the weight is smaller as the image is closer to the adjacent image, so that the pixels are mixed in a way that the adjacent images are merged with each other, thereby making the boundary line (boundary) between the background image and the screen image inconspicuous. In addition, when multiple camera images are used in the generation of the screen projection image, boundary areas can also be set at the boundary parts of the images and mixed. In this way, by mixing the boundary parts of the images that sometimes become discontinuous, it is possible to avoid the boundary parts becoming unnatural and attracting the attention of the driver.

[0118] (Example 2)

[0119] <Example 2-1: Dynamic Control of Boundary Area>

[0120] In the second embodiment, an embodiment is described in which the problem of the disappearance of the following vehicle 11 is solved by dynamically controlling the boundary portion (boundary areas 17a, 17b) between the image on the screen projection surface 13A and the image on the projection surface 10 instead of setting the screen projection surface 13A.

[0121] Fig.14 This is a diagram for explaining a method of dynamically controlling the boundary regions 17a and 17b. For example, when the following vehicle 11 enters the region near the boundary between the rear image and the right image, the following vehicle 11 disappears.

[0122] The display control device 200 is configured so that the boundary regions 17a and 17b include regions where the following vehicle 11 may disappear. Fig.14 When the image of the following vehicle 11 is located near the boundary area 17a as shown in the lower figure in FIG, the width of the boundary area 17a is expanded so that the image of the following vehicle 11 is included in the boundary area 17a. In this boundary area, the pixels of the rear image and the pixels of the right image are mixed to generate the pixels of the display image. Therefore, if the image of the following vehicle 11 is included in at least one of the rear image and the right image, the image of the following vehicle 11 is displayed in the display image. In this way, it is possible to prevent the following vehicle 11 from disappearing near the boundary between the rear image and the right image.

[0123] exist Fig.14In the embodiment, the right end of the boundary area 17a is fixed, and the left end of the boundary area 17a is moved to the left, so that the image of the following vehicle 11 is included in the boundary area. However, the right end of the boundary area 17a may also be controlled to move according to the position of the image of the following vehicle 11. For example, the right end of the boundary area 17a may also be moved to the left, so that the image of the following vehicle 11 is located in the center of the boundary area 17a, so as to move the position of the entire boundary area. Fig.14 In the figure, an example is shown in which the image of the following vehicle 11 is located near the boundary between the rear image and the right image. When the image of the following vehicle 11 is located near the boundary between the rear image and the left image, the boundary area 17b is expanded to include the image of the following vehicle 11, and the pixels of the rear image and the pixels of the left image are mixed in the boundary area to generate pixels of the display image. In this way, the display control device 200 includes the image of the following vehicle 11 in the boundary areas 17a and 17b, and mixes the boundary areas 17a and 17b, so that the unnaturalness of the boundary areas 17a and 17b is not noticeable, and the disappearance of the following vehicle 11 near the boundary between the rear image and the left and right images can be avoided. As the area near the boundary between the rear image and the left and right images, it is also possible to set the same as using Fig.11 The area to which the screen projection surface 13A is applied (screen projection application area 16 ) described above is the same area.

[0124] <Example 2-2: Control of the Width of Boundary Regions 17a and 17b Depending on the Presence or Absence of Following Vehicle 11>

[0125] The display control device 200 may also be configured to not expand the width of the boundary regions 17a and 17b when the following vehicle 11 is not detected, but to expand the width of the boundary regions when the following vehicle 11 is detected. In addition, the display control device 200 may detect the width of the image of the following vehicle 11, and determine whether to expand the width of the boundary regions 17a and 17b based on the width of the image of the following vehicle 11.

[0126] For example, when the following vehicle 11 is detected, the display control device 200 determines the lateral width of the following vehicle 11 and the range reflecting the following vehicle 11. When the following vehicle 11 is narrower than the prescribed width, the display control device 200 determines that the following vehicle 11 is a two-wheeled vehicle, and when the following vehicle 11 is wider than the prescribed width, the display control device 200 determines that the following vehicle 11 is a four-wheeled vehicle.

[0127] Since the width of a four-wheeled vehicle is wider than that of a two-wheeled vehicle, when the following vehicle 11 is a four-wheeled vehicle, the entire image of the four-wheeled vehicle will not disappear from the display image, and even if a part of the four-wheeled vehicle disappears from the display image, it can be considered that the following vehicle 11 will not be missed. Therefore, when the lateral width of the following vehicle 11 is wide and it is determined to be a four-wheeled vehicle, the display control device 200 can also control the width of the boundary area near the four-wheeled vehicle to remain unchanged.

[0128] Since the width of a two-wheeled vehicle is narrower than that of a four-wheeled vehicle, the entire image of the two-wheeled vehicle is likely to disappear from the display image when the following vehicle 11 is a two-wheeled vehicle. Therefore, when the lateral width of the following vehicle 11 is narrow and it is determined to be a two-wheeled vehicle, the display control device 200 controls the width of the boundary area near the two-wheeled vehicle to be increased.

[0129] In addition, the display control device 200 may evaluate the distance from the image of the following vehicle 11 on the image to the boundary between the rear image and the left and right images, and widen the boundary area only when the image of the following vehicle 11 is near the boundary. If the position of the image of the following vehicle 11 is far from the boundary, the subsequent processing of determining whether widening is necessary can be omitted, thereby reducing the load caused by determining whether widening of the boundary area is necessary.

[0130] It should be noted that the display control device 200 may also be configured to evaluate the distance and approach speed to the following vehicle 11, and implement the above-mentioned mixing in the boundary area near the following vehicle 11 only when the possibility of interference with the host vehicle 100 is detected. For example, when the inter-vehicle distance with the following vehicle 11 is larger than a specified value and the inter-vehicle distance has not decreased, it is determined that there is no possibility of interference between the following vehicle 11 and the host vehicle 100, and it is determined that there is no need to widen the boundary area. In this way, it is possible to prevent the image from being unclear (blurred) due to unnecessary double display of images in the boundary area.

[0131] <Example 2-3: Control of the width of the boundary area according to the position of the following vehicle 11>

[0132] Fig.15 This is a diagram for explaining control of the width of the boundary region according to the position of the following vehicle 11 .

[0133] In the boundary area, the image is displayed in duplicate and the image is not clear, so it is preferable that the width of the boundary area is narrow. On the other hand, when the following vehicle 11 near the host vehicle moves in the lateral direction, the movement amount of the following vehicle 11 on the image is larger than that of the following vehicle 11 far from the host vehicle. At this time, if the width of the boundary area is not controlled to follow the lateral movement of the following vehicle 11, the image of the following vehicle 11 is likely to deviate from the boundary area and disappear from the displayed image.

[0134] Therefore, the display control device 200 can be controlled in the following manner: for example, when a narrower following vehicle 11 is located farther away from the vehicle, the width of the boundary area near the following vehicle 11 is narrowed, and as the following vehicle 11 approaches the vehicle, the width of the boundary area is continuously or phasedly expanded.

[0135] This can suppress image blurring due to double display of images, and prevent the image of the following vehicle 11 from deviating from the boundary area and disappearing from the display image.

[0136] Furthermore, even when the detection process of the following vehicle 11 is delayed, by increasing the width of the boundary region near the following vehicle 11 , it is possible to prevent the following vehicle 11 from disappearing from the boundary region.

[0137] <Example 2-4: Control of Mixing Ratio in Boundary Region>

[0138] Fig.16 This is a diagram for explaining the control of the mixing ratio in the boundary area.

[0139] The display control device 200 may also be controlled in the following manner: the center position of the image of the following vehicle 11 or the position of the face (or head) of the driver of the following vehicle 11 is detected, and the intersection point where the mixing ratio of the image on the right side and the image on the left side of the boundary area is 1:1 is made consistent with the center position of the following vehicle 11 or the position of the face (or head) of the driver of the following vehicle 11. The technology of face recognition for detecting a facial image from an image is well known, so a detailed description is omitted. Face recognition is performed by pattern matching that evaluates the degree of matching with a dictionary image, or it may be an algorithm that evaluates the degree of matching with multiple dictionary images and recognizes it as a human face if it matches any one of them. If a helmet is worn, the driver's face will not be visible, but by adding a dictionary image that matches the image above the shoulders of the driver wearing the helmet, the face (or head) of the driver wearing the helmet can also be detected and the position can be determined.

[0140] The blending ratio in the boundary area varies depending on the position of the image relative to the boundary area. For example, when the image of the two-wheeled vehicle is located at a portion with a low blending ratio (e.g., close to 0%) near the end of the boundary area, the image of the two-wheeled vehicle is blended very thinly, and an image obtained by blending the background image without the two-wheeled vehicle more densely becomes the displayed image. As a result, the two-wheeled vehicle is displayed very thinly, and the driver may miss the two-wheeled vehicle.

[0141] At this time, by setting the mixing ratio to 1:1 at the center position of the image of the following vehicle 11 or the position of the face of the driver of the following vehicle 11, that is, the position where the driver looks when visually recognizing the two-wheeled vehicle, the image of the two-wheeled vehicle is displayed at about half the density, thereby reducing the risk of missing the following vehicle 11. In addition, since the face of the driver of the two-wheeled vehicle can be visually recognized, it is also easy to predict the next action of the two-wheeled vehicle.

[0142] <Effects of Example 2>

[0143] As described above, when a two-wheeled vehicle enters a blind spot formed between the field of view of the left and right camera images on the displayed image and the field of view of the camera image of the rear camera, and the image of the two-wheeled vehicle is likely to disappear from the displayed image, the width of the boundary area where the left and right camera images are mixed with the camera image of the rear camera for display is expanded to include the image of the two-wheeled vehicle, thereby preventing the image of the two-wheeled vehicle from disappearing.

[0144] In addition, by controlling the boundary area, the image can be displayed using only one projection surface 10. Therefore, compared with the case of using both the screen projection surface 13A and the projection surface 10, the image processing amount is reduced, and the display delay time from the detection of the following vehicle 11 to the display image being displayed on the display unit 7 can be shortened.

[0145] Furthermore, since the display delay time is shortened, the time from when the surrounding conditions of the host vehicle change to when the driving operation corresponding to the changed surrounding conditions is performed can be shortened.

[0146] (Example 3)

[0147] Next, refer to Fig.17 etc. will be described below. It should be noted that the third embodiment can be combined with the first embodiment or the second embodiment.

[0148] <Example 3-1: Position Control of Virtual Viewpoint 20 Depending on the Presence or Absence of Following Vehicle 11>

[0149] Fig.17 This is a diagram for explaining a method of controlling the position of the virtual viewpoint 20 according to the presence or absence of the following vehicle 11 .

[0150] The display control device 200 is controlled in the following manner: when there is a following vehicle 11, the virtual viewpoint 20 is set at a higher position relative to the road surface; when there is no following vehicle 11, the virtual viewpoint 20 is set at a lower position relative to the road surface compared to the case where there is a following vehicle 11.

[0151] By setting the virtual viewpoint 20 at a higher position than the road surface, an image as if the vehicle is viewed from a higher position can be provided, so that the movements of the host vehicle and the following vehicle 11 on the road surface can be easily understood.

[0152] On the other hand, if the virtual viewpoint 20 is set at a higher position relative to the road surface, the movement of fixed objects such as road surface markings caused by the movement of the vehicle appears larger on the display screen, so this will attract the driver's attention. This situation can be explained as if the virtual viewpoint 20 is raised, the sight line guidance becomes higher. During driving, the driver must look forward, so it is best to avoid unnecessary attention. When there is no following vehicle 11, by setting the virtual viewpoint 20 at a lower position relative to the road surface, the movement of the road surface on the display unit 7 becomes less, so it is possible to prevent unnecessary attention.

[0153] <Example 3-2: Position Control of Virtual Viewpoint 20 According to Vehicle Speed>

[0154] Fig.18 This is a diagram for explaining a method of controlling the position of the virtual viewpoint 20 according to the vehicle speed of the host vehicle.

[0155] The display control device 200 may be configured to set the virtual viewpoint 20 at a higher position relative to the road surface or to set the virtual viewpoint 20 at a lower position according to the vehicle speed of the host vehicle.

[0156] When the speed of the host vehicle is low, the movement of the road surface on the screen becomes less, so it is difficult to guide the line of sight to the moving road surface. Therefore, when the speed of the host vehicle is below the specified speed, the display control device 200 sets the virtual viewpoint 20 at a higher position relative to the road surface, and when the speed of the host vehicle exceeds the specified speed, the virtual viewpoint 20 is set at a lower position relative to the road surface compared to the case where the speed of the host vehicle is below the specified speed.

[0157] Thus, it is possible to prevent the sight from being unnecessarily attracted to the moving road surface, and it is easy to grasp the actions of the vehicle and the following vehicle 11. This control corresponding to the speed of the vehicle can also be implemented in combination with the control corresponding to the presence or absence of the following vehicle 11. For example, the following control can be performed: when the speed of the vehicle is low and there is a following vehicle 11, the virtual viewpoint 20 is set at a higher position, and in other cases, the virtual viewpoint 20 is set at a lower position. The following control can also be performed: only when the speed of the vehicle is high and there is no following vehicle 11, the virtual viewpoint 20 is set at a lower position, and in other cases, the virtual viewpoint 20 is set at a higher position. It can be said that the former is a control that emphasizes suppressing the sight guidance, and the latter is a control that emphasizes easy monitoring of the rear, and any one of the controls can be adopted according to the design concept of the vehicle.

[0158] <Example 3-3: Tilt Control of Projection Surface 10 According to Movement of Virtual Viewpoint 20>

[0159] Fig.19 and Fig. 20 It is a diagram for explaining a method of controlling the inclination of the projection surface 10 .

[0160] like Fig.19 As shown in FIG. 1 , when the projection plane 10 extends vertically to the road surface, the image of the portion of the following vehicle 11 close to the road surface is projected onto the projection plane 10 below the road surface. If the image on the projection plane 10 is observed from a virtual viewpoint 20 set at a position higher than the actual camera position, the image of the following vehicle 11 appears to be crushed (distorted) in the vertical direction.

[0161] Therefore, if Fig. 20 As shown, when the virtual viewpoint 20 is set at a higher position by the control of Example 3-1 or Example 3-2, the display control device 200 sets the projection plane 10 to be at the position where the following vehicle 11 is assumed to be located, and makes the projection plane 10 inclined relative to the road surface. At this time, the display control device 200 sets the projection plane 10 to intersect perpendicularly with the virtual sight line 15 extending from the virtual viewpoint 20 to the position where the following vehicle 11 is assumed to be located.

[0162] It should be noted that the distance from the vehicle 100 to the projection surface 10 can be appropriately changed according to the speed of the vehicle 100. For example, when the vehicle 100 is traveling at high speed, the distance from the vehicle 100 to the projection surface 10 can be lengthened. The standard vehicle-to-vehicle distance is longer when the vehicle speed is high and shorter when the vehicle speed is low. Therefore, it is reasonable to change the distance from the vehicle 100 to the projection surface 10 according to the vehicle speed.

[0163] In order to facilitate the understanding of the movements of the host vehicle and the following vehicle 11 on the road surface, when providing an image as if the vehicles were viewed from a high position, the image distortion of the following vehicle 11 can be eliminated by tilting the projection plane 10 with respect to the road surface.

[0164] In addition, when the virtual viewpoint 20 is set at a lower position through the control of Example 3-1 or Example 3-2, the display control device 200 can also set the projection plane 10 to be at the estimated position of the subsequent vehicle 11 and set the projection plane 10 to be perpendicular to the road surface.

[0165] By setting the projection plane 10 to be perpendicular to the road surface, the movement of the road surface on the display unit 7 is reduced, thereby preventing unnecessary attention from being drawn.

[0166] In this way, the display control device 200 changes the inclination angle of the projection surface 10 according to the presence or absence of the following vehicle 11 or the speed of the host vehicle, thereby making it difficult to guide the driver's line of sight and reducing the distortion of the image projected on the projection surface.

[0167] <Example 3-4: Control of including the road surface in the projection plane 10>

[0168] Fig.21 This is a diagram for explaining a control method for including a road surface in the projection plane 10 .

[0169] like Fig.21 As shown, the display control device 200 may be configured to include the road surface in the projection plane 10 .

[0170] By including the road surface in the projection plane 10, the position of the road surface markings (speed limit markings on the road surface, etc.) does not change due to the projection, so the distortion (distortion) caused by the projection can be eliminated. In addition, by observing the projection plane 10 including the road surface from the virtual viewpoint 20 set at a higher position, the road surface markings become easier to see.

[0171] In addition, the display control device 200 can also be configured to include the image of the road surface in the projection plane 10 when the height of the virtual viewpoint 20 from the ground is set to be higher, and when the height of the virtual viewpoint 20 from the ground is set to be lower than this, the range of the road surface included in the projection plane 10 is smaller than the range of the road surface included in the projection plane 10 when the height of the virtual viewpoint 20 from the ground is set to be higher.

[0172] It should be noted that when the virtual viewpoint 20 is set at a lower position due to the absence of a following vehicle 11, even if the image is projected onto a projection surface 10 that does not include a road surface, a display image that is the same as what the driver actually sees can be obtained. Therefore, in this case, the road surface does not need to be added to the projection surface 10.

[0173] <Example 3-5: Control of adding a gentle slope surface 18 to the lower part of the projection surface 10>

[0174] Fig. 22 This is a diagram for explaining a control method for adding a gentle slope surface 18 to the lower portion of the projection surface 10 .

[0175] The display control device 200 may be configured to add a gentle slope surface 18 between the projection surface 10 and the road surface, and set the entire projection surface 10 and the gentle slope surface 18 as the projection surface.

[0176] The inclination of the gentle slope surface 18 relative to the road surface is smaller than the inclination of the projection surface 10 relative to the road surface. Fig. 22 2 shows a case where the gentle slope surface 18 is a flat surface, but the gentle slope surface 18 is not limited thereto, and may be a curved surface that smoothly connects the projection surface 10 and the road surface.

[0177] If the image of the road surface further away from the projection plane 10 is projected onto the projection plane 10 tilted in a vertical manner relative to the road surface, the image will be distorted and difficult to see. As a result, the image of the road surface closer to the projection plane 10 and the image of the road surface further away from the projection plane 10 are discontinuous, resulting in a greater sense of incongruity in appearance.

[0178] By adding the gentle slope 18 between the projection surface 10 and the road surface, the discontinuity between the road surface and the projection surface 10 is alleviated, thereby reducing the distortion of the image of the road surface located farther than the projection surface 10 .

[0179] <Example 3-6: Control of Setting the Position of the Gentle Slope Surface 18 in a Way Not to Impede Visibility of the Following Vehicle 11>

[0180] Depending on the position where the gentle slope 18 is set, it is possible to hinder the visual recognition of the following vehicle 11. For example, when there is a following vehicle 11 near the projection surface 10, the portion of the image of the following vehicle 11 projected on the gentle slope 18 will be distorted, and the straight line extending over the portion projected on the gentle slope 18 and the portion projected on the projection surface above it will appear to be bent. As a countermeasure, it is possible to consider limiting the height of the upper end of the gentle slope 18 to the range where the lower structure of the assumed following vehicle exists. The lower structure of the following vehicle refers to, for example, a portion below the bumper of a four-wheeled vehicle or a portion below the axle of a two-wheeled vehicle.

[0181] Therefore, the distorted portion is limited to the lower portion of the image of the following vehicle 11. Since the driver visually recognizes the following vehicle from the upper portion of the vehicle body, even if the lower portion of the image of the following vehicle 11 is distorted, visual recognition of the following vehicle 11 is not greatly hindered.

[0182] Therefore, the display control device 200 may also set the upper end of the gentle slope 18 to be located at the bumper or axle of the following vehicle 11 at the assumed position of the following vehicle 11 , and connect the gentle slope 18 to the projection surface 10 .

[0183] Thus, the bent portion (gentle slope 18 ) of the projection plane 10 is set below the following vehicle 11 , so that the image above the following vehicle 11 is not discontinuous, and visibility of the following vehicle 11 is improved.

[0184] In addition, for the bird's-eye view display, in order to maintain continuity with when the following vehicle 11 is detected, the road surface may be included in the projection plane 10 when the following vehicle 11 is not detected. However, since there is a need to reduce the visual guidance, when the road surface is included in the projection plane 10, the visual stimulation may be reduced by narrowing the range of the road surface included in the projection plane 10 or blurring the road surface portion.

[0185] (Effects of the Invention)

[0186] The effects of the present invention are described below in comparison with the prior art.

[0187] <Example 1>

[0188] For example, the following method is disclosed in Japanese Patent Publication No. 2014-531078, which is characterized by comprising the following steps: when a protruding object is detected in the vicinity of a vehicle, a projection surface is deformed in the vicinity of the protruding object in such a manner that the projection surface is raised over the entire width of the protruding object.

[0189] In the technology of Japanese Patent Table 2014-531078, if a specific part of the projection surface is greatly deformed, the inclination of the projection surface becomes larger around the deformed part, thereby causing the projected image to be greatly deformed, and its shape is distorted and becomes difficult to visually recognize. According to the above-mentioned embodiment 2, the image of the approaching follow-up vehicle is projected on the screen projection surface 13A, and the image of the background is projected on a distant projection surface independent of the screen projection surface 13A, and finally the images on the two projection surfaces are synthesized to obtain the display image, so that the image of the follow-up vehicle and the image of the background will not be distorted, and thus will not hinder visual recognition due to distortion. Even in the boundary area set at the edge of the screen projection surface 13A, where the image is double-displayed due to mixing, the effect of not being distorted and not hindering visual recognition can be obtained. In other words, there is no part that is distorted due to the deformation of the projection surface, thereby becoming difficult to visually recognize.

[0190] <Example 2>

[0191] Human vision has the ability to separate and individually recognize dual displayed images, such as an exhibit in a display case and one's own face reflected in the display case.

[0192] The second embodiment above is a method that actively utilizes the visual ability of the human being, and adds the image of the following vehicle to the boundary area where the image is double-displayed due to blending, so as to solve the problem that the image of the following vehicle disappears at the seam of the field of view of the two camera images. For example, even if the face of the driver of the following two-wheeled vehicle enters the boundary area to which blending is applied and the driver's face is blended with the background, if the image of the face is not deformed, it can be visually recognized, and it can also be expected that the face direction can be distinguished.

[0193] According to the method of Japanese Patent Publication No. 2014-531078, if, for example, the face part is greatly deformed, it is not possible to expect to be able to distinguish whether it is a face or the direction of the face. In other words, if the image is displayed in double, the human visual function will assist in recognition, while if deformation occurs, the visual function will not work and hinder recognition. Therefore, the method of the present application can achieve a more excellent effect.

[0194] <Example 2-4>

[0195] For example, according to the vehicle surrounding image generation device disclosed in Japanese Patent Laid-Open No. 2019-185381, when the direction of movement of the three-dimensional object is close to the direction of one overhead image, the range for weighted arithmetic averaging is moved to one side of the other overhead image, and when the direction of movement of the three-dimensional object is close to the direction of another overhead image, the range for weighted arithmetic averaging is moved to one side of the overhead image. However, the weighted distribution in the range for weighted arithmetic averaging is fixed and has nothing to do with the movement of the three-dimensional object. Therefore, according to the control of the range for weighted arithmetic averaging, it is possible that a part that is important in controlling the movement of the vehicle 100, such as the face of the driver of the following vehicle, is located at the end of the range for weighted arithmetic averaging and the weight becomes smaller, so that it is displayed very thin on the displayed image and becomes difficult to visually recognize.

[0196] In contrast, in the second embodiment, control is performed as follows: the center position of the image of the following vehicle 11 is detected, or, when the following vehicle 11 is a two-wheeled vehicle, the position of the face (or head) of the driver of the two-wheeled vehicle is detected, and at the center position of the image or the position of the face, the mixing ratio of the image on the right side and the image on the left side of the boundary area is set to, for example, 1:1. As a result, the image of the following vehicle 11 or the face of the driver is displayed at about half the density, which can reduce the risk of being missed, and when the mixing ratio is controlled based on the position of the driver's face, it is easy to predict the next action based on the direction of the driver's face.

[0197] Fig.23 This is a diagram showing an example of a bowl-shaped projection surface 10 . Fig.24 It is a diagram showing an example of a curved cylindrical projection plane 10 and a screen projection plane 13A. By using such a projection plane 10 or screen projection plane 13A, the distance from the vehicle 100 to the projection plane 10 or the screen projection plane 13A is constant at any position behind the vehicle 100, thereby reducing the distortion of the image of the following vehicle 11, etc.

[0198] In addition, for example, it can be understood that the modes described below also belong to the technical scope of the present invention.

[0199] (1) It comprises: a vehicle detection unit for detecting surrounding vehicles existing around the vehicle based on a plurality of captured images of the surroundings of the vehicle captured by a plurality of camera devices mounted on the vehicle; and an image processing unit for generating a display image displayed on a display unit mounted on the vehicle by synthesizing the plurality of captured images, wherein the image processing unit performs: a projection transformation for projecting the plurality of captured images onto a projection plane set to be located behind the vehicle; and a viewpoint transformation for transforming the captured images projected onto the projection plane into images observed from a virtual viewpoint located at a position different from that of the camera devices, and the image processing unit makes the height above the ground of the virtual viewpoint when the surrounding vehicles are not detected in a specified area lower than the height above the ground of the virtual viewpoint when the surrounding vehicles are detected in the specified area.

[0200] (2) When the running speed of the vehicle is equal to or less than a predetermined value, the image processing unit sets the height of the virtual viewpoint above the ground to be higher than the height above the ground when the running speed of the vehicle exceeds the predetermined value.

[0201] (3) The inclination of the projection plane with respect to the road surface when the height of the virtual viewpoint from the ground is low is set larger than the inclination of the projection plane with respect to the road surface when the height of the virtual viewpoint from the ground is high.

[0202] (4) When the height of the virtual viewpoint above the ground is set high, the image processing unit includes the road surface in the projection plane, and when the height of the virtual viewpoint above the ground is set low, the image processing unit does not include the road surface in the projection plane.

[0203] (5) When the height of the virtual viewpoint above the ground is set high, the image processing unit includes the road surface in the projection plane. When the height of the virtual viewpoint above the ground is set low, the image processing unit makes the range of the road surface included in the projection plane smaller than the range of the road surface included in the projection plane when the height of the virtual viewpoint above the ground is set higher.

[0204] (6) When a position on the projection plane corresponding to a predetermined height above the ground is set as a reference point, an angle formed by a straight line connecting the reference point and the virtual viewpoint and a straight line passing through the reference point and perpendicular to the projection plane is less than a predetermined value.

[0205] (7) The inclination of the projection plane with respect to the road surface at a portion below the reference point in the projection plane is smaller than the inclination of the projection plane with respect to the road surface at a portion above the reference point in the projection plane.

[0206] (8) The image processing unit generates a background image of the vehicle by projecting a plurality of the captured images onto a first projection surface set to be located behind the vehicle. When the surrounding vehicles are detected within a specified area, the second projection surface is set to be located at a position where the surrounding vehicles are detected, and a surrounding vehicle image as an image of the surrounding vehicles is generated by projecting at least one of the plurality of images onto the second projection surface, and the surrounding vehicle image is overlapped with the background image.

[0207] (9) The image processing unit generates a rear image of the vehicle by projecting the plurality of captured images onto a projection surface set to be located behind the vehicle, and sets a boundary area in the rear image in which pixel values ​​of two adjacent captured images among the plurality of captured images are mixed. When the surrounding vehicles are detected within a specified area, at least one of a mixing ratio of the pixel values ​​and a width of the boundary area is changed according to at least one of a position and a width of the surrounding vehicles.

[0208] (10) The specified area is set based on the following area: when a plurality of captured images of the periphery of the vehicle are projected onto a first projection surface set to be located behind the vehicle, if the range of projection of the plurality of captured images of the periphery of the vehicle is limited in such a manner that the background image becomes a continuous image, the background image will not be displayed in the area of ​​the background image.

[0209] (11) A vehicle includes the above-mentioned display control device.

[0210] (12) The display control method includes the following steps: a step of detecting surrounding vehicles existing around the vehicle based on a plurality of captured images of the surroundings of the vehicle captured by a plurality of camera devices mounted on the vehicle; a step of projecting the plurality of captured images onto a projection surface set to be located behind the vehicle; and a step of transforming the captured images projected onto the projection surface into images observed from a virtual viewpoint located at a position different from that of the camera devices, wherein the step of moving the virtual viewpoint includes the step of making the height of the virtual viewpoint above the ground when the surrounding vehicles are not detected within a specified area lower than the height of the virtual viewpoint above the ground when the surrounding vehicles are detected within the specified area.

[0211] Although specific examples of the present invention have been described in detail above, these are merely examples and do not limit the claims. The technology described in the claims includes various modifications and changes made to the specific examples described above.

[0212] The disclosure of Japanese Patent Application No. 2020-093481 filed on May 28, 2020, including the specification, drawings, and abstract, is incorporated herein by reference in its entirety.

[0213] Industrial Applicability

[0214] An embodiment of the present invention is suitable for use in a display control device and a vehicle.

[0215] Description of Reference Numerals

[0216] 1: Right camera

[0217] 1a: Field of view

[0218] 1a1: Cut out range

[0219] 2: Left camera

[0220] 2a: Field of view

[0221] 2a1: Cut out range

[0222] 3: Rear camera

[0223] 3a: Field of view

[0224] 3a1: Cut out range

[0225] 4: Operation Department

[0226] 5: Navigation device

[0227] 6: Wall

[0228] 7: Display unit

[0229] 8: Sensor

[0230] 9: CAN

[0231] 10: Projection surface

[0232] 11: Follow-up vehicle

[0233] 12: Blind spot area

[0234] 13: Objects

[0235] 13A: Screen projection surface

[0236] 13a: Objects

[0237] 13b: Objects

[0238] 15: Virtual Sight

[0239] 16: Screen projection application area

[0240] 17a: Border area

[0241] 17b: Border area

[0242] 17c: Border area

[0243] 18: Gentle slope

[0244] 20: Virtual Viewpoint

[0245] 50: Cube

[0246] 51: Positive

[0247] 52: Side

[0248] 61: Image processing unit

[0249] 62: Vehicle Inspection Department

[0250] 63: Situation Assessment Unit

[0251] 64: Image control unit

[0252] 65: CAN Communication Department

[0253] 100: Vehicles

[0254] 200: Display control device

Claims

1. A display control device, comprising: a vehicle detection unit that detects surrounding vehicles existing around the vehicle based on a plurality of captured images of the surroundings of the vehicle captured by a plurality of camera devices mounted on the vehicle; and an image processing unit that generates a display image to be displayed on a display unit mounted on the vehicle by synthesizing the plurality of captured images; The image processing unit performs: projecting transformation of projecting the plurality of captured images onto a projection surface set to be located behind the vehicle; and A viewpoint conversion process of converting the captured image projected onto the projection surface into an image observed from a virtual viewpoint located at a position different from that of the imaging device, The image processing unit makes the height of the virtual viewpoint above the ground lower when the surrounding vehicle is not detected in the predetermined area than when the surrounding vehicle is detected in the predetermined area. The image processing unit generates a background image of the vehicle by projecting a plurality of the captured images onto a first projection surface set to be located behind the vehicle. When the surrounding vehicles are detected within a specified area, the second projection surface is set to be located at a position where the surrounding vehicles are detected, and a surrounding vehicle image as an image of the surrounding vehicles is generated by projecting at least one of the plurality of images onto the second projection surface, and the surrounding vehicle image is overlapped with the background image.

2. The display control device according to claim 1, wherein: The image processing unit sets the height above the ground of the virtual viewpoint to be higher when the running speed of the vehicle is equal to or lower than a predetermined value than when the running speed of the vehicle exceeds the predetermined value.

3. The display control device according to claim 1, wherein: The specified area is set based on the following area: when a plurality of captured images of the periphery of the vehicle are projected onto a first projection surface set to be located behind the vehicle, if the range of projection of the plurality of captured images of the periphery of the vehicle is limited in such a manner that the background image of the vehicle becomes a continuous image, the images will not be displayed in the area of ​​the background image. 4 . A vehicle comprising the display control device according to claim 1 .

5. A display control device comprising: a vehicle detection unit that detects surrounding vehicles existing around the vehicle based on a plurality of captured images of the surroundings of the vehicle captured by a plurality of camera devices mounted on the vehicle; and an image processing unit that generates a display image to be displayed on a display unit mounted on the vehicle by synthesizing the plurality of captured images; The image processing unit performs: projecting transformation of projecting the plurality of captured images onto a projection surface set to be located behind the vehicle; and A viewpoint conversion process of converting the captured image projected onto the projection surface into an image observed from a virtual viewpoint located at a position different from that of the imaging device, The image processing unit makes the height of the virtual viewpoint above the ground lower when the surrounding vehicle is not detected in the predetermined area than when the surrounding vehicle is detected in the predetermined area. The image processing unit generates a rear image of the vehicle by projecting a plurality of the captured images onto a projection surface set to be located behind the vehicle, and sets a boundary area in the rear image in which pixel values ​​of two adjacent captured images among the plurality of captured images are mixed. When the surrounding vehicles are detected within a specified area, at least one of a mixing ratio of the pixel values ​​and a width of the boundary area is changed according to at least one of a position and a width of the surrounding vehicles.

6. The display control device according to claim 5, wherein: The image processing unit sets the height above the ground of the virtual viewpoint to be higher when the running speed of the vehicle is equal to or lower than a predetermined value than when the running speed of the vehicle exceeds the predetermined value.

7. A vehicle comprising the display control device according to claim 5.

8. A display control method, comprising the following steps: The first step is to detect surrounding vehicles existing around the vehicle based on a plurality of captured images of the surroundings of the vehicle captured by a plurality of camera devices mounted on the vehicle; A second step is to project the plurality of captured images onto a projection surface set to be located behind the vehicle; The third step is to transform the captured image projected onto the projection surface into an image observed from a virtual viewpoint located at a position different from that of the camera device; as well as The fourth step is to display the captured image transformed into an image observed from the virtual viewpoint on a display unit mounted on the vehicle. In the third step, the following steps are also included: The fifth step is to make the height of the virtual viewpoint above the ground when the surrounding vehicles are not detected in the specified area lower than the height of the virtual viewpoint above the ground when the surrounding vehicles are detected in the specified area; A sixth step is to generate a background image of the vehicle by projecting a plurality of the captured images onto a first projection surface set to be located behind the vehicle; The seventh step is to set the second projection surface to be at a position where the surrounding vehicles are detected when the surrounding vehicles are detected in the specified area; In an eighth step, a surrounding vehicle image is generated as an image of the surrounding vehicles by projecting at least one of the plurality of images onto the second projection surface; as well as The ninth step is to overlap the surrounding vehicle image with the background image.

9. A display control method, comprising the following steps: The first step is to detect surrounding vehicles existing around the vehicle based on a plurality of captured images of the surroundings of the vehicle captured by a plurality of camera devices mounted on the vehicle; A second step is to project the plurality of captured images onto a projection surface set to be located behind the vehicle; The third step is to transform the captured image projected onto the projection surface into an image observed from a virtual viewpoint located at a position different from that of the camera device; as well as The fourth step is to display the captured image transformed into an image observed from the virtual viewpoint on a display unit mounted on the vehicle. In the third step, the following steps are also included: The fifth step is to make the height of the virtual viewpoint above the ground when the surrounding vehicles are not detected in the specified area lower than the height of the virtual viewpoint above the ground when the surrounding vehicles are detected in the specified area; A sixth step is to generate a rear image of the vehicle by projecting the plurality of captured images onto a projection surface set to be located behind the vehicle; In a seventh step, a boundary area is set in the rear image, where pixel values ​​of two adjacent captured images are mixed; as well as In the eighth step, when the surrounding vehicle is detected in a predetermined area, at least one of a mixing ratio of the mixing of the pixel values ​​and a width of the boundary area is changed according to at least one of a position and a width of the surrounding vehicle.

Images