Control unit, display system, and image generation method
The control device generates a perspective image to address image distortions in bird's-eye views, providing a transparent view of hidden areas and highlighting potential distortions, enhancing driver awareness of the vehicle's surroundings.
Patent Information
- Application Number
- JP2023086356
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-12-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Distortion or image loss occurs when combining partial bird's-eye views to generate an entire view, making it difficult for drivers to accurately recognize the vehicle's surroundings.
A control device generates a perspective image showing hidden areas behind the vehicle's body, complementing an overhead image with a transparent view to highlight potential distortion zones, using multiple cameras and display devices to provide a comprehensive view of the vehicle's surroundings.
Enhances the driver's understanding of the vehicle's surroundings by reducing misinterpretation due to image distortions or losses, ensuring a clearer perception of the environment through combined overhead and perspective images.
Smart Images

Figure 2024169137000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a control device, a display system, and an image generating method. [Background technology]
[0002] 2. Description of the Related Art Conventionally, there is known a system that generates a plurality of partial overhead images based on images captured around a vehicle, and displays an overall overhead image based on the plurality of partial overhead images (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2020-145579 A Summary of the Invention [Problem to be solved by the invention]
[0004] When multiple partial overhead images are synthesized to generate an overall overhead image, distortion or image loss may occur, making it difficult for a driver to properly recognize the situation around the vehicle from the overhead image alone.
[0005] An object of the present disclosure is to provide a control device, a display system, and an image generating method that generate information that allows a driver to appropriately recognize the situation around the vehicle. [Means for solving the problem]
[0006] A control device according to an embodiment of the present disclosure generates an image of the surroundings of a vehicle. The control device generates a see-through image, which is an image of a scene in an external area of the vehicle that is blocked by a part of the body of the vehicle when viewed from a specific spatial area inside the vehicle. The control device generates the see-through image so as to show an attention range in an overhead image, which is an image of a scene around the vehicle and is displayed on a display device inside the vehicle.
[0007] A display system according to an embodiment of the present disclosure includes the above-described control device and a display device mounted on a vehicle.
[0008] An image generating method according to an embodiment of the present disclosure displays an image on a display device mounted on a vehicle. The image generating method generates a see-through image that is an image of a scene in an external area of the vehicle that is obstructed by a part of the body of the vehicle when viewed from a specific spatial area inside the vehicle. The image generating method generates the see-through image so as to reflect an attention range in an overhead image that is an image of a scene around the vehicle and is displayed on a display device inside the vehicle. Effect of the Invention
[0009] According to the control device, the display system, and the image generating method according to an embodiment of the present disclosure, information is generated that allows the driver to appropriately recognize the situation around the vehicle. [Brief description of the drawings]
[0010] [Figure 1] 1 is a block diagram showing a schematic configuration example of a display system according to an embodiment. [Diagram 2] 1 is a side view showing a configuration example of a vehicle equipped with a display system. [Diagram 3] FIG. 3 is a diagram showing the vehicle of FIG. 2 from the driver's point of view. [Figure 4] FIG. 3 is a plan view of the vehicle of FIG. 2 viewed from above toward the ground. [Diagram 5] 1 is a diagram showing an example of a relationship between a viewpoint of a driver of a vehicle and a range in which the driver can see ahead of the vehicle; [Figure 6] FIG. 1 is a diagram showing an example of an overhead view image. [Figure 7] FIG. 13 is a diagram showing an example of a perspective image. [Figure 8] 1 is a flowchart illustrating an example of a procedure of a video generating method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] (Overview of Display System 1) 1, a display system 1 according to an embodiment of the present disclosure includes a first display device 10, a second display device 20, an image capturing device 30, and a control device 40. At least a part of the display system 1 according to the present embodiment is mounted on a vehicle 50 (see FIG. 2, etc.).
[0012] The display system 1 displays information including information about a blind spot when the driver of the vehicle 50 visually recognizes the surroundings of the vehicle 50. Specifically, the display system 1 displays an overhead image 80 (see FIG. 6). The overhead image 80 is an image generated from a viewpoint looking down toward the ground from a viewpoint above the vehicle, looking down toward the ground, at the vehicle 50 and objects around the vehicle 50 that are located on the ground.
[0013] The overhead image 80 is generated by the control device 40 based on an image of the surroundings of the vehicle 50 captured by the image capturing device 30. The overhead image 80 may include distortion caused by image processing for converting the image of the surroundings of the vehicle 50 into the overhead image 80. Distortion of the overhead image 80 may occur at various positions, such as the edge of the image.
[0014] When the overhead view video 80 is generated based on multiple images, it may contain distortion caused by image processing when combining the multiple images. The overhead view video 80 may also contain distortion that occurs at the boundaries between the multiple images.
[0015] In addition, due to image processing for converting an image taken around the vehicle 50 into the overhead view image 80, an area around the vehicle 50 that should be shown in the overhead view image 80 may disappear from the overhead view image 80. In this case, an object present in the disappeared area is not included in the overhead view image 80. The disappearance of an area or object may occur at various positions, such as the edge of an image. The disappearance of an area or object may also occur at the boundary between multiple images.
[0016] As described above, display defects such as image distortion or disappearance of areas or objects may occur in the overhead image 80. Display defects in the overhead image 80 may lead to the driver of the vehicle 50 misperceiving the situation around the vehicle 50.
[0017] Therefore, the display system 1 further displays a see-through image 70 (see FIG. 7) as information that complements the overhead image 80. The see-through image 70 is an image generated on the assumption that, for example, when the driver of the vehicle 50 views the front of the vehicle 50, the driver can see through a range that the driver cannot see due to the presence of an object of an opaque vehicle body such as the dashboard 55 (see FIG. 2, etc.). The see-through image 70 is an image of a scene that is blocked by a part of the vehicle body of the vehicle 50 in the external area of the vehicle 50 when viewed from a specific spatial area inside the vehicle 50. In other words, the see-through image 70 is an image of a scene that the driver can see if the dashboard 55 was transparent. The see-through image 70 is generated by the control device 40 based on an image of the surroundings of the vehicle 50 captured by the imaging device 30. The display system 1 can reduce the possibility that the driver will erroneously recognize the situation around the vehicle 50 due to a display defect in the overhead image 80 by having the driver visually recognize the range in which a display defect may occur in the overhead image 80 using the see-through image 70. The range in which a display defect may occur in the overhead image 80, or the range in which a defect has occurred, is also referred to as the attention range of the overhead image 80. By having the driver visually recognize this attention range using the see-through image 70, the possibility that the driver will erroneously recognize the situation around the vehicle 50 can be reduced.
[0018] (Example of the configuration of display system 1) An example of the configuration of each part of the display system 1 will be described below.
[0019] <Vehicle 50> 2 and 3, the vehicle 50 includes a first display device 10. When the driver of the vehicle 50 views the front of the vehicle 50 through the windshield 52, the first display device 10 displays a see-through image 70 (see FIG. 7) generated from a viewpoint where the driver sees through the range outside the range of the windshield 52. The see-through image 70 is generated as an image that sees through a range that is blocked by, for example, the dashboard 55 below the windshield 52 and cannot be seen.
[0020] The vehicle 50 further includes a second display device 20. The second display device 20 displays an overhead image 80 (see FIG. 6) generated from a viewpoint looking down toward the ground, of the vehicle 50 located on the ground and objects around the vehicle 50. The direction toward the ground corresponds to the negative direction of the Z axis.
[0021] The vehicle 50 has tires 56 and is configured to move forward in the positive direction of the Y axis, as shown in Fig. 4. The vehicle 50 has side mirrors 53 and 54, as shown in Fig. 3 or Fig. 4.
[0022] The vehicle 50 is equipped with an imaging device 30 that captures images used to generate the see-through image 70 and the overhead image 80. As shown in Fig. 4, the vehicle 50 is equipped with a front camera 31 that captures an image of the front of the vehicle 50 as the imaging device 30. The front of the vehicle 50 corresponds to the positive direction of the Y axis. The front camera 31 may be installed on, for example, a bumper at the front of the vehicle 50.
[0023] The vehicle 50 is equipped with, as the imaging device 30, a side camera 32 that images the right side facing the front of the vehicle 50, and a side camera 33 that images the left side facing the front of the vehicle 50. The right side facing the front of the vehicle 50 corresponds to the positive direction of the X-axis. The left side facing the front of the vehicle 50 corresponds to the negative direction of the X-axis. The side camera 32 may be installed on, for example, the right side mirror 53. The side camera 33 may be installed on, for example, the left side mirror 54.
[0024] The vehicle 50 is equipped with a rear camera 34 that captures an image behind the vehicle 50. The rear of the vehicle 50 corresponds to the negative direction of the Y axis. The rear camera 34 may be installed on a rear bumper of the vehicle 50, for example.
[0025] The vehicle 50 is equipped with an oblique camera 35 that captures an image in a direction inclined at 45 degrees to the right from the front of the vehicle 50, and an oblique camera 36 that captures an image in a direction inclined at 45 degrees to the left from the front of the vehicle 50. The oblique cameras 35 and 36 may be installed on the front bumper of the vehicle 50, for example.
[0026] <Display device> As described above, the first display device 10 displays the perspective image 70 (see FIG. 7). The first display device 10 includes a projector 12 and a screen 14, as shown in FIG.
[0027] The projector 12 may be installed inside the vehicle 50 so as to project image light for forming an image including the see-through image 70 on the screen 14. The projector 12 may be provided, for example, on the ceiling inside the vehicle 50 or on the headrest of the driver's seat.
[0028] The screen 14 is configured so that the driver can view an image including the see-through image 70 formed by the image light projected from the projector 12. That is, the screen 14 functions as a display surface for the image. The screen 14 is arranged in the traveling direction of the vehicle 50 as seen from the driver of the vehicle 50 so that the driver can view the image while facing the traveling direction of the vehicle 50. The screen 14 may be formed by laminating a retroreflective material and a diffusion plate that diffuses light in the vertical direction of the vehicle 50. The vertical direction of the vehicle 50 corresponds to the Z-axis direction. Note that the position of the screen that displays the see-through image 70 is not limited to the traveling direction of the vehicle 50. It may be arranged on any member in the vehicle that the vehicle passenger needs to see through and view. For example, the screen may be arranged on a pillar of the vehicle body. The screen may be arranged on any pillar among the A to D pillars of the vehicle body. The projector 12 may project the see-through image 70 onto each screen.
[0029] The first display device 10 is not limited to a device that projects an image onto a display surface, which is a combination of the projector 12 and the screen 14, but may be configured to include a display that forms an image on the display surface. The display may include various types of displays, such as an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or an inorganic EL display.
[0030] As described above, the second display device 20 is configured to allow the driver to view the overhead image 80 (see FIG. 6). The second display device 20 is disposed in the traveling direction of the vehicle 50 as seen by the driver of the vehicle 50. The second display device 20 includes a display that forms an image on a display surface. The second display device 20 may include a device that projects an image on the display surface. The second display device 20 may include various types of displays, such as an LCD, or an organic EL or inorganic EL display.
[0031] The second display device 20 may be configured as a part of the first display device 10. That is, the first display device 10 may include the second display device 20. For example, the first display device 10 may display an overhead image 80 on a part of the screen 14. For example, the first display device 10 may synthesize the overhead image 80 with a part of the see-through image 70 and display it.
[0032] Regardless of whether the second display device 20 is configured as part of the first display device 10, or whether the first display device 10 and the second display device 20 are configured as separate entities, the first display device 10 and the second display device 20 are collectively referred to as a display device.
[0033] <Camera Unit 30> The photographing device 30 includes cameras that photograph the surroundings of the vehicle 50. As described above, the photographing device 30 includes the front camera 31, the side cameras 32 and 33, the rear camera 34, and the oblique cameras 35 and 36. The cameras are configured to be able to photograph RGB images. The cameras may include photographing elements that perform photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.
[0034] The number of cameras mounted on the vehicle 50 as the image capturing device 30 is not limited to six as described above, and may be five or less, or seven or more. The image capturing device 30 may include a camera that is not mounted on the vehicle 50. The image capturing device 30 may or may not be mounted on the vehicle 50. In other words, the vehicle 50 does not need to be equipped with a part of the display system 1 according to an embodiment of the present disclosure.
[0035] The combination of cameras mounted on the vehicle 50 as the imaging device 30 is not limited to the combination of six cameras exemplified in Fig. 4. For example, when the vehicle 50 is provided with the front camera 31 and the side camera 32, it may not be provided with the oblique camera 35. For example, when the vehicle 50 is provided with the front camera 31 and the side camera 33, it may not be provided with the oblique camera 36. For example, when the vehicle 50 is provided with the oblique cameras 35 and 36, it may not be provided with the front camera 31 and the side cameras 33 and 34.
[0036] <Control device 40> The control device 40 acquires images of the surroundings of the vehicle 50 from the image capturing device 30. The control device 40 generates an overhead image 80 and a perspective image 70 based on the images of the surroundings of the vehicle 50.
[0037] The control device 40 generates the overhead image 80 based on images captured around the vehicle 50. The control device 40 may generate the overhead image 80 based on a known algorithm.
[0038] The control device 40 generates a see-through image 70 based on the viewpoint of the driver of the vehicle 50. As shown in Fig. 5, the viewpoint of the driver of the vehicle 50 is represented by an eye box eb. For example, a range in front of the vehicle 50 that the driver cannot directly see is referred to as a first range s1. On the other hand, a range in front of the vehicle 50 that the driver can directly see through the windshield 52 is referred to as a second range s2. The first range s1 and the second range s2 are separated by a line connecting the eye box eb and the bottom end of the windshield 52.
[0039] The control device 40 generates a perspective image 70 as an image of the scenery of the first range s1 seen from the driver's viewpoint based on an image of the first range s1 captured by the front camera 31. The control device 40 may generate the perspective image 70 based on an image captured by the oblique camera 35 or 36.
[0040] The control device 40 may generate the perspective image 70 so that the perspective image 70 appears to be naturally connected to the scenery in the second range s2 viewed by the driver. For example, the control device 40 may generate the perspective image 70 so that the perspective image 70 appears to be continuous with the scenery in the second range s2. The control device 40 may generate the perspective image 70 so that the brightness or saturation that the driver feels when viewing the perspective image 70 approaches the brightness or saturation of the scenery in the second range s2 viewed by the driver.
[0041] The control device 40 may determine the position of the eye box eb to be the position of the driver's eyes, which is determined based on data such as the driver's physique. The control device 40 may determine the position of the eye box eb to be the position of the driver's eyes detected by eye tracking. The display system 1 may further include a device for eye tracking.
[0042] The controller 40 may include at least one processor to provide control and processing power for performing various functions. The functions of the controller 40 may be implemented in one processor or in several processors. The processor may be implemented as a single integrated circuit (IC). The processor may be implemented as multiple communicatively connected integrated circuits or discrete circuits. The processor may be implemented based on various other known technologies.
[0043] The processor may include a general-purpose processor that loads a specific program to execute a specific function, or a dedicated processor specialized for a specific process. The general-purpose processor may include, for example, a central processing unit (CPU) or a digital signal processor (DSP). The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control device 40 may include either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together.
[0044] The control device 40 may include a memory unit. The memory unit may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The memory unit stores various information. The memory unit stores programs and the like executed by a processor or the like. The memory unit may be configured as a non-transitory readable medium. The memory unit may function as a work memory for the processor or the like. At least a part of the memory unit may be configured integrally with the processor or the like.
[0045] The control device 40 may include a communication interface for wired or wireless communication with the first display device 10, the second display device 20, and the image capture device 30. The communication interface may be configured to enable communication using a communication method based on various communication standards. The communication interface may be configured based on a known communication technology. The first display device 10, the second display device 20, or the image capture device 30 may include a communication interface for communicating with the control device 40.
[0046] The control device 40 may include an input device that accepts input from a user. The input device may include, for example, a keyboard or physical keys, or may include a touch panel or touch sensor, or a pointing device such as a mouse. The input device is not limited to these examples and may include various other devices. The control device 40 may be configured to be able to communicate with an external input device.
[0047] The control device 40 may or may not be mounted on the vehicle 50 .
[0048] (Example of operation of display system 1) The display system 1 displays an overhead image 80 and a see-through image 70. The overhead image 80 may include display defects such as image distortion or object loss caused by image processing that converts an image of the surroundings of the vehicle 50 into the overhead image 80. Hereinafter, an operation example will be described in which the display system 1 generates the overhead image 80 and the see-through image 70 so that the driver of the vehicle 50 can visually confirm the attention range of the overhead image 80 in the see-through image 70 when the vehicle 50 is parked forward in a parking space.
[0049] <Display of overhead image 80> The control device 40 generates an overhead view image 80 as shown in Fig. 6, and displays it on the second display device 20. The parking space is defined by a white line 63 and has a car stopper 62. In addition, a block 61 exists outside the parking space.
[0050] The overhead image 80 includes a vehicle image 88 looking down on the vehicle 50, a forward image 81, a right-hand image 82, and a left-hand image 83. The overhead image 80 may further include an image of the rear of the vehicle 50, which is on the opposite side of the traveling direction of the vehicle 50.
[0051] The control device 40 uses an image prepared in advance as an image specific to the vehicle 50 as the vehicle image 88. The control device 40 generates a front image 81 based on an image of the area in front of the vehicle 50 captured by the front camera 31. The control device 40 generates a right image 82 based on an image of the area to the right of the vehicle 50 captured by the side camera 32. The control device 40 generates a left image 83 based on an image of the area to the left of the vehicle 50 captured by the side camera 33.
[0052] The overhead view image 80 includes a boundary portion 86 located at the boundary between a front image 81 and a right image 82 , and a boundary portion 87 located at the boundary between the front image 81 and a left image 83 .
[0053] When the overhead view image 80 is generated based on a plurality of images, display defects in the overhead view image 80 are likely to occur at the boundaries between the plurality of images. Therefore, in the overhead view image 80 illustrated in FIG. 6, the boundaries 86 and 87 and the range including the boundaries 86 and 87 correspond to the attention range.
[0054] The control device 40 may superimpose and display a frame 84 that defines the attention area to alert the driver to possible display problems such as image distortion or object loss in the boundaries 86 and 87 included in the overhead view image 80 and in the attention area including the boundaries 86 and 87. The frame 84 may be displayed with a dashed line as shown in the figure, but may also be displayed with other line types such as a solid line or a dashed dotted line. The frame 84 may be displayed in a color that is likely to attract the driver's attention, such as red or yellow. The frame 84 may also be displayed in a color that has a high contrast with the overhead view image 80.
[0055] The control device 40 may superimpose and display a mark 85 on the forward image 81 of the overhead image 80 to alert the driver to the area ahead. The mark 85 is not limited to an "!" as shown in the figure, and may be displayed in various other ways. The mark 85 may be displayed in a color that easily alerts the driver, such as red or yellow. The mark 85 may be displayed in a color that has a high contrast with the overhead image 80. The control device 40 may also display a text explanation in addition to the mark 85. Specifically, the control device 40 may display a message suggesting that a check should be made visually or by using the fluoroscopic image 70, at a location on the overhead image 80 that corresponds to the attention range.
[0056] An image such as the frame 84 or the mark 85 superimposed on the overhead image 80 is also referred to as an additional image. The control device 40 may superimpose the additional image on the overhead image 80 regardless of whether a display defect such as image distortion or object disappearance actually occurs at the boundary portion 86 or 87 of the overhead image 80. The control device 40 may superimpose the additional image on the overhead image 80 without determining whether a display defect such as image distortion or object disappearance occurs at the boundary portion 86 or 87 of the overhead image 80. In other words, the control device 40 may superimpose the additional image on the overhead image 80 to notify the driver of the possibility of a display defect occurring at the boundary portion 86 or 87 of the overhead image 80.
[0057] <Display of fluoroscopic image 70> The control device 40 generates a see-through image 70 as shown in Fig. 7, and displays it on the screen 14 of the first display device 10. The see-through image 70 as shown in Fig. 7 is generated as an image that shows the scenery in a first range s1 (see Fig. 5) that cannot be directly seen by the driver in a natural way with the scenery in a second range s2 (see Fig. 5) that can be directly seen by the driver.
[0058] 7, for example, a perspective image 73 of a white line 63 in the perspective image 70 is drawn so as to appear to be naturally connected to the white line 63 that the driver directly sees through the windshield 52. In addition, a perspective image 74 of a car stopper 62 in the perspective image 70 is drawn so as to appear as if the driver can directly see the car stopper 62, even though the car stopper 62 is located in a first range s1 that the driver cannot directly see.
[0059] As described above, there is a possibility that a display defect such as a distortion of the display or disappearance of an object occurs in the boundary portions 86 and 87 of the overhead-view image 80 and in the attention range including the boundary portions 86 and 87. When a display defect occurs in the boundary portions 86 or 87 of the overhead-view image 80, the driver of the vehicle 50 may not notice the display defect in the overhead-view image 80 and may mistakenly understand the situation around the vehicle 50, resulting in driving the vehicle 50 incorrectly.
[0060] The control device 40 generates the perspective image 70 so that the perspective image 70 reflects the attention range of the overhead image 80, so that the driver of the vehicle 50 can confirm the attention range of the overhead image 80 in the perspective image 70.
[0061] In other words, the control device 40 generates a see-through image 70 of the area ahead of the vehicle 50 and displays it on at least a portion of the display device or the first display device 10, and generates an overhead image 80 of the surroundings of the vehicle 50 and displays it on at least a portion of the display device or the second display device 20. The control device 40 generates the see-through image 70 so that the see-through image 70 reflects the attention range of the overhead image 80. When generating the overhead image 80 based on multiple images captured around the vehicle 50, the control device 40 may generate the see-through image 70 with the boundary between the multiple images constituting the overhead image 80 as the attention range.
[0062] The control device 40 may generate the perspective image 70 so that the perspective image 70 shows at least a part of the range of the surroundings of the vehicle 50 that is shown in the overhead image 80. The control device 40 may generate the overhead image 80 and the perspective image 70 so that the range of the surroundings of the vehicle 50 that is shown in the overhead image 80 is wider than the range that is shown in the perspective image 70.
[0063] In this embodiment, the see-through image 70 is described as displaying a view in front of the vehicle 50, but the present invention is not limited to this and may display a view around the vehicle 50. Specifically, the see-through image 70 may be an image including views behind, to the sides, above (i.e., the ceiling), and below (i.e., the floor) of the vehicle 50.
[0064] <<Overlaying additional images>> The control device 40 may superimpose, on the perspective image 70, an image indicating which part of the perspective image 70 should be looked at in order to check the attention range of the overhead image 80.
[0065] 7, for example, the control device 40 may display a frame 71 surrounding a portion of the perspective image 70 that corresponds to the attention ranges corresponding to the boundaries 86 and 87 in the overhead image 80 in FIG 6, superimposed on the perspective image 70. The driver of the vehicle 50 can check the status of the attention range of the overhead image 80 by looking at the portion of the perspective image 70 that is surrounded by the frame 71.
[0066] The frame 71 may be displayed in a dashed line as shown in the figure, but may also be displayed in other line types such as a solid line or a dashed line. The frame 71 may be displayed in a color that easily attracts the driver's attention, such as red or yellow. The frame 71 may also be displayed in a color that has a high contrast with the fluoroscopic image 70.
[0067] The control device 40 may display a mark 72 superimposed near the frame 71 so as to make the driver of the vehicle 50 aware of the frame 71 superimposed on the fluoroscopic image 70. The mark 72 is not limited to being displayed as an "!" as shown in the figure, and may be displayed in various other forms. The mark 72 may be displayed in a color that is likely to attract the driver's attention, such as red or yellow. The mark 72 may be displayed in a color that has a high contrast with the fluoroscopic image 70.
[0068] As described above, the control device 40 may superimpose an additional image on a portion of the perspective image 70 that shows the attention range of the overhead image 80. By superimposing an additional image on the perspective image 70, the control device 40 can make the driver of the vehicle 50 aware of the possibility that a display malfunction is occurring in the attention range of the overhead image 80.
[0069] The control device 40 may superimpose an additional image such as a frame 84 or a mark 85 on the overhead image 80 to make the driver aware of the possibility of a display malfunction occurring in the attention range of the overhead image 80. When the control device 40 superimposes an additional image such as a frame 71 or a mark 72 on the perspective image 70, the control device 40 may not superimpose an additional image such as a frame 84 or a mark 85 on the overhead image 80. Conversely, when the control device 40 superimposes an additional image such as a frame 84 or a mark 85 on the overhead image 80, the control device 40 may not superimpose an additional image such as a frame 71 or a mark 72 on the perspective image 70. In other words, it is sufficient for the control device 40 to superimpose an additional image on at least one of the perspective image 70 or the overhead image 80.
[0070] <Example of how to display the procedure> The control device 40 may execute a display method including the steps of the flowchart illustrated in Fig. 8. The display method may be realized as a display program executed by a processor constituting the control device 40. The display program may be stored in a non-transitory computer-readable medium. Among the steps included in the display method, the step of generating an image is also referred to as an image generating method.
[0071] The control device 40 acquires an image of the surroundings of the vehicle 50 from the photographing device 30 (step S1). The control device 40 generates an overhead view image 80 of the vehicle 50 based on the image of the surroundings of the vehicle 50, and displays it on the second display device 20 (step S2). The control device 40 generates a perspective image 70 assuming that the first range s1 (see FIG. 5) is seen through from the viewpoint of the driver of the vehicle 50 based on the image of the front of the vehicle 50, and displays it on the first display device 10 (step S3). The control device 40 superimposes an image such as a frame 71 or a mark 72 on the perspective image 70 at a position corresponding to an attention range such as a boundary portion 86 or 87 of the overhead view image 80 (step S4). After executing the procedure of step S4, the control device 40 ends the execution of the procedure of the flowchart of FIG. 8. After executing the procedure of step S4, the control device 40 may return to the procedure of step S1 to update the overhead view image 80 and the perspective image 70.
[0072] (summary) As described above, according to the display system 1 and the control device 40 of this embodiment, the overhead image 80 and the see-through image 70 are generated and displayed so that the driver of the vehicle 50 can appropriately recognize the situation around the vehicle 50. The control device 40 generates the see-through image 70 so that the driver of the vehicle 50 can check the attention range of the overhead image 80 with the see-through image 70. In this way, even if a display malfunction occurs in the attention range of the overhead image 80, information for the driver of the vehicle 50 to appropriately recognize the situation around the vehicle 50 is generated and displayed. As a result, the driver of the vehicle 50 is less likely to misrecognize the situation around the vehicle 50.
[0073] Furthermore, the control device 40 superimposes an additional image on the perspective image 70 or the overhead image 80 to prompt the driver of the vehicle 50 to check the attention range of the overhead image 80 using the perspective image 70. By superimposing the additional image, the driver of the vehicle 50 can appropriately recognize the situation around the vehicle 50, taking into consideration the possibility of a display malfunction in the attention range of the overhead image 80.
[0074] In the above-described embodiment, the control device 40 generates the perspective image 70 and the perspective image 80, and determines the position at which the additional image is superimposed on the perspective image 70 or the perspective image 80, assuming that the boundary between the multiple images constituting the perspective image 80 is the attention range of the perspective image 80. The attention range of the perspective image 80 is not limited to the boundary between the multiple images. Therefore, the control device 40 may set the attention range to a range where a display defect in the perspective image 80 may occur, not limited to the boundary between the multiple images. The control device 40 may set the attention range to a range including a position where a display defect due to a singularity inherent to the algorithm of image processing is likely to occur in the perspective image 80. The singularity inherent to the algorithm of image processing may include, for example, a point where an image expands to infinity in the calculation of image processing, or a point where an image disappears in the calculation of image processing.
[0075] (Other embodiments) Other embodiments of the display system 1 and the control device 40 will be described below.
[0076] <Caution for X-ray image 70> The control device 40 may generate the see-through image 70 based on a plurality of images captured in front of the vehicle 50. In this case, the see-through image 70 is generated as an image obtained by synthesizing the plurality of images. Therefore, the see-through image 70 has boundaries between the plurality of images. The boundaries between the plurality of images in the see-through image 70 and a predetermined range including the boundaries are also referred to as an attention range of the see-through image 70.
[0077] The control device 40 may generate the perspective image 70 such that the attention range of the perspective image 70 does not overlap with the attention range of the overhead image 80. Conversely, the control device 40 may generate the overhead image 80 such that the attention range of the perspective image 70 does not overlap with the attention range of the overhead image 70. In this manner, the attention range of the perspective image 70 and the attention range of the overhead image 80 complement each other. As a result, the driver of the vehicle 50 can appropriately recognize the situation around the vehicle 50.
[0078] For example, the control device 40 may generate the perspective image 70 based on an image of the vehicle 50 diagonally forward to the right taken by the oblique camera 35 and an image of the vehicle 50 diagonally forward to the left taken by the oblique camera 36. The control device 40 may also generate the overhead image 80 based on an image of the vehicle 50 in front taken by the front camera 31 and images of the vehicle 50 on the left and right taken by the side cameras 32 and 33. In this case, the attention range of the perspective image 70 including the boundaries of the two images constituting the perspective image 70 is located directly in front of the vehicle 50. On the other hand, the attention range of the overhead image 80 including the boundaries of the three images constituting the overhead image 80 is located diagonally forward to the right and diagonally forward to the left of the vehicle 50. As a result, the control device 40 can generate the perspective image 70 and the overhead image 80 such that the attention range of the perspective image 70 and the attention range of the overhead image 80 do not overlap each other.
[0079] The control device 40 may generate the fluoroscopic image 70 based on one image captured at a wide angle ahead of the vehicle 50. In this manner, the control device 40 can generate the fluoroscopic image 70 so that the fluoroscopic image 70 does not have an attention range. Since the fluoroscopic image 70 does not have an attention range, the attention range of the fluoroscopic image 70 and the attention range of the overhead image 80 do not overlap with each other.
[0080] <Changing the Position of Boundaries Between Multiple Images in the Overhead View 80> For example, the control device 40 may generate an overhead image 80 based on an image of the vehicle 50 diagonally forward to the right taken by the oblique camera 35 and an image of the vehicle 50 diagonally forward to the left taken by the oblique camera 36. In this case, the range corresponding to the boundaries between the multiple images in the overhead image 80 is located near the center of the perspective image 70. Here, as illustrated in FIG. 7, when the second display device 20 that displays the overhead image 80 is disposed near the center of the screen 14 that displays the perspective image 70, the display of the range corresponding to the boundaries between the multiple images in the overhead image 80 in the perspective image 70 is obstructed by the second display device 20. Therefore, the control device 40 may generate the overhead image 80 by changing the positions of the boundaries between the multiple images in the overhead image 80 so that the range corresponding to the boundaries between the multiple images in the overhead image 80 is displayed at a position outside the second display device 20 in the perspective image 70.
[0081] Furthermore, when the first display device 10 realizes the function of the second display device 20 instead and displays the overhead image 80 on a part of the screen 14, the control device 40 may change the position on the screen 14 where the overhead image 80 is displayed to a position outside the range corresponding to the boundary between the multiple images in the overhead image 80. In other words, the control device 40 may generate the overhead image 80 on the display device such that the portion of the see-through image 70 reflecting the attention range of the overhead image 80 does not overlap the portion displaying the overhead image 80.
[0082] <Combination of cameras capturing images used to generate overhead image 80 or perspective image 70> The control device 40 may generate the overhead image 80 or the see-through image 70 based on an image of the surroundings of the vehicle 50 captured by a camera not mounted on the vehicle 50. For example, in order to assist driving operations when parking the vehicle 50 in a parking lot, a garage, or the like, the control device 40 may generate the overhead image 80 or the see-through image 70 based on an image captured by a camera that captures the scenery of the parking lot, the garage, or the like. The display system 1 may include a camera not mounted on the vehicle 50 as the imaging device 30.
[0083] The control device 40 may generate the overhead image 80 or the perspective image 70 based only on images of the surroundings of the vehicle 50 captured by a camera not mounted on the vehicle 50. The control device 40 may generate the overhead image 80 or the perspective image 70 based on both images of the surroundings of the vehicle 50 captured by a camera not mounted on the vehicle 50 and images of the surroundings of the vehicle 50 captured by a camera mounted on the vehicle 50.
[0084] The control device 40 may generate the overhead image 80 or the perspective image 70 based on images of the surroundings of the vehicle 50 captured by any combination of the multiple cameras mounted on the vehicle 50. For example, the control device 40 may generate the perspective image 70 based on images captured by each of multiple cameras including the oblique camera 35 or 36 in addition to the forward camera 31. The control device 40 may generate the perspective image 70 based on images captured by a camera not mounted on the vehicle 50 in addition to the image captured by the forward camera 31.
[0085] <Superimposing an additional image onto the scenery in the second range s2> An attention range such as a boundary between multiple images in the overhead image 80 may be included in a second range s2 that the driver of the vehicle 50 can view through the windshield 52. In this case, the display system 1 may display a virtual image of the additional image on the windshield 52 using, for example, a HUD (Head Up Display) or the like, and superimpose the additional image on the scenery in the second range s2 viewed by the driver, so as to encourage the driver of the vehicle 50 to directly view the attention range of the overhead image 80 through the windshield 52. In this case, the display device may include a device such as a HUD that displays a virtual image on the windshield 52.
[0086] The drawings for explaining the embodiments according to the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.
[0087] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to be logically inconsistent, and multiple components can be combined into one or divided. It should be understood that these are also included in the scope of the present disclosure.
[0088] All of the features described in this disclosure and / or all steps of any method or process disclosed herein may be combined in any combination except those combinations in which the features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is merely one example of a generic series of the same or equivalent features.
[0089] Furthermore, the embodiments of the present disclosure are not limited to any of the specific configurations of the embodiments described above, but rather extend to any novel feature or combination of features described herein, or any novel method or process step or combination of features described herein.
[0090] The vehicle 50 according to the present disclosure may include, for example, automobiles, industrial vehicles, railroad vehicles, lifestyle vehicles, or fixed-wing aircraft traveling on runways. The automobiles may include, for example, passenger cars, trucks, buses, motorcycles, or trolley buses. The industrial vehicles may include, for example, industrial vehicles for agriculture or construction. The industrial vehicles may include, for example, forklifts or golf carts. The industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, or lawnmowers. The industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, or road rollers. The vehicle 50 may include vehicles that run by human power. The classification of the vehicle 50 is not limited to the above examples. For example, the automobile may include industrial vehicles that can run on roads. The same type of vehicle 50 may be included in multiple classifications.
[0091] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. Configurations distinguished by descriptions such as "first" and "second" in this disclosure may exchange numbers in the configuration. For example, the first display device 10 may exchange identifiers "first" and "second" with the second display device 20. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the exchange of identifiers. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a symbol. Descriptions of identifiers such as "first" and "second" in this disclosure alone should not be used to interpret the order of the configuration or to justify the existence of an identifier with a smaller number.
[0092] The above describes an embodiment of a display method using display system 1. However, embodiments of the present disclosure can also be embodied as a method or program for implementing the device, or as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).
[0093] The form of implementation of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may be a program module incorporated into an operating system, etc. Furthermore, the program may or may not be configured so that all processing is performed only by the CPU on the control board. The program may be configured so that a part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as necessary.
[0094] As described above, the display system 1 according to the present disclosure is configured to display the attention area in the overhead image 80 also in the perspective image 70. The display system 1 may be configured to display, not only the overhead image 80, but also an area corresponding to a display defect included in various images such as a bird's-eye view image generated from a viewpoint different from the viewpoint of the driver, in the perspective image 70.
[0095] In one embodiment, (1) a control device that generates an image of the surroundings of a vehicle generates a see-through image that is an image of a scene within an external area of the vehicle that is obstructed by a part of the body of the vehicle when viewed from a specific spatial area inside the vehicle, and generates the see-through image so as to reflect a range of attention in an overhead image that is an image of a scene around the vehicle and is displayed on a display device inside the vehicle.
[0096] (2) The control device described in (1) above may superimpose an additional image on a portion of the perspective image that includes an object or area that corresponds to the attention range.
[0097] (3) The control device described in (1) or (2) above may generate the overhead image based on a plurality of images taken of the surroundings of the vehicle, and may generate the perspective image with the boundary between the plurality of images constituting the overhead image as the attention range.
[0098] (4) The control device described in any one of (1) to (3) above may generate the overhead image on the display device such that the portion of the perspective image reflecting the attention range of the overhead image does not overlap the portion displaying the overhead image.
[0099] (5) The control device according to any one of (1) to (4) above may generate the perspective image so that the perspective image captures at least a portion of the range captured in the overhead image.
[0100] (6) The control device described in any one of (1) to (5) above may generate the overhead image and the perspective image so that the range captured in the overhead image is wider than the range captured in the perspective image.
[0101] In one embodiment, (7) a display system includes the control device according to any one of (1) to (6) above, and a display device mounted on a vehicle.
[0102] (8) In the display system described in (7) above, the display device may include a first display device that displays the perspective image and a second display device that displays the overhead image.
[0103] (9) The display system described in (7) or (8) above may further include an imaging device that captures images of the surroundings of the vehicle.
[0104] In one embodiment, (10) an image generation method for generating an image of the surroundings of a vehicle generates a see-through image that is an image of a scene within an external area of the vehicle that is obstructed by a part of the body of the vehicle when viewed from a specific spatial area inside the vehicle, and generates the see-through image so as to reflect a range of attention in an overhead image that is an image of a scene around the vehicle and is displayed on a display device inside the vehicle. [Explanation of symbols]
[0105] 1 Display System 10 first display device (12: projector, 14: screen) 20 2nd display device 30 Photographing device (31: front camera, 32, 33: side camera, 34: rear camera, 35, 36: oblique camera) 40 Control device 50 Vehicle (52: Windshield, 53, 54: Side mirror, 55: Dashboard, 56: Tire) 61 Block 62 Car stop 63 White Line 70 Transparent image (71: frame, 72: mark, 73, 74: transparent image) 80 Bird's-eye view image (81: forward image, 82: right image, 83: left image, 84: frame, 85: mark, 86, 87: boundary, 88: vehicle image) eb Eyebox s1 First range s2 Second range
Claims
1. A control device that generates an image of the surroundings of a vehicle, generating a perspective image that is an image of a scene in an external region of the vehicle that is occluded by a part of a body of the vehicle when viewed from a specific spatial region inside the vehicle; generating the perspective image so as to reflect a range of attention in an overhead image, which is an image of a scene around the vehicle and is displayed on a display device inside the vehicle; Control device.
2. The control device according to claim 1 , wherein an additional image is superimposed on a portion of the perspective image that includes an object or area corresponding to the attention range.
3. generating the overhead image based on a plurality of images captured around the vehicle; The control device according to claim 1 , wherein the perspective image is generated by setting a boundary between a plurality of images constituting the overhead image as the attention range.
4. The control device according to claim 1 or 2, wherein the display device generates the overhead image such that a portion of the perspective image that shows an attention range of the overhead image does not overlap a portion displaying the overhead image.
5. The control device according to claim 1 or 2, wherein the perspective image is generated so that the perspective image shows at least a part of a range shown in the overhead image.
6. The control device according to claim 1 , further comprising: generating the overhead image and the perspective image such that a range captured in the overhead image is wider than a range captured in the perspective image.
7. A display system comprising: the control device according to claim 1 or 2; and a display device mounted on a vehicle.
8. The display system according to claim 7 , wherein the display devices include a first display device that displays the perspective image and a second display device that displays the overhead image.
9. The display system according to claim 7 , further comprising an image capturing device that captures an image of the surroundings of the vehicle.
10. 1. An image generating method for generating an image of an area around a vehicle, comprising: generating a perspective image that is an image of a scene in an external region of the vehicle that is occluded by a part of a body of the vehicle when viewed from a specific spatial region inside the vehicle; generating the perspective image so as to reflect a range of attention in an overhead image, which is an image of a scene around the vehicle and is displayed on a display device inside the vehicle; Video generation method.
Citation Information
Patent Citations
Display control device, display system, and display control method
JP2020145579A
