Holographic sideview mirror system and method for controlling same

Abstract

The present invention relates to a sideview mirror alternative system that displays a sideview mirror image as a 3D holographic image to provide a driver with multiple foci. The present invention provides a holographic sideview mirror system comprising: a side camera installed on the side of a vehicle to capture an image; a display module installed inside the vehicle to output the captured image; and a controller that relays the image between the side camera and the display module, wherein the controller generates a holographic image on the basis of the image captured by the side camera, and the display module can output the holographic image.

Description

Holographic side mirror system and its control method

[0001] The present invention relates to a holographic side mirror system and a control method thereof, and more specifically, to a holographic side mirror system using a side camera that replaces a conventional side mirror and a holographic display that implements an image of the side camera as a three-dimensional image, and a control method thereof.

[0002] Unlike images (i.e. photographs), which store only two-dimensional information (light intensity) about a three-dimensional object, 3D holograms store the amplitude and phase of light information using interference patterns. This allows holographic images, obtained by restoring a hologram, to effectively express depth, similar to viewing an actual three-dimensional object. Therefore, holographic images are expected to be highly useful in fields such as augmented reality and virtual reality, and active research is currently underway.

[0003] Holograms, a display technology that most perfectly restores the three-dimensional information of an object, have long been reproduced only as still images in the form of photographs. However, with the recent rapid development of 5G-level data communication technology and ultra-high-resolution display technology, they are reaching a level where video playback is possible. Spatial Light Modulators (SLMs), which can be said to be display devices that optically transmit computer-generated hologram patterns created through digital hologram technology, are recently being implemented in various ways, such as LCoS and DMD.

[0004] Holographic stereoscopic images can be produced by using a method that separates images provided to the viewer's left and right eyes using a light separation element, or by reproducing an electronically generated hologram using a spatial light modulator (SLM).

[0005] The method using the above spatial light modulator utilizes the interference phenomenon of light, and thus can provide natural parallax. However, the method using the spatial light modulator requires a coherent light source, and therefore the light source unit of the spatial light modulator can only output straight light, such as a laser. Consequently, the viewing angle of the holographic display using straight light is limited.

[0006] Meanwhile, vehicles are equipped with various mirrors, such as side-view mirrors and rear-view mirrors, to check the flow of surrounding vehicles located behind or to the side when changing lanes while driving, or to secure a rear view when reversing.

[0007] Recently, camera-type digital side mirrors are being actively commercialized in electric vehicles and luxury vehicles.

[0008] The Digital Side Mirror (DSM) system provides the driver with a view of the vehicle's surroundings, captured by cameras. The driver can secure a view of the vehicle's sides and rear through images captured by cameras positioned on either side of the vehicle.

[0009] Digital side mirrors offer the advantage of analyzing camera footage to provide more precise driving information and accident prevention. Furthermore, while conventional mirrors can sometimes obstruct the driver's view at night or in rainy weather, digital mirrors digitally post-process these limitations before displaying them, providing drivers with a more visible screen.

[0010] However, despite these advantages, digital side mirrors have the disadvantage of not providing the multi-focus field of view found in regular mirrors.

[0011] In order to solve the problems described above, the present invention proposes a digital holographic side mirror system using a holographic display that displays a 3D image capable of providing a multi-focus field of view.

[0012] Meanwhile, the present invention aims to provide a holographic side mirror system and control method that uses information on the driver's head direction or gaze direction to control the driver's gaze so that it is included within the field of view of a holographic image.

[0013] The problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0014] One embodiment of the present invention provides a holographic side mirror system including a side camera installed on the side of a mobile device to capture images, a display module installed inside the mobile device to output captured images, and a controller relaying images between the side camera and the display module, wherein the controller generates a holographic image based on the image captured by the side camera, and the display module can output the holographic image.

[0015] The above controller can receive an image from a side camera, convert the received image into phase and amplitude data through a real-time hologram generation algorithm operation, and encode the phase and amplitude data into phase or amplitude data to generate a hologram image.

[0016] The controller further includes a distance sensor installed adjacent to the side camera, and receives an image from the side camera and distance information from the distance sensor, converts the received image and distance information into phase and amplitude data through a real-time hologram generation algorithm operation, and encodes the phase and amplitude data into phase or amplitude single data to generate a hologram image.

[0017] The controller switches the image displayed on the display module to either a hologram mode that displays the image as a 3D image or a normal mode that displays the image as a 2D image, and the display module can use a straight light source in the hologram mode and a diffuse light source in the normal mode.

[0018] Switching between holographic mode and normal mode may involve switching the positions of the orthogonal light source and the diffuse light source, or may involve switching the orthogonal light source and the diffuse light source arranged around the orthogonal light source on and off.

[0019] The controller analyzes at least one factor among the driving speed of the mobile device or the analysis content of the input image, and determines whether the mobile device is in a first environment in which movement must be careful or a second environment distinct from the first environment based on the analysis, and if the mobile device is in the first environment, it can switch to a hologram mode, and if it is in the second environment, it can switch to a normal mode.

[0020] The mobile device further includes a tracking camera installed inside the device to collect information about the driver's head, and the controller calculates the direction of the driver's head using the collected information about the driver's head, and controls the direction of the light source of the display module so that the direction of the driver's head is included within the field of view of the holographic image.

[0021] The tracking camera collects information on the driver's gaze direction when the driver's head direction does not change for a preset period of time, and the controller calculates the driver's gaze direction using the collected gaze direction information, and can control the direction of the light source of the display module so that the driver's gaze direction is included within the field of view of the holographic image.

[0022] The controller can be controlled in a personalized mode that includes changing at least one of the following: changing the distance expression method for an object in a holographic image, changing the size, changing the depth, and changing the color, depending on the user.

[0023] One embodiment of the present invention provides a control method for a holographic side mirror system, including the steps of capturing an image with a side camera installed on the side of a mobile device, image processing the captured image with a controller to create a holographic image, and outputting the holographic image from a display module installed inside the mobile device.

[0024] The image displayed on the display module can be switched to either a hologram mode that displays the image as a 3D image or a normal mode that displays the image as a 2D image, and this switching can be performed.

[0025] The controller analyzes at least one factor among the driving speed of the mobile device or the analysis content of the input image, and based on this analysis, determines whether the mobile device is in a first environment in which movement must be careful or a second environment distinct from the first environment, and if the mobile device is in the first environment, switches to hologram mode, and if it is in the second environment, switches to general mode. The analysis content of the input image may include analysis of at least one of the number of mobile devices, the number of pedestrians, lanes, whether it is indoors or outdoors, and the number of objects.

[0026] A tracking camera collects information about the driver's head, and a controller can calculate the driver's head direction using the collected driver's head information, and control the direction of a light source of a display module so that the driver's head direction is included within the field of view of a holographic image. If the driver's head direction does not change for a preset period of time, information about the driver's gaze direction is collected, and the controller can calculate the driver's gaze direction using the collected gaze direction information, and control the direction of a light source of the display module so that the driver's gaze direction is included within the field of view of a holographic image.

[0027] A user can be identified, and a holographic image can be output in a personalized mode according to the identified user, and this personalized mode can include at least one change among a change in the distance expression method for an object in the holographic image, a change in size, a change in depth, and a change in color.

[0028] First, according to one embodiment of the present invention, by providing a side camera image as a 3D hologram, it is possible to provide a multi-focus field of view like a general mirror even though it is a digital side mirror system, thereby enabling an intuitive sense of distance to an object.

[0029] Additionally, convenience can be improved by automatically switching between holographic image mode and general image mode depending on the environment, allowing holographic images to be used only in complex and attention-requiring situations.

[0030] In addition, by tracking the driver's head direction and gaze direction and adjusting the direction of the viewing angle of the holographic image, it is possible to provide the convenience of viewing the holographic image from any direction.

[0031] Further scope of the applicability of the present invention will become apparent from the detailed description below. However, since various modifications and variations within the spirit and scope of the present invention will become apparent to those skilled in the art, it should be understood that the detailed description and specific examples, such as preferred embodiments of the present invention, are given by way of example only.

[0032] FIG. 1 is a configuration diagram of a holographic side mirror system according to one embodiment of the present invention.

[0033] FIG. 2 is a drawing for explaining the configuration of a holographic side mirror system installed in a vehicle according to one embodiment of the present invention.

[0034] Figure 3 is a drawing for explaining a 3D image display method of a holographic side mirror system.

[0035] Figure 4 is a flowchart showing a control method of a holographic side mirror system.

[0036] Figure 5 is a flowchart for explaining the image processing process of the controller.

[0037] Figure 6 is a flowchart for explaining the display mode switching process.

[0038] Figure 7 is a flowchart for explaining mode determination according to driving speed and image analysis.

[0039] Figure 8 is a flowchart illustrating one embodiment of an automatic mode switching process.

[0040] Figure 9 is a diagram showing an example of an image analyzed during the automatic mode switching process.

[0041] Figure 10 is a flowchart illustrating one embodiment of a mode switching algorithm based on image analysis during an automatic mode switching process.

[0042] Figure 11 is a schematic diagram showing the configuration of a display module to explain the switching of a light source in hologram mode and normal mode.

[0043] Figure 12 is a flowchart for explaining the adjustment of the hologram generation direction according to the head direction of the holographic side mirror system.

[0044] Figure 13 is an example diagram for explaining an example of head direction tracking and gaze direction tracking.

[0045] Figure 14 is a diagram illustrating an example of a user input section for switching holographic mode.

[0046] Figure 15 is a drawing for explaining an example of a personalized mode according to a user of the hologram mode.

[0047] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be assigned the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles.

[0048] In addition, in explaining the embodiments disclosed in this specification, if it is judged that a specific description of related known technology may obscure the gist of the embodiments disclosed in this specification, the embodiments disclosed in this specification will be described in detail with reference to the attached drawings, and regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted.

[0049] The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, it should be noted that the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification should not be construed as being limited by the attached drawings.

[0050] Furthermore, for the convenience of explanation, each drawing is described, but it is also within the scope of the present invention for a person skilled in the art to implement another embodiment by combining at least two drawings.

[0051] Additionally, when an element such as a layer, region or substrate is referred to as existing "on" another element, it will be understood that this may be directly on the other element, or that there may be intermediate elements in between.

[0052] Hereinafter, the terms mobile device and vehicle or automobile may be used interchangeably. The term "system" below refers to a holographic side mirror system according to one embodiment of the present invention.

[0053] Figure 1 is a configuration diagram of a holographic side mirror system (10) according to one embodiment of the present invention.

[0054] As illustrated in FIG. 1, the holographic side mirror system (10) may include an input unit (100), a controller (200), and a display module (300).

[0055] The input unit (100) can receive information and transmit it to the controller (200). The input unit (100) can include a side camera (110), a distance sensor (120), an external camera (130), a tracking camera (140), and a user interface (150).

[0056] The side camera (110) can serve as a replacement for the conventional side mirror of the mobile device (1). The side camera (110) according to one embodiment of the present invention can be installed on the side of the mobile device (1). The side camera (110) can capture images of the side and rear of the mobile device and transmit the captured images to the controller. The side camera, which monitors the side and rear of the mobile device, can generally be attached to the front door or A-pillar of the vehicle. The structure of the side camera (110) is not limited.

[0057] The image transmitted from the side camera (110) is displayed by the display module (300) so that the driver can accurately recognize the side and rear situations. The display module may be an existing display device in the vehicle or a separately installed display module. The preferred location is close to the side mirror (10), and preferably located on the interior side of the A-pillar of the vehicle.

[0058] The distance sensor (120) can measure the distance to an object captured by the side camera (110). The distance sensor (120) according to one embodiment of the present invention can be installed adjacent to the side camera (110). By adding distance information to the visual information provided by the side camera (110), the distance information can be utilized in the image processing process.

[0059] An external camera (130) is installed on the outside of the mobile device and can capture the surrounding environment of the mobile device. That is, it is installed on the outside of the mobile device and can capture the surrounding environment of the vehicle with a wide field of view.

[0060] The tracking camera (140) may be installed inside the mobile device. The tracking camera (140) may be installed in front of the driver's seat or passenger seat, and may be installed in a location capable of tracking the driver's head or gaze direction. Using the tracking information, the system can adjust the direction of the holographic display according to the driver's head or gaze direction.

[0061] The user interface (150) is installed inside the mobile device and can receive input directly from the driver. The user interface (150) can be installed in various forms, such as a touchscreen, buttons, or voice input. The user interface allows the driver to adjust the settings of the holographic display to suit his or her personal preferences.

[0062] The controller (200) can be responsible for complex image processing and analysis functions. The controller (200) can include an image processing unit (210) and an image analysis unit (220).

[0063] The image processing unit (210) can process the captured image data and convert it into a format suitable for transmission to the display module (300). This process can generate a holographic image. Alternatively, noise removal, contrast adjustment, color correction, etc. may be included in a general image.

[0064] The image analysis unit (220) can extract and interpret necessary information from collected image data. It can include analysis functions such as object recognition, distance measurement, and motion detection within the image.

[0065] The display module (300) can generate and display a holographic image. The display module (300) can include a directional light source (310), a diffusive light source (320), an optical system (330), and a spatial light modulator (SLM) (340).

[0066] A straight-line light source (310) can output straight-line light for generating a holographic image. Since the straight-line light plays an important role in the formation of a hologram, it can be a key element in generating a holographic image.

[0067] The diffuse light source (320) can output diffuse light for generating a general 2D image. The diffuse light source (320) can evenly distribute light over a wide area, thereby providing the optical characteristics required for a traditional 2D display.

[0068] The optical system (330) can appropriately control the light output from the light source and transmit it to the spatial light modulator (340) in an optimized state. The optical system (330) can be composed of at least one optical component, such as a lens or a mirror, and can precisely control the path of light to output a final image.

[0069] The spatial light modulator (340) can generate a final holographic image by modulating the phase and amplitude of the input light. Of course, it can also output a general 2D image.

[0070] Through the above configuration, the display module (300) can provide the driver with a real-time holographic image of the surrounding environment. This can contribute to improving driving safety, particularly by providing the driver with a sense of distance. Furthermore, it can provide the driver with the flexibility to adjust the depth, size, and position of the holographic image according to their preferences or needs.

[0071] FIG. 2 is a drawing for explaining the configuration of a holographic side mirror system (10) installed in a vehicle according to one embodiment of the present invention. It shows a simplified representation of the interior front portion of the vehicle (1) and illustrates the arrangement of major components of the holographic side mirror system (10).

[0072] A holographic side mirror system (10) may include a side camera (110) that photographs the side and rear environment of a vehicle, a display module (300) that displays image information, and a tracking camera (140) that tracks the position and direction of the driver's head.

[0073] A side camera (110) can be installed on the side of the vehicle, i.e., in the area where a traditional side mirror is located. This camera can replace a conventional physical side mirror and collect visual information from outside the vehicle.

[0074] The display module (300) is positioned inside the vehicle and can visually display images collected from the side camera (110) to the driver. The display module may be installed in a location with easy access for the driver to easily view the image information. Preferably, it may be installed near a window on the A-pillar or the inside of the vehicle door.

[0075] Tracking cameras (140) may be installed on both sides of the vehicle interior to collect driver head information. Preferably, they may be located on the A-pillar, and at least two cameras may be positioned at different locations, along with an infrared camera (not shown), for distance calculation and 3D position tracking.

[0076] This system configuration aims to provide drivers with a safer and more efficient driving environment by converting the function of traditional side mirrors into holographic displays.

[0077] Below, the control method of the holographic side mirror system is described in detail.

[0078] FIG. 3 is a drawing for explaining a 3D image display method of a holographic side mirror system (10). FIG. 3 (a) is a drawing showing the relative positions of display modules (300) located on the driver's and passenger's side (hereinafter referred to as passenger's display in the description of FIG. 3). FIG. 3 (b) is a drawing schematically showing an image being output as a 3D holographic image from the display module (300). Hereinafter, the display refers to a display module.

[0079] As illustrated in Fig. 3 (a), the driver can look at the passenger seat display (300) while driving. The passenger seat display (300) can receive an image captured from the rear side of the passenger seat of the vehicle from a side camera (110) installed on the outer side of the vehicle on the passenger seat side, process the image, and then output the image. As illustrated in the drawing, the image that has undergone image processing at this time can be a 3D holographic image. The 3D holographic image can express a sense of distance by outputting different 3D layers depending on the distance from the object captured by the side camera (110).

[0080] For example, as shown in Fig. 3 (b), when expressed in 3D, the closest vehicle (3001) and road can be output at the front so that they feel close, the vehicle (3003, 3003) following behind can be output at the back so that they feel further away, and the background (3004) can be output at the back so that they feel farther away.

[0081] Figure 4 is a flowchart showing a control method of a holographic side mirror system.

[0082] First, the side camera (110) or distance sensor (120) can capture an external image of the vehicle and collect distance information (S410). The captured image and distance information can be transmitted to the controller. According to one embodiment of the present invention, the controller can receive the captured image and distance information and perform an image processing process (S420).

[0083] At this time, there may be differences in the image processing process depending on the presence or absence of distance information. This will be described in detail later in FIG. A holographic image can be generated through the image processing process in the controller (200).

[0084] The generated holographic image can be output from a display module installed inside the mobile device (S430). The display module may include both a configuration capable of outputting a general 2D image and a configuration capable of outputting a holographic 3D image.

[0085] Figure 5 is a flowchart for explaining the image processing process of the controller.

[0086] First, the controller can receive video or distance information captured from the input unit (S510). At this time, there may be cases where the distance sensor (120) is not included or the distance information is incomplete. Therefore, the controller determines whether the input information includes distance information (S520).

[0087] If distance information exists, a real-time hologram generation algorithm that converts the input image (i.e., RGB image) and distance information into amplitude and phase is executed (S530). If distance information does not exist, a real-time hologram generation algorithm that converts only the input image into amplitude and phase is executed (S540). Through the above algorithm operations, the converted phase and amplitude data can be obtained as a result.

[0088] The transformed phase and amplitude data can be encoded into phase-only data or amplitude-only data (S550). Since holographic images require only phase or amplitude data, a process of encoding them into a single data type is required.

[0089] Next, the encoded phase data or amplitude data as described above is transmitted to the display (S560). The transmitted data can be displayed as a 3D holographic image through the display's optical system (330).

[0090] Figure 6 is a flowchart for explaining the display mode switching process.

[0091] First, the controller can receive data, such as captured video or collected distance information (S610). The video data may be obtained from the vehicle's side camera (110). Such data can be transmitted to the next step and prepared for processing.

[0092] The controller can analyze the input image and data and execute a mode determination algorithm that determines the mode based on the analysis (S620). During this process, the controller analyzes the characteristics of the input image data and determines the display mode to be displayed on the screen. Details of the analysis algorithm for each data type are described below in Figures 7 through 10.

[0093] Based on the results of the above algorithm, the controller can make a decision to switch the display mode (S630). The display mode can be either a normal mode in which the image is displayed as a 2D image or a hologram mode in which the image is displayed as a 3D image using a hologram.

[0094] When the controller determines to switch the display mode, an image processing process may be performed according to the determined display mode (S640). That is, pre-processed or real-time processed image data may be generated to reconstruct the input captured image into an image to be ultimately displayed to the driver. For example, when switched to general mode, the image may be composed as a general 2D image, and an image processing process such as image processing for a clear image or a process for displaying additional information about an object on the screen may be performed. As another example, when switched to hologram mode, the captured image may be displayed as a 3D holographic image through an image processing process as described above in FIG. 5.

[0095] Finally, the controller transmits the converted image to the display module for display on the screen (S650). Ultimately, the driver can view the image through the display module. This process simplifies the complex image processing process in the holographic side mirror system, providing the driver with optimized visual information.

[0096] Figure 7 is a flowchart for explaining mode determination according to driving speed and image analysis.

[0097] First, the controller can determine the vehicle's driving speed (S710). This determination process can be performed based on information received from a sensor measuring the vehicle's driving speed. According to one embodiment of the present invention, the system can automatically change the display mode based on the speed.

[0098] As described above, after determining the speed, if the vehicle's driving speed exceeds a predetermined speed, the display mode can be switched to normal mode (S740). The predetermined speed is preferably 60 km / h, and if it exceeds this speed, the display mode can be switched to normal mode.

[0099] When the vehicle's driving speed exceeds a certain speed, the vehicle is moving quickly, so displaying the image of the side camera (110) in the normal mode, which is a 2D image, can be clearer and more helpful for the driver's instantaneous judgment. On the other hand, when the vehicle is moving at a high speed, the background also moves quickly, so if it is displayed in the hologram mode, which is a 3D image, there is a problem that the driver may not be able to quickly recognize it or may feel dizzy. Therefore, to prevent this, the display mode can be automatically changed according to the vehicle's speed.

[0100] If the vehicle's driving speed is below a predetermined speed, a step of analyzing the side camera (110) image may be performed (S720). For example, if the vehicle's driving speed is below 60 km / h, the image input to the side camera (110) may be analyzed to switch to normal mode or hologram mode.

[0101] After analyzing the camera footage, a mode switching decision algorithm can be executed based on the analysis (S730). For example, in situations where there are no vehicles or complexities within the side camera (110) footage, the footage can be displayed in normal mode (S740). Alternatively, in situations where there are many vehicles or objects within the side camera footage or the footage is complex, the footage can be displayed in hologram mode (S750). The mode switching decision algorithm will be described in detail later with reference to FIG. 10.

[0102] Figure 8 is a flowchart illustrating one embodiment of an automatic mode switching process.

[0103] A holographic side mirror system (hereinafter, “system”) according to one embodiment of the present invention can automatically switch display modes based on data input. FIG. 8 is a diagram illustrating a process for initiating mode switching in an automatic mode switching mode.

[0104] First, the system may be in automatic mode switching mode (S810). This mode can be activated by driver input or specific conditions, and the system enters an initial state for performing automated tasks.

[0105] In automatic mode switching, the system can first determine whether the vehicle is in reverse, stopped, or moving. If the vehicle is in reverse (S820), the display mode can be switched directly to hologram mode (S870). When driving in reverse, the reduced speed and increased attention demands can place a complex cognitive burden on the driver. In such situations, holographic technology can be used to provide distance information in 3D, enhancing the driver's ability to perceive the surroundings. The distance information displayed in three dimensions through holograms can help drivers more intuitively perceive the relative distance to objects behind them, contributing to a reduction in the risk of accidents.

[0106] Additionally, if it is determined that the vehicle is not moving backwards, it can be determined whether the vehicle is stopped (S830). At this time, the order of the backward and stopped determinations may be unrelated. If the vehicle is determined to be stopped, the driver's gaze information collected through a tracking camera installed inside the vehicle can be used to determine whether the driver is looking at the display for a predetermined period of time (S840). Preferably, the next step can be performed if the vehicle is stopped and the driver is looking at the display for more than one second. If the driver is not looking at the screen, the determination can be made again by returning to the beginning (S820).

[0107] If the driver is watching the screen, a mode switching decision algorithm can be executed (S860). The mode switching decision algorithm is described in detail in FIG. 10.

[0108] If the vehicle is not stopped, it can be determined to be in motion. While in motion, it can be determined whether the speed is N or higher (S850). If the speed is N or higher, the vehicle can automatically switch to normal mode as described above in Figure 8 (S880). The speed N can preferably be 60 km / h.

[0109] If the driving speed is less than N, it can be determined whether the driver is watching the screen for a predetermined period of time (S840). If the driver is watching the screen, a mode switching determination algorithm can be executed (S860). Finally, the system can execute the mode switching determination algorithm to switch to hologram mode (S870) or normal mode (S880). An embodiment of the mode switching determination algorithm is described in detail below.

[0110] Figure 9 is a diagram showing an example of an image analyzed during the automatic mode switching process.

[0111] The holographic side mirror system requires sophisticated environmental awareness, which means switching between standard and holographic modes based on the complexity of the surroundings. In simple traffic situations, such as when there are few vehicles or obstacles in the camera image, the system operates in default mode, providing a basic 2D image display (901). This mode provides the driver with the minimum necessary information, helping them maintain focus while driving.

[0112] On the other hand, in situations where complexity increases due to vehicles, pedestrians, or other objects within the camera image, the system should switch to holographic mode (902). Holographic mode provides the driver with detailed environmental information along with a sense of spatial depth, allowing them to perceive more variables and respond appropriately. This provides significant benefits to the driver, especially when performing maneuvers requiring careful attention, such as backing up or parking.

[0113] Accordingly, the image analysis unit (220) of the controller (200) can continuously analyze the complexity of the surrounding environment and, accordingly, support the driver in making optimal decisions in a given situation. This system's dynamic adjustment capabilities can maximize driver safety and vehicle operational efficiency.

[0114] Figure 10 is a flowchart illustrating an embodiment of a mode switching algorithm based on image analysis during an automatic mode switching process. The algorithm evaluates specific conditions based on input from a vehicle's side camera image to switch between normal mode and hologram mode.

[0115] First, as described above, the system can receive video input from the side camera (110) (S1010). The system can then go through an image analysis process to analyze the input video (S1020). The video can be analyzed, and a mode transition can be determined based on the analysis results.

[0116] The system can analyze the video to determine whether the vehicle is located indoors or outdoors (S1030). If the vehicle is located indoors, such as in an indoor parking lot, and not outdoors, the system can switch to hologram mode, as the environment requires high levels of attention and parking (S1080).

[0117] Next, it can be determined whether the lane is recognized within a predetermined range in the video environment (S1040). In the uncomplicated situation (901) of Fig. 10, the lane is recognized as long, and in the complex situation (902), part of the lane is obscured by the vehicle and thus invisible. Therefore, in a situation where the lane is obscured, i.e., a situation where the lane is not recognized within a predetermined range, the environment can be recognized as complex and switched to hologram mode (S1080).

[0118] Additionally, the number of pedestrians in the video environment can be determined (S1050). If the number of pedestrians is determined to be n or more, the environment is considered complex and can be switched to hologram mode (S1080). For example, if the number of pedestrians is three or more, the system can switch to hologram mode.

[0119] Finally, the number of vehicles and objects (such as obstacles) larger than a certain size can be determined within the video environment (S1060). For example, it can be determined whether the environment contains three or more vehicles and obstacles. If there are many vehicles or obstacles, the environment can be considered complex and switched to hologram mode (S1080).

[0120] If the conditions described above are not met and the environment is determined to be non-complicated, the system can switch to normal mode (S1070). Specifically, if the vehicle is located outdoors, lanes are recognized within a predetermined range, the number of pedestrians in the image is below a predetermined number, and the number of vehicles and obstacles is also below a predetermined number, the image can be displayed in normal mode.

[0121] The conditions described above are merely exemplary and are not intended to be limiting. They may be omitted or added. Furthermore, the judgment order may also be changed.

[0122] Figure 11 is a schematic diagram showing the configuration of a display module to explain the switching of a light source in hologram mode and normal mode.

[0123] Switching between hologram mode (1101) and normal mode (1102) may include switching the light source. The display module (300) may use a linear light source (310) in hologram mode and a diffuse light source in normal mode. That is, when switching from normal mode to hologram mode, the diffuse light source may be switched to a linear light source (310), and when switching from hologram mode to normal mode, the linear light source may be switched to a diffuse light source.

[0124] Fig. 11 (a) is a schematic diagram showing a structure for switching the physical positions of a straight light source (310) and a diffuse light source (320). Fig. 11 (b) is a schematic diagram showing a structure for switching on and off a straight light source (310) and a diffuse light source (320).

[0125] The display module may include an SLM display (340), an optical system (330), a directional light source (310), and a diffuse light source (320).

[0126] Referring to Fig. 11 (a), in the hologram mode (1101), a straight light source (310) is positioned at the location of the central light source, and a diffuse light source (320) can be positioned at an adjacent location that can be switched. The straight light source (310) emits precisely controlled light to form the basis of a holographic image. The emitted light passes through an optical system (330) including a diffuse lens (331). The diffuse lens (331) can evenly disperse the light output from the straight light source (310).

[0127] As light passes through the optical system (330), it is evenly distributed and the quality of light transmitted to the spatial light modulator (SLM) (340) is optimized. The spatial light modulator (340) modulates the phase or amplitude of the incoming light to generate a holographic image. Consequently, the driver can perceive the holographic image generated through the modulated light in three dimensions.

[0128] In the general mode (1102), the positions of the straight light source (310) and the diffuse light source (320) are switched so that the diffuse light source (320) can be positioned at the position of the central light source. At this time, the diffuse lens can move together with the straight light source (310). The diffuse light source (320) can allow light to reach the spatial light modulator (340) directly without passing through the diffuse lens. The spatial light modulator (340) does not modulate the phase and amplitude of specific light, and instead omits the generation of complex interference patterns for forming a holographic image, and can instead generate a general 2D image.

[0129] As a result, light emitted from the diffuse light source (320) can be converted into a 2D image form by the spatial light modulator (340) and displayed to the driver. This process meets general image display requirements and can be ideal for providing information to the driver in situations where the complexity of a holographic mode is not required.

[0130] Referring to Fig. 11 (b), a light source may be installed in a structure in which a plurality of diffuse light sources (320) are arranged around a straight-line light source (310). In this case, switching between the hologram mode (1101) and the general mode (1102) can be achieved by switching the straight-line light source (310) and the diffuse light source (320) on and off. That is, in the hologram mode (1101), the straight-line light source (310) is activated, the diffuse light source (320) is deactivated, and when switching to the general mode (1102), the straight-line light source (310) is deactivated, and the conversion light source (320) is activated.

[0131] Figure 12 is a flowchart for explaining the adjustment of the hologram generation direction according to the head direction of the holographic side mirror system.

[0132] A holographic side mirror system according to one embodiment of the present invention can detect and track the direction of a driver's head and gaze direction to adjust the viewing angle direction of a hologram.

[0133] First, the driver's head direction can be input using a tracking camera and an infrared camera (S1210). Referring to Fig. 1, the tracking cameras (140) can be positioned on both sides of the vehicle, and the location and distance of the head box can be calculated based on the information input from the tracking cameras on both sides (S1220). Through the above calculation, the driver's head direction can be detected and tracked (S1230). The infrared camera can be installed adjacent to the tracking camera, enabling continuous tracking in situations such as night driving.

[0134] At this time, it is determined whether the driver's head direction has not changed for more than a preset period of time (S1240). If the head direction has not changed in a certain direction, a step is taken to collect and calculate the driver's gaze direction information using a tracking camera (S1250).

[0135] Based on the result of collecting and calculating the gaze direction information, it is determined whether the gaze direction is within the hologram's field of view (S1260). Since the hologram mode uses straight light and has a narrow field of view for the driver to perceive the screen in three dimensions, it is necessary to align the driver's gaze direction with the hologram's generation direction. Therefore, if the driver's gaze direction is not within the hologram's field of view, the image output unit can change the hologram's generation direction of the display module so that the driver's gaze direction is included within the hologram's field of view (S1280).

[0136] If the driver's head direction changes, it can be determined whether the head direction is within the hologram's field of view (S1270). If the head direction is within the hologram's field of view, the driver's head direction detection and tracking continues (S1230). If the head direction is not within the hologram's field of view, the image output unit can adjust the hologram generation direction of the display module so that the driver's head direction is included within the hologram's field of view (S1280).

[0137] Tracking the driver's gaze direction only when the driver's head orientation remains unchanged for a certain period of time can reduce the system's processing load. This means the system tracks the driver's gaze direction based on static head position rather than continuously detecting head movement. This ensures system responsiveness while preventing unnecessary data processing, reducing power consumption and increasing overall system efficiency.

[0138] Figure 13 is an example diagram for explaining an example of head direction tracking and gaze direction tracking.

[0139] Figure 13 (a) is a schematic diagram of the configuration of a holographic side mirror system placed inside a vehicle, showing the essential elements for detecting the driver's head position and tracking its direction.

[0140] In a system according to one embodiment of the present invention, the "head box (160)" is defined as the space where the driver's head is positioned, and a tracking camera (140) can be utilized within this space to detect the driver's head movement and direction. The tracking camera monitors the driver's gaze and head direction in real time, and this data can be used to adjust a holographic display that reflects the driver's intentions and the surrounding environment.

[0141] Tracking cameras can be positioned on the left and right sides of the vehicle, preferably on the A-pillar. To clearly define the head box and enable more precise distance calculations, at least two cameras, such as infrared cameras, can be installed in different locations. This camera arrangement allows for the collection of data from multiple angles for distance measurement and 3D position tracking, providing a more accurate estimate of the driver's head position and orientation.

[0142] The display module (110) can be installed inside the vehicle adjacent to the side camera. Preferably, it can be located below the A-pillar or inside the vehicle door, to provide holographic information within the driver's natural line of sight. It can also be installed in other potential locations within the vehicle, such as the center fascia or instrument panel (cluster), and can be flexibly positioned according to driver preference and the specific vehicle interior configuration.

[0143] Fig. 13 (b)-1 and Fig. 13 (b)-2 illustrate examples of the driver's head direction within the head box (160). In a case such as Fig. 13 (b)-1, the head direction may be recognized as not being in the front direction with respect to the screen of the display module, and functions such as gaze tracking described above in Fig. 12 may not be executed. In the case of Fig. 13 (b)-2, the head direction may be recognized as being facing the display module frontally, and gaze tracking may be executed if there is no change in the head direction for a preset period of time or longer.

[0144] Figure 14 is a diagram illustrating an example of a user interface for switching holographic modes.

[0145] As described above, the holographic side mirror system according to one embodiment of the present invention can switch the display mode between normal mode and holographic mode. It may include an algorithm that analyzes various factors to automatically switch the display mode, and of course, manual switching is also possible. Figure 14 illustrates an example of a user interface for manual switching and a user interface that controls a personalized mode for the driver in holographic mode.

[0146] In one embodiment of the present invention, a user interface (hereinafter, driver input unit) (1420) may be placed in a space within the handle (1410). This is merely an example of a space that the driver can easily access to perform mode switching.

[0147] The driver input unit (1420) may include a mode change button (1421), a hologram personalization setting button (1422), etc.

[0148] For example, the mode change button (1421) may be an on-off button that can execute hologram mode change. The button may be used when the driver activates or deactivates the hologram mode and switches to the normal mode.

[0149] The hologram personalization setting button (1422) may be a button for adjusting, for example, the depth expression level of a hologram when in hologram mode. For example, the hologram depth expression level can be adjusted via a stick-type dial. The dial can be easily rotated to finely adjust the depth of the hologram, allowing the driver to adjust the depth of the hologram image to their personal preference even while driving. This configuration can contribute to providing a safer driving environment by improving the driver experience and enabling more effective use of holograms while driving.

[0150] Meanwhile, to allow the driver to intuitively perceive and adjust the depth expression level of the hologram, a display (1423) indicating the depth expression level may be placed next to the button. Preferably, the display may use various colors to visually indicate levels such as 2, 4, 8, 16, and 32, allowing the driver to easily identify the currently set depth expression level.

[0151] Figure 15 is a drawing for explaining an example of a personalized mode according to the driver in hologram mode.

[0152] As described above in FIG. 14, the holographic side mirror system according to one embodiment of the present invention can provide a personalized mode that allows the driver to change the personalized settings of the holographic image. The controller can control the personalized mode, which includes changing at least one of the following: changing the distance expression method for an object in the holographic image, changing the size, changing the depth, and changing the color, depending on the driver. In this case, the driver can be identified, and the holographic image can be output in the personalized mode according to the identified driver.

[0153] Figure 15 (a) schematically illustrates an example of a personalized mode, where the distance between the driver and the hologram is set to a preferred distance for the driver. For example, the distance from the hologram image can be set to 40 cm, 45 cm, or 50 cm, depending on the driver.

[0154] Figure 15 (b) schematically illustrates an example of a personalized mode, in which the depth level (i.e., the number of layers) of a holographic image is set. The driver can then determine the number of layers to use based on distance. The drawing illustrates an example of four layers.

[0155] Figure 15 (c) schematically illustrates one example of a personalized mode, where the size of an object in a holographic image is set. The driver can set the holographic image to their preferred size. This allows the driver to easily interpret visual information and more accurately perceive the vehicle's surroundings.

[0156] Figure 15 (d) schematically illustrates one example of a personalized mode, where color expression settings can be changed to distinguish distances in a holographic image. For example, closer objects can be set to appear darker or redder. By changing the color settings, distance distinctions can be made clearer, allowing drivers to easily recognize their relative positions with respect to surrounding objects.

[0157] These features could include the ability to recognize a driver's individual profile and automatically apply optimal settings. As soon as the driver enters the car, the system can retrieve and instantly apply their previously stored preferences. This reduces the driver's need to manually adjust settings each time, contributing to a safer and more comfortable driving environment.

[0158] It will be apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit and essential characteristics of the present invention.

[0159] The above detailed description should not be construed as limiting in any respect and should be considered illustrative only. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included within the scope of the present invention.

Claims

1. A side camera installed on the side of the mobile device to capture video; A display module installed inside the above mobile device and outputting the image; and A controller that relays images between the side camera and the display module Including, The above controller Generate a holographic image based on the image captured by the above side camera, The above display module Characterized in that it can output the above holographic image Holographic side mirror system.

2. In paragraph 1, The above controller, Receive video from the above side camera, The input image is converted into phase and amplitude data through a real-time hologram generation algorithm. Characterized in that the hologram image is generated by encoding the phase and amplitude data into phase or amplitude data. Holographic side mirror system.

3. In paragraph 1, Further comprising a distance sensor installed adjacent to the side camera, The above controller Receive video from the above side camera Receive distance information from the above distance sensor, The input image and distance information are converted into phase and amplitude data through the operation of a real-time hologram generation algorithm, Characterized in that the hologram image is generated by encoding the phase and amplitude data into single phase or amplitude data. Holographic side mirror system.

4. In paragraph 1, The above controller Switching the image displayed on the above display module to either a hologram mode that displays the image as a 3D image or a normal mode that displays the image as a 2D image, The above display module In the above hologram mode, a straight light source is used, The above general mode is characterized by using a diffuse light source. Holographic side mirror system.

5. In paragraph 4, Switching between the above hologram mode and the above normal mode characterized by including a position change of the above straight light source and the above diffuse light source. Holographic side mirror system.

6. In paragraph 4, Switching between the above hologram mode and the above normal mode It is characterized by including on-off switching of the straight light source and the diffuse light source arranged surrounding the straight light source. Holographic side mirror system.

7. In paragraph 4, The above controller Analyze at least one factor among the driving speed of the above-mentioned mobile device or the analysis content of the input image, By the above analysis, it is determined whether the mobile device is in a first environment in which movement must be taken care of or a second environment distinct from the first environment, If the above mobile device is in the first environment, it switches to hologram mode, When the above mobile device is in a second environment, it is characterized by switching to a normal mode, The analysis content of the input video above is Characterized in that it includes analysis of at least one of the number of moving devices, number of pedestrians, lanes, whether indoor or outdoor, and number of objects in the above image. Holographic side mirror system.

8. In paragraph 1, It further includes a tracking camera installed inside the mobile device to collect the driver's head information, The above controller Using the driver's head information collected above, the driver's head direction is calculated, Characterized in that the direction of the light source of the display module is controlled so that the direction of the driver's head is included within the field of view of the holographic image. Holographic side mirror system 9. In paragraph 8, The above tracking camera If the driver's head direction does not change for more than a preset period of time. It is characterized by collecting information on the driver's gaze direction, The above controller Using the above collected gaze direction information, the driver's gaze direction is calculated, It is characterized in that the direction of the light source of the display module is controlled so that the direction of the driver's gaze is included within the field of view of the holographic image. Holographic side mirror system.

10. In paragraph 1, The above controller It is characterized in that it controls in a personalized mode including at least one change among a change in the distance expression method for an object of the holographic image, a change in size, a change in depth, and a change in color according to the user. Holographic side mirror system.

11. Step of shooting video using the side camera installed on the side of the mobile device; A step of generating a hologram image by image processing the above-described captured image in the controller; A step of outputting the hologram image from a display module installed inside the mobile device; Control method of holographic side mirror system.

12. In paragraph 11, It is characterized by further including a step of switching the image displayed in the display module to one of a hologram mode that displays the image as a 3D image or a general mode that displays the image as a 2D image. Control method of holographic side mirror system.

13. In paragraph 12, The above conversion steps are A step of analyzing at least one factor among the driving speed of the mobile device or the analysis contents of the input image in the controller; A step of determining whether the mobile device is in a first environment in which movement must be taken care of or a second environment distinct from the first environment through the above analysis; A step of switching to hologram mode when the above-mentioned mobile device is in a first environment; characterized in that it includes a step of switching to a normal mode when the above mobile device is in a second environment; The analysis content of the input video above is Characterized in that it includes analysis of at least one of the number of moving devices, number of pedestrians, lanes, whether indoor or outdoor, and number of objects in the above image. Control method of holographic side mirror system.

14. In paragraph 11, Step where the tracking camera collects the driver's head information; A step in which the controller calculates the driver's head direction using the driver's head information collected above; A step of controlling the direction of the light source of the display module so that the direction of the driver's head is included within the field of view of the holographic image; A step of collecting information on the driver's gaze direction through the tracking camera when the driver's head direction does not change for a preset period of time; A step in which the controller calculates the driver's gaze direction using the collected gaze direction information; A step of controlling the direction of the light source of the display module so that the driver's line of sight direction is included within the field of view of the holographic image; characterized in that it includes; Control method of holographic side mirror system.

15. In paragraph 11, Steps for identifying the user; and A step of outputting the holographic image in a personalized mode according to the identified user, wherein the personalized mode is It is characterized by including at least one change among a change in the distance expression method for an object of the above holographic image, a change in size, a change in depth, and a change in color. Control method of holographic side mirror system.

Images