Method and apparatus for providing user interface

Abstract

Disclosed is a method, performed by a computing device, for providing a user interface of a vehicle surrounding environment monitoring system. The method may include the steps of: acquiring driver gaze direction information; and displaying, on the basis of the driver gaze direction information, at least one of a plurality of vehicle environment images on a vehicle environment image display unit.

Description

Method and device for providing a user interface

[0001] The present invention relates to a user interface, and more particularly, to a vehicle user interface utilizing eye tracking technology.

[0002] While the practice of handing over driving tasks to computers is becoming a reality, the process is becoming increasingly complex and carries with it potential risks. Despite their convenience, autonomous vehicles must operate safely and reliably under real-world road conditions. The Driver Monitoring System (DMS) is considered a key safety technology in the automotive industry for the development of semi-autonomous and autonomous vehicles.

[0003] In 2017, the National Transportation Safety Board recommended installing driver monitoring systems (DMS) in autonomous and semi-autonomous vehicle testing to improve safety and minimize accidents related to backup driver fatigue, concentration, and distraction. Furthermore, with Euro NCAP including DMS as a key safety feature in its "Road Map 2025" from 2020, the adoption of this feature is expected to expand further.

[0004] DMS is a system that monitors and evaluates the behavior and condition of vehicle drivers. DMS can be primarily used in autonomous vehicles and commercial vehicles to enhance vehicle safety technology and driver security.

[0005] DMS collects and analyzes various information by monitoring the driver's status and behavior through, for example, facial recognition technology, eye tracking technology, blink detection technology, driver movement detection technology, voice recognition technology, and biometric recognition technology.

[0006] Surround View Monitoring (SVM) systems primarily monitor a vehicle's surroundings and provide visual information to the driver. SVM systems provide a 360-degree view of obstacles, parking spaces, and the surrounding environment, making driving and parking safer and more convenient. First unveiled at the 2007 Tokyo Motor Show, SVM systems are expected to continue their growth in the global market.

[0007] DMS and SVM technologies are part of advanced driver assistance systems (ADAS) designed to enhance vehicle and driver safety and convenience. Both systems aim to enhance vehicle safety and driver convenience. While both systems share the goal of enhancing vehicle safety, they function differently. DMS focuses on monitoring the driver's condition and detecting and preventing dangerous driving situations, while SVM provides visual information about the vehicle's surroundings, assisting the driver's vision during parking and low-speed driving. Collectively, these two systems can work collaboratively to enhance driver safety and convenience.

[0008] Therefore, there is a demand for technology that links the two systems.

[0009] [Prior Art Literature]

[0010] (Patent Document 0001) Republic of Korea Patent No. 10-2131717

[0011] The present disclosure seeks to provide an improved user interface for a vehicle surrounding environment monitoring system, such as an SVM system, using DMS technology.

[0012] A method for providing a user interface of a vehicle environment monitoring system performed by a computing device for realizing the aforementioned task is disclosed. The method may include: a step of obtaining driver gaze direction information; and a step of displaying at least one of a plurality of vehicle environment images on a vehicle environment image display unit based on the driver gaze direction information.

[0013] Alternatively, the step of displaying at least one of a plurality of vehicle environment images on the vehicle environment image display unit based on the driver gaze direction information may include: a step of displaying an image of a camera corresponding to the driver gaze direction information among a plurality of vehicle environment cameras on the vehicle environment image display unit;

[0014] Alternatively, the step of displaying at least one of the plurality of vehicle environment images on the vehicle environment image display unit based on the driver gaze direction information may include: displaying at least one of the plurality of vehicle environment images on a display corresponding to the driver gaze direction information among a plurality of displays installed at different locations;

[0015] Alternatively, the step of displaying at least one of the plurality of vehicle environment images on the vehicle environment image display unit based on the driver gaze direction information may include: displaying at least one of the plurality of vehicle environment images on a display area corresponding to the driver gaze direction information among the plurality of display areas of the vehicle environment image display unit;

[0016] Alternatively, the step of obtaining the driver gaze direction information may include: a step of identifying trigger information; a step of obtaining gaze tracking information in response to the identification of the trigger information; and a step of determining the driver gaze direction information based on the obtained gaze tracking information.

[0017] Alternatively, the trigger information may include at least one of voice information, gesture information, vehicle driving information, or vehicle location information.

[0018] Alternatively, the step of acquiring the gaze tracking information may include: acquiring the gaze tracking information during a gaze tracking information acquisition period corresponding to the trigger information;

[0019] Alternatively, the step of acquiring gaze tracking information during the gaze tracking information acquisition period corresponding to the trigger information may include: a step of identifying that the trigger information is short-term gaze direction determination information; and a step of acquiring gaze tracking information at a time corresponding to the short-term gaze direction determination information.

[0020] Alternatively, the trigger information identified by the short-term gaze direction determination information may include at least one of predetermined voice information or gesture information that operates the vehicle surrounding environment monitoring system using gaze tracking information at the time of identification of the trigger information.

[0021] Alternatively, the step of acquiring gaze tracking information during a gaze tracking information acquisition period corresponding to the trigger information may include: a step of identifying that the trigger information is long-term gaze direction determination information; and a step of acquiring a representative value for gaze tracking information of a period corresponding to the long-term gaze direction determination information.

[0022] Alternatively, the trigger information, which is the long-term gaze direction determination information, may include vehicle driving information or vehicle position information that operates the vehicle surrounding environment monitoring system using gaze tracking information that is continuously acquired in a time series manner.

[0023] A computer program stored on a computer-readable storage medium for achieving the above-described task is disclosed. When the computer program is executed on one or more processors, the computer program performs the following operations for providing a user interface for a vehicle surrounding environment monitoring system, wherein the operations may include: obtaining driver gaze direction information; and displaying at least one of a plurality of vehicle surrounding images on a vehicle surrounding image display based on the driver gaze direction information.

[0024] A computing device providing a user interface for a vehicle environment monitoring system for realizing the aforementioned task is disclosed. The computing device includes: a memory including computer-executable components; a processor executing the following computer-executable components stored in the memory; wherein the processor is capable of obtaining driver gaze direction information and displaying at least one of a plurality of vehicle environment images on a vehicle environment image display based on the driver gaze direction information.

[0025] The present disclosure can provide an improved user interface for a vehicle surrounding environment monitoring system, such as an SVM system, using DMS technology.

[0026] FIG. 1 is a block diagram of a vehicle surrounding environment monitoring system according to some embodiments of the present disclosure.

[0027] Figure 2a is a drawing for explaining driver gaze direction information.

[0028] Figure 2b is another drawing for explaining driver gaze direction information.

[0029] Figure 3 illustrates an exemplary vehicle surrounding environment display unit that is operated according to the driver's gaze direction.

[0030] Figure 4 is a diagram illustrating an operation of manipulating multiple display areas of a vehicle environment image display unit that displays a vehicle environment image according to the driver's gaze direction. Figure 5 illustrates an exemplary vehicle environment image.

[0031] FIG. 6 is a flowchart of a method for providing a user interface of a vehicle surrounding environment monitoring system according to some embodiments of the present disclosure.

[0032] FIG. 7 is a block diagram of a computing device according to some embodiments of the present disclosure.

[0033] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to facilitate understanding of the present disclosure. However, it will be apparent that these embodiments may be practiced without these specific details.

[0034] As used herein, the terms "component," "module," "system," and the like refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and / or a thread of execution. A component may be localized within a single computer. A component may be distributed between two or more computers. Furthermore, these components may execute from various computer-readable media having various data structures stored therein. Components may communicate via local and / or remote processes, for example, by signals comprising one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and / or data transmitted to another system via a network such as the Internet via signals).

[0035] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean either of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, "X employs A or B" can apply to any of these cases. Furthermore, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated items listed.

[0036] Additionally, the terms "comprises" and / or "comprising" should be understood to imply the presence of the features and / or components in question. However, it should be understood that the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other features, components, and / or groups thereof. Furthermore, unless otherwise specified or clear from the context to refer to the singular form, the singular in the specification and claims should generally be construed to mean "one or more."

[0037] And, the term "at least one of A or B" should be interpreted to mean "if it includes only A", "if it includes only B", or "if it is combined in the composition of A and B".

[0038] Those skilled in the art should further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0039] The description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present invention is not limited to the embodiments disclosed herein. The present invention is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

[0040] FIG. 1 is a block diagram of a vehicle surrounding environment monitoring system (1000) according to some embodiments of the present disclosure.

[0041]

[0042] *In some examples, the vehicle surrounding environment monitoring system (1000) may be a system that provides convenience to the driver by collecting and processing various information about the vehicle surrounding environment. For example, the vehicle surrounding environment monitoring system (1000) may include a surround view monitoring (SVM) system or an around view monitoring (AVM) system that monitors the vehicle surrounding environment and provides visual information to the driver. For example, the vehicle surrounding environment monitoring system (1000) may provide a 2D or 3D surround view image that appears to look down on the 360-degree situation around the vehicle from above by combining multiple vehicle environment images acquired by the vehicle environment camera unit (300). In addition, the vehicle surrounding environment monitoring system (1000) may include a driver monitoring system (DMS) that monitors the driver's status and behavior through facial recognition technology, eye tracking technology, eye blink detection technology, driver movement detection technology, voice recognition technology, biometrics technology, etc. The vehicle surrounding environment monitoring system (1000) can assist the driver by detecting drowsiness or loss of attention of the driver using information about the driver's condition and behavior obtained through facial recognition technology, eye tracking technology, eye blink detection technology, driver movement detection technology, voice recognition technology, biometric recognition technology, etc.

[0043] In some examples, as illustrated in FIG. 1, the computing device (100) may include a control unit (100), an eye tracking unit (200), a vehicle environment camera unit (300), a vehicle environment image display (400), and a trigger information identification unit (500). The configuration of the vehicle surrounding environment monitoring system (1000) illustrated in FIG. 1 is merely a simplified example. In some embodiments of the present disclosure, the vehicle surrounding environment monitoring system (1000) may include other configurations for implementing the SVM and DVM functions of the vehicle surrounding environment monitoring system (1000), and only some of the disclosed configurations may constitute the vehicle surrounding environment monitoring system (1000).

[0044] The control unit (100) can typically process the overall operation of the vehicle surrounding environment monitoring system (1000). The control unit (100) can process signals, data, information, etc. input or output through components of the vehicle surrounding environment monitoring system (1000) or run application programs to provide or process appropriate information or functions to the user.

[0045] According to some embodiments of the present disclosure, the control unit (100) may provide a user interface for a vehicle environment monitoring system. For example, the control unit (100) may obtain driver gaze direction information and provide a user interface that displays at least one of a plurality of vehicle environment images on a vehicle environment image display unit (400) based on the driver gaze direction information. Exemplary operations of the control unit (100) for providing the user interface are described with reference to FIGS. 2 to 5 .

[0046] The eye tracking unit (200) can obtain driver gaze direction information indicating the driver's gaze direction by processing gaze tracking information that tracks the driver's gaze at a certain point in time and for a certain period of time. The driver gaze direction information obtained by the body tracking unit (200) can be used as input for a user interface. In some examples, the eye tracking unit (200) may be a camera-based eye tracking device that obtains driver gaze direction information by tracking the driver's pupils and gaze using a camera. In another example, the eye tracking unit (200) may be an infrared laser-based eye tracking device that tracks eye movements by shining an infrared laser on the pupils. In yet another example, the eye tracking unit (200) may be a device that obtains driver gaze direction information by detecting the direction of the driver's face using a camera. However, the present invention is not limited thereto, and the eye tracking unit (200) can obtain driver gaze direction information in various ways.

[0047] The vehicle environment camera unit (300) may include one or more cameras or sensing devices that capture images of the vehicle's surroundings. In some examples, the vehicle environment camera unit (300) may include multiple cameras installed to capture images of the vehicle's four directions: front, rear, left, and right. The multiple vehicle environment images acquired by the vehicle environment camera unit (300) may be reconstructed by the control unit (100) in consideration of the shape and space of the vehicle, thereby being synthesized into a surround view image of the vehicle viewed from the sky. The vehicle environment camera unit (300) may be composed of various types of 2D and 3D cameras and sensing devices, or a combination thereof.

[0048] The vehicle environment display unit (400) may include one or more displays mounted inside the vehicle surrounding environment monitoring system (1000). The vehicle environment display unit (400) may display various information of the vehicle surrounding environment monitoring system (1000). For example, the vehicle environment display unit (400) may display a vehicle environment image captured around the vehicle, a surround view image generated by combining multiple vehicle environment images, etc. The vehicle environment display unit (400) may display an image of the vehicle surroundings in a display manner determined through the user interface of the vehicle surrounding environment monitoring system (1000), as described with reference to FIGS. 2 to 5 . However, the present invention is not limited thereto, and the vehicle environment display unit (400) may be implemented in various manners.

[0049] The trigger information identification unit (500) can detect various information used as input for the user interface along with driver gaze direction information. For example, the trigger information identification unit (500) can identify voice information, gesture information, vehicle driving information, or vehicle location information as trigger information. The trigger information identification unit (500) can be implemented through DMS technology. For example, the trigger information identification unit (500) can obtain information monitoring the driver through facial recognition technology, gaze tracking technology, eye blink detection technology, driver movement detection technology, voice recognition technology, biometrics technology, etc. In addition, the trigger information identification unit (500) can obtain information related to vehicle operation, such as vehicle driving information or vehicle location information, by using an electronic device related to vehicle operation. The trigger information obtained by the trigger information identification unit (500) can be used as input for the user interface of the vehicle surrounding environment monitoring system (1000) along with driver gaze direction information.

[0050] Hereinafter, exemplary operations of a control unit (100) for providing a user interface of a vehicle surrounding environment monitoring system (1000) are described with reference to FIGS. 2 to 5.

[0051] Fig. 2a is a diagram illustrating driver gaze direction information. Fig. 2b is another diagram illustrating driver gaze direction information. Fig. 3 illustrates an exemplary vehicle surrounding environment display unit (400) operated according to the driver gaze direction. Fig. 4 is a diagram illustrating an operation of operating multiple display areas of a vehicle environment image display unit (400) that displays a vehicle environment image according to the driver gaze direction. Fig. 5 illustrates an exemplary vehicle environment image.

[0052] According to some embodiments of the present disclosure, the control unit (100) may obtain driver gaze direction information through the gaze tracking unit (200). In this case, the control unit (100) may display at least one of a plurality of vehicle environment images on the vehicle environment image display unit based on the driver gaze direction information.

[0053] Specifically, the control unit (100) may provide a user interface having a function of operating the vehicle environment display unit (400) using the driver's gaze direction information as input. In some examples, the driver's gaze direction information may be information indicating the direction the driver looks at inside the vehicle through the gaze tracking unit (200). For example, as illustrated in FIG. 3A, the driver's gaze direction information may be information indicating that the driver looks at the right side mirror of the vehicle. In this case, the control unit (100) may operate the vehicle environment image display unit (400) to display information necessary when the driver looks at the right side mirror of the vehicle among a plurality of vehicle environment images. As another example, as illustrated in FIG. 3B, the driver's gaze direction information may be information indicating that the driver looks at the left side mirror of the vehicle. In this case, the control unit (100) may operate the vehicle environment image display unit (400) to display information necessary when the driver looks at the left side mirror of the vehicle among a plurality of vehicle environment images. Hereinafter, specific embodiments are described in which a control unit (100) provides a user interface for displaying at least one of a plurality of vehicle environment images on a vehicle environment image display unit (400) based on driver gaze direction information.

[0054] According to some embodiments of the present disclosure, the control unit (100) can display an image of a camera corresponding to driver gaze direction information among a plurality of vehicle environment cameras on the vehicle environment image display unit (400).

[0055] Specifically, the control unit (100) can determine an image corresponding to driver gaze direction information from among a plurality of vehicle environment images. In some examples, the plurality of vehicle environment images may include four images (310, 320, 330, 340) acquired by a vehicle environment camera unit (300) including a plurality of vehicle environment cameras installed to capture four directions of the front, rear, left, and right of the vehicle, as illustrated in FIG. 5. In another example, the plurality of vehicle environment images may include four images that display one of four areas (front, rear, left, and right areas of the vehicle) on a surround view generated by combining the four images (310, 320, 330, 340) acquired by the vehicle environment camera unit (300). In this case, the control unit (100) can determine an image of a camera positioned in the direction the driver is looking (or an area on the surround view) as an image of a camera corresponding to the driver gaze direction information. For example, as illustrated in FIG. 3A, driver gaze direction information may be information indicating that the driver is looking at the vehicle's right side mirror. In this case, the control unit (100) may determine the image (310) of the camera positioned to the right of the driver as the image of the camera corresponding to the driver gaze direction information. Furthermore, the control unit (100) may operate the vehicle environment image display unit (400) to display the image (310) of the camera positioned to the right of the driver.

[0056] Referring to FIG. 3, examples of the control unit (100) operating the vehicle environment display unit (400) to display an image of a camera positioned in the direction viewed by the driver among a plurality of vehicle environment cameras are illustrated. For example, the vehicle environment image display unit (400) indicated by '400-1' displays an image (330) of a camera positioned in front of the driver. In addition, the vehicle environment image display unit (400) indicated by '400-2' displays an image (320) of a camera positioned in the left direction of the driver. In addition, the vehicle environment image display unit (400) indicated by '400-3' displays an image (330) of a camera positioned in the right direction of the driver. In addition, the vehicle environment image display unit (400) indicated by '400-4' displays an image (340) of a camera positioned in the rear direction of the driver.

[0057] In some other examples, the control unit (100) may operate the vehicle environment display unit (400) to display an image from a camera positioned in a direction other than the direction in which the driver is looking, among the plurality of vehicle environment cameras. For example, the control unit (100) may operate the vehicle environment display unit (400) to display an image from a camera positioned in a direction opposite to the direction in which the driver is looking, among the plurality of vehicle environment cameras.

[0058] Additionally, instead of an image from a single camera, the control unit (100) may determine images from two cameras corresponding to the driver's gaze direction information as corresponding to the driver's gaze direction information. For example, the control unit (100) may determine an image from one of the left or right cameras positioned in the direction the driver is looking, and an image from one of the front or rear cameras, as images from the two cameras corresponding to the driver's gaze direction information. In addition, the control unit (100) may operate the vehicle environment image display unit (400) to simultaneously display images from the two cameras corresponding to the driver's gaze direction information.

[0059] According to some embodiments of the present disclosure, the control unit (100) may display at least one of a plurality of vehicle environment images on a display corresponding to driver gaze direction information among a plurality of displays installed at different locations of the vehicle environment image display unit (400).

[0060] In some examples, the vehicle environment image display unit (400) may include multiple displays installed in different locations. For example, the vehicle environment image display unit (400) may include a center display located in the center of the dashboard located at the front of the vehicle, a left display located on the left side of the dashboard or around the left side mirror, and a right display located on the right side of the dashboard or around the right side mirror. In this case, the control unit (100) may determine a display located in the direction the driver is looking from among the multiple displays as the display corresponding to the driver's gaze direction information. In another example, the control unit (100) may determine a display located in the opposite direction from the driver's gaze direction from among the multiple displays as the display corresponding to the driver's gaze direction information. However, the present invention is not limited thereto, and the control unit (100) may determine a display area corresponding to the driver's gaze direction information in various ways.

[0061] The control unit (100) may operate the vehicle environment display unit (400) to display a vehicle environment image through a display corresponding to the driver's gaze direction information. In this case, the image displayed through the display corresponding to the driver's gaze direction information may include an image of a camera corresponding to the driver's gaze direction information described above. For example, the control unit (100) may operate the vehicle environment display unit (400) to display an image of a camera positioned in the opposite direction from the driver's gaze direction on a display positioned in the direction the driver is looking. However, the present invention is not limited thereto, and the control unit (100) may operate the vehicle environment display unit (400) to display various images through the display corresponding to the driver's gaze direction information.

[0062] According to some embodiments of the present disclosure, the control unit (100) may display at least one of the plurality of vehicle environment images on a display area corresponding to the driver's gaze direction information among the plurality of display areas of the vehicle environment image display unit (400).

[0063] Specifically, the vehicle environment image display unit (400) may be a display having multiple display areas, each of which can display a vehicle environment image. Referring to FIG. 4, the vehicle environment image display unit (400) may be a single display mounted across the left and right sides of the dashboard. Here, the vehicle environment image display unit (400) may include a central display area (410), a left display area (420), and a right display area (430). In this case, the control unit (100) may determine a display area positioned in the direction the driver is looking from among the multiple display areas as the display area corresponding to the driver's gaze direction information. As another example, the control unit (100) may determine a display area positioned in the opposite direction from the driver's gaze direction from among the multiple display areas as the display area corresponding to the driver's gaze direction information. However, the present invention is not limited thereto, and the control unit (100) may determine the display area corresponding to the driver's gaze direction information in various ways.

[0064] The control unit (100) may operate the vehicle environment display unit (400) to display a vehicle environment image through a display area corresponding to the driver's gaze direction information. In this case, the image displayed through the display area corresponding to the driver's gaze direction information may include an image of a camera corresponding to the driver's gaze direction information described above. For example, the control unit (100) may operate the vehicle environment display unit (400) to display an image of a camera positioned in the opposite direction from the driver's gaze direction on a display area positioned in the direction the driver is looking. However, the present invention is not limited thereto, and the control unit (100) may operate the vehicle environment display unit (400) to display various images through the display area corresponding to the driver's gaze direction information.

[0065] According to some embodiments of the present disclosure, the trigger information identification unit (500) can identify trigger information. Furthermore, the gaze tracking unit (200) can acquire gaze tracking information in response to the identification of the trigger information. Furthermore, the gaze tracking unit (200) can determine driver gaze direction information based on the acquired gaze tracking information.

[0066] In order to prevent malfunction of the vehicle surrounding environment monitoring system (1000), the control unit (100) may operate the vehicle surrounding environment display unit (400) using driver gaze direction information when trigger information is identified. Specifically, since the driver observes the vehicle surroundings while driving, the driver's actions used as inputs for the user interface of the vehicle surrounding environment monitoring system (1000) and the driver's general actions while driving need to be distinguished from each other. To this end, the control unit (100) may operate the vehicle surrounding environment display unit (400) using driver gaze tracking information only when trigger information is identified, thereby preventing malfunction.

[0067] According to some embodiments of the present disclosure, the trigger information may include at least one of voice information, gesture information, vehicle driving information, or vehicle location information.

[0068] Specifically, the trigger information identification unit (500) can detect various information used as input for the user interface along with driver gaze direction information. For example, the trigger information identification unit (500) can obtain driver monitoring information through facial recognition technology, eye tracking technology, eye blink detection technology, driver movement detection technology, voice recognition technology, biometric recognition technology, etc. In addition, the trigger information identification unit (500) can obtain information related to vehicle operation, such as vehicle driving information or vehicle location information, using an electronic device related to vehicle operation. The trigger information can be defined to perform a specific operation of the vehicle surrounding environment monitoring system (1000) according to information that can be identified from voice information, gesture information, vehicle driving information, or vehicle location information. For example, when the trigger information identification unit (500) identifies voice information such as “SVM image automatic selection mode,” the control unit (100) can display an image corresponding to the driver gaze direction information through the vehicle environment display (400). However, the present invention is not limited thereto, and the trigger information can be defined in various ways.

[0069] According to some embodiments of the present disclosure, the gaze tracking unit (200) may acquire gaze tracking information during a gaze tracking information acquisition period corresponding to the trigger information in response to identification of trigger information.

[0070] Specifically, the control unit (100) can operate the eye tracking unit (200) to acquire driver gaze direction information at a specific point in time and for a specific period in response to the identification of trigger information. Since the driver gaze direction information may vary depending on the point in time and period of tracking the driver's gaze, the eye tracking unit (200) can prevent malfunction by acquiring gaze tracking information at a point in time and for a specific period defined by individual trigger information.

[0071] For example, the gaze tracking unit (200) can identify that the trigger information is short-term gaze direction determination information. For example, the trigger information identified as short-term gaze direction determination information may include at least one of predetermined voice information or gesture information that operates the vehicle surrounding environment monitoring system (1000) using eye tracking information at the time of identification of the trigger information (or for a short period (e.g., 0.1 second)). For example, the trigger information identified as short-term gaze direction determination information may be predetermined voice information such as “now.” When the eye tracking unit (200) identifies the voice information “now” as the trigger information, the eye tracking information may be acquired at a time corresponding to the short-term gaze direction determination information. In some examples, the time corresponding to the short-term gaze direction determination information may mean a time at which the eye tracking unit (200) can immediately acquire the eye tracking information after identifying the trigger information. For example, the eye tracking unit (200) may determine the gaze direction determined by the eye tracking information acquired earliest after identifying the trigger information (or the eye tracking information acquired for a relatively short period of time) as the driver gaze direction information. Therefore, the control unit (100) may use the driver gaze direction information at a time desired by the user. A user interface can be provided that instantly displays the vehicle environment image desired by the user.

[0072] As another example, the eye tracking unit (200) may identify that the trigger information is long-term gaze direction determination information. For example, the trigger information, which is long-term gaze direction determination information, may include vehicle driving information or vehicle position information that operates the vehicle surrounding environment monitoring system using gaze tracking information continuously acquired in time series. For example, the trigger information identified as short-term gaze direction determination information may include vehicle driving information indicating that the vehicle is parked (e.g., the vehicle is driven for a predetermined period of time at a predetermined speed or lower) or vehicle position information (e.g., the parking lot is identified through GPS information or image analysis), or a combination thereof. When the eye tracking unit (200) identifies vehicle driving information or vehicle position information identified as long-term gaze direction determination information as the trigger information, the eye tracking information for a period corresponding to the long-term gaze direction determination information may be continuously acquired. In some examples, the period corresponding to the long-term gaze direction determination information may be set to various periods, such as 5 seconds or 10 seconds. In this case, the gaze tracking unit (200) can determine the gaze direction determined by a representative value (e.g., an average value, etc.) of the gaze tracking information acquired during a period corresponding to the long-term gaze direction determination information as the driver gaze direction information. Accordingly, the vehicle surrounding environment monitoring system (1000) can provide a user interface that displays a vehicle environment image desired by the user by utilizing the driver gaze direction information that changes over a certain period of time depending on the vehicle's location or driving status.

[0073] FIG. 6 is a flowchart of a method for providing a user interface of a vehicle surrounding environment monitoring system (1000) according to some embodiments of the present disclosure.

[0074] According to some embodiments of the present disclosure, a method for providing a user interface of a vehicle surrounding environment monitoring system may include a step (s100) of obtaining driver gaze direction information.

[0075] According to some embodiments of the present disclosure, a method for providing a user interface of a vehicle surrounding environment monitoring system may include a step of displaying at least one of a plurality of vehicle surrounding environment images on a vehicle surrounding image display unit based on the driver gaze direction information.

[0076] Alternatively, the step (s200) of displaying at least one of a plurality of vehicle environment images on the vehicle environment image display unit based on the driver gaze direction information may include a step of displaying an image of a camera corresponding to the driver gaze direction information among a plurality of vehicle environment cameras on the vehicle environment image display unit.

[0077] Alternatively, the step (s200) of displaying at least one of the plurality of vehicle environment images on the vehicle environment image display unit based on the driver gaze direction information may include a step of displaying at least one of the plurality of vehicle environment images on a display corresponding to the driver gaze direction information among a plurality of displays installed at different locations.

[0078] Alternatively, the step (s200) of displaying at least one of a plurality of vehicle environment images on a vehicle environment image display unit based on the driver gaze direction information may include a step of displaying at least one of the plurality of vehicle environment images on a display area corresponding to the driver gaze direction information among a plurality of display areas of the vehicle environment image display unit.

[0079] Alternatively, the step (s100) of obtaining the driver gaze direction information may include a step of identifying trigger information, a step of obtaining gaze tracking information in response to the identification of the trigger information, and a step of determining the driver gaze direction information based on the obtained gaze tracking information.

[0080] Alternatively, the trigger information may include at least one of voice information, gesture information, vehicle driving information, or vehicle location information.

[0081] Alternatively, the step of acquiring the gaze tracking information may include a step of acquiring the gaze tracking information during a gaze tracking information acquisition period corresponding to the trigger information.

[0082] Alternatively, the step of acquiring gaze tracking information during a gaze tracking information acquisition period corresponding to the trigger information may include a step of identifying that the trigger information is short-term gaze direction determination information, and a step of acquiring gaze tracking information at a time corresponding to the short-term gaze direction determination information.

[0083] Alternatively, the trigger information identified by the short-term gaze direction determination information may include at least one of predetermined voice information or gesture information that operates the vehicle surrounding environment monitoring system using gaze tracking information at the time of identification of the trigger information.

[0084] Alternatively, the step of acquiring gaze tracking information during a gaze tracking information acquisition period corresponding to the trigger information may include a step of identifying that the trigger information is long-term gaze direction determination information, and a step of acquiring a representative value for gaze tracking information for a period corresponding to the long-term gaze direction determination information.

[0085] Alternatively, the trigger information, which is the long-term gaze direction determination information, may include vehicle driving information or vehicle position information that operates the vehicle surrounding environment monitoring system using gaze tracking information that is continuously acquired in a time series manner.

[0086] The steps described in the above-described embodiments of the present disclosure are presented for illustrative purposes only, and some steps may be omitted or additional steps may be added. Furthermore, the steps described above may be performed in any order.

[0087] FIG. 7 illustrates a simplified, general schematic diagram of an exemplary computing environment in which some embodiments of the present disclosure may be implemented.

[0088] Although the present disclosure has been described above generally in the context of computer-executable instructions that may be executed on one or more computers, those skilled in the art will appreciate that the present disclosure may be implemented in combination with other program modules and / or as a combination of hardware and software.

[0089] Generally, program modules include routines, programs, components, data structures, and the like that perform particular tasks or implement particular abstract data types. Furthermore, those skilled in the art will appreciate that the methods of the present disclosure can be implemented with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which may be operatively connected to one or more associated devices.

[0090] The described embodiments of the present disclosure can also be practiced in distributed computing environments, where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0091] Computers typically include a variety of computer-readable media. Any media that can be accessed by a computer can be a computer-readable medium. Computer-readable media includes volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media includes volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store the desired information.

[0092] Computer-readable transmission media typically includes any information delivery media that embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as another transport mechanism. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information within the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, or other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

[0093] An exemplary environment (1100) implementing various aspects of the present disclosure is illustrated, including a computer (1102), which includes a processing unit (1104), a system memory (1106), and a system bus (1108). The system bus (1108) couples system components, including but not limited to the system memory (1106), to the processing unit (1104). The processing unit (1104) may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be utilized as the processing unit (1104).

[0094] The system bus (1108) may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. The system memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). A basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, or EEPROM, and includes basic routines that help transfer information between components within the computer (1102), such as during start-up. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0095] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—which may also be configured for external use within a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from or writing to a CD-ROM disk (1122) or other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. The interface (1124) for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

[0096] These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. For the computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will appreciate that other types of media readable by a computer, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

[0097] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or portions of the operating system, applications, modules, and / or data may also be cached in RAM (1112). It will be appreciated that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0098] A user may enter commands and information into the computer (1102) via one or more wired / wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) that is connected to the system bus (1108), but may be connected by other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.

[0099] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface, such as a video adapter (1146). In addition to the monitor (1144), the computer typically includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0100] The computer (1102) may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) (1148) via wired and / or wireless communications. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and typically include many or all of the components described for the computer (1102), but for simplicity, only the memory storage device (1150) is shown. The logical connections shown include wired / wireless connections to a local area network (LAN) (1152) and / or a larger network, such as a wide area network (WAN) (1154). These LAN and WAN networking environments are common in offices and companies, facilitating enterprise-wide computer networks such as intranets, all of which can be connected to computer networks worldwide, such as the Internet.

[0101] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communications to the LAN (1152), which may also include a wireless access point installed therein for communicating with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communications computing device on the WAN (1154), or have other means of establishing communications over the WAN (1154), such as via the Internet. The modem (1158), which may be internal or external and wired or wireless, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, program modules described for the computer (1102) or portions thereof may be stored in a remote memory / storage device (1150). It will be appreciated that the network connections depicted are exemplary and other means of establishing a communications link between the computers may be used.

[0102] The computer (1102) operates to communicate with any wireless device or object that is arranged and operates via wireless communication, such as a printer, a scanner, a desktop and / or portable computer, a portable data assistant (PDA), a communication satellite, any equipment or location associated with a radio-detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or may simply be an ad hoc communication between at least two devices.

[0103] Wi-Fi (Wireless Fidelity) enables connections to the Internet and other devices without wires. Wi-Fi is a wireless technology that allows devices, such as computers, to send and receive data anywhere within the coverage area of ​​a base station, both indoors and outdoors, similar to cell phones. Wi-Fi networks use wireless technologies called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual-band).

[0104] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0105] Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein, for convenience, as “software”), or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0106] The various embodiments presented herein can be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" includes a computer program or media accessible from any computer-readable device. For example, computer-readable media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Furthermore, various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0107] It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not intended to be limited to the specific order or hierarchy presented.

[0108] The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein.

[0109] As described above, the relevant contents have been described in the best form for carrying out the invention.

[0110] The present disclosure can provide an improved user interface for a vehicle surrounding environment monitoring system, such as an SVM system, using DMS technology.

Claims

1. A method for providing a user interface of a vehicle surrounding environment monitoring system performed by a computing device, A step of obtaining driver gaze direction information; and A step of displaying at least one of a plurality of vehicle environment images on a vehicle environment image display unit based on the driver's gaze direction information; including, method.

2. In paragraph 1, The step of displaying at least one of a plurality of vehicle environment images on a vehicle environment image display unit based on the driver's gaze direction information is: A step of displaying an image of a camera corresponding to the driver's gaze direction information among a plurality of vehicle environment cameras on the vehicle environment image display unit; including, method.

3. In paragraph 1, The step of displaying at least one of a plurality of vehicle environment images on a vehicle environment image display unit based on the driver's gaze direction information is: A step of displaying at least one of the plurality of vehicle environment images on a display corresponding to the driver's gaze direction information among a plurality of displays installed at different locations; including, method.

4. In paragraph 1, The step of displaying at least one of a plurality of vehicle environment images on a vehicle environment image display unit based on the driver's gaze direction information is: A step of displaying at least one of the plurality of vehicle environment images on a display area corresponding to the driver's gaze direction information among the plurality of display areas of the vehicle environment image display unit; including, method.

5. In paragraph 1, The steps for obtaining the above driver gaze direction information are: Step of identifying trigger information; In response to the identification of the above trigger information, a step of obtaining gaze tracking information; and A step of determining the driver's gaze direction information based on the acquired gaze tracking information; including, method.

6. In paragraph 5, The trigger information includes at least one of voice information, gesture information, vehicle driving information, or vehicle location information. method.

7. In paragraph 5, The steps for obtaining the above gaze tracking information are: A step of acquiring gaze tracking information during a gaze tracking information acquisition period corresponding to the above trigger information; including, method.

8. In paragraph 7, The steps of acquiring gaze tracking information during the gaze tracking information acquisition period corresponding to the above trigger information are: A step of identifying that the above trigger information is short-term gaze direction determination information; and A step of acquiring gaze tracking information at a time corresponding to the short-term gaze direction determination information; including, method.

9. In paragraph 8, The trigger information identified by the short-term gaze direction determination information includes at least one of predetermined voice information or gesture information that operates the vehicle surrounding environment monitoring system using gaze tracking information at the time of identification of the trigger information. method.

10. In paragraph 5, The steps of acquiring gaze tracking information during the gaze tracking information acquisition period corresponding to the above trigger information are: A step of identifying that the above trigger information is long-term gaze direction determination information; and A step of obtaining a representative value for gaze tracking information of a period corresponding to the above long-term gaze direction determination information; including, method.

11. In paragraph 10, The trigger information, which is the long-term gaze direction determination information, includes vehicle driving information or vehicle location information that operates the vehicle surrounding environment monitoring system using gaze tracking information continuously acquired in time series. method.

12. A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs the following operations to provide a user interface for a vehicle surrounding environment monitoring system, wherein the operations are: A step of obtaining driver gaze direction information; and A step of displaying at least one of a plurality of vehicle environment images on a vehicle environment image display based on the driver's gaze direction information; including, A computer program stored on a computer-readable storage medium.

13. A computing device providing a user interface for a vehicle surrounding environment monitoring system, wherein the computing device: Memory containing computer executable components; A processor that executes the following computer-executable components stored in memory; The above processor, Obtain driver gaze direction information, and Displaying at least one of a plurality of vehicle environment images on a vehicle environment image display based on the driver's gaze direction information. Computing device.

Images