DEVICE FOR CONTROLLING AUTONOMOUS DRIVING AND A METHOD THEREFOR

The autonomous driving control device addresses the lack of notification algorithms by identifying and warning users of uncontrollable situations, ensuring user safety through proactive communication.

DE102023133192A1Pending Publication Date: 2024-12-24HYUNDAI MOTOR CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE102023133192
Authority / Receiving Office
DE · DE
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-20
Filing Date
2023-11-28
Publication Date
2024-12-24

AI Technical Summary

Technical Problem

Existing autonomous driving technologies lack effective algorithms for providing notifications when uncontrollable situations arise, leading to user inconvenience or danger.

Method used

An autonomous driving control device identifies situational information about target controls, determines if they are executable, and provides advance warnings or notifications to users through visual, auditory, or haptic means when necessary, using sensors, a notification device, and a control system.

Benefits of technology

Enhances user safety by informing users of uncontrollable driving situations, allowing them to take over control proactively, thus preventing potential dangers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

An autonomous driving control device (100) comprises a sensor device (110), a notification device (120), a memory (130), and a control device (140). The autonomous driving control device (100) identifies situation information about at least one target control item for a host vehicle (301, 401) using the sensor device (110). The autonomous driving control device (100) also determines whether the at least one target control item is executable based on the situation information or whether advance warning regarding the situation information is required. The autonomous driving control device (100) also provides at least one notification to the user using the notification device (120) if it is determined that the at least one target control item is not executable or advance warning is required regarding the situation information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical field

[0001] The present disclosure and invention relates to an autonomous driving control device and a method thereof, and more particularly relates to a technology for providing a notification of a driving situation to a user. background

[0002] As autonomous vehicles gradually become more widespread, various technologies related to autonomous driving are being developed. Depending on the degree of control, autonomous driving can be divided into semi-autonomous driving, conditionally autonomous driving, highly autonomous driving, and / or fully autonomous driving.

[0003] Meanwhile, various types of control algorithms (or functions) for driving control of a host vehicle are being developed among the control functions for autonomous driving.

[0004] For example, an autonomous driving control device based on smart cruise control (SCC) may maintain a distance between vehicles using at least one of a relative distance between a host vehicle and another vehicle (e.g., a preceding vehicle traveling in front of the host vehicle), a speed, an acceleration, or any combination thereof. The autonomous driving control device may provide a function for adjusting and controlling (in particular, regulating) the traveling speed of the host vehicle.

[0005] For example, an autonomous driving control device may perform SCC control of the driving of a host vehicle based on information obtained by various types of components. For example, the autonomous driving control device may identify at least one of information about a specific point in a forward range of the host vehicle, information about a specific section (e.g., a speed limit), or any combination thereof based on navigation information obtained by a navigation system. The autonomous driving control device may control the traveling speed (e.g., a longitudinal speed) of the host vehicle based on the identification result.The SCC function based on such navigation information can be defined as navigation-based smart cruise control (NSCC).

[0006] However, if an uncontrollable control element is identified or a situation that requires user awareness occurs while the host vehicle's driving control is being performed based on NSCC, the provision of a notification function must be considered in advance. Due to the lack of a notification provision algorithm, the user may experience inconvenience or be placed in a dangerous situation. Brief explanation

[0007] The present disclosure has been made to solve the above-mentioned problems of the prior art while maintaining the advantages of the prior art.

[0008] In embodiments of the present disclosure (hereinafter referred to as "disclosure"), an autonomous driving control device may identify (hereinafter referred to as "identify") situation information about at least one target control item for a host vehicle (i.e., a vehicle having the autonomous driving control device). The autonomous driving control device may also identify, based on the situation information, whether the target control item is executable. The autonomous driving control device may also provide a notification to a user when it is identified that the target control item is not executable.

[0009] According to one embodiment of the present disclosure, an autonomous driving control device may identify situation information about at least one target control element for the host vehicle. The autonomous driving control device may also determine, based on the situation information, whether advance warning is required regarding the situation information. The autonomous driving control device may also provide a notification to the user if it determines that advance warning is required.

[0010] According to an embodiment of the present disclosure, if it is identified that longitudinal speed control is required while the host vehicle is in a speed control section or speed restriction section or speed monitoring section (hereinafter referred to as: speed control section), an autonomous driving control device determines whether advance warning notification is required using a sensor device based on various information about the speed control section and the host vehicle.

[0011] According to an embodiment of the present disclosure, an autonomous driving control device may provide a pre-warning notification with information about a change in road type to the user when it is identified that the type of a road on which the host vehicle is traveling changes within a certain distance (e.g., distance).

[0012] According to an embodiment of the present disclosure, an autonomous driving control device may provide, using a notification device, a notification including positions of a start point and an end point of a speed control section, a real-time location of the host vehicle, a real-time traveling speed of the host vehicle, a speed limit, a guidance for changing the control authority for the host vehicle, a first road type of the traveling road, a certain distance, a real-time remaining distance until a road type change, a second road type of a road ahead, or any combination thereof.

[0013] The technical problems to be solved by the present disclosure are not limited to the problems mentioned above. Any other technical problems not mentioned herein should be clearly understood from the following description by those skilled in the art to which the present disclosure belongs.

[0014] According to one aspect of the present disclosure, an autonomous driving control device comprises: a sensor device, a notification device, a memory configured to store instructions, and a controller operatively connected to the sensor device, the notification device, and the memory. The instructions, when executed by the controller, cause the autonomous driving control device to identify situation information about at least one target control item for the host vehicle using the sensor device. The instructions, when executed by the controller, cause the autonomous driving control device to determine whether the at least one target control item is executable based on the situation information or whether advance warning regarding the situation information is required.The instructions, when executed by the control device, cause the autonomous driving control device to provide at least one notification to the user using the notification device when it is determined that the at least one target control item is not executable or the advance warning regarding the situation information is required.

[0015] According to one embodiment, the instructions, when executed by the controller, may cause the autonomous driving controller to determine whether the at least one target control element is executable based on navigation-based intelligent cruise control (NSCC).

[0016] According to one embodiment, the instructions, when executed by the controller, may cause the autonomous driving control device, while controlling the (e.g., fully or semi-autonomously) driving of the host vehicle, when it is determined that longitudinal speed control is required, to identify the situation information about the at least one target control element that has the longitudinal speed control (e.g., that is involved in and / or performs a longitudinal speed control).

[0017] According to one embodiment, the instructions, when executed by the controller, may cause the autonomous driving control device to identify the situation information including at least one of a start point and an end point of the speed control section, a real-time location of the host vehicle, a lowest target speed of the host vehicle by the NSCC controller, or a combination of the start point and the end point of the speed control section, the real-time location of the host vehicle, and the lowest target speed of the host vehicle by the NSCC controller when it is identified that the host vehicle is traveling in a speed control section.The instructions, when executed by the control device, may also cause the autonomous driving control device to identify, based on the situation information, whether NSCC-based control is possible such that a traveling speed of the host vehicle at the end point is lower than a speed limit of the speed control section. The instructions, when executed by the control device, may also cause the autonomous driving control device to provide a first notification to the user using the notification device if it is determined that control is not possible.

[0018] According to one embodiment, the instructions, when executed by the controller, may cause the autonomous driving control device to identify the speed limit of the speed control section and to identify a speed corresponding to a certain proportion / ratio of the speed limit as the lowest target speed of the host vehicle.

[0019] According to one embodiment, the first notification may include at least one of the positions of the start point and the end point of the speed control section, the real-time location of the host vehicle, a real-time traveling speed of the host vehicle, the speed limit, an instruction for changing (e.g., a handover) the control authority for the host vehicle, or a combination of the positions of the start point and the end point of the speed control section, the real-time location of the host vehicle, the real-time traveling speed of the host vehicle, the speed limit, and the instruction for changing the control authority for the host vehicle.

[0020] According to one embodiment, the instructions, when executed by the controller, may cause the autonomous driving controller to identify a first road type of a traveling road on which the host vehicle is traveling. The instructions, when executed by the controller, may also cause the autonomous driving controller to provide a second notification to the user using the notification device when it is identified that a second road type of a road ahead, which is within a certain distance from a real-time location of the host vehicle, is different from the first road type of the traveling road.

[0021] According to one embodiment, the second notification may include at least one of the first road type of the traveled road, the determined distance, a real-time distance remaining until the first road type is changed, the second road type of the road ahead, a real-time location of the host vehicle, a real-time traveling speed of the host vehicle, an instruction for changing the control authority for the host vehicle, or a combination of the first road type of the traveled road, the determined distance, the real-time distance remaining until the first road type is changed, the second road type of the road ahead, the real-time location of the host vehicle, the real-time traveling speed of the host vehicle, and the instruction for changing the control authority for the host vehicle.

[0022] According to one embodiment, the instructions, when executed by the controller, may cause the autonomous driving controller to determine whether NSCC is being performed on / applied to the host vehicle based on navigation information when it is determined that the at least one target control item is not executable or the advance warning is required based on the situation information. The instructions, when executed by the controller, may also cause the autonomous driving controller to provide the at least one notification to the user using the notification device when it is determined that NSCC is being performed.

[0023] According to one embodiment, the instructions, when executed by the controller, may cause the autonomous driving controller to provide the at least one notification comprising at least one of a visual content, an auditory content, a haptic function, or a combination of the visual content, the auditory content, and the haptic function to the user using the notification device.

[0024] According to one aspect of the present disclosure, an autonomous driving control method may include: identifying, by a controller, situation information about at least one target control item for a host vehicle using a sensor device. The autonomous driving control method may also include: determining, by the controller, based on the situation information, whether the at least one target control item is executable or whether advance warning is required regarding the situation information. The autonomous driving control method may also include: providing, by the controller, at least one notification to a user using a notification device when it is determined that the at least one target control item is not executable or advance warning is required regarding the situation information.

[0025] According to one embodiment, determining by the control device based on the situation information whether the at least one target control element is executable or whether the advance warning is required with regard to the situation information may further comprise: determining by the control device whether the at least one target control element is executable based on NSCC.

[0026] According to one embodiment, identifying, by the controller, the situation information about the at least one target control element for the host vehicle using the sensor device may further comprise: identifying, by the controller, the situation information about the at least one target control element that has the longitudinal speed control (e.g., that participates in and / or performs a longitudinal speed control) during the controller's control of the (e.g., fully or semi-autonomously) driving of the host vehicle when it is determined that a longitudinal speed control is required.

[0027] According to one embodiment, the autonomous driving control method may further comprise: identifying, by the controller, the situation information including at least one of a start point and an end point of the speed control section, a real-time location of the host vehicle, a lowest target speed of the host vehicle by NSCC control, or a combination of the start point and the end point of the speed control section, the real-time location of the host vehicle, and the lowest target speed of the host vehicle by NSCC control when it is identified that the host vehicle is traveling in a speed control section.The autonomous driving control method may further include: identifying, by the control device, whether NSCC-based control is possible such that a traveling speed of the host vehicle at the end point is lower than a speed limit of the speed control section based on the situation information. The autonomous driving control method may further include: providing, by the control device, a first notification to the user using the notification device when it is determined that control is not possible.

[0028] According to one embodiment, the identification, by the controller, of the lowest target speed of the host vehicle by the NSCC controller may further comprise: identifying, by the controller, the speed limit of the speed control section, and identifying, by the controller, a speed corresponding to a certain proportion / ratio of the speed limit as the lowest target speed of the host vehicle.

[0029] According to one embodiment, the first notification may include at least one of the positions of the start point and the end point of the speed control section, the real-time location of the host vehicle, the real-time traveling speed of the host vehicle, the speed limit, an instruction for changing the control authority for the host vehicle, or a combination of the positions of the start point and the end point of the speed control section, the real-time location of the host vehicle, the real-time traveling speed of the host vehicle, the speed limit, and the instruction for changing the control authority for the host vehicle.

[0030] According to one embodiment, the autonomous driving control method may further comprise: identifying, by the control device, a first road type of a traveling road on which the host vehicle is traveling. The autonomous driving control method may further comprise: providing, by the control device, a second notification to the user using the notification device when it is identified that a second road type of a road ahead, which is located at a certain distance from a real-time location of the host vehicle, is different from the first road type of the traveling road.

[0031] According to one embodiment, the second notification may include at least one of the first road type of the traveled road, the determined distance, a real-time distance remaining until the first road type is changed, the second road type of the road ahead, a real-time location of the host vehicle, a real-time traveling speed of the host vehicle, an instruction for changing the control authority for the host vehicle, or a combination of the first road type of the traveled road, the determined distance, the real-time distance remaining until the first road type is changed, the second road type of the road ahead, the real-time location of the host vehicle, the real-time traveling speed of the host vehicle, and the instruction for changing the control authority for the host vehicle.

[0032] According to one embodiment, the autonomous driving control method may further comprise: determining, by the control device, whether NSCC is being performed on the host vehicle based on navigation information / is being applied to the host vehicle when it is determined that the at least one target control item is not executable or the advance warning is required in view of the situation information. The autonomous driving control method may further comprise: providing, by the control device, the at least one notification to the user using the notification device when it is identified that the NSCC is being performed.

[0033] According to one embodiment, the autonomous driving control method may further comprise: providing, by the control device, the at least one notification comprising at least one of a visual content, an auditory content, a haptic function, or a combination of the visual content, the auditory content, and the haptic function to the user using the notification device. Short description of the drawings

[0034] The above and other objects, features and advantages of the present disclosure should become more apparent from the following detailed description taken in conjunction with the accompanying drawings: Fig. 1 is a block diagram showing components of an autonomous driving control device according to an embodiment of the present disclosure, Fig. 2 is a block diagram showing components of an autonomous driving control device according to an embodiment of the present disclosure, Fig. 3 is a conceptual diagram illustrating the operation of an autonomous driving control device according to an embodiment of the present disclosure, Fig. 4 is a conceptual diagram illustrating the operation of an autonomous driving control device according to an embodiment of the present disclosure, Fig. 5 is an operation flowchart of an autonomous driving control device according to an embodiment of the present disclosure, and Fig. 6 is an operation flowchart of an autonomous driving control device according to an embodiment of the present disclosure.

[0035] With regard to the description of the drawings, identical or similar components are identified by identical or similar reference numerals. Detailed description

[0036] Below, some examples (embodiments) of the present disclosure will be described in detail with reference to the exemplary drawings. Throughout the drawings, the same reference numerals are used to identify the same or equivalent elements. Furthermore, a detailed description of well-known features or functions has been omitted in order not to unnecessarily obscure the essence of the present disclosure.

[0037] In describing the components of the embodiments according to the present disclosure, terms such as first, second, A, B, (a), (b), and the like may be used. These terms are merely used to distinguish the components from other components, and the terms do not limit the type, order, or sequence of the components. Unless otherwise defined, all terms used herein, including technical and scientific terms, are to be understood as commonly understood in the art to which the present disclosure belongs. It is further to be understood that terms should be interpreted to have a meaning consistent with their meanings in the context of this disclosure and the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined herein.When a component, device, element, or the like of the present disclosure is described as having a purpose or performing an operation, function, or the like, the component, device, element, or the like should be considered herein as "configured" to fulfill that purpose or to perform that operation or function. Each component, device, element, and the like may separately form a processor and a memory, such as a non-transitory computer-readable medium, or may be included as part of the device. The term "operationally connected" among hardware parts may hereinafter mean that a direct connection or an indirect connection is established between the hardware parts in a wired or wireless manner, such that the second hardware is controlled by the first among the hardware parts.

[0038] Hereinafter, examples of the present disclosure are described with reference to the Fig. 1 to 6 are described in detail.

[0039] Fig. 1 is a block diagram showing components of an autonomous driving control device according to an embodiment of the present disclosure.

[0040] According to one embodiment, an autonomous driving control device 100 may include at least one of a sensor device 110, a notification device 120, a memory 130, a control device 140, or any combination thereof. Fig. The configuration of the autonomous driving control device 100 shown in Figure 1 is an example, and the embodiments of the present disclosure are not limited thereto. For example, the autonomous driving control device 100 may further include components (e.g., at least one of an interface device, a communication device, a display device, or any combination thereof) that are not shown in Fig. 1 are shown.

[0041] According to one embodiment, the sensor device 110 may acquire (or identify) various information (e.g., information including at least one of biometric information of a user, internal information of a host vehicle, driving environment information, or any combination thereof) used for driving a host vehicle.

[0042] In one embodiment, the sensor device 110 may include at least one sensor comprising at least one of a camera, a radar, a LiDAR, or any combination thereof.

[0043] In another embodiment, the sensor device 110 may also include a navigation system. In one embodiment, the controller 140 may perform or apply driving control (e.g., navigation-based intelligent cruise control (NSCC)) to a host vehicle using navigation information identified using the navigation system.

[0044] In one embodiment, the sensor device 110 may identify or determine or recognize (herein also briefly: identify) situation information about at least one target control element for the host vehicle.

[0045] For example, when it is identified that driving control (e.g., longitudinal speed control, e.g., longitudinal speed regulation) is required for the host vehicle in a situation where the host vehicle is driven under control (e.g., control based on NSCC) by the autonomous driving control apparatus 100, the sensor device 110 may identify situation information about at least one target control element that has the longitudinal speed control (e.g., that participates in and / or performs longitudinal speed control).

[0046] For example, if it is identified that the host vehicle is in the speed control section or speed restriction section or speed monitoring section (hereinafter referred to as: speed control section), the sensor device 110 may identify situation information including at least one of a start and end point of the speed control section, a real-time location of the host vehicle, the lowest target speed of the host vehicle by NSCC control (or NSCC regulation), or any combination thereof.

[0047] In one embodiment, the sensor device 110 may acquire information using at least one sensor and may identify the acquired information based on other external components (e.g., a communication device).

[0048] For example, the sensor device 110 may identify driving environment information including information about at least one of a traveling speed of the host vehicle, a traveling acceleration of the host vehicle, a traveling direction of the host vehicle, a destination of the host vehicle, a traveling route of the host vehicle, a traveling mode of the host vehicle, a traveling road of the host vehicle, a real-time location of the host vehicle, a real-time remaining distance to a certain point, an area adjacent to the host vehicle, or any combination thereof.

[0049] In one embodiment, the sensor device 110 may identify information about a first road type of a traveled road on which the host vehicle is traveling.

[0050] In another embodiment, the sensor device 110 may identify a second road type of a road ahead that is located at a certain distance (e.g., 200 m) from the real-time location of the host vehicle.

[0051] According to one embodiment, the notification device 120 may include various types of alarm providing devices and provide a user with a notification including information about controlling a host vehicle for controlling using the alarm providing devices.

[0052] For example, the notification device 120 may include a display device. In one embodiment, the autonomous driving control device 100 may provide a user with a notification having visual content and / or auditory content through the display device.

[0053] For example, the notification device 120 may include an audio output device. For example, the autonomous driving control device 100 may provide a user with a notification with auditory content through the audio output device.

[0054] In one embodiment, the notification device 120 may include a haptic device (e.g., a vibration device). For example, the autonomous driving control device 100 may provide the user with a notification having a haptic function through the haptic device. For example, the haptic device may include at least one device that provides the user with a vibration through at least one of at least one seat, at least one belt, at least one steering wheel, or any combination thereof in the host vehicle.

[0055] According to one embodiment, memory 130 may store instructions or data. For example, memory 130 may store one or more instructions that cause autonomous driving control device 100 to perform various operations when executed by controller 140.

[0056] For example, memory 130 and controller 140 may be implemented as a single chipset. Controller 140 may include a communications processor and / or a modem.

[0057] For example, memory 130 may store various information related to autonomous driving control device 100. For example, memory 130 may store information about the operating history of controller 140. For example, memory 130 may store information related to states and / or operations of components (e.g., at least one of an engine control unit (ECU), sensor device 110, notification device 120, controller 140, or any combination thereof) of the host vehicle.

[0058] According to one embodiment, the control device 140 may be operatively connected to the sensor device 110, the notification device 120, and / or the memory 130. For example, the control device 140 may control operations for controlling the sensor device 110, the notification device 120, and / or the memory 130.

[0059] For example, the controller 140 may identify situational information about at least one target control element for the host vehicle using the sensor device 110.

[0060] For example, the controller 140 may determine whether at least one target control element for the host vehicle is executable or can be executed (e.g., is capable of being executed) based on the situation information identified using the sensor device 110.

[0061] For example, if it is identified that longitudinal speed control (e.g., longitudinal speed regulation) is required while the controller 140 is controlling the control of the host vehicle, the controller 140 may identify situation information about at least one target control element that has the longitudinal speed control (e.g., that is involved in and / or performing longitudinal speed control).

[0062] For example, when it is identified that the host vehicle is traveling in the speed control section, the controller 140 may, using the sensor device 140, identify situation information including at least one of a start and end point of the speed control section, the real-time location of the host vehicle, the lowest target speed of the host vehicle by NSCC control, or any combination thereof.

[0063] For example, the controller 140 may identify a speed limit of a speed control section by using a navigation system present in the sensor device 110. The controller 140 may identify that a speed corresponding to a certain (e.g., predetermined) proportion / ratio (e.g., 90%) of the speed limit of the speed control section is the lowest target speed of the host vehicle. For example, if the speed limit in the speed control section is 100 km / h, the controller 140 may identify that the lowest target speed is 90 km / h. For example, when the host vehicle is controlled in the corresponding section, the lowest target speed may be the lowest value among the controllable driving speeds of a host vehicle by the controller 140.

[0064] For example, the controller 140 may identify a first road type of a traveled road on which the host vehicle is traveling using the sensor device 110.

[0065] For example, using the sensor device 110, the controller 140 may identify a second road type of a road ahead that is located at a specific distance (e.g., 200 m) from the real-time location of the host vehicle. Numerical values ​​for the specific or predetermined (herein referred to as specific) distance are examples, and embodiments of the present disclosure are not limited thereto.

[0066] For example, the controller 140 may determine whether the at least one target control element can be executed based on situation information or whether advance warning with regard to situation information is required.

[0067] For example, the controller 140 may determine whether at least one target control element is executable or can be executed based on NSCC.

[0068] For example, if it is identified that the host vehicle is traveling within the speed control section, the controller 140 may identify the situation information including at least one of a start and end point of the speed control section, a real-time location of the host vehicle, the lowest target speed of the host vehicle through NSCC control, or any combination thereof. If it is identified that the host vehicle is traveling within the speed control section, the controller 140 may also identify, based on the situation information, whether NSCC-based control is possible such that the traveling speed of the host vehicle at the end point is below the speed limit of the speed control section.

[0069] For example, if it is identified that at least one target control is not executable or advance warning regarding the situation information is required, the controller 140 may provide at least one notification to the user using the notification device 120.

[0070] For example, if it is identified that at least one target control item is not executable or advance warning is required regarding the situation information, the controller 140 may determine whether NSCC based on navigation information is being performed on the host vehicle or is being applied to the host vehicle. If it is identified that NSCC is being performed, the controller 140 may provide a notification to the user using the notification device 120. In other words, only while performing control of the host vehicle based on NSCC may the controller 140 perform a notification providing function using the notification device 120.

[0071] For example, if it is determined that at least one target control cannot be executed or advance warning regarding the situation information is required, the controller 140 may provide at least one notification to the user using the notification device 120.

[0072] For example, based on the situation information, the controller 140 may identify whether NSCC-based control is possible such that the traveling speed of the host vehicle at an endpoint is lower than the speed limit of the speed control section. If it is identified that control is not possible, the controller 140 may provide a first notification to the user using the notification device 120.

[0073] For example, the first notification may include at least one of positions of a start and end point of the speed control section, a real-time location of the host vehicle, a real-time travel speed of the host vehicle, a speed limit, instructions for changing control authority for the host vehicle, or any combination thereof.

[0074] For example, the controller 140 may identify the first road type of the traveling road on which the host vehicle is traveling. If it is identified that the second road type of a road ahead, which is within a certain distance (e.g., 200 m) from a real-time location of the host vehicle, is different from the first road type of the traveling road, the controller 140 may provide a second notification to the user using the notification device 120. For example, if the controller 140 identifies that the second road type is one of the road types that the host vehicle (e.g., the host vehicle) cannot control based on a certain operating mode (e.g., intelligent cruise control (SCC) and / or navigation-based intelligent cruise control (NSCC)) (e.g.,on which (in particular fully or partially autonomous) driving control by means of SCC and / or NSSC is not feasible), the control device 140 can provide a second notification to the user using the notification device 120.

[0075] For example, the second notification may include at least one of a first road type of a traveled road, a specific distance, a real-time distance remaining until a road type change, a second road type of a road ahead, a real-time location of the host vehicle, a real-time travel speed of the host vehicle, instructions for changing control authority for the host vehicle, or any combination thereof.

[0076] For example, the controller 140 may provide the user with at least one notification including at least one of visual content, audible content, a haptic function, or any combination thereof using the notification device 120. The at least one notification may include, for example, the first notification and / or the second notification.

[0077] Fig. 2 is a block diagram showing components of an autonomous driving control device according to an embodiment of the present disclosure.

[0078] According to one embodiment, an autonomous driving control device (e.g., the autonomous driving control device 100 of Fig. 1) a condition determination device 210 and / or a warning notification device 220 (e.g., the notification device 120 in Fig. 1).

[0079] The Fig. The structure of the components shown in Figure 2 is an example, and the embodiments of the present disclosure are not limited thereto. For example, in Fig. 2 the condition determination device 210 and / or the warning notification device 220 as a part of the control device 140 and the sensor device 110 of Fig. 1 or be implemented as a single chip with the control device 140. In other words, at least part of the operations of the condition determining device 210 and / or the warning notification device 220, which are described below, can be performed by the control device 140 and / or the sensor device 110 of Fig. 1 (or under control by the control device 230).

[0080] The state determination device 210 may, for example, determine whether the control target element (e.g., also target control element) is executable and / or may determine a situation in which advance warning is required during the drive control for the host vehicle.

[0081] For example, the state determination device 210 may determine situation information about at least one target control element for the host vehicle and, based on the situation information, determine whether at least one target control element is executable or can be executed (e.g., is capable of being executed).

[0082] For example, the condition determination device 210 can determine whether the advance warning is necessary based on the situation information.

[0083] For example, the state determination device 210 may communicate to the warning notification device 220 whether a control state and / or a pre-warning notification is required.

[0084] For example, the warning notification device 220 may determine whether advance warning notification is required during control. If it determines that advance warning notification is required, the warning notification device 220 may transmit a warning notification signal.

[0085] For example, the warning notification device 220 may transmit the warning notification signal to various types of notification devices (e.g., at least one of a display device, an audio output device, a haptic device, or any combination thereof) installed in the host vehicle.

[0086] For example, the warning notification device 220 may transmit a warning notification signal containing a content of at least one notification including a first notification and a second notification, information, or the like, to a notification device (e.g., the notification device 120 of Fig. 1) deliver.

[0087] Fig. 3 is a conceptual diagram illustrating the operation of an autonomous driving control device according to an embodiment of the present disclosure.

[0088] According to one embodiment, a host vehicle 301 may be Fig. 3 as a driving situation under the control of an autonomous driving control device (e.g., the autonomous driving control device 100 in Fig. 1). An operation of the host vehicle 301 described below may be defined as an operation under the control of an autonomous driving control device. In other words, an operation of the host vehicle 301 described below may include a driving control operation (e.g., an NSCC operation) of the autonomous driving control device.

[0089] According to one embodiment, host vehicle 301 may identify situation information about at least one target control element. For example, if it is identified that longitudinal speed control (e.g., longitudinal speed regulation) is required, host vehicle 301 may identify situation information about at least one target control element that has longitudinal speed control (e.g., that participates in and / or performs longitudinal speed control).

[0090] For example, if it is identified that the host vehicle 301 is in a Fig. 3 (e.g., an area with speed restrictions, for example, controlled at individual checkpoints, controlled by a section control, also known as “section control,” or the like), the host vehicle 301 can identify situation information about a driving state of the host vehicle 301 and / or the speed control section.

[0091] For example, when it is identified that the host vehicle 301 is traveling in the speed control section, the situation information may include at least one of a start point and an end point of the speed control section, a real-time location of the host vehicle 301, the lowest target speed of the host vehicle 301 by the NSCC controller, or any combination thereof.

[0092] For example, the host vehicle 301 may identify that the host vehicle 301 is currently traveling by moving a first distance t1 from a starting point 310 of the speed control section.

[0093] For example, the host vehicle 301 may identify that the host vehicle 301 is currently traveling in a state in which it remains a second distance t2 from an end point 320 of the speed control section.

[0094] For example, based on the situation information, the host vehicle 301 may identify whether NSCC control is possible such that the travel speed of the host vehicle at the end point 320 is less than the speed limit of the speed control section.

[0095] For example, the host vehicle 301 may identify the speed limit of the speed control section using at least one of situational information, detection information obtained by a sensor device, or any combination thereof. The host vehicle 301 may also identify a speed corresponding to a certain proportion / ratio (e.g., 90%) of the speed limit as the lowest target speed of the host vehicle.

[0096] For example, when the host vehicle 301 reaches the end point 320, the host vehicle 301 may identify whether the host vehicle 301 is capable of traveling at a speed below a speed limit by performing control based on the lowest target speed from the current location.

[0097] For example, if it is identified that NSCC control that allows the host vehicle's travel speed to be below the speed limit of the speed control section at the end point 320 is not possible, the host vehicle 301 may provide a first notification to the user using a notification device.

[0098] For example, the first notification may include at least one of positions / locations of the starting point 310 and the ending point 320 of the speed control section, a real-time location of the host vehicle 301, a real-time travel speed of the host vehicle 301, a speed limit, instructions for a change (e.g., a handover) of control authority for the host vehicle 301, or any combination thereof.

[0099] For example, the control authority change instructions may include information that allows a user to take over driving control of the host vehicle (e.g., to start driving in a manual mode).

[0100] Fig. 4 is a conceptual diagram illustrating the operation of an autonomous driving control device according to an embodiment of the present disclosure.

[0101] According to one embodiment, a host vehicle 401 may be Fig. 4 as a driving situation under control by an autonomous driving control device (e.g., the autonomous driving control device 100 in Fig. 1). An operation of the host vehicle 401 described below may be defined as an operation under the control of the autonomous driving control device. In other words, an operation of the host vehicle 401 described below may include a driving control operation (e.g., NSCC operation) of the autonomous driving control device.

[0102] According to one embodiment, the host vehicle 401 may identify a road type of a traveled road. For example, when identifying the road type of the traveled road, the host vehicle 401 may identify that a road with a different road type than the identified road type is within a certain distance ahead.

[0103] For example, the host vehicle 401 may select a first road type of a currently traveled road 440, which is Fig. 4 shown.

[0104] The first road type can be, for example, a traffic junction, in particular a motorway interchange (IC for short), or a junction (JC for short), although this is only an example. Embodiments of the present disclosure are not limited thereto. The first road type can be, for example, a motorway and / or an expressway.

[0105] For example, the host vehicle 401 may identify a second road type of an upcoming road 460 that is located at a certain distance (e.g., 200 m) from the real-time location of the host vehicle 401 traveling on the traveling road 440.

[0106] For example, the host vehicle 401 may define a boundary 450 between the road 440 being traveled and the road 460 ahead according to Fig. 4 identify.

[0107] The second road type may, for example, be a federal highway, but this is only an example; embodiments of the present disclosure are not limited thereto. The second road type may, for example, be a main road, an IC, or a JC.

[0108] For example, if it is identified that the second road type of the upcoming road 460 is different from the first road type of the traveled road 440, the host vehicle 401 may send a second notification using a notification device (e.g., the notification device 120 in Fig. 1) to the user. For example, if the autonomous driving control device identifies that the second road type is one of the road types that its own vehicle cannot control based on a particular operation mode (e.g., intelligent cruise control (SCC) and / or navigation-based intelligent cruise control (NSCC)) (e.g., on which (in particular, fully or partially autonomous) driving control using SCC and / or NSSC is not feasible), then the autonomous driving control device may provide a second notification to the user using the notification device.

[0109] For example, the second notification may include at least one of a road type of the traveled road 440, a specific distance, a real-time remaining distance until a road type change, a road type of the road ahead 460, a real-time location of the host vehicle 401, a real-time traveling speed of the host vehicle 401, instructions for changing control authority for the host vehicle 401, or any combination thereof.

[0110] For example, the control authorization change instructions may include information that allows a user to perform driving control of the host vehicle (e.g., to begin driving in a manual mode).

[0111] Fig. 5 is an operation flowchart of an autonomous driving control device according to an embodiment of the present disclosure.

[0112] According to one embodiment, an autonomous driving control device (e.g., the autonomous driving control device 100 of Fig. 1) the Fig. 5. For example, at least one of components (e.g., the sensor device 110, the notification device 120, the memory 130, and / or the control device 140 of Fig. 1), which are included in the autonomous driving control device, are designed to control the operations of Fig. 5 to be carried out.

[0113] In the following embodiment, S510 to S530 may be executed sequentially, but are not always executed sequentially. For example, the order of the operations may be changed, and at least two operations may be executed in parallel. Furthermore, descriptions relating to the above may be Fig. 5 corresponding to or identical with the descriptions relating to them, are briefly described or omitted to avoid redundancy.

[0114] According to one embodiment, the autonomous driving control device may identify situation information about at least one target control element using a sensor device (S510).

[0115] For example, if it is identified that longitudinal speed control (e.g., longitudinal speed regulation) is required while the autonomous driving control device controls the driving of the host vehicle, the autonomous driving control device may identify situation information about at least one target control element that has the longitudinal speed control (e.g., that participates in and / or performs longitudinal speed control).

[0116] For example, when it is identified that the host vehicle is traveling in the speed control section, the autonomous driving control device may identify situation information including at least one of a start and end point of the speed control section, a real-time location of the host vehicle, the lowest target speed of the host vehicle by an NSCC controller, or any combination thereof.

[0117] For example, the autonomous driving control device may identify the speed limit of the speed control section and identify a speed corresponding to a certain proportion / ratio of the speed limit as the lowest target speed of the host vehicle.

[0118] For example, the autonomous driving control device may identify a first road type of a traveled road on which the host vehicle is traveling and / or a second road type of a road ahead that is located at a certain distance from a real-time location of the host vehicle.

[0119] According to one embodiment, the autonomous driving control device may determine whether the at least one target control item is executable based on the situation information or whether advance warning regarding the situation information is required (S520).

[0120] For example, the autonomous driving control device may determine whether at least one target control item can be executed based on NSCC.

[0121] For example, if it is identified that the host vehicle is traveling within the speed control section, the autonomous driving control device may identify the situation information including at least one of a start point and an end point of the speed control section, a real-time location of the host vehicle, the lowest target speed of the host vehicle through NSCC control, or any combination thereof. If it is identified that the host vehicle is traveling within the speed control section, the autonomous driving control device may also identify, based on the situation information, whether NSCC-based control is possible such that the traveling speed of the host vehicle at the end point is lower than the speed limit of the speed control section.

[0122] For example, the autonomous driving control device may determine whether the second road type of a road ahead, which is within a certain distance from the real-time location of the host vehicle, is different from the first road type of a road being traveled on which the host vehicle is currently traveling.

[0123] For example, the autonomous driving control device may determine whether the first road type is different from the second road type. For example, the first road type may be a road type on which the autonomous driving control device can control the host vehicle based on a particular operation mode (e.g., SCC (Smart Cruise Control) and / or NSCC (Navigation-based Smart Cruise Control)). For example, in contrast to the first road type, the second road type may be one of the road types on which the autonomous driving control device cannot control the host vehicle based on a particular operation mode (e.g., SCC (Smart Cruise Control) and / or NSCC (Navigation-based Smart Cruise Control)).

[0124] According to one embodiment, when it is identified that at least one target control item is not executable or advance warning is required regarding the situation information, the autonomous driving control device may provide at least one notification to the user using a notification device (S530).

[0125] For example, when it is identified that longitudinal speed control (e.g., longitudinal cruise control) is required while the autonomous driving control device controls the driving of the host vehicle, the autonomous driving control device may identify situation information about at least one target control element that includes the longitudinal speed control.

[0126] For example, if it is determined that at least one target control item is not executable or advance warning is required based on the situational information, the autonomous driving control device may determine whether NSCC is being performed on / applied to the host vehicle based on navigation information. If it is determined that NSCC is being performed, the autonomous driving control device may provide a notification to the user using the notification device.

[0127] For example, the autonomous driving control device may provide at least one notification comprising at least one of a visual content, an auditory content, a haptic function, or any combination thereof to the user using the notification device.

[0128] For example, the at least one notification may include a first notification and / or a second notification.

[0129] For example, the first notification may include at least one of positions of a start and end point of the speed control section, a real-time location of the host vehicle, a real-time travel speed of the host vehicle, a speed limit, instructions for changing control authority for the host vehicle, or any combination thereof.

[0130] For example, the second notification may include at least one of a road type of a road being traveled, a specific distance, a real-time distance remaining until the road type is changed, a road type of a road ahead, a real-time location of the host vehicle, a real-time travel speed of the host vehicle, instructions for changing control authority for the host vehicle, or any combination thereof.

[0131] Fig. 6 illustrates a data processing system related to an autonomous driving control method according to an embodiment of the present disclosure.

[0132] Referring to Fig.6, a data processing system (e.g., a computer system, a computing system) 1000 relating to an autonomous driving control method may comprise at least a processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a (data) storage device (e.g., a mass storage device) 1600, and a network interface 1700, which are interconnected via a bus 1200.

[0133] Processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in memory 1300 and / or storage device 1600. Each of memory 1300 and storage device 1600 may include various types of volatile or non-volatile storage media. For example, memory 1300 may include read-only memory (ROM) 1310 and random access memory (RAM) 1320.

[0134] Accordingly, the operations of the method or algorithm described in connection with the embodiments disclosed in this specification may be implemented directly with a hardware module, a software module, or a combination of the hardware module and the software module executed by the processor 1100. The software module may be located on a storage medium (i.e., the memory 1300 and / or the storage device 1600), such as random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrical EPROM (EEPROM), a register, a hard disk, a removable disk, and / or a compact disk (CD-ROM).

[0135] The storage medium may be connected to the processor 1100. The processor 1100 may read information from the storage medium and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor 1100. The processor and the storage medium may be implemented with an application-specific integrated circuit (ASIC). The ASIC may be provided in a user terminal. Alternatively, the processor and the storage medium may be implemented with separate components in the user terminal.

[0136] The above description is merely an example of the technical idea of ​​the present disclosure, and various modifications and variations can be made by a person skilled in the art without departing from the essential features of the present disclosure.

[0137] Accordingly, the embodiments of the present disclosure are not intended to limit the technical idea of ​​the present disclosure, but to explain it, and the scope and spirit of the present disclosure are not limited by the above embodiments. The scope of the present disclosure should be interpreted by the appended claims, and all equivalents thereto should be construed as being included within the scope of the present disclosure.

[0138] An autonomous driving control apparatus according to an embodiment of the present disclosure and a method thereof are described as follows.

[0139] Further, according to at least one of embodiments of the present disclosure, when a situation is identified in which control of driving based on a certain control algorithm (e.g., NSCC) is impossible or difficult, an autonomous driving control device may identify information about a situation before the situation occurs and then perform a function of providing a notification of the corresponding information in advance to a user.

[0140] Furthermore, according to at least one embodiment of the present disclosure, an autonomous driving control device can provide information about a driving limit situation of a host vehicle to the user in advance. Consequently, usability including a function of providing information to a user can be increased, and the validity / effectiveness of driving control for a host vehicle, including autonomous driving control, can be ensured.

[0141] Further, according to at least one of embodiments of the present disclosure, when the continuation of the driving control is inevitably interrupted due to a change in an environment while the driving control for the host vehicle is being performed, an autonomous driving control device can perform a function of providing information to cope with the corresponding situation in advance by providing a warning notification in advance to the user.

[0142] In addition, a variety of effects can be provided which are understood directly or indirectly through this disclosure.

[0143] Although the present disclosure has been described with reference to embodiments and the accompanying drawings, the present disclosure is not limited thereto. The present disclosure may be modified and altered in numerous ways by those skilled in the art to which the present disclosure belongs, without departing from the spirit and scope of the present disclosure and invention as claimed in the following claims. List of reference symbols 100 Autonomous driving control device 110 Sensor device 120 Notification setup 130 storage 140 Control device 301 Host Vehicle 310 starting point 320 Endpoint 401 Host vehicle 440 Currently used road 450 limit 460 Road ahead

Claims

[1] Autonomous driving control device (100), comprising: a sensor device (110), a notification device (120), a memory (130) configured to store instructions, and a control device (140) operatively connected to the sensor device (110), the notification device (120) and the memory (130), wherein the instructions, when executed by the controller (140), cause the autonomous driving control device (100) to: to identify situation information about at least one target control element for a host vehicle (301, 401) using the sensor device (110), to determine whether the at least one target control element is executable on the basis of the situation information or whether advance warning regarding the situation information is required, provide at least one notification to the user using the notification device (120) when it is determined that the at least one target control is not executable or the advance warning regarding the situation information is required. [2] The autonomous driving control device (100) according to claim 1, wherein the instructions, when executed by the controller (140), cause the autonomous driving control device (100) to: to determine whether the at least one target control element can be executed on the basis of a navigation-based intelligent cruise control, or NSCC for short. [3] The autonomous driving control device (100) according to claim 1 or 2, wherein the instructions, when executed by the controller (140), cause the autonomous driving control device (100) to: while controlling the driving of the host vehicle (301, 401), when it is determined that longitudinal speed control is required, identifying the situation information about the at least one target control element having the longitudinal speed control. [4] The autonomous driving control device (100) according to any one of claims 1 to 3, wherein the instructions, when executed by the control device (140), cause the autonomous driving control device (100) to: when it is identified that the host vehicle (301) is traveling in a speed control section, to identify the situation information comprising at least one of a start point (310) and an end point (320) of the speed control section, a real-time location of the host vehicle (301), a lowest target speed of the host vehicle (301) by an NSCC controller, or a combination of the start point (310) and the end point (320) of the speed control section, the real-time location of the host vehicle (301), and the lowest target speed of the host vehicle (301) by the NSCC controller, to identify, based on the situation information, whether an NSCC-based control is possible such that a driving speed of the host vehicle (301) at the end point (320) is lower than a speed limit of the speed control section, and if it is determined that the controller is not able to provide a first notification to the user using the notification device (120). [5] The autonomous driving control device (100) according to claim 4, wherein the instructions, when executed by the controller (140), cause the autonomous driving control device (100) to: to identify the speed limit of the speed control section, and to identify a speed corresponding to a certain proportion of the speed limit as the lowest target speed of the host vehicle (301). [6] The autonomous driving control device (100) according to claim 4 or 5, wherein the first notification includes at least one of the positions of the start point (310) and the end point (320) of the speed control section, the real-time location of the host vehicle (301), a real-time traveling speed of the host vehicle (301), the speed limit, a guide for a change, in particular for a handover, the control authority for the host vehicle (301), or a combination of the positions of the start point (310) and the end point (320) of the speed control section, the real-time location of the host vehicle (301), the real-time traveling speed of the host vehicle (301), the speed limit, and the guide for changing the control authority for the host vehicle (301). [7] The autonomous driving control device (100) according to any one of claims 1 to 6, wherein the instructions, when executed by the control device (140), cause the autonomous driving control device (100) to: to identify a first road type of a traveled road (440) on which the host vehicle (401) is traveling, and when it is identified that a second road type of a road ahead (460) located within a certain distance from a real-time location of the host vehicle (401) is different from the first road type of the road being traveled (440), provide a second notification to the user using the notification device (120). [8] The autonomous driving control device (100) according to claim 7, wherein the second notification includes at least one of the first road type of the traveling road (440), the determined distance, a real-time distance remaining until the first road type is changed, the second road type of the road ahead (460), a real-time location of the host vehicle (401), a real-time traveling speed of the host vehicle (401), a guidance for changing the control authority for the host vehicle (401), or a combination of the first road type of the traveling road (440), the determined distance, the real-time distance remaining until the first road type is changed, the second road type of the road ahead (460), the real-time location of the host vehicle (401), the real-time traveling speed of the host vehicle (401), and the guidance for changing the control authority for the host vehicle (401). [9] The autonomous driving control device (100) according to any one of claims 1 to 8, wherein the instructions, when executed by the control device (140), cause the autonomous driving control device (100) to: if it is determined that the at least one target control element is not executable or advance warning is required regarding the situation information, determine whether NSCC is performed based on navigation information on the host vehicle (301, 401), and if it is determined that the NSCC is being performed, provide the at least one notification to the user using the notification device (120). [10] The autonomous driving control device (100) according to any one of claims 1 to 9, wherein the instructions, when executed by the control device (140), cause the autonomous driving control device (100) to: providing the at least one notification comprising at least one of a visual content, an auditory content, a haptic function, or a combination of the visual content, the auditory content, and the haptic function to the user using the notification device (120). [11] Autonomous driving control method, the method comprising: Identifying (S510), by a control device (140), situation information about at least one target control element for a host vehicle (301, 401) using a sensor device (110), Determining (S520) by the control device (140) on the basis of the situation information whether the at least one target control element is executable or whether a pre-warning with regard to the situation information is required, and Providing, by the control device (140), at least one notification to a user using a notification device (120) when it is determined that the at least one target control element is not executable or the advance warning regarding the situation information is required. [12] The method according to claim 11, wherein determining (S520) by the control device (140) on the basis of the situation information whether the at least one target control element is executable or whether the advance warning is required with regard to the situation information further comprises: Determining by the control device (140) whether the at least one target control element is executable on the basis of NSCC. [13] The method according to claim 11 or 12, wherein identifying (S510), by the control device (140), the situation information about the at least one target control element for the host vehicle (301, 401) using the sensor device (110) further comprises: while controlling the driving of the host vehicle (301, 401), when it is determined that longitudinal speed control is required, identifying, by the control device (140), the situation information about the at least one target control element having the longitudinal speed control. [14] Method according to one of claims 11 to 13, further comprising: when it is identified that the host vehicle (301) is traveling in a speed control section, identifying, by the control device (140), the situation information comprising at least one of a start point (310) and an end point (320) of the speed control section, a real-time location of the host vehicle (301), a lowest target speed of the host vehicle (301) by NSCC control, or a combination of the start point (310) and the end point (320) of the speed control section, the real-time location of the host vehicle (301), and the lowest target speed of the host vehicle (301) by the NSCC control, Identifying by the control device (140) whether an NSCC-based control is possible such that a driving speed of the host vehicle (301) at the end point (320) is lower than a speed limit of the speed control section, based on the situation information, if it is determined that control is not possible, providing, by the control device (140), a first notification to the user using the notification device (120). [15] The method of claim 14, wherein identifying, by the controller (140), the lowest target speed of the host vehicle (301) by the NSCC controller further comprises: Identifying, by the control device (140), the speed limit of the speed control section, and Identifying, by the controller (140), a speed corresponding to a certain portion of the speed limit as the lowest target speed of the host vehicle (301). [16] The method according to claim 14 or 15, wherein the first notification comprises at least one of the positions of the start point (310) and the end point (320) of the speed control section, the real-time location of the host vehicle (301), the real-time traveling speed of the host vehicle (301), the speed limit, an instruction for changing the control authority for the host vehicle (301), or a combination of the positions of the start point (310) and the end point (320) of the speed control section, the real-time location of the host vehicle (301), the real-time traveling speed of the host vehicle (301), the speed limit, and the instruction for changing the control authority for the host vehicle (301). [17] Method according to one of claims 11 to 16, further comprising: Identifying, by the control device (140), a first road type of a traveled road (440) on which the host vehicle (401) is traveling, when it is identified that a second road type of a road ahead (460) located at a certain distance from a real-time location of the host vehicle (401) is different from the first road type of the road being traveled (440), providing, by the control device (140), a second notification to the user using the notification device (120). [18] The method of claim 17, wherein the second notification comprises at least one of the first road type of the traveled road (440), the determined distance, a real-time distance remaining until the first road type is changed, the second road type of the road ahead (460), a real-time location of the host vehicle (401), a real-time traveling speed of the host vehicle (401), an instruction for changing the control authority for the host vehicle (401), or a combination of the first road type of the traveled road (440), the determined distance, the real-time distance remaining until the first road type is changed, the second road type of the road ahead (460), the real-time location of the host vehicle (401), the real-time traveling speed of the host vehicle (401), and the instruction for changing the control authority for the host vehicle (401). [19] Method according to one of claims 11 to 18, further comprising: if it is determined that the at least one target control element is not executable or the advance warning is required with regard to the situation information, determining by the control device (140) whether NSCC is performed on the basis of navigation information on the host vehicle (301, 401), and if it is identified that the NSCC is being performed, providing, by the control device (140), the at least one notification to the user using the notification device. [20] Method according to one of claims 11 to 19, further comprising: Providing, by the control device (140), the at least one notification comprising at least one of a visual content, an auditory content, a haptic function, or a combination of the visual content, the auditory content, and the haptic function to the user using the notification device (120).