Apparatus and method for user discomfort management for AR / VR applications in smart home environment

Abstract

A method of a device supporting augmented reality (AR) or virtual reality (VR) includes detecting an occurrence of at least one sensory discomfort of a user when the user uses the device supporting AR or VR; determining, by an operating state of a source Internet of Things (IoT) device, a source IoT device that triggers the occurrence of the at least one sensory discomfort of the user; providing at least one suggestion to the user through a user interface of the device; and adjusting the operating state of the source IoT device based on a response of the user to the provided at least one suggestion.

Description

Technical Field

[0001] This disclosure generally relates to the field of wearable display devices, and more specifically to an apparatus and method for managing user discomfort in augmented reality (AR) / virtual reality (VR) applications, such as in smart home environments. Background Technology

[0002] AR / VR devices have undergone significant evolution over the years, leading to increased demand. Technological advancements in areas such as display technology, computing power, and motion tracking have fueled this increased demand. These advancements have resulted in more immersive and realistic experiences.

[0003] There are several key aspects users need to consider when using AR / VR devices. One of the most important aspects is user comfort, which plays a crucial role in enhancing the overall experience. Several factors can affect user comfort when using AR / VR devices. One of these factors is the user's physical environment. The user's physical environment encompasses the surrounding environment where the content is presented to the user. The user's physical environment can also include other devices or objects that may affect the user experience. Unexpected events related to other devices can cause user discomfort when the user is immersed in AR / VR devices.

[0004] For example, a user's physical environment could be a smart home environment with multiple Internet of Things (IoT) devices. The operational status of these IoT devices could cause user discomfort. These IoT devices could include an oven, a music system, an air conditioner, a blender, and other IoT-enabled devices. In a non-limiting example, a strong odor from an oven could cause user discomfort during a VR session, thus affecting the user experience corresponding to the VR activity.

[0005] In another unrestricted example, when a user is participating in an online meeting during a VR session, loud and varied music from a music system can cause tactile discomfort and potentially negatively impact the user's experience. In yet another unrestricted example, when a user is exercising during a VR session, air conditioning running at a high temperature in economy mode can cause skin discomfort due to sweating. In this scenario, the user needs to stop exercising to manually change the air conditioning fan speed setting, which disrupts their rhythm and could potentially cause annoyance.

[0006] Therefore, there is a need for an improved method and apparatus that can overcome all the limitations and problems associated with existing AR / VR devices in the user's physical environment discussed above. Summary of the Invention

[0007] Technical solution This summary is provided to introduce, in a simplified format, aspects of this disclosure that will be further described in the detailed embodiments thereof. This summary is not intended to identify any critical or essential aspects of this disclosure, nor is it intended to define the scope of this disclosure.

[0008] According to one aspect of this disclosure, a method is provided for a device supporting augmented reality (AR) or virtual reality (VR). The method may include: detecting the occurrence of at least one sensory discomfort experienced by a user when using the AR- or VR-supporting device. The method may include: determining, through the operational state of a source Internet of Things (IoT) device, the source IoT device that triggered the occurrence of the at least one sensory discomfort in the user. The method may include: providing at least one suggestion to the user via a user interface of the device. The method may include: adjusting the operational state of the source IoT device based on the user's response to the provided at least one suggestion.

[0009] According to one aspect of this disclosure, an apparatus supporting augmented reality (AR) or virtual reality (VR) is provided. The apparatus may include: a memory storing instructions; and at least one processor configured to cause the apparatus to perform operations when the instructions are executed. The operations may include: detecting the occurrence of at least one sensory discomfort of the user when the user uses the AR- or VR-enabled apparatus. The operations may include: determining the source IoT device that triggered the occurrence of the at least one sensory discomfort of the user based on the operational state of the source Internet of Things (IoT) device. The operations may include: providing at least one suggestion to the user through the user interface of the apparatus. The operations may include: adjusting the operational state of the source IoT device based on the user's response to the provided at least one suggestion.

[0010] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided for storing instructions. The instructions, when executed by at least one processor of a device supporting augmented reality (AR) or virtual reality (VR), cause the device to perform operations. The operations may include: detecting the occurrence of at least one sensory discomfort of the user when the user uses the AR or VR-enabled device. The operations may include: determining, through the operational state of a source Internet of Things (IoT) device, the source IoT device that triggered the occurrence of the at least one sensory discomfort of the user. The operations may include: providing at least one suggestion to the user via a user interface of the device. The operations may include: adjusting the operational state of the source IoT device based on the user's response to the at least one suggestion provided.

[0011] To further illustrate the advantages and features of this disclosure, a more specific description of the disclosure will be presented with reference to specific embodiments of the disclosure shown in the accompanying drawings. It should be understood that these drawings depict only exemplary embodiments of the disclosure and are therefore not intended to limit its scope. The disclosure will be described and explained in the accompanying drawings with additional features and details. Attached Figure Description

[0012] These and other features, aspects, and advantages of this disclosure will become better understood when the following detailed description is read with reference to the accompanying drawings, in which the same reference numerals denote the same parts throughout the drawings, wherein: Figure 1 A block diagram of an augmented reality (AR) / virtual reality (VR) device for user discomfort management for AR / VR applications in a smart home environment, according to one or more embodiments disclosed herein, is shown. Figure 2 One or more embodiments according to the disclosure herein are shown. Figure 1 Detailed block diagram of the modules of the AR / VR device; Figure 3 A detailed block diagram of a sensory discomfort engine in a module of an AR / VR device according to one or more embodiments disclosed herein is shown; Figure 4 A flowchart is shown illustrating a method for user discomfort management for AR / VR applications in a smart home environment, according to one or more embodiments disclosed herein; Figure 5 A flowchart illustrating operations for adjusting the operational state of a source IoT device in one or more Internet of Things (IoT) devices in a smart home environment, according to one or more embodiments disclosed herein; Figure 6 A first use case scenario for user discomfort management when an unpleasant odor from burnt food causes discomfort to a user of an AR / VR device, according to one or more embodiments disclosed herein, is illustrated. Figure 7 A second use case scenario is illustrated, based on one or more embodiments disclosed herein, for user discomfort management when noise from a juicer causes discomfort to the user of an AR / VR device; Figure 8 A third use case scenario is illustrated, based on one or more embodiments disclosed herein, for user discomfort management when a loud music system causes discomfort to a user of an AR / VR device; Figure 9A fourth use case scenario is illustrated, based on one or more embodiments disclosed herein, for user discomfort management when a temperature setting in an air conditioner in normal mode causes discomfort to a user of an AR / VR device; and Figure 10 A fifth use case scenario is illustrated, based on one or more embodiments disclosed herein, for user discomfort management when discomfort is caused to a user of an AR / VR device due to numbness caused by overcooling.

[0013] Furthermore, those skilled in the art will understand that the elements in the accompanying drawings are shown for simplicity and may not necessarily be drawn to scale. For example, flowcharts illustrate methods according to the most prominent operations involved to aid in understanding various aspects of this disclosure. Additionally, in terms of the construction of the apparatus, one or more components of the apparatus may have been indicated by conventional symbols in the drawings, and the drawings may show only specific details relevant to understanding embodiments of this disclosure, so as not to obscure the details that would be readily understood by those of ordinary skill in the art who benefit from the description herein. Detailed Implementation

[0014] To facilitate understanding of the principles of this disclosure, reference will now be made to the embodiments shown in the accompanying drawings, and these embodiments will be described using specific language. However, it will be understood that this is not intended to limit the scope of the disclosure, and such changes and further modifications to the illustrated systems, as well as such further applications of the principles of the disclosure as illustrated herein, are expected to occur normally to those skilled in the art to which this disclosure pertains.

[0015] Those skilled in the art will understand that the foregoing general description and the following detailed description are for the purpose of interpreting this disclosure and not limiting it.

[0016] References to “aspect,” “on the other hand,” or similar language throughout this specification indicate that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. Therefore, the appearance of phrases such as “in an embodiment,” “in another embodiment,” and similar language throughout this specification may, but not necessarily all, refer to the same embodiment.

[0017] The terms “comprising,” “including,” or any other variation thereof are intended to cover non-exclusive inclusion, such that a process or method that includes the listed operations includes not only those operations but may also include other operations not expressly listed or inherent to such a process or method. Similarly, one or more devices, subsystems, elements, structures, or components preceding “comprising…” do not exclude the presence of other devices or subsystems or elements or structures or components, or additional devices or subsystems or elements or structures or components, without further constraints.

[0018] The embodiments described herein, along with their various features and beneficial details, are explained more fully with reference to the non-limiting embodiments illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques have been omitted to avoid unnecessarily obscuring the embodiments herein. Furthermore, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments. Unless otherwise indicated, the term "or" as used herein means non-exclusive or. The examples used herein are intended only to facilitate understanding of how the embodiments described herein can be practiced and to further enable those skilled in the art to practice the embodiments described herein. Therefore, these examples should not be construed as limiting the scope of the embodiments described herein.

[0019] As is conventional in the art, embodiments can be described and illustrated based on modules that perform one or more functions as described. These modules, which may be referred to herein as units or blocks, or may include blocks or units, are physically implemented by analog or digital circuitry (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuitry, etc.) and may optionally be driven by firmware and software. The circuitry may, for example, be implemented in one or more semiconductor chips, or on a substrate support such as a printed circuit board. The circuitry constituting a block may be implemented by dedicated hardware, processors (e.g., one or more programmed microprocessors and associated circuitry), or a combination of dedicated hardware for performing certain functions of the block and processors for performing other functions of the block. Without departing from the scope of this disclosure, each block of an embodiment may be physically divided into two or more interactive and separate blocks. Similarly, without departing from the scope of this disclosure, the blocks of an embodiment may be physically combined into more complex blocks.

[0020] The accompanying drawings are provided to aid in the easy understanding of the various technical features, and it should be understood that the embodiments presented herein are not limited to the drawings. Therefore, this disclosure should be construed as extending to any changes, equivalents, and alternatives other than those specifically set forth in the drawings. Although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are generally used only to distinguish one element from another.

[0021] The term “coupled” and its derivatives refer to any direct or indirect communication between two or more elements, regardless of whether these elements are physically in contact with each other. The terms “send,” “receive,” and “communicate,” and their derivatives include both direct and indirect communication. The phrase “associated with” and its derivatives refer to including, being included in, interconnected with, containing, being contained within, connected to or connected to, coupled to or coupled to, able to communicate with, cooperate with, interleaved, juxtaposed, proximate, bound to or bound to, having, possessing the properties of, having a relationship to or with, etc. The term “controller” refers to any device, system, or part thereof that controls at least one operation. The functionality associated with any particular controller can be centralized or distributed, local or remote. When used with listed items, the phrase “at least one of” indicates that different combinations of one or more of the listed items can be used, and only one of the listed items may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C, and any variations thereof. As an additional example, the expression "at least one of a, b, or c" can indicate only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof. Similarly, the term "group" means one or more. Therefore, a group of items can be a single item or a collection of two or more items.

[0022] Furthermore, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed and implemented in a computer-readable medium by computer-readable program code. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof adapted to be implemented in suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium accessible by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, compact disc (CD), digital video optical disc (DVD), or any other type of storage. "Non-transitory" computer-readable media does not include wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer-readable media includes media that can permanently store data and media that can store data and be rewritten later (such as rewritable optical discs or erasable memory devices).

[0023] The embodiments will now be described in detail with reference to the accompanying drawings.

[0024] Figure 1A block diagram of an augmented reality (AR) / virtual reality (VR) device 100 for user discomfort management for AR / VR applications in a smart home environment, according to one or more embodiments disclosed herein, is shown.

[0025] In one or more embodiments, the AR / VR device 100 is configured to receive input 102 from an Internet of Things (IoT) device 104 (also referred to as "one or more IoT devices 104"). Input 102 may correspond to IoT device data associated with one or more IoT devices 104. The AR / VR device 100 may also be configured to send output 106 to one or more IoT devices 104. Output 106 may correspond to control commands for controlling the functions of one or more IoT devices 104.

[0026] AR / VR device 100 includes a processor 108, one or more sensors 112, a memory 114, and an input / output (I / O) interface 120. The processor 108 includes one or more modules 110 (hereinafter referred to as "module 110") for performing operations for user comfort management. The memory 114 may include a database 116 and an operating system 118. AR / VR device 100 may correspond to a wearable display device, such as a VR device, an AR device, a mixed reality (MR) device, or any other similar electronic device.

[0027] In one or more embodiments, processor 108 may be operatively coupled to module 110 for processing, running, or performing a set of operations. In another embodiment, processor 108 may include at least one data processor for running processes in a virtual memory area network. Processor 180 may correspond to at least one (or more) processors. Processor 108 may include dedicated processing units such as integrated system (bus) controllers, memory management control units, floating-point units, graphics processing units, digital signal processing units, etc. In another embodiment, processor 108 may include a central processing unit (CPU), a graphics processing unit (GPU), or both. Processor 108 may be one or more general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, servers, networks, digital circuits, analog circuits, combinations thereof, or other known or later-developed means for analyzing and processing data. Processor 108 may execute one or more instructions (such as manually generated (i.e., programmed) code) to perform one or more operations disclosed herein.

[0028] In one or more embodiments, processor 108 includes module 110 for performing specific operations. The term "module" or "multiple modules" as used herein can refer to a unit comprising, for example, one or more of hardware, software, and firmware, or combinations of two or more of them. "Module" or "multiple modules" can be used interchangeably with terms such as logic, logic block, component, etc. A "module" or "multiple modules" can be a minimal device component for performing one or more functions, or can be a part of such a component. Processor 108 can control module 110 to perform a specific set of operations described in this disclosure.

[0029] In one or more embodiments, one or more sensors 112 may include a heart rate (HR) sensor, a breathing pattern sensor, a skin conductivity sensor, an odor sensor, a light intensity sensor, an audio sensor, and one or more image sensors. One or more image sensors are configured to capture a user's facial expressions, blink rate, eye movements, and pupil dilation. The HR sensor is configured to measure the user's heart rate. In a non-limiting example, the HR sensor may be an electronic or optical HR sensor. In a non-limiting example, the HR monitor may be a built-in or Bluetooth heart rate monitor. The breathing pattern sensor is configured to measure the user's breathing pattern. In a non-limiting example, the breathing pattern sensor used in the AR / VR device 100 may be a camera-based adaptive breathing sensor, a wearable breathing pattern sensor, or a non-contact breathing pattern sensor. These HR sensors and breathing pattern sensors are used in the AR / VR device 100 to enhance the immersive experience and provide additional data for health and fitness applications.

[0030] A skin conductivity sensor is configured to measure a user's skin conductivity. The skin conductivity sensor measures changes in the conductivity of the user's skin. These changes can be caused by sweat gland activity, which is controlled by the sympathetic nervous system and is closely related to arousal. The skin conductivity sensor can be used in an AR / VR device 100 to estimate a user's emotional state and stress level. In a non-limiting example, the skin conductivity sensor may include a self-adjusting skin conductance response sensor or a silver-silver chloride Velcro fastener electrode.

[0031] An odor sensor is configured to measure odors around the user. The odor sensor senses unpleasant odors or smells in the user's surrounding environment. A light intensity sensor is configured to measure the light intensity level of the glasses screen of the AR / VR device 100. The light intensity sensor measures the brightness of light. The light intensity sensor can be used in VR devices to enhance the immersive experience by providing accurate and realistic lighting, or to measure the light intensity level of the AR / VR device 100. In a non-limiting example, the light intensity sensor may include an imaging photometer and a colorimeter. An audio sensor is configured to capture the user's voice and the audio environment around the user.

[0032] In one or more embodiments, memory 114 may include any non-transitory computer-readable medium known in the art, including, for example, volatile memory (such as static random access memory (SRAM) and dynamic random access memory (DRAM)) and / or non-volatile memory (such as read-only memory (ROM), erasable programmable ROM, flash memory, hard disk, optical disk, and magnetic tape). Memory 114 is operatively coupled to processor 108 to store bit streams or processing instructions for performing one or more processes. Furthermore, memory 114 includes an operating system 118 for performing one or more tasks of the AR / VR device 100 (such as those performed by a general-purpose operating system in the communications domain). Additionally, database 116 stores information required by module 110 and processor 108 to perform one or more functions. Furthermore, memory 114 may store one or more values, such as, but not limited to, one or more intermediate data generated by module 110, parameters required by module 110, thresholds, etc. Memory 114 may store one or more models for performing operations as disclosed throughout this disclosure.

[0033] In one or more embodiments, I / O interface 120 refers to a hardware or software component that enables data communication between AR / VR device 100 and any other device or system. I / O interface 120 serves as a communication medium for exchanging information, commands, or data with other devices or systems. I / O interface 120 may be part of processor 108 or may be a separate component. I / O interface 120 may be created in software or may be a physical connection in hardware. I / O interface 120 may be configured to connect to an external network, external media, display, or any other component or combination thereof. The external network may be a physical connection (such as a wired Ethernet connection) or may be established wirelessly. In a non-limiting example, AR / VR device 100 may be configured to communicate with a cloud database via I / O interface 120 to store output 106. In another non-limiting example, AR / VR device 100 may be configured to communicate with one or more external devices or display units via I / O interface 120 to send output 106.

[0034] Figure 2 One or more embodiments according to the disclosure herein are shown. Figure 1 Detailed block diagram of module 110 of AR / VR device 100.

[0035] Module 110 includes a VR device and scene data aggregator module 201, an IoT device data aggregator module 203, a user history discomfort data aggregator module 207, a discomfort-related engine 205, and an action recommender and feature control module 209.

[0036] VR device and scene data aggregator module 201 includes VR sensor data module 211 and VR activity performance consistency tracker module 213. VR sensor data module 211 is configured to receive VR data from one or more sensors 112. VR data includes information associated with at least one of the following: user's heart rate, user's breathing pattern, user's skin conductance, odor around the user, brightness level of the AR / VR device 100's glasses screen, user's facial expression, user's blink rate, user's eye movement, user's pupil dilation, user's voice, and the audio environment around the user.

[0037] The VR sensor data module 211 can also be configured to aggregate received VR data and send the aggregated VR data to the discomfort-related engine 205. Examples of aggregated VR data are shown in Table 1 below.

[0038] [Table 1]

[0039] In some embodiments, the VR activity performance consistency tracker module 213 is configured to (continuously) monitor a user's VR screen activity while using the AR / VR device 100. The VR activity performance consistency tracker module 213 may also be configured to generate VR activity performance data based on (continuous) monitoring of the user's VR screen activity. The user's VR activity performance data includes content information and user movement information during the user's VR screen activity. Content information may include the type of content, content resolution, or frame rate of change associated with the content. In a non-limiting example, the type of content may be game content, movie, conference content, or training content. In a non-limiting example, the content resolution may be SD, HD, or HDR. Information on user movement during the user's VR screen activity may include the frequency of the user's VR screen activity start / stop, information on the user's head movement, information on the user's hand movement, and information on the user's body tremors.

[0040] For example, the VR activity performance consistency tracker module 213 continuously monitors the user's VR screen activity, and if any consistency interruption is observed during the user's VR screen activity (such as if the user is playing a game and then suddenly his movement slows down (reduced frame rate) or he has a start / stop activity pattern within a short interval), the VR activity performance consistency tracker module 213 records this information as information about the user's movement during the user's VR screen activity.

[0041] VR Activity Performance Consistency Tracker Module 213 can also be configured to send the user's VR activity performance data to the Discomfort-Related Engine 205. Examples of the user's VR activity performance data are shown in Table 2 below.

[0042] [Table 2]

[0043] In some embodiments, the IoT device data aggregator module 203 is configured to (periodically) monitor the operational status of each of one or more IoT devices in a smart home environment. The IoT device data aggregator module 203 is also configured to generate IoT device context data for each of the one or more IoT devices based on the monitored operational status of the respective IoT device. The IoT device context data includes the monitored operational status of the one or more IoT devices and operational features of each of the one or more IoT devices that modify the monitored operational status. The IoT device context data includes profiles of the one or more IoT devices, the current operating activities of the one or more IoT devices, and features that can be used to modify the current operating activities. The IoT device data aggregator module 203 may also be configured to send the IoT device context data to the discomfort-related engine 205. Examples of IoT device data are shown in Table 3 below.

[0044] [Table 3]

[0045] In some embodiments, the user history discomfort data aggregator module 207 is configured to detect sensory discomfort events associated with the user over a period of time. Sensory discomfort events are detected when the user stops AR / VR activity, manually operates one or more IoT devices, and resumes AR / VR activity. The user history discomfort data aggregator module 207 is also configured to generate historical discomfort data for the user based on the detected sensory discomfort events over a period of time. The user history discomfort data may include the type of sensory discomfort caused to the user, information corresponding to sensors among the one or more sensors that detect the sensory discomfort, the user's facial expression when the sensory discomfort occurs, the IoT device operated by the user among the one or more IoT devices when the sensory discomfort occurs, and at least one operational feature of the IoT device modified by the user. In a non-limiting example, the type of sensory discomfort may be tactile discomfort, olfactory discomfort, numbness, auditory discomfort, visual discomfort, and other discomforts caused to the user. In a non-limiting example, the type of facial expression of the user when the sensory discomfort occurs may be an expression of annoyance, disgust, pallor, or any other facial expression of the user.

[0046] User history discomfort data aggregator module 207 can also be configured to send user history discomfort data to the discomfort-related engine 205. Examples of user history discomfort data are shown in Table 4 below.

[0047] [Table 4]

[0048] In some embodiments, the discomfort-related engine 205 includes a discomfort engine 215 and a device-to-discomfort-related engine 217. The discomfort engine 215 is configured to detect at least one instance of discomfort experienced by the user while using the AR / VR device 100. Figure 3 A detailed block diagram of a sensory discomfort engine 215 in a module of an AR / VR device 100 according to one or more embodiments disclosed herein is shown. The sensory discomfort engine 215 includes a machine learning (ML) model 301 and a sensory discomfort data normalization module 303.

[0049] ML model 301 is an example of an artificial intelligence model. The AI-related functions according to this disclosure are operated by a processor (e.g., processor 108) and memory (e.g., memory 114). The processor may include one or more processors. Here, one or more processors may include general-purpose processors (such as central processing units (CPUs), application processors, or digital signal processors (DSPs)), graphics-specific processors (such as graphics processing units (GPUs) or vision processing units (VPUs)), or AI-specific processors (such as neural processing units (NPUs)). One or more processors control the processing of input data according to predefined operating rules or AI models stored in memory. Optionally, when one or more processors are AI-specific processors, the AI-specific processors may be designed with a hardware architecture dedicated to processing a specific AI model.

[0050] Predefined operating rules or artificial intelligence models are made through training. Here, "made through training" means that a base artificial intelligence model is trained using a learning algorithm with a large amount of training data, thereby making predefined operating rules or artificial intelligence models configured to perform desired characteristics (or purposes). This training can be performed within the apparatus itself that performs the artificial intelligence according to this disclosure, or it can be performed via a separate server and / or a separate system. Examples of learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0051] Artificial intelligence models can include multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and neural network computation is performed by calculating the results of the previous layer and these multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized using the training results of the artificial intelligence model. For example, multiple weight values ​​can be updated to minimize the loss or cost values ​​obtained from the artificial intelligence model during the training process. Artificial neural networks can include deep neural networks (DNNs), and examples of artificial neural networks can include, but are not limited to, convolutional neural networks (CNNs), DNNs, recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), and deep Q-networks.

[0052] In some embodiments, ML model 301 receives user historical discomfort data from user historical discomfort data aggregator module 207. ML model 301 uses the user historical discomfort data as training data for training itself. ML model 301 also receives VR data from VR sensor data module 211 and user VR activity performance data from VR activity performance consistency tracker module 213. ML model 301 uses the VR data and user VR activity performance data as input data.

[0053] The ML model 301 detects at least one sensory discomfort of the user based on the input data fed into the ML model 301 trained on the training data. The sensory discomfort data normalization module 303 normalizes the detected sensory discomfort to values ​​between [0-1] and outputs the sensory discomfort data to the device to the discomfort-related engine 217.

[0054] Examples of data on feelings of discomfort are shown in Table 5 below.

[0055] [Table 5]

[0056] In some embodiments, the device-to-discomfort correlation engine 217 is configured to determine the source IoT device that triggers at least one user's discomfort based on the operational state of one or more IoT devices in a smart home environment. For example, the device-to-discomfort correlation engine 217 may determine the correlation between one or more IoT devices, operational characteristics, and at least one discomfort based on IoT device context data and discomfort data. The device-to-discomfort correlation engine 217 may also determine the source IoT device and operational characteristics that modify the operational state of the source IoT device. The device-to-discomfort correlation engine 217 may determine the source IoT device and operational characteristics based on the determined correlation. In a non-limiting example, the determined correlation may be as shown in Table 6.

[0057] [Table 6]

[0058] In some embodiments, the motion recommender and function control module 209 includes a motion widget renderer and gesture mapper module 219 and a device controller module 221. The motion widget renderer and gesture mapper module 219 is configured to provide actionable suggestions to the user as a user interface in the AR / VR device 100. The motion widget renderer and gesture mapper module 219 can generate AR objects for controlling operational features of the source IoT device. For example, the motion widget renderer and gesture mapper module 219 can generate AR objects and map user control gestures to the AR objects. User control gestures include gestures for controlling operational features of the source IoT device. Controlling operational features can include turning a feature on or off, increasing or decreasing a feature value, or any other control operation. The motion widget renderer and gesture mapper module 219 can also provide the generated AR objects to the user as a user interface in the AR / VR device 100.

[0059] In some embodiments, the device controller module 221 adjusts the operating state of the source IoT device based on detected user responses to provided operable suggestions. The operating state of the source IoT device is adjusted to reduce at least one level of user discomfort. For example, the device controller module 221 may detect user gestures as user responses to provided operable suggestions. The device controller module 221 may also map detected user gestures to operation control commands for the source IoT device. The device controller module 221 may adjust the operating state of the source IoT device based on the mapping of detected user gestures to operation control commands for the source IoT device. The device controller module 221 may send operation control commands to the source IoT device to adjust the operating state of the source IoT device.

[0060] Figure 4 A flowchart is shown of a method 400 for user discomfort management of AR / VR applications in a smart home environment, according to one or more embodiments disclosed herein. Method 400 includes a series of operations 401 to 407 performed by a processor 108 of an AR / VR device 100.

[0061] In operation 401, processor 108 detects the occurrence of at least one sensory discomfort experienced by the user while using AR / VR device 100. Processor 108 may use an ML model to detect the occurrence of at least one sensory discomfort based on at least one of the user's historical discomfort data, VR data associated with AR / VR device 100, and the user's VR activity performance data. The flow of method 400 now proceeds to operation 403.

[0062] In operation 403, processor 108 identifies the source IoT device whose operational state in one or more IoT devices within the smart home environment triggers at least one user's perceived discomfort. Processor 108 also identifies operational characteristics of the source IoT device that modify its operational state. For example, processor 108 may determine the correlation between one or more IoT devices, operational characteristics, and at least one perceived discomfort based on IoT device context data and the detected occurrence of at least one perceived discomfort. Processor 108 may also determine the source IoT device and operational characteristics based on the determined correlation. The flow of method 400 now proceeds to operation 405.

[0063] In operation 405, processor 108 provides the user with operable suggestions as a user interface in AR / VR device 100. For example, processor 108 generates AR objects for controlling operational features of the source IoT device. Processor 108 may also provide the generated AR objects as a user interface in AR / VR device 100. The flow of method 400 now proceeds to operation 407.

[0064] In operation 407, processor 108 adjusts the operating state of the source IoT device to reduce at least one level of discomfort experienced by the user. Processor 108 may adjust the operating state of the source IoT device based on detected user responses to provided actionable suggestions.

[0065] Figure 5 A flowchart of a method 500 for adjusting the operational state of a source IoT device in one or more IoT devices in a smart home environment, according to one or more embodiments disclosed herein, is shown. Method 500 includes a series of operations 501 to 505 performed by a processor 108 of an AR / VR device 100.

[0066] In operation 501, processor 108 detects the user gesture as a user response to a provided actionable suggestion. The user gesture can be detected using any known gesture detection method. The flow of method 500 now proceeds to operation 503.

[0067] In operation 503, processor 108 maps the detected user gesture to an operation control command from the source IoT device. The process of method 500 now proceeds to operation 505.

[0068] In operation 505, processor 108 adjusts the operating state of the source IoT device based on the mapping of detected user gestures to operation control commands of the source IoT device. Processor 108 can send operation control commands to the source IoT device to adjust the operating state of the source IoT device.

[0069] Figure 6A first use case scenario is illustrated for user discomfort management when an unpleasant odor from burnt food causes discomfort to the user of AR / VR device 100, according to one or more embodiments disclosed herein.

[0070] In the first use case scenario, the user performs a VR screen activity and has placed food to be prepared in an oven. If the oven temperature is high, it may cause an irritating odor from the food. During the VR screen activity, the irritating odor from the oven can make the user uncomfortable, thus affecting the user's experience in the VR screen activity. The irritating odor is followed by the food burning and may cause the user to sneeze. In addition, the user may feel disgusted by the smell, and the feeling of disgust may be visible on their face.

[0071] In this scenario, the VR sensor data module 211 collects information from one or more sensors 112. The VR sensor data module 211 aggregates the sensor data into VR data, which may include information such as irregular breathing flow, high odor levels around the user, facial expressions of disgust, twitching of the user's nose, and audio environmental indications of sneezing. The VR sensor data module 211 sends the aggregated VR data to the discomfort engine 215. Furthermore, the IoT device data aggregator module 203 generates IoT device context data indicating the monitored operational status of one or more IoT devices. For example, the IoT device context data may indicate that a heating, ventilation, and air conditioning system, an oven, and a music system are operating.

[0072] The discomfort engine 215 detects the occurrence of odor discomfort in the user while using the AR / VR device 100. The device-to-discomfort engine 217 determines that the oven is the source IoT device that triggered the user's odor discomfort during operation. The motion widget renderer and gesture mapper module 219 can generate AR objects for controlling the oven temperature. The motion widget renderer and gesture mapper module 219 can provide the generated AR objects as a user interface in the AR / VR device 100. The AR objects provide triggers for using gestures to manipulate the problematic features (e.g., temperature), which reduces the user's olfactory discomfort and allows the user to continue their VR screen activity without interruption.

[0073] Figure 7 A second use case scenario is illustrated for user discomfort management when noise from a juice blender causes user discomfort to AR / VR device 100, according to one or more embodiments disclosed herein.

[0074] In the second use case scenario, a user performs a VR screen activity, and someone starts a juicer in the house. If the noise from the juicer is loud, it can cause auditory discomfort to the user, affecting their experience of the VR screen activity. The user may feel annoyed, and this annoyance can be seen on their face.

[0075] In this scenario, the VR sensor data module 211 collects information from one or more sensors 112. The VR sensor data module 211 aggregates the sensor data into VR data, which may include information such as the user's facial expression being distressed, the user's surrounding audio environment indicating high noise, and the user's audio input having a high decibel level and being abnormal. The VR sensor data module 211 sends the aggregated VR data to the discomfort engine 215. Furthermore, the IoT device data aggregator module 203 generates IoT device context data indicating the monitored operational status of one or more IoT devices. For example, the IoT device context data may indicate that the oven is on, the robot cleaner is charging, the juicer is running, the television is running at normal volume, and the pressure cooker is off.

[0076] The discomfort engine 215 detects auditory discomfort experienced by the user while using the AR / VR device 100. The device-to-discomfort engine 217 determines that the juicer is the source IoT device triggering the user's auditory discomfort during operation. The motion widget renderer and gesture mapper module 219 can generate AR objects for controlling the speed of the juicer. The motion widget renderer and gesture mapper module 219 can provide the generated AR objects as a user interface in the AR / VR device 100. The AR objects provide triggers for using gestures to operate the problematic feature (motor speed), which reduces the user's auditory discomfort and allows the user to continue their VR screen activity without interruption.

[0077] Figure 8 A third use case scenario is illustrated, based on one or more embodiments disclosed herein, for user discomfort management when a loud music system causes discomfort to the user of AR / VR device 100.

[0078] In the third use case scenario, a user performs a VR screen activity (such as a meeting), and someone plays loud music with varying pitches in the room. The high and low pitches in the music can cause tactile discomfort for the user, affecting their experience during the VR screen activity. The user may feel disgusted, and this disgust can be seen on their face.

[0079] In this scenario, the VR sensor data module 211 collects information from one or more sensors 112. The VR sensor data module 211 aggregates the sensor data into VR data, which may include information such as the user's facial expression of disgust, the surrounding audio environment indicating high volume and high-frequency sounds, and observed pupil dilation. The VR sensor data module 211 sends the aggregated VR data to the discomfort engine 215. Furthermore, the IoT device data aggregator module 203 generates IoT device context data indicating the monitored operational status of one or more IoT devices. For example, the IoT device context data may indicate that the HVAC system is running, the robot cleaner is charging, the blender is off, and the music system is on.

[0080] The discomfort engine 215 detects tactile discomfort experienced by the user while using the AR / VR device 100. The device-to-discomfort engine 217 determines that the music system is the source IoT device triggering the user's tactile discomfort during operation. The motion widget renderer and gesture mapper module 219 can generate AR objects for controlling the volume and bass settings of the music system. The motion widget renderer and gesture mapper module 219 can provide the generated AR objects as a user interface in the AR / VR device 100. The AR objects provide triggers for using gestures to operate the problematic features (volume and bass), which reduces the user's tactile discomfort and allows the user to continue their VR screen activity without interruption.

[0081] Figure 9 A fourth use case scenario is illustrated, based on one or more embodiments disclosed herein, for user discomfort management when the temperature setting in the air conditioner in normal mode causes discomfort to the user of AR / VR device 100.

[0082] In the fourth use case scenario, a user is exercising in VR, but the air conditioner is operating at normal temperature and in economy mode. The air conditioner's temperature setting can cause the user to sweat, and the user may experience skin discomfort due to sweating. Furthermore, the user may feel annoyed by the skin discomfort, and this annoyance is visible on their face.

[0083] In this scenario, the VR sensor data module 211 collects information from one or more sensors 112. The VR sensor data module 211 aggregates the sensor data into VR data, which may include information such as irregular breathing rate, moderate odor around the user, a distressed facial expression, high skin conductivity, high heart rate, and abnormal blink rate. The VR sensor data module 211 sends the aggregated VR data to the discomfort engine 215. Furthermore, the IoT device data aggregator module 203 generates IoT device context data indicating the monitored operational status of one or more IoT devices. For example, the IoT device context data may indicate that the HVAC system is at a normal temperature setting and the range hood is on.

[0084] The discomfort engine 215 detects numbness or skin discomfort experienced by the user while using the AR / VR device 100. The device-to-discomfort engine 217 determines that the HVAC system is the source IoT device triggering the numbness or skin discomfort. The motion widget renderer and gesture mapper module 219 can generate AR objects for controlling the temperature of the HVAC system. The motion widget renderer and gesture mapper module 219 can provide the generated AR objects as a user interface in the AR / VR device 100. The AR objects provide triggers for using gestures to operate the problematic features (temperature and settings), which reduces the user's numbness or skin discomfort and allows the user to continue their VR screen activity without interruption.

[0085] Figure 10 A fifth use case scenario is illustrated, based on one or more embodiments disclosed herein, for user discomfort management when numbness caused by overcooling causes discomfort to the user of AR / VR device 100.

[0086] In the fifth use case scenario, a user engages in VR screen activities for an extended period, but the air conditioning is running at a low temperature. Due to the excessive cold, the user's hands become numb, causing a loss of touch ability. Additionally, the user may experience skin discomfort and a pallor on their face.

[0087] In this scenario, the VR sensor data module 211 collects information from one or more sensors 112. The VR sensor data module 211 aggregates the sensor data into VR data, which may include information such as a high heart rate, heavy respiratory flow, decreased skin conductance, pale facial expression, low ambient temperature, and abnormal pupil dilation. The VR sensor data module 211 sends the aggregated VR data to the discomfort engine 215. Furthermore, the VR activity performance consistency tracker module 213 generates VR activity performance data indicating user trembling and sends this data to the discomfort correlation engine 205. Additionally, the IoT device data aggregator module 203 generates IoT device context data indicating the monitored operational status of one or more IoT devices. For example, the IoT device context data might indicate that the HVAC system is operating at a low temperature and that a music system is running.

[0088] The discomfort engine 215 detects numbness in the user while using the AR / VR device 100. The device-to-discomfort engine 217 determines that the HVAC system is the source IoT device triggering the user's numbness. The motion widget renderer and gesture mapper module 219 can generate AR objects for controlling the temperature of the HVAC system. The motion widget renderer and gesture mapper module 219 can provide the generated AR objects as a user interface in the AR / VR device 100. The AR objects provide triggers for using gestures to operate the problematic feature (temperature), which reduces the user's numbness and discomfort and allows the user to continue their VR screen activity without interruption.

[0089] In the examples, modules and / or units and / or models may include programs, subroutines, parts of programs, software components, or hardware components capable of performing the stated tasks or functions. As used herein, modules and / or units and / or models may be implemented independently of other modules on a hardware component such as a server, or modules may reside on the same server as other modules, or within the same program. Modules and / or units and / or models may be implemented on hardware components such as processors, one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any means of manipulating signals based on operating instructions. When executed by a processor, modules and / or units and / or models may be configured to perform any of the functions described.

[0090] The methods disclosed herein in one or more embodiments provide various technical benefits and advantages. These benefits and advantages include improving the user experience when using AR / VR devices 100 in a smart home environment by identifying IoT devices that cause discomfort to the user and allowing the user to adjust the operating state of the IoT devices via AR interaction to reduce user discomfort.

[0091] The various actions, behaviors, boxes, operations, etc. in the flowchart can be executed in the order they are presented, in different orders, or simultaneously. Furthermore, in some embodiments, without departing from the scope of this disclosure, some of the actions, behaviors, boxes, operations, etc., may be omitted, added, modified, or skipped.

[0092] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The systems, methods, and examples provided herein are illustrative only and not restrictive.

[0093] While specific language has been used to describe the subject matter, it is not intended to create any limitation. As will be apparent to those skilled in the art, various working modifications can be made to the methods to achieve the inventive concepts taught herein. The accompanying drawings and the foregoing description provide examples of embodiments. Those skilled in the art will understand that one or more of the described elements can be well combined into a single functional element. Alternatively, certain elements can be separated into multiple functional elements. Elements from one embodiment can be added to another embodiment.

[0094] The description of the specific embodiments above will fully reveal the general nature of the embodiments herein. Others can readily modify and / or adapt these specific embodiments for various applications by applying existing knowledge without departing from the general concept. Therefore, such modifications and adaptations should and are intended to be understood within the meaning and scope of equivalent forms of the disclosed embodiments. It will be understood that the wording or terminology used herein is for descriptive purposes and not for limitation. Therefore, although the embodiments herein have been described according to preferred embodiments, those skilled in the art will recognize that modifications can be used to practice the embodiments herein within the scope of the embodiments described herein.

Claims

1. A method for supporting an augmented reality (AR) or virtual reality (VR) device (100), the method comprising: Detection (401) When the user uses the device that supports AR or VR, at least one instance of discomfort experienced by the user occurs; The source IoT device is determined (403) by the operating state of the source IoT device, which triggers the occurrence of the user's at least one feeling of discomfort; At least one suggestion (405) is provided to the user through the user interface of the device; as well as The operating state of the source IoT device is adjusted (407) based on the user's response to the at least one suggestion provided.

2. The method according to claim 1, wherein, The step of detecting the occurrence of at least one feeling of discomfort in the user includes: Based on at least one of the user's historical discomfort data, VR data associated with the device, and the user's VR activity performance data, a machine learning (ML) model is used to detect the occurrence of the user's at least one sensory discomfort.

3. The method according to claim 2, further comprising: Detect sensory discomfort events associated with the user during a time period when the user interrupts AR / VR activity, manually operates one of the one or more IoT devices, and resumes AR / VR activity; as well as The user's historical discomfort data is generated based on the detected discomfort events within the time period.

4. The method according to claim 3, wherein, For each of the aforementioned discomfort events, the generated historical discomfort data for the user includes: The types of discomfort experienced by the user. Information corresponding to the sensor that detected the discomfort. The user's facial expressions when the discomfort occurs. The IoT device operated by the user when the discomfort occurs, and At least one operational feature of the IoT device modified by the user.

5. The method according to claim 2, further comprising: Monitor the user's VR screen activity when the user uses the device; as well as VR activity performance data of the user is generated based on the monitored VR screen activity of the user, wherein the generated VR activity performance data of the user includes at least one of the content information and the user's movement information during the VR screen activity of the user.

6. The method according to claim 2, further comprising: The VR data is received from one or more sensors of the device. The VR data includes at least one piece of information associated with at least one of the following: the user's heart rate, the user's breathing pattern, the user's skin conductance, the odor around the user, the brightness level of the device's glasses screen, the user's facial expression, the user's blink rate, the user's eye movement, or the user's pupil dilation.

7. The method according to claim 1, further comprising: Monitor the operational status of each IoT device in one or more IoT devices; as well as Based on the monitored operational status, generate IoT device context data for the corresponding IoT device among the one or more IoT devices. The IoT device context data includes the monitored operating state of the corresponding IoT device among the one or more IoT devices, and the operating characteristics of each of the one or more IoT devices configured to modify the monitored operating state.

8. The method according to claim 7, wherein, The steps for identifying the source IoT device include: Based on the context data of the IoT devices and the detected occurrence of at least one sensory discomfort, determine the correlation between the one or more IoT devices, the operational characteristics, and the at least one sensory discomfort; and The source IoT device and its operational characteristics are determined based on the determined correlation, and the operational characteristics of the source IoT device are configured to modify the operational state of the source IoT device.

9. The method according to claim 8, wherein, The steps of providing the user with the at least one suggestion include: Generate an AR object for controlling the operational features of the source IoT device; and The generated AR object is provided to the user as the user interface in the device.

10. The method according to claim 1, wherein, The steps for adjusting the operating state of the source IoT device include: The user's gesture is detected as the user's response to the at least one suggestion provided; Mapping the detected user gestures to operation control commands from the source IoT device; and The operating state of the source IoT device is adjusted based on the mapping of the detected user gesture to the operation control command of the source IoT device.

11. A device (100) supporting augmented reality (AR) or virtual reality (VR), the device comprising: Memory (114), store instructions; At least one processor (108) is configured to cause the device to perform operations when executing the instructions, the operations including: Detect the occurrence of at least one feeling of discomfort in the user when the user uses the device that supports AR or VR. The source IoT device is determined by its operational status, and the source IoT device triggers the occurrence of at least one of the user's sensory discomforts. Provide at least one suggestion to the user via the user interface of the device, and The operating state of the source IoT device is adjusted based on the user's response to the at least one suggestion provided.

12. The apparatus according to claim 11, wherein, The operation further includes at least one operation of the method according to any one of claims 2 to 10.

13. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor (108) of a device (100) supporting augmented reality (AR) or virtual reality (VR), cause the device to perform operations, the operations including: Detect the occurrence of at least one feeling of discomfort of the user when the user uses the device that supports AR or VR; The source IoT device is determined by the operating status of the source IoT device, and the source IoT device triggers the occurrence of the user's at least one feeling of discomfort; At least one suggestion is provided to the user through the user interface of the device; as well as The operating state of the source IoT device is adjusted based on the user's response to the at least one suggestion provided.

14. The non-transitory computer-readable storage medium according to claim 13, wherein, The operation further includes at least one operation of the method according to any one of claims 2 to 10.

Images