Temperature regulation feedback system responsive to user input

Abstract

A temperature adjustment feedback system responsive to user input is disclosed, comprising: receiving user input regarding a virtual object; updating the virtual object based at least in part on the user input; determining a temperature adjustment based at least in part on the user input; and outputting the temperature adjustment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 860,687 (Attorney Docket No. ALIBP411+), filed on June 12, 2019, and entitled AERIAL THERMAL FEEDBACK SYSTEM FOR 3D INTERACTION, which is incorporated herein by reference for all purposes. Background Art

[0003] Typically, in order to experience a product firsthand, a person must go to a physical store to test the item. However, going to a physical store to try out a product is not always convenient or practical. For example, a person can interact with a product remotely by browsing photos on a website, but this experience is not immersive and does not come close to actually using the product. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0005] Figure 1A is a diagram illustrating an embodiment of a system for providing a mid-air temperature regulation output in response to a user input.

[0006] Figure 1B is a functional diagram illustrating an embodiment of a host computer for providing temperature regulation responsive to user input.

[0007] Figure 2 is a diagram showing an example of a semi-air temperature adjustment device.

[0008] Figure 3 is a diagram illustrating an example of a system for providing a mid-air temperature regulation output in response to a user input.

[0009] Figure 4 is a flow chart illustrating an embodiment of a process for providing a mid-air temperature regulation output in response to a user input.

[0010] Figure 5 is a flow chart illustrating an embodiment of a process for providing a mid-air temperature regulation output in response to a user input.

[0011] Figure 6 is a flow chart illustrating an example of a process for providing a mid-air temperature adjustment output in response to a user input.

[0012] Fig. 7A and Figure 7B Examples of 3D virtual objects displayed by a system for providing a mid-air temperature regulation output in response to user input are described. DETAILED DESCRIPTION

[0013] The present invention may be implemented in a variety of ways, including as a process; an apparatus; a system; a combination of substances; a computer program product presented on a computer-readable storage medium; and / or a processor, for example, a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. In this specification, these implementations or any other form that the present invention may take may be referred to as technology. In general, within the scope of the present invention, the order of the steps of the disclosed process may be changed. Unless otherwise stated, a component described as being configured to perform a task (e.g., a processor or memory) may be implemented as a general component that is temporarily configured to perform a task at a given time, or a specific component that is manufactured to perform a task. The term "processor" used herein refers to one or more devices, circuits, and / or processing cores configured to process data (e.g., computer program instructions).

[0014] A detailed description of one or more embodiments of the present invention is provided below with reference to the accompanying drawings that illustrate the principles of the present invention. The present invention is described in relation to such embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many replacements, modifications and equivalents. Many specific details are listed in the following description to provide a thorough understanding of the invention. These details are provided for illustrative purposes, and the present invention can be implemented according to the claims without some or all of these specific details. For the sake of clarity, technical materials known in the technical field related to the present invention are not described in detail, so that the present invention will not be unnecessarily obscured.

[0015] Embodiments of providing a mid-air temperature adjustment output in response to user input are described herein. User input about a virtual object is received. In some embodiments, the virtual object is a three-dimensional (3D) object, which is rendered and presented by an automatic stereoscopic display so that the virtual object is presented as a 3D hologram. In some embodiments, the user input does not require the user to physically contact the physical object, and is detected using a sensor device. For example, the user input can be a hand movement, a foot movement, a head movement, and / or an eye movement. In some embodiments, due to the collision between the user input and the virtual object detected in the 3D space, the user input determines the virtual object. Determine the mid-air temperature adjustment at least in part based on the user input. In various embodiments, "mid-air temperature adjustment" includes temperature adjustment (e.g., heat, cold, and / or air) that can be perceived (e.g., by a user) without direct / physical contact with a physical object (e.g., a source of temperature adjustment). In some embodiments, the temperature adjustment is determined at least in part based on a measurement, and the measurement is determined based on a user input associated with at least a portion of the virtual object in the 3D space. Output mid-air temperature adjustment. For example, the mid-air temperature adjustment feedback includes heat blowing from at least one mid-air temperature adjustment device in the direction of the user's hand. In some embodiments, the appearance of the virtual object is updated in response to the user input. In some embodiments, feedback other than mid-air temperature adjustment is provided in response to the user input, for example, tactile feedback.

[0016] Figure 1A 1 is a diagram showing an embodiment of a system for providing a mid-air temperature adjustment output in response to a user input. In this example, the system 100 includes a host computer 112, a display device 102, a user input detection device 104, a mid-air temperature adjustment device 106, a mid-air tactile feedback device 108, and an additional feedback device 110. Figure 1A As shown, each of the display device 102, the mid-air tactile feedback device 108, the mid-air temperature adjustment device 106, and the additional feedback device 110 is connected to the host computer 112. Figure 1A As shown, the user input detection device 104 is connected to the display device 102. In some other embodiments, the user input detection device 104 can be directly connected to the host computer 112 instead of the display device 102. In some embodiments, one or more of the display device 102, the user input detection device 104, the mid-air temperature adjustment device 106, the mid-air tactile feedback device 108, and the additional feedback device 110 includes a corresponding driver that interfaces with it and another device that communicates with it.

[0017] The host computer 112 is configured to execute a software environment (e.g., a game engine) that is configured to render 3D virtual objects in a 3D space. For example, the software environment is a Unity game engine. Files / information for rendering 3D virtual objects can be stored locally on the host computer 112, or obtained from a remote server (not shown) via a network (not shown). In some embodiments, the 3D virtual objects rendered by the host computer 112 include products. In some embodiments, the 3D virtual objects rendered by the host computer 112 are at least partially animated. For example, the 3D virtual object may include a sprinkler head that emits an animation of a stream of water. The host computer 112 is configured to output at least one 3D virtual object to the display device 102.

[0018] The display device 102 is configured to present a 3D virtual object. In some embodiments, the display device 102 is configured to present the 3D virtual object as a 3D hologram. In various embodiments, the display device 102 is an autostereoscopic display device (e.g., a Seefront TM ) or Dimenco TM )), virtual reality display devices, or devices that display images that can give a 3D depth perception without requiring the viewer to use special headgear or glasses.

[0019] The user input detection device 104 includes one or more sensors that track user input / motion. For example, the user input detection device 104 is configured to track eye movement, hand movement, leg / foot movement and / or head movement. In response to each movement detected, the user input detection device 104 is configured to send a corresponding message to the display device 102, and the display device 102 can send a corresponding message to the host computer 112 in turn. Based on the message of the detected user input, the software environment executed on the host computer 112 is configured to determine whether the user input collides with at least a portion of the 3D virtual object presented on the display device 102, if so, then the appearance of the 3D virtual object is updated accordingly and / or causes the mid-air temperature adjustment device 106, the mid-air tactile feedback device 108, and / or other feedback devices 110 to output mid-air feedback. In some embodiments, when it is detected that the point in the 3D space where the user input is detected overlaps with the volume in the 3D space defined for the 3D virtual object, it is determined that the user input "collides" with at least part of the 3D virtual object. In other words, the user gesture interaction sensed by the user input detection device 104 may cause the host computer 112 to generate an updated appearance of the 3D virtual display presented by the display device 102 and / or cause mid-air feedback to be output by at least one peripheral feedback device (e.g., the mid-air temperature adjustment device 106, the mid-air tactile feedback device 108, and / or the additional feedback device 110).

[0020] For example, the host computer 112 can render a plurality of 3D virtual objects, including products downloaded from a remote server associated with an online shopping platform. The host computer 112 is configured to send the rendered plurality of 3D virtual objects to the display device 102, wherein the 3D virtual products are presented in the form of 3D holograms. The user input detection device 104 tracks user gestures regarding the 3D virtual products. The specific detected user gesture detected by the user input detection device 104 is determined by the host computer 112 to match a predetermined user gesture associated with selecting a specific 3D virtual product. In response to the detected predetermined selected user gesture, the software environment executed on the host computer 112 is configured to cause the selected 3D virtual product to be presented on the display device 102 and interact with it via user input. In a specific example, the initially presented 3D virtual product includes a 3D shower head and / or a faucet, and the selected specific item is a specific model of the shower head. Once the shower head is selected, the user can continue to interact with the selected shower head via user input to obtain an immersive mid-air sensory experience, as described below.

[0021] The mid-air temperature regulation device 106 is configured to receive instructions from the host computer 112 to output temperature regulation feedback. In various embodiments, the mid-air temperature regulation device 106 includes a heating component and a fan. In some embodiments, the mid-air temperature regulation device 106 includes a heating component, a cooling component, and a fan. In some embodiments, the mid-air temperature regulation device 106 is instructed by the host computer 112 to activate the fan to generate a cool breeze, thereby reducing the temperature near the system 100 and / or causing the user to feel the wind (e.g., the user's hand). In some embodiments, the mid-air temperature regulation device 106 is configured to activate a heating component (e.g., a positive temperature coefficient (PTC)) to generate heat, thereby increasing the temperature near the system 100. In some embodiments, the mid-air temperature regulation device 106 is configured to activate a cooling component (e.g., a thermoelectric cooler) to cool, thereby reducing the temperature near the system 100. In some embodiments, the mid-air temperature regulation device 106 is configured to activate the fan together with at least one of the heating component and the cooling component to distribute heat and / or cold air to a specific direction away from the mid-air temperature regulation device 106 and toward the user. In some embodiments, the degree (e.g., temperature) of temperature conditioning (e.g., heat or cold) produced by the mid-air temperature conditioning device 106 is determined and indicated by the host computer 112. In some embodiments, the strength and / or speed of wind produced by the mid-air temperature conditioning device 106 is determined and indicated by the host computer 112.

[0022] When the user perceives the mid-air temperature regulation feedback, the user may feel warm or cool, as if they were close to or even directly in contact with the temperature regulation source. In some embodiments, the mid-air temperature regulation device 106 is configured to output temperature regulation feedback in response to user interaction / input with respect to the 3D virtual object presented by the display device 102. For example, if the user interacts with the presented 3D virtual object in a manner that allows the user to experience heat emanating from the 3D virtual object, the mid-air temperature regulation device 106 is configured to activate its heating component and its fan to transfer heat to the user as if the heat was emanating from the 3D virtual object.

[0023] Although only one instance of the mid-air temperature adjustment device 106 is shown in the system 100, multiple instances of the mid-air temperature adjustment device 106 can be connected to the host computer 112. For example, each instance of the mid-air temperature adjustment device 106 can be arranged at a different position (e.g., relative to another peripheral device of the host computer 112, such as the display device 102 and the mid-air tactile feedback device 108) to provide temperature adjustment feedback from different orientations / positions. For example, multiple instances of the mid-air temperature adjustment device 106 can provide different temperature feedback for their different orientations / positions. In some embodiments, the host computer 112 can select all instances of the mid-air temperature adjustment device 106, or only a subset of the instances of the mid-air temperature adjustment device 106, to output temperature adjustment feedback based on factors such as the position of the user (e.g., the position of the user's palm, which is more sensitive to detecting wind and / or temperature changes than the back of the hand) and / or the direction of the selected 3D virtual object currently interacting with the user.

[0024] The mid-air tactile feedback device 108 is configured to receive instructions from the host computer 112 to output tactile feedback. Multiple tactile (or single tactile) is the science and engineering of applying tactile in a computer system. In some embodiments, mid-air tactile includes non-contact tactile feedback that a user senses in mid-air. There are several types of mid-air tactile feedback devices including, for example, ultrasonic vibration using a two-dimensional (2D) array of ultrasonic transducers, laser-based tactile feedback, and air vortex using a driven flexible nozzle with a subwoofer. In various embodiments, the mid-air tactile feedback device 108 is configured to emit ultrasonic waves to generate tactile sensations at one or more focal points. For example, the focal point of the ultrasonic wave in 3D space corresponds to the detected position of the user's hand, thereby causing the user to perceive tactile feedback. In some embodiments, the degree (e.g., pressure) and / or position of the tactile feedback generated by the mid-air tactile feedback device 108 is determined and indicated by the host computer 112.

[0025] When the user perceives the mid-air tactile feedback, they may feel pressure if the user directly touches the physical object. In some embodiments, the mid-air tactile feedback device 108 is configured to output tactile feedback in response to user interaction / input about the 3D virtual object presented by the display device 102. In one example, if the user browses multiple 3D virtual objects or 3D environments / scenes (e.g., menus), the mid-air tactile feedback device 108 is configured to output tactile feedback, allowing the user to perceive that they are physically moving items or to make selections (e.g., specific 3D virtual objects). In another example, if the user interacts with the presented 3D virtual object in a manner that allows the user to touch the 3D virtual object, the mid-air tactile feedback device 108 is configured to output tactile feedback, causing the user to feel that his or her hand is contacting the object.

[0026] Although only one instance of the mid-air haptic feedback device 108 is shown in the system 100, multiple instances of the mid-air haptic feedback device 108 can be connected to the host computer 112. For example, each instance of the mid-air haptic feedback device 108 can be arranged at a different position (e.g., relative to another peripheral device of the host computer 112, such as the display device 102 and the mid-air temperature adjustment device 106) to provide haptic feedback from different orientations / positions. In some embodiments, the host computer 112 can select all instances of the mid-air haptic feedback device 108, or only a subset of the instances of the mid-air haptic feedback device 108, to output haptic feedback based on factors such as the position of the user (e.g., the position of the user's palm, which is more sensitive to detecting wind and / or temperature changes than the back of the hand) and / or the direction of the selected 3D virtual object currently interacting with the user.

[0027] In an embodiment, the host computer 112 is configured to cause the mid-air temperature adjustment device 106 and the mid-air haptic feedback device 108 to simultaneously output feedback so that the user perceives that he or she is touching or contacting a tangible object that provides coolness or warmth without ever touching a physical object or source of heat / cold. Returning to the above example, in which the user selects a 3D virtual object including a shower head to further interact with it, the display device 102 is configured to amplify the presentation of the 3D virtual shower head or otherwise make it more conspicuous so that the user can more easily try it. For example, the user can use his or her hands to cause the 3D hologram of the shower head to spray water, and the 3D hologram of the shower head will be updated to show water spraying from the 3D virtual shower head in a manner close to the physical version of the same shower head. For example, the user's hand movements will be tracked by the user input detection device 104 and ultimately input to the host computer 112. If the software environment executed on the host computer 112 determines that the user input matches a predetermined user gesture associated with causing the 3D virtual shower head to spray water (e.g., turning the palm in a certain direction), the software environment is configured to update the appearance of the 3D virtual shower head by causing the display device 102 to additionally present an animation of water spraying from the shower head. For example, the user can interact with the 3D virtual shower head and the animated water flow to change the shape of the water flow and trigger temperature adjustment feedback and tactile feedback output, so that the user's hand can feel pressure and warmth from the water sprayed by the 3D virtual shower head. In another example, if the 3D virtual object includes both a 3D virtual shower head and a 3D virtual bathtub faucet, the 3D virtual shower head and the 3D virtual bathtub faucet can spray water at the same time to interact with the user. For example, the user's hand movement will be tracked by the user input detection device 104 and ultimately input to the host computer 112. If the software environment executing on the host computer 112 determines that the user input collides with the animation of the 3D virtual shower head and / or 3D virtual bathtub faucet spraying water in the 3D space, the software environment is configured to update the appearance of the 3D virtual shower head and / or 3D virtual bathtub faucet by causing the display device 102 to display a changing animation of water spraying from the shower head and / or bathtub faucet (where the water flow is deformed due to the collision with the user's hand). In addition, if the software environment executing on the host computer 112 determines that the user input collides with the animation of the 3D virtual shower head and / or 3D virtual bathtub faucet spraying water in the 3D space, the host computer 112 is configured to send instructions to the mid-air temperature regulation device 106 to cause the mid-air temperature regulation device 106 to generate heat and activate a fan to distribute the heat and is configured to send instructions to the mid-air haptic feedback device 108 to cause the mid-air haptic feedback device 108 to generate tactile sensations at a focal point corresponding to the position of the user's hand in the 3D space. As a result of temperature regulation and haptic feedback, users can experience an immersive simulated attempt of water spraying from a selected sprinkler head without having to directly contact or even touch the physical item.If multiple instances of the mid-air temperature regulating device 106 and the mid-air haptic feedback device 108 are used, the host computer 112 may send different signals to each instance of the device to cause the instance of the device to emit different feedback. For example, if the 3D virtual object can produce hot water and cold water from different nozzles at the same time, the host computer 112 may send a set of signals to cause one instance of the mid-air temperature regulating device 106 to emit hot air, while the host computer 112 may send another set of signals to cause another instance of the mid-air temperature regulating device 106 to emit cold air.

[0028] Additional feedback device 110 is configured to provide additional feedback except tactile or based on temperature. In one example, additional feedback device 110 includes a speaker, which is configured to provide feedback based on audio. In another example, additional feedback device 110 is configured to provide smell or feedback based on fragrance. In certain embodiments, in response to being tracked by user input detection device 104 and forwarded to user input of host computer 112, host computer 112 is configured to determine whether user input and / or the 3D virtual object currently presented by display device 102 will trigger audio and / or feedback based on fragrance.

[0029] Figure 1B is a functional diagram illustrating an embodiment of a host computer for providing temperature adjustment in response to user input. It will be apparent that other computer system architectures and configurations may be used to provide temperature adjustment in response to user input. In some embodiments, Figure 1A The host computer 112 of the system 100 can be implemented using a computer system 150. The computer system 150, which includes various subsystems as described below, includes at least one microprocessor subsystem (also referred to as a processor or central processing unit (CPU)) 152. For example, the processor 152 can be implemented by a single-chip processor or multiple processors. In some embodiments, the processor 152 is a general-purpose digital processor that controls the operation of the computer system 150. Using instructions retrieved from the memory 160, the processor 152 controls the receipt and manipulation of input data, and the output and display of data on an output device (e.g., a display 168).

[0030] The processor 152 is bidirectionally coupled to a memory 160, which may include a first main storage area, typically a random access memory (RAM), and a second main storage area, typically a read-only memory (ROM). As is well known in the art, the main storage can be used as a general storage area and scratch-pad memory, and can also be used to store input data and processing data. The main storage can also store programming instructions and data in the form of data objects and text objects, as well as other data and instructions for processes running on the processor 152. As is also well known in the art, the main storage generally includes basic operating instructions, program code, data, and objects used by the processor 152 to perform its functions (e.g., programming instructions). For example, the memory 160 can include any suitable computer-readable storage medium, as described below, depending on, for example, whether data access requires bidirectional or unidirectional. For example, the processor 152 can also directly and very quickly retrieve and store frequently needed data in a cache memory (not shown).

[0031] Removable mass storage device 162 provides additional data storage capacity for computer system 150 and is coupled bidirectionally (read / write) or unidirectionally (read-only) to processor 152. Storage 162 may also include computer readable media such as magnetic tape, flash memory, PC cards, portable mass storage devices, holographic storage devices, and other storage devices, for example. Fixed mass storage 170 may also provide additional data storage capacity, for example. The most common example of fixed mass storage 170 is a hard disk drive. Mass storage 162, 170 typically stores additional programming instructions, data, etc., which are not typically actively used by processor 152. It should be understood that the information retained in mass storage 162 and 170 may be used as virtual memory in a standard manner as part of memory 160 (e.g., RAM), if desired.

[0032] In addition to providing access to the storage subsystem for the processor 152, the bus 164 can also be used to provide access to other subsystems and devices. As shown, these can include a display 168, a network interface 166, a keyboard 154, and a pointing device 158, as well as auxiliary input / output (I / O) device interfaces, a sound card, speakers, and other required subsystems. For example, the pointing device 158 can be a mouse, a stylus, a trackball, or a tablet computer, and is very useful for interacting with a graphical user interface.

[0033] The network interface 166 allows the processor 152 to be coupled to another computer, computer network, or telecommunication network using a network connection as shown. For example, through the network interface 166, the processor 152 can receive information (e.g., data objects or program instructions) from another network or output information to another network during the execution of the method / process steps. Information, usually represented as a sequence of instructions executed on the processor, can be received from another network and output to another network. An interface card or similar device and appropriate software implemented (e.g., executed / implemented) by the processor 152 can be used to connect the computer system 150 to an external network and transmit data according to a standard protocol. For example, the various process embodiments disclosed herein can be executed on the processor 152 or on a network with a remote processor that shares part of the processing, such as the Internet, an intranet, or a local area network. Additional mass storage devices (not shown) can also be connected to the processor 166 through the network interface 152.

[0034] An auxiliary I / O device interface (not shown) may be used in conjunction with the computer system 150. The auxiliary I / O device interface may include general and custom interfaces that allow the processor 152 to send and typically receive data from other devices, such as microphones, touch-sensitive displays, sensor readers, tape readers, voice or handwriting recognizers, biometric readers, cameras, portable mass storage devices, and other computers.

[0035] Figure 2 is a diagram showing an example of a mid-air temperature adjustment device. In some embodiments, Figure 1A The mid-air temperature regulating device 106 of the system 100 is used Figure 2 This is achieved by the mid-air temperature regulating device 206.

[0036] As shown in FIG. 2 , the mid-air temperature regulating device 206 includes a fan 202 and a heating element 204, such as a positive temperature coefficient (PTC), which are respectively connected to a drive circuit 208 with a microcontroller. The drive circuit 208 with a microcontroller is in turn connected to a host computer (e.g., Figure 1AThe main computer 112 of the system 100 is provided. The drive circuit 208 with a microcontroller is configured to interface with the main computer. For example, instructions from the main computer are received on the drive circuit 208 with a microcontroller, and the drive circuit 208 with a microcontroller is configured to send corresponding instructions to one or both of the fan 202 and the heating component 204. In some embodiments, the fan 202 and the heating component 204 can be instructed to be activated at different times. For example, the heating component 204 can be instructed to be activated (e.g., turned on) when a user makes a specific selection of a user input detection device connected to the main computer. For example, the heating component 204 can be activated when a subsequent output of the temperature adjustment feedback is expected, because the heating component 204 cannot generate heat immediately. Therefore, at some time after the heating component 204 is activated, the drive circuit 208 with a microcontroller receives instructions from the main computer to activate the fan 202. Since the heating component 204 is activated in advance and has generated heat, the subsequent activation of the fan 202 will cause the heat to be distributed / transferred from the mid-air temperature adjustment device 206.

[0037] The following is an example process by which the mid-air temperature regulation device 206 is configured to generate temperature regulation: When the driver circuit 208 with a microcontroller receives an activation signal from the host computer, it can activate the heating component 204 and / or the fan 202 at the same time or later, depending on the type of the activation signal. When the driver circuit 208 with a microcontroller receives a deactivation signal from the host computer, it can deactivate the heating component 204 and / or the fan 202 at the same time, depending on the type of the deactivation signal. The host computer can only send an activation signal to the fan 202 in case the heated air needs to be cooled. In some embodiments, the host computer includes a timer to track the time that the heating component 204 is turned on, and sends a deactivation signal after the timer expires (e.g., to prevent the heating component 204 from overheating). For example, the timer is configured (e.g., by a user) to start at a certain value (e.g., 10) and start counting down when the heating component 204 begins to generate heat. Once the timer reaches zero, the host computer sends a deactivation signal to the driver circuit 208 with a microcontroller to cause the heating component 204 and the fan 202 to be deactivated.

[0038] In some embodiments, the drive circuit 208 with a microcontroller is configured to receive instructions related to a specific degree and / or intensity at which one or both of the fan 202 or the heating assembly 204 will operate. The drive circuit 208 with a microcontroller is configured to convert the specific degree and / or intensity into corresponding computer instructions, which are configured to cause the corresponding fan 202 or one or both of the heating assembly 204 to operate accordingly and send the corresponding computer instructions to one or both of the fan 202 or the heating assembly 204.

[0039] Figure 3 is a diagram illustrating an example of a system for providing a mid-air temperature adjustment output in response to a user input. Figure 3 In the example of FIG. 3 , the system 300 includes a display device 310, a motion detection device 308, a mid-air tactile feedback device 306a, a mid-air tactile feedback device 306b, a mid-air temperature adjustment device 304, and a host computer (not shown). Figure 3 As shown in FIG. 1 , the host computer is connected to each of the display device 310 , the motion detection device 308 , the mid-air tactile feedback device 306 a , the mid-air tactile feedback device 306 b , and the mid-air temperature adjustment device 304 , and receives and sends instructions thereto.

[0040] In this example, the display device 310 includes an autostereoscopic display that presents a 3D hologram of an object to the user 302, and the user does not need to wear special glasses to experience the 3D depth of the hologram. In other embodiments, the display of the virtual display requires special glasses, goggles, or other equipment can be used. The virtual object presented by the display device 310 is rendered by a software environment (e.g., a game engine) executed on a host computer. For example, the virtual object includes two stereo images, which can be displayed in an autostereoscopic 3D display. The motion detection device 308 is built into the display device 310 and is configured to track the user's movements, including, for example, eye and / or hand movements. The motion detection device 308 is configured to track the orientation and posture of the user's 302 hand. The mid-air tactile feedback device 306a is placed in front of and above the display device 310, and the mid-air tactile feedback device 306b is placed in front of and below the display device 310 to provide tactile feedback to the user 302 from the top and bottom, respectively, to match the binocular illusion (e.g., 3D virtual object) presented in mid-air. In Figure 3 In FIG. 3 , each mid-air tactile feedback device 306a and 306b includes an ultrasonic tactile device that includes a 2D array of multiple transducers, and these transducers produce focused ultrasonic vibrations so that the user 302 can feel the touch when the hand is placed near it. As shown in FIG. 3 , the mid-air temperature adjustment device 304 is placed next to the mid-air tactile feedback device 306a to provide heated air in conjunction with (or independently of) tactile feedback. For example, the angle of the mid-air temperature adjustment device 304 is between 30 degrees and 45 degrees, facing the focus of the mid-air tactile feedback device 306a. Although not shown in FIG. Figure 3306a or 306b is activated to generate focused tactile feedback, heated air is also generated by the corresponding mid-air temperature regulation device to provide temperature regulation feedback, which can be sensed by the palm of the user 302, for example, the user 302 interacts with the 3D object presented by the display device 310. For example, based on the predicted current position of the palm of the user 302 and / or the direction of the 3D virtual object presented by the display device 310, the tactile and / or temperature regulation feedback can be output from either the mid-air tactile feedback device 306a and the mid-air temperature regulation device 304 or the mid-air tactile feedback device 306b and the corresponding mid-air tactile feedback device.

[0041] Figure 4 is a flow chart illustrating an embodiment of a process for providing a mid-air temperature adjustment output in response to a user input. In some embodiments, process 400 may be performed in Figure 1A is implemented on the system 100.

[0042] At 402, information associated with a plurality of virtual objects is obtained from an online platform server. In some embodiments, information corresponding to a virtual object corresponding to a new product is obtained from a server associated with an online shopping platform. For example, the information associated with the virtual object includes an image that can be used to render a 3D hologram of a corresponding new physical product sold on the online shopping platform. The information associated with the virtual object also includes a description of the features and / or specifications of the new product corresponding to the virtual object.

[0043] At 404, a plurality of virtual objects are caused to be presented. The virtual objects are presented as 3D holograms on a display device, and a user can browse through them via user input (e.g., hand movements). For example, user interaction with the presented 3D virtual objects can allow a cursor to highlight any one of the 3D virtual objects.

[0044] At 406, a selection is received associated with the presented virtual object. In some embodiments, a user selection gesture is determined for selecting a particular presented 3D virtual object (e.g., an action by a user performing a pushing motion with his or her hand). If a predetermined user selection gesture is detected for a 3D virtual object currently highlighted by the cursor, the 3D virtual object is selected for further interaction only. In some embodiments, the 3D virtual object is automatically selected if no 3D virtual object has been detected within a predetermined time since the cursor highlighted the 3D virtual object (e.g., in anticipation of a user's potential interest in the 3D virtual object and / or the system may not be able to detect the predetermined user selection gesture that the user is attempting to perform).

[0045] An example application of mid-air temperature-regulated output in response to user input is a holographic signage kiosk setup to showcase virtual products. Users can interact with virtual products using mid-air gestures and movements as well as eye movements, and their hands receive simulated warmth or coldness when they collide with the 3D virtual objects in different ways.

[0046] Figure 5 is a flow chart illustrating an embodiment of a process for providing a mid-air temperature adjustment output in response to a user input. In some embodiments, the process 500 may be performed in Figure 1A is implemented on the system 100.

[0047] At 502, user input regarding a virtual object is received. The user input includes user movement, such as eye movement, hand movement, leg movement, foot movement, and / or head movement. In some embodiments, the virtual object is presented on a display. For example, the virtual object is a user's movement in a Figure 4 The object selected in process 400. For example, the display includes an autostereoscopic display. In various embodiments, if it is determined that the location of the user input collides with the virtual object in 3D space (e.g., is located within the collision area of ​​), it is determined that the user input is associated with the virtual object.

[0048] At 504, the virtual object is updated based at least in part on the user input. In some embodiments, the appearance of the virtual object is updated based on the user input. For example, the shape and / or associated animation of the virtual object is updated based on the location of the user input associated with the location of the virtual object in 3D space (e.g., a collision area), and the updated virtual object is presented. For example, the user input may have deformed at least a portion of the virtual object because the virtual object is configured to be compressible (e.g., the virtual object is a sofa cushion). In another example, the user input may have changed the associated animation of the virtual object (e.g., the virtual object includes a shower head with an animation of water spraying from the shower head).

[0049] At 506, mid-air temperature adjustment feedback is determined based at least in part on the user input. A corresponding mid-air temperature adjustment feedback including at least one of air and warm or cold is determined based on the user input. For example, the corresponding mid-air temperature adjustment feedback is generated based on a user input position regarding a position of at least a portion of the virtual object in the 3D space.

[0050] At 508, a mid-air temperature adjustment feedback is output. The corresponding mid-air temperature adjustment feedback is output in the direction of the user input and / or the user's current position. In some embodiments, the corresponding mid-air temperature adjustment feedback is output together with another half-air feedback (e.g., tactile feedback) also determined based on the user input.

[0051] An example application for providing temperature adjustment feedback in response to user input is in a shopping area, where a user may wish to interact with a virtual product before purchasing the physical version of the product. The user can view not only the product details, but also the features that the product can provide. For example, the user may want to try and feel the pressure of the new water jet characteristics of a newly released luxury jet bathtub. In another example, the user may also want to try and feel the different types of hot waterfall settings of a new shower head product. By applying the temperature adjustment feedback system according to some embodiments described herein, the user can interact with the characteristics of the 3D virtual product and receive multi-sensory feedback to experience a more realistic and intuitive product interaction simulation.

[0052] Figure 6 is a flow chart illustrating an example of a process for providing a mid-air temperature adjustment output in response to a user input. In some embodiments, process 600 may be performed at Figure 1A In some embodiments, Figure 4 The process 400 may be implemented at least in part by the process 600. In some embodiments, Figure 5 Process 500 may be implemented at least in part by process 600 .

[0053] At 602, a generated 3D virtual environment is presented. The 3D virtual environment includes a 3D virtual scene. In some embodiments, one or more 3D virtual objects are presented in the 3D virtual environment. The 3D virtual environment and / or the 3D virtual objects are presented by a display device (e.g., autostereoscopic). In some embodiments, the 3D virtual environment and / or the 3D virtual objects are created using a game engine (e.g., Unity). For example, the 3D virtual scene includes a bathroom, and the 3D virtual objects in the scene are different bathroom facilities (e.g., a bathtub, a shower head, and / or a toilet).

[0054] At 604, a user selection of a 3D virtual object is detected in the 3D virtual environment. The user selects a particular 3D virtual object based on, for example, detected user input matching a predetermined selection gesture. The selected 3D virtual object can be enlarged in the presentation via a display device to make its features more visible to the user.

[0055] In some embodiments, the 3D virtual object includes animation. For example, the 3D virtual object is a combination of a bathroom accessory and a liquid (e.g., water) released from the accessory. Specifically, the 3D virtual object includes a stream of water flowing from a faucet, shower head, or nozzle. In some embodiments, the 3D virtual object is animated. For example, if the 3D virtual object is a stream of water, the animation is that water is continuously sprayed from the source (e.g., a faucet, shower head, or nozzle).

[0056] At 606, user input is detected. The motion detection device detects the motion / input of the user (e.g., hand, eye, head, foot, and / or leg). In some embodiments, one or more points in 3D space associated with the user input are determined. For example, coordinates associated with the position of the user's hand in 3D space are determined.

[0057] At 608, it is determined whether the user input collides with the 3D virtual object. If the user input collides with the 3D virtual object, control will be transferred to 610. Otherwise, if the user input does not collide with the 3D virtual object, control will return to 606 to wait for user input.

[0058] In some embodiments, in order to determine whether the user input collides with the 3D virtual object, the point associated with the user input in the 3D space is associated with the collision area of ​​the 3D virtual object. For example, the collision area of ​​the 3D virtual object can be defined as a portion of the 3D virtual object or the entire volume of the 3D virtual object. For example, if the coordinates associated with the position of the user's hand in the 3D space overlap with any defined collision area of ​​the 3D virtual object, it is determined that there is a collision between the user input and the 3D virtual object. However, if the coordinates associated with the position of the user's hand in the 3D space do not overlap with any defined collision area of ​​the 3D virtual object, it is determined that there is no collision between the user input and the 3D virtual object, and wait for the next user input to be detected in step 606.

[0059] At 610, a measurement is determined based at least in part on the user input and a reference point. In some embodiments, the reference point is defined as a portion of a 3D virtual object. In some embodiments, the reference point is defined as a location of a motion detection device. For example, a distance between a location of a user input and a reference point in 3D space can be calculated.

[0060] At 612, the appearance of the 3D virtual object is updated based at least in part on the measurement. In some embodiments, the 3D virtual object can be updated by changing the shape and / or animation of the 3D virtual object. For example, if the detected user's hand approaches a reference point of the 3D virtual object including a source of water flow, the distance between the source of the water flow and the user's hand will decrease, and the 3D virtual object may be displayed as a shorter water flow that appears to be compressed. In another example, depending on the position of the user's hand, the water flow can be displayed as both colliding with the object (user's hand) and flowing around the object.

[0061] At 614, temperature regulation feedback is generated based at least in part on the measurements. In some embodiments, the temperature regulation feedback can be updated by turning on a heating component, changing the temperature of the heating component, turning on a fan, and / or a combination thereof. Because the heating component of the temperature regulation feedback device cannot generate heat instantaneously (e.g., it may take two to three seconds to generate heat), it can be turned on before detecting a collision between the user input and the 3D virtual object in anticipation of such a collision. For example, when the user chooses to interact with the 3D virtual object providing thermal feedback, or even when the 3D virtual environment is first presented (e.g., in steps 604 or 602, respectively), the heating component of the temperature regulation feedback device can be turned on. For example, later, when a collision between the user's hand and the 3D virtual object is detected, the fan portion of the temperature regulation feedback device can be turned on to direct the generated heat to the position of the user's hand (e.g., the palm) to simulate warmth (or other temperature regulation) feedback provided by the user engaging the 3D virtual object.

[0062] Thermoregulation feedback is output at 616. If multiple thermoregulation feedback devices are used, thermoregulation feedback may be provided by a combination of thermoregulation feedback devices based on the position of the user's hand (eg, palm).

[0063] At 618, non-temperature related feedback is generated based at least in part on the measurements.

[0064] At 620 , non-temperature related feedback is output.

[0065] In some embodiments, the mid-air haptic feedback can be updated by changing the intensity / pressure and / or area of ​​the mid-air focus position. For example, if the user's hand were to be closer to the 3D virtual object that includes the source of the water flow, the haptic feedback would provide greater pressure to the user's hand (e.g., palm) than if the user's hand were to be farther away from the source of the water flow. Additionally, for example, if the user's hand were to be closer to the reference point of the 3D virtual object that includes the source of the water flow, the mid-air haptic feedback would provide pressure to a smaller contact area of ​​the user's hand (e.g., palm) than if the user's hand were farther away from the source of the water flow. If multiple mid-air haptic feedback systems are used, the haptic feedback can be provided by a combination of haptic feedback systems depending on the position of the user's hand (e.g., palm).

[0066] In some embodiments, the audio feedback may be updated by playing multiple recorded audio data and / or at different volumes based on the position of the user's hand.

[0067] In some embodiments, the scent-based feedback may be updated by emitting a plurality of scents based on detected user input and / or a product type associated with a selected 3D virtual object.

[0068] In some embodiments, non-temperature related feedback is coordinated with temperature regulation feedback so that a user can perceive multiple feedbacks together. For example, mid-air tactile feedback can be output from a mid-air tactile feedback device in the same direction and at least partially synchronized with output from a mid-air temperature regulation feedback device. In a specific example, if a user performs a user input to a 3D showerhead object spraying water, mid-air thermal and mid-air tactile feedback (in addition to audio feedback) can be output to the location of the user's palm so that the user can experience a simulated feeling of warm water touching his or her palm.

[0069] In some embodiments, the type of feedback updated to the 3D virtual object and / or generated by one or more feedback devices is determined based on a predetermined mapping between the current position of the user's hand and the corresponding update and / or feedback rules. In some embodiments, the type of feedback updated to the 3D virtual object and / or generated by one or more feedback devices is dynamically determined based on entity-based simulation. For example, the type of feedback updated to the 3D virtual object includes how the shape and animation of the 3D virtual object changes under different user inputs. For example, the feedback type generated by one or more feedback devices includes length, temperature, intensity, volume, and / or output direction of each feedback device (e.g., mid-air temperature adjustment feedback device, mid-air tactile feedback device).

[0070] At 622, it is determined whether to continue to provide the mid-air temperature regulation output. If the mid-air temperature regulation output is to continue to be provided, control returns to 606. Otherwise, if the mid-air temperature regulation output is no longer to be provided, process 600 ends.

[0071] For example, if the system implementing process 600 is shut down and / or powered off, the mid-air temperature regulation output is no longer provided.

[0072] Fig. 7A and Figure 7B Examples of 3D virtual objects displayed by a system for providing a mid-air temperature adjustment output in response to user input are described. Fig. 7A and Figure 7B In the example of the system 700, the system 700 includes a display device 710, an eye movement detection device 708, a manual detection device 706, a mid-air tactile feedback device 702a, a mid-air tactile feedback device 702b, a mid-air temperature adjustment device 704a, a mid-air temperature adjustment device 704b, and a host computer (not shown). In the example of the system 700, the eye movement detection device 708 is embedded in the display device 710, but the manual detection device 706 is not embedded in the display device 710, but is arranged adjacent to the mid-air tactile feedback device 702b. Fig. 7A and Figure 7BAs shown, the mid-air tactile feedback device 702a and the mid-air temperature adjustment device 704a are placed above the display device 710, and the mid-air tactile feedback device 702b and the mid-air temperature adjustment device 704b are placed below the display device 710. Depending on the position of the detected user input (for example, with respect to the eye movement detection device 708 and the manual detection device 706) and / or the direction of the 3D virtual sprinkler 712, feedback can be provided by different combinations of the mid-air tactile feedback device 702a, the mid-air tactile feedback device 702b, the mid-air temperature adjustment device 704a, and the mid-air temperature adjustment device 704b. Although not shown in FIG. Fig. 7A and Figure 7B As shown in FIG. 7 , the host computer is connected to a display device 710, an eye movement detection device 708, a manual detection device 706, a mid-air tactile feedback device 702a, a mid-air tactile feedback device 702b, a mid-air temperature adjustment device 704a, and a mid-air temperature adjustment device 704b, and receives instructions from and sends instructions to them.

[0073] exist Fig. 7A In the example of , display device 710, which is an autostereoscopic display, is displaying 3D virtual shower head 712. For example, 3D virtual shower head 712 is displayed by selection by a user from a menu previously displayed on display device 710. In a specific example, a virtual bathroom scene was previously presented on display device 710, and the user selected 3D virtual shower head 712 to interact with features of the shower head model. 3D virtual shower head 712 includes an animation of water flowing from shower head 716. For example, the animation of water flow 712 of 3D virtual shower head 716 can be modeled based on how water actually sprays out of a physical shower head on which 3D virtual shower head 712 is based. Fig. 7A As shown, without any user interaction with the 3D virtual sprinkler 712, the animation of the water stream 716 flows continuously because it does not collide with any object. Figure 7B As described, user input (e.g., gestures and / or eye movements) detected by the system 700 may cause the 3D virtual shower head 712 to be presented in different manners and / or cause the system 700 to output feedback via at least one feedback device (e.g., mid-air tactile feedback device 702a, mid-air tactile feedback device 702b, mid-air temperature adjustment device 704a, and mid-air temperature adjustment device 704b).

[0074] exist Figure 7BIn the example of FIG. 7 , a user hand 714 is placed “under” a 3D virtual shower head 712 to “touch” the animated water flowing from the 3D virtual shower head 712. The posture and position of the user hand 714 (in the 3D space where the 3D virtual shower head 712 exists) are detected by the hand detection device 706 (and, in some embodiments, the user's eye movement by the eye movement detection device 708). For example, the current position of the user hand 714 is detected to collide with a defined collision area of ​​the 3D virtual shower head 712 and its accompanying water animation. Therefore, the length / distance between the current position of the user hand 714 and the position of the hand detection device 706 (or a reference point on the 3D virtual shower head 712, e.g., the center of the face of the 3D virtual shower head 712) is measured by the host computer. The measured length / distance is used by the host computer to cause water to spray out of the 3D virtual showerhead 712, the water stream animation 720, it appears as if the water stream collides with the object (user hand 714), and now appears to be deformed and flowing around the detected object (user hand 714) instead of flowing continuously, as shown. Fig. 7A shown.

[0075] In addition, the measured length / distance is used by the host computer to cause the mid-air tactile feedback device 702a to provide tactile feedback 722 downwardly toward the user's hand 714, and the mid-air temperature regulation device 704a to provide heated air feedback 718 downwardly toward the user's hand 714. The focal position (e.g., ultrasonic) tactile feedback 722 is determined by the host computer to be near or the current position of the user's hand 714 (e.g., based on the measured length / distance) and / or the known water pressure associated with the physical version of the 3D virtual showerhead 712. The intensity of the heat / air flow of the heated air feedback 718 is also determined by the host computer to be near or the current position of the user's hand 714 (e.g., based on the measured length / distance). The combination of the tactile feedback 722 and the heated air feedback 714 on the user's hand 718 will simulate the feeling of warm water spraying from the 3D virtual showerhead 712 and contacting the palm of the user's hand 714. For example, feedback is caused to be output by the mid-air haptic feedback device 702a and the mid-air temperature adjustment device 704a, both of which point downward, because the 3D virtual shower head 712 points downward, and it is expected that the user's palm will face upward to contact the warm water flowing downward from the 3D virtual shower head 712. In a different example, water sprays upward from the 3D virtual object (e.g., a water jet in a hot tub), causing feedback to be output by the mid-air haptic feedback device 702b and the mid-air temperature adjustment device 704b, both of which point upward, because it is expected that the user's palm will face downward to contact the warm water flowing upward from the water jet. In addition to the haptic feedback 722 and the hot air feedback 718, additional feedback, such as water sounds and / or fragrance, can be output from corresponding feedback devices (not shown) through the system 700 to obtain an immersive simulation experience.

[0076] As the user's hand 714 moves, the focus and / or pressure of the tactile feedback 722 and the heat and / or wind intensity of the heated air feedback 718 can be updated accordingly by the host computer based on the current position of the user's hand 714 to better simulate the different levels of water pressure and warmth that the user should experience when using the physical version of the 3D virtual shower head 712.

[0077] Embodiments of providing a mid-air temperature adjustment output in response to user input are disclosed. User input regarding a virtual object is detected. At least mid-air temperature adjustment feedback (sometimes in addition to mid-air tactile feedback) is provided in response to the detected user input to provide the user with a temperature adjustment (e.g., including cold, heat, and / or airflow) simulated by / from the virtual object so that the user can have an immersive virtual experience without having to make direct contact with the physical object.

[0078] Although the foregoing embodiments have been described in detail for the purpose of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A system, characterized in that: include: a motion detection device configured to detect user input with respect to the virtual object; A display device configured to display a virtual object; Temperature regulation equipment; as well as A processor is coupled to the motion detection device, the display device and the temperature adjustment device, the processor being configured to: Acquire information of a virtual object corresponding to a product sold on the online shopping platform from a server of the online shopping platform, the information including characteristics of the product corresponding to the virtual object and an image that can be used to render the virtual object; Controlling display of the virtual object on the display device according to the image; determining a temperature adjustment based at least in part on a characteristic of a product corresponding to the virtual object and the user input regarding the virtual object; and The temperature adjustment device is caused to output the temperature adjustment so that a user can interact with a virtual object of the product before purchasing a physical version of the product.

2. The system according to claim 1, characterized in that The determining the temperature adjustment based at least in part on the user input regarding the virtual object comprises: determining a measurement based at least in part on the user input and a reference point associated with the motion detection device; using the measurements to determine instructions related to generating the temperature adjustment; and The instructions associated with generating the temperature adjustment are sent to the temperature adjustment device.

3. The system according to claim 1, characterized in that The temperature regulating device includes one or more of: a fan, a heating component and a cooling component.

4. The system according to claim 3, characterized in that The causing the temperature adjustment device to output the temperature adjustment comprises: sending a first instruction to the temperature regulating device to activate the heating component for the first time; and A second instruction is sent to the temperature adjustment device for activating the fan for a second time, wherein the second instruction is delayed from the first instruction.

5. The system according to claim 1, characterized in that Also includes: a mid-air haptic feedback device configured to generate mid-air haptic feedback; Wherein the processor is coupled to the mid-air tactile feedback device and is further configured to control the mid-air tactile feedback device in response to the user input.

6. The system according to claim 5, characterized in that The mid-air haptic feedback device includes an ultrasonic tactile display configured to emit ultrasonic waves to produce tactile sensations at one or more focal points using a plurality of transducers.

7. The system according to claim 5, characterized in that The controlling the mid-air tactile feedback device in response to the user input comprises: determining a measurement based at least in part on the user input and a reference point associated with the motion detection device; using the measurements to determine instructions related to generating the mid-air haptic feedback; and The instructions related to generating the mid-air tactile feedback are sent to the mid-air tactile feedback device.

8. The system according to claim 5, characterized in that The processor is configured to control the temperature adjustment device to output the temperature adjustment at least partially in synchronization with the mid-air haptic feedback device outputting the mid-air haptic feedback.

9. The system according to claim 1, characterized in that Also includes: The processor is further configured to determine whether the user input collides with the virtual object; and Wherein the processor is configured to determine the temperature adjustment based at least in part on the user input regarding the virtual object based at least in part on whether the user input collides with the virtual object.

10. The system according to claim 9, characterized in that The display device comprises an autostereoscopic display device.

11. The system according to claim 1, characterized in that The user input is obtained without requiring the user to physically contact the physical object.

12. A method, characterized in that include: Acquire information of a virtual object corresponding to a product sold on the online shopping platform from a server of the online shopping platform, the information including characteristics of the product corresponding to the virtual object and an image that can be used to render the virtual object; displaying the virtual object according to the image; receiving user input regarding the virtual object; updating the virtual object based at least in part on the user input; determining a temperature adjustment based at least in part on a characteristic of a product corresponding to the virtual object and the user input; as well as The temperature adjustment is output so that a user can interact with a virtual version of the product before purchasing a physical version of the product.

13. The method according to claim 12, characterized in that The updating of the virtual object includes updating at least one of a shape and an animation related to the virtual object.

14. The method according to claim 12, characterized in that Determining the temperature adjustment based at least in part on the user input comprises: determining a measurement based at least in part on the user input and a reference point; and The temperature adjustment is determined using the measurements.

15. The method according to claim 14, characterized in that The reference point is associated with a portion of the virtual object.

16. The method according to claim 14, characterized in that The reference point is related to the position of a motion detection device that detects the user input.

17. The method according to claim 12, characterized in that The information related to multiple virtual objects was obtained from the server of the online shopping platform, including: causing presentation of the plurality of virtual objects; and A selection is received regarding the presented virtual object.

18. The method according to claim 12, characterized in that Also included is determining that the user input collides with a collision area associated with the virtual object.

19. The method according to claim 12, characterized in that The temperature regulation includes airflow, and at least one of heat regulation and cold regulation.

20. The method according to claim 12, characterized in that Also includes: determining mid-air haptic feedback based at least in part on the user input; and The mid-air haptic feedback is output.

Images