Extended reality system based on multimodal interface
By transmitting output data with device identification and limited time between a computer and an extended reality output device, the time delay problem of synchronous reproduction of audio and tactile content in the prior art is solved, and the extended reality is continuously expanded during multi-device output. The experience improves wearability and addiction, and improves data transmission stability in 5G environment.
Patent Information
- Application Number
- CN202110625680.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-09-17
- Filing Date
- 2021-06-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-06-04
AI Technical Summary
When the existing extended reality system is output by multiple devices, the time delay calculation of synchronous reproduction of audio and tactile content is complicated, resulting in additional delays, and the transmission method is not suitable for single-channel communication methods such as WiFi, resulting in interruption of user experience.
By transmitting output data with device identification and limited time between computers and extended reality output devices, we ensure that our own data is reproduced only within a limited time, adopt asynchronous transmission method, and optimize data transmission in a 5G communication environment to reduce data loss .
It realizes an uninterrupted extended real experience during multi-device output, improves wearability and addiction, reduces user experience interruption, and improves data transmission stability in 5G communication environment.
Smart Images

Figure CN113760089B_ABST
Abstract
Description
[0001] References to related applications
[0002] This application claims priority to Korean Patent Application No. 10-2020-0068521 filed on June 5, 2020 and Korean Patent Application No. 10-2020-0119884 filed on September 17, 2020, which are incorporated herein by reference in their entirety. Technical Field
[0003] The present invention relates to an extended reality system, and more particularly to an extended reality system based on a multimodal interface. Background Art
[0004] Extended reality (XR) is a concept that includes virtual reality (VR), augmented reality (AR), and mixed reality (MR), as a combination of all virtualities generated by computer technology and wearable devices. As the XR market grows, the demand for multimodal systems that input and reproduce data related to the five senses of humans is increasing. In particular, multimodal systems can be effectively used in a variety of virtual training systems, such as medical fields such as surgery, defense fields such as shooting training and flight training, and educational fields such as driving or playing musical instruments.
[0005] In order to convey the five senses of human beings to users, multimodal systems require multiple augmented reality output devices such as display devices, headphones, and tactile devices. In order to reproduce the five senses more realistically, each augmented reality output device needs to synchronize and reproduce video, audio, tactile and other contents.
[0006] As a method for synchronously reproducing data, Korean Patent Publication No. 10-2019-0079613 calculates the delay associated with transmitting the audio stream and includes the time-to-play based on the calculated delay in the audio data packet, thereby inserting a time delay into the audio stream to synchronize the audio content with the tactile content. However, the problem with this method is that when there are many output devices, the time delay to be inserted is complicated to calculate, resulting in additional delay.
[0007] In addition, when providing content such as audio or video to users, in order to increase the sense of reality and immersion, a method of processing the content itself such as 3D audio mixing is used.
[0008] Prior art literature
[0009] Patent Literature
[0010] (Patent Document 1) Korean Patent Publication No. 10-2019-0079613 (“Multi-device audio streaming system with synchronization”, Sony Perception Co., Ltd., published on March 22, 2018.)
[0011] (Patent Document 2) US 2018 / 0095534 A1 (2018.04.05) Summary of the invention
[0012] The object of the present invention is to provide an extended reality system and a data transmission method therefor, so that a user can uninterruptedly experience the reproduction of the five senses with a sense of reality.
[0013] Yet another object of the present invention is to provide a temperature transfer device that provides content in a form that allows the user to experience a sense of reality and immersion, and that can improve wearability.
[0014] However, the problems to be solved by the present invention are not limited thereto, and the present invention can be expanded in various ways without departing from the spirit and scope of the present invention.
[0015] According to one embodiment of the present invention, an extended reality system based on a multimodal interface includes: a computer, which transmits output data including at least one device identification (Device ID), at least one limited time and at least one reproduction data to at least one extended reality output device; at least one extended reality output device, which receives the output data and reproduces only its own reproduction data within the limited time based on the at least one device identification.
[0016] According to one aspect, the computer may broadcast the output data to at least one augmented reality output device.
[0017] According to one aspect, at least one augmented reality output device may not reproduce its own reproduction data if a limited time has passed.
[0018] According to one aspect, the output data also includes a command flag, which includes information indicating whether to reproduce the reproduced data. At least one extended reality output device can reproduce its own reproduced data within a limited time based on the command flag.
[0019] According to one aspect, the output data may be configured in an NFXM (NUI / NUX Format for XR Machine) format having a mirror structure.
[0020] According to one aspect, the NFXM format may include a command, a device number, a device identifier, a data packet size, and a data packet.
[0021] According to one aspect, the computer can selectively encrypt the entire area or a portion of the area in the NFXM format and broadcast the output data.
[0022] According to one aspect, a computer encrypts the entire area and the partial area in the NFXM format, and the encrypted entire area and the partial area can be accessed using different keys.
[0023] According to one aspect, the reproduction data may include at least one of visual, auditory, tactile, olfactory and taste reproduction data.
[0024] According to one aspect, the extended reality output device may be any one of a head mounted display (HMD, Head Mounted Display), headphones, and a tactile device.
[0025] According to one aspect, the computer may synchronously transmit the output data based on at least one of the reproduction data and the command mark.
[0026] According to another embodiment of the present invention, an extended reality system based on a multimodal interface includes: a computer, which transmits output data including at least one device identification, at least one limited time and at least one reproduction data to at least one extended reality output device; at least one extended reality output device, which receives the output data and reproduces only its own reproduction data within the limited time based on at least one device identification, and the computer is located inside any one of the at least one extended reality output device.
[0027] According to another embodiment of the present invention, a method for transmitting reproduction data from a computer to at least one extended reality output device includes the following steps: synchronizing the action time between the computer and the at least one extended reality output device; the computer transmits output data including at least one device identifier, at least one limited time and at least one reproduction data to the at least one extended reality output device.
[0028] According to another embodiment of the present invention, a method for an extended reality output device to receive reproduction data from a computer includes the following steps: synchronizing the action time between the computer and the extended reality output device; receiving output data transmitted by the computer, the output data including at least one device identifier, at least one limited time and at least one reproduction data; and reproducing its own reproduction data within the limited time based on the at least one device identifier.
[0029] According to another embodiment of the present invention, a wearable temperature transfer device includes: a content output device that transmits content to a user; a wearing tool that enables the user to wear the wearable temperature transfer device on his or her body; at least one temperature sensor that measures the ambient temperature of the wearable temperature transfer device; at least one thermoelectric element that transmits temperature to the user; and a processor that considers at least one of the content, the ambient temperature, and a temperature pre-set by the user to determine the temperature to be transmitted to the user by at least one thermoelectric element.
[0030] According to one aspect, the ambient temperature includes an external temperature and a user's body temperature, wherein the external temperature includes at least one of air temperature and room temperature, and the processor may modify the temperature determined based on the content of the consideration based on the ambient temperature.
[0031] According to one aspect, at least one temperature sensor is attached to the inner side and the outer side of the wearable temperature transfer device, and can measure the surrounding temperature. The inner side is the side that contacts the user's body when the user wears the wearable temperature transfer device, and the outer side is the side that does not contact the user's body when the user wears the wearable temperature transfer device.
[0032] According to one aspect, the processor may estimate the temperature of a portion of the inner side surface and the outer side surface to which the at least one temperature sensor is not attached, using the temperature measured by the at least one temperature sensor.
[0033] According to one aspect, at least one thermoelectric element can heat the user via an exothermic reaction or cool the user via an endothermic reaction.
[0034] According to one aspect, in the case where the content is audio, the processor determines the temperature to be delivered to the user based on at least one of the frequency, speed, and tempo of the audio.
[0035] According to one aspect, in a case where the content is an extended reality simulation, the processor may determine the temperature to be delivered to the user based on the relative positions of the objects and the user included in the extended reality simulation.
[0036] The disclosed technology may have the following effects. However, it does not mean that a specific embodiment must include all or only the following effects, and therefore, the scope of rights of the disclosed technology cannot be understood as being limited thereto.
[0037] According to the aforementioned extended reality system based on a multimodal interface of the embodiment of the present invention, the user can continuously experience the reproduction of the five senses with a sense of reality.
[0038] According to the wearable temperature transfer device of the embodiment of the present invention described above, the wearability can be improved by providing content in a form in which the user can feel a sense of reality and immersion. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 An NFXM for an extended reality system according to an embodiment of the present invention is schematically shown.
[0040] Figure 2 An extended reality system and a data transmission method based on a multimodal interface according to an embodiment of the present invention are schematically illustrated.
[0041] Figure 3 An extended reality system and a data transmission method based on a multimodal interface according to another embodiment of the present invention are schematically shown.
[0042] Figure 4A and Figure 4B A protocol format of a multimodal communication interface according to an embodiment of the present invention is shown.
[0043] Figure 5 Shown as Figure 4A and Figure 4B The input data format of an example of the protocol format.
[0044] Figure 6 Shown as Figure 4A and Figure 4B The output data format of an example of the protocol format.
[0045] Figure 7 A data encryption method in a protocol format of a multimodal communication interface according to an embodiment of the present invention is shown.
[0046] FIG. 8A to FIG. 8C An example in which an extended reality system according to an embodiment of the present invention is applied to a surgery training system is shown.
[0047] Fig. 9 Show FIG. 8A to FIG. 8C The data processing flow of the surgical training system.
[0048] Fig.10 An example in which an extended reality system according to an embodiment of the present invention is applied to a welding training system is shown.
[0049] Fig.11 An example in which an extended reality system according to an embodiment of the present invention is applied to an excavator training system is shown.
[0050] Fig.12 Detailed description is a block diagram showing the structure of a wearable temperature transfer device that can be used in an extended reality system based on a multimodal interface according to another embodiment of the present invention.
[0051] Fig.13The structure of a temperature transfer headset according to another embodiment of the present invention is schematically shown.
[0052] Fig.14 The data processing flow in the temperature transfer headset according to another embodiment of the present invention is shown.
[0053] Fig.15 The structure of a temperature transfer HMD according to another embodiment of the present invention is schematically shown.
[0054] Fig.16A and Fig. 16B It is speculation Fig.15 A method of transmitting the temperature of a location in the HMD where no temperature sensor is attached will be described.
[0055] Fig.17 The structure of a temperature transfer garment according to another embodiment of the present invention is schematically shown.
[0056] Description of symbols
[0057] 200: Computer
[0058] 300: Display
[0059] 400:Headphones
[0060] 500:Haptic Devices DETAILED DESCRIPTION
[0061] While the present invention is susceptible to various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail.
[0062] However, this is not intended to limit the present invention to specific embodiments, and it should be understood that all changes, equivalents, and substitutes within the concept and technical scope of the present invention are included.
[0063] The terms "first" and "second" can be used to describe various components, but the components shall not be limited by the terms. The terms are only used to distinguish one component from other components. For example, without exceeding the scope of the present invention, the first component can be named as the second component, and similarly, the second component can also be named as the first component.
[0064] When it is mentioned that a certain component is “connected” or “joined” to other components, although it can be directly connected or joined to other components, it should be understood that other components may exist in between. On the contrary, when it is mentioned that a certain component is “directly connected” or “directly joined” to other components, it should be understood that other components do not exist in between.
[0065] The terms used in this application are only used to illustrate specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions as long as the context does not clearly indicate a different meaning. In this application, terms such as "including" or "having" are intended to specify the existence of features, numbers, steps, operations, constituent elements, parts or combinations thereof recorded in the specification, and it should be understood that the existence or additional possibility of one or more other features or numbers, steps, operations, constituent elements, parts or combinations thereof is not excluded in advance.
[0066] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art having ordinary knowledge in the art to which the present invention belongs. Terms with the same meaning as defined in commonly used dictionaries should be interpreted as having the same meaning as that in the context of the relevant art and should not be interpreted as having an ideal or excessive formal meaning unless explicitly defined in this application.
[0067] The preferred embodiments of the present invention will be described clearly and in detail below with reference to the accompanying drawings so that a person having ordinary knowledge in the technical field to which the present invention belongs can easily implement the present invention.
[0068] Figure 1 The NFXM (NUI / NUX Format for XR Machine) used in an extended reality (XR) system according to an embodiment of the present invention is schematically shown.
[0069] Reference Figure 1 NFXM processes input data related to the five senses of human beings through voice recognition, gesture recognition, space recognition, grasp recognition, motion tracking, head tracking, etc., and reproduces the five senses of human beings through audio, video, touch, etc. to provide users with extended reality.
[0070] For example, the user's action data collection through NFXM can be performed by the following methods. Voice recognition can use a voice analysis algorithm to process the user's voice data input through a microphone, and gesture recognition can use a gesture algorithm to process the user's actions input through a camera. Spatial recognition can be analyzed and processed using a camera and a spatial algorithm, and grasping recognition can be analyzed and processed using MEMS (micro electro mechanical system) and a grasping algorithm. Motion tracking can analyze and track data collected through a gyro sensor through a motion tracking algorithm, and head tracking can use a head tracking algorithm to process data about the user's head movement received through a gyro sensor and a compass sensor.
[0071] In addition, for example, the five senses reproduction through NFXM can be performed by the following method. Reproducing hearing through audio can provide stereoscopic and spatial sound through binaural effect, where binaural effect is output through dual speakers, and reproducing vision through pictures can provide pictures that can feel actual and realistic through HMD (head mounted display) and extended reality technology. Touch can provide users with touch according to the physical properties, shape, size, etc. of the object through the physical force and vibration, pressure and other technologies of the tactile device and the object tactile algorithm.
[0072] Figure 2 An extended reality system and a data transmission method based on a multimodal interface according to an embodiment of the present invention are schematically illustrated.
[0073] Reference Figure 2 The computer 200 receives and processes data about the five senses of human beings from the input device, and transmits the output data according to the NFXM to the extended reality output device. The computer 200 includes any computing device that can receive and process the input of data, for example, a server, a PC, a mobile device, etc. The output data includes at least one device identification, at least one limited time, and at least one reproduction data.
[0074] According to an embodiment of the present invention, an extended reality system based on a multimodal interface can generally use an asynchronous transmission method, wherein the asynchronous transmission method is a transmission device (for example: Figure 2 Computer 200 or Figure 3 The computer 200 in the display device 300) sends a signal to all receiving devices (for example: Figure 2 The display device 300, the headset 400, the haptic device 500 or Figure 3 The headset 400 and the tactile device 500) broadcast data.
[0075] In the existing XR system, due to the limitation of transmission bandwidth, when the size of output data increases, data is transmitted through other additional channels according to the output data, so at least as many channels as the number of XR output devices are required. However, this transmission is not suitable for communication methods such as WiFi that process data in one channel.
[0076] The device identifier indicates the extended reality output device that will reproduce the reproduction data. That is, the recipient's extended reality output device reproduces the received reproduction data only when its own device identifier exists in at least one device identifier included in the output data, and does not reproduce the received reproduction data when there is no device identifier belonging to itself.
[0077] The limited time indicates the time required to reproduce the reproduction data. In other words, the augmented reality output device reproduces the reproduction data within the limited time included in the output data, and does not reproduce the reproduction data if the limited time has passed.
[0078] According to an embodiment, the limited time may be a time range in which the reproduction data is valid. For example, in the case where specific audio data needs to be played at t1, the limited time may be set to (t1, t2). At this time, if the headset receives the corresponding output data before t1, the audio data is played at t1, if the corresponding output data is received between t1 and t2, the audio data is played immediately, and if the corresponding output data is received after t2, the audio data is not played. At this time, the difference between t1 and t2 may be within 1 ms.
[0079] In addition, if the transmission time of the output data is long due to the long distance between the computer 200 and the augmented reality output device or the communication restriction, the limited time is appropriately set in consideration of this point, so that the uninterrupted augmented reality content can be reproduced for the remote user. For example, if 1 second is required when considering the transmission time, the limited time is set to (t1+1, t2+1) in the above example, so that the user can be provided with uninterrupted streaming.
[0080] In addition, the time limit may also be set in consideration of the size of the buffer of the augmented reality output device. In order to provide the user with uninterrupted augmented reality content, the time limit may be set in the form of reproducing the reproduced data after a certain period of time has passed after the reproduction data of a length that can be stored in the buffer of the augmented reality output device is transmitted in advance.
[0081] In the transmission method used by the existing extended reality system, in order to prevent the reproduction data such as video from being damaged, there are many cases where the reproduction is stopped if the buffering is not completed without skip logic. Or, in the case where the content cannot be reproduced in real time due to distance or communication constraints, the reproduction data received during this period is paused and reset to give the opportunity to receive again. The problem in this case is that the user will experience that the content is being reproduced and then suddenly stopped.
[0082] In the case of an extended reality system and a data transmission method according to an embodiment of the present invention, the reproduced data is reproduced only within a limited time, and the reproduced data is not reproduced after the limited time has passed, so that the user can experience the extended reality content without a pause in the middle. In particular, in a sufficiently fast communication environment such as 5G, the loss of reproduced data during transmission can be minimized, so the user can experience uninterrupted extended reality in a real-time environment.
[0083] The reproduction data refers to data that the extended reality output device needs to reproduce. For example, the reproduction data may include at least one of 3D video, stereo, 3D audio, vibration, weight, volume, pressure, and gesture.
[0084] The extended reality output device receives the output data broadcasted by the computer 200 and reproduces only its own reproduction data within a limited time based on the device identification.
[0085] Figure 2 Although the display 300, the headset 400, and the haptic device 500 are illustrated as the augmented reality output device, the present invention is not limited thereto and may include any device that can reproduce the five senses for providing the user with augmented reality. Figure 2 Although the display 300, the headset 400, and the tactile device 500 are shown as having one each, the extended reality system based on the multimodal interface according to an embodiment of the present invention may include all of the devices or only a part of the devices, and the devices may also include multiple ones.
[0086] For example, the display 300 may be a HMD (head mounted display), a CAVE (cave-type) display, a 3D display, etc., and the tactile device 500 may be a glove, a strap, a formal suit, an accessory, a wearable device, etc.
[0087] In addition, in order to synchronize the extended reality contents (e.g., video, audio, and tactile) with each other through a limited time, the computer 200 and / or the extended reality output device need to be initialized to synchronize the action time. To this end, the computer 200 and the extended reality output device include an internal RTC (real-time clock) or timer for asynchronous communication. The synchronized action time can be performed by repeatedly sending and receiving test data including time information based on any device of the computer 200 and the extended reality output device. Generally, the test data is repeatedly transmitted 2 to 5 times, thereby completing the synchronization of the action time between the computer 200 and the extended reality output device.
[0088] If the action time synchronization is completed, the computer 200 and the extended reality output device can transmit data including the limited time to each other based on the synchronized action time, confirm the limited time in the data received from other devices and confirm the order of reproducing the data, and provide the user with extended reality content with synchronized reproduction time.
[0089] In addition, in addition to the above-mentioned broadcasting method (i.e., asynchronous transmission), the extended reality system based on the multimodal interface according to one embodiment of the present invention can selectively use the synchronous transmission method according to the type or setting of the data. In the case of using the synchronous transmission method, the reproduction data can be reproduced immediately regardless of the limited time, and on the contrary, the reproduction data may not be reproduced even within the limited time.
[0090] For example, data related to user safety may need to be reproduced immediately before a specified time has elapsed (e.g., before t1 in the above example) or after a specified time has elapsed (e.g., after t2 in the above example). On the contrary, even when the specified time has elapsed (e.g., between t1 and t2 in the above example), it may not be reproduced. Therefore, the synchronous transmission method may be used for specific data (e.g., safety or security related data) depending on the application field.
[0091] In addition, the user can directly determine whether to reproduce the data using the command flag (CmdFlag) described later. Figure 4A and Figure 4B This will be described later.
[0092] Figure 3 An extended reality system and a data transmission method based on a multimodal interface according to another embodiment of the present invention are schematically shown.
[0093] Figure 3 The system shown and Figure 2The extended reality system shown is different in that the computer 200 is located inside the display 300. In the case of an application that requires a small amount of data to be processed, instead of the computer 200 as another external device to process and transmit the data, the data can be processed and transmitted in the form of a processor or the like inside any of the extended reality output devices. Figure 3 Although the computer 200 is shown to be located inside the display 300, the present invention is not limited thereto and can be located in any one of many augmented reality output devices.
[0094] Figure 4A and Figure 4B A protocol format of a multimodal communication interface according to an embodiment of the present invention is shown.
[0095] Reference Figure 4A and Figure 4B The protocol format has a mirror structure (i.e., command MSB-device number MSB-device number LSB-command LSB, device ID-packet size-packet-packet size-device ID). The mirror structure is a form in which the same data is transmitted twice, and a pair of data received during asynchronous communication between the computer 200 and the augmented reality output device is compared, which can automatically check the data validity and provide a high level of data communication stability and reliability.
[0096] Command indicates relevant information about the type of data and is allocated 1 byte. I / O indicates whether the corresponding data is input data or output data. It is set to 1 in the case of input data and to 0 in the case of output data. The next 5 bits indicate whether the corresponding data includes relevant reproduction data of a certain sense among vision, hearing, touch, smell, and taste. Reserve bits are prepared bits that can be used as needed. For example, in the case of including reproduction data about other senses (e.g., temperature) in addition to the five senses, reserve bits can be used.
[0097] The command flag indicates information showing whether to reproduce the reproduction data. It is necessary for the extended reality system to limit the extended reality output device for the purpose of safety or security according to the applicable field so that it does not reproduce even if it receives the reproduction data, or to limit the reproduction of the reproduction data to only specific users. Alternatively, it is also necessary to impose restrictions to prevent specific users from reproducing the reproduction data in the output devices of other users. For example, in the case of using the extended reality system for surgical education, it is necessary to impose restrictions to prevent the teaching staff from moving away from a specific position of the tactile gloves worn by the learner. In order to achieve this, the command flag includes information showing whether to reproduce the reproduction data, so that the extended reality output device can reproduce or not reproduce the reproduction data based on the command flag.
[0098] The device count indicates the number of devices that send / receive data and is allocated 1 byte.
[0099] The device ID indicates whether the data belongs to a certain augmented reality output device and is allocated 2 bytes.
[0100] Packet Size indicates the size of the packet and is allocated 4 bytes.
[0101] The data packet includes at least one of measurement data, reproduction data and a limited time. In the case of input data (i.e., the I / O bit of the command is 1), the measurement device (e.g., microphone, camera, MEMS, etc.) includes the measurement data transmitted to the computer 200, and in the case of output data (i.e., the I / O bit of the command is 0), the output device (e.g., display 200, headset 300, tactile device 500, etc.) includes the reproduction data to be reproduced.
[0102] Below, with Figure 5 and Figure 6 Take two examples as examples Figure 4A and Figure 4B The protocol format is described below.
[0103] Figure 5 Shown as Figure 4A and Figure 4B The input data format of an example of the protocol format. Figure 5 , the input data format shown is input data including visual, auditory and tactile related measurement data (0x90), which is transmitted by two devices (0x02), indicating that three types of measurement data and time information are transmitted from the nine-axis sensor (0x7001), and one type of measurement data and time information is transmitted from the tactile glove (0x1001).
[0104] Figure 6 Shown as Figure 4A and Figure 4B The output data format of the protocol format is an example. Figure 6 , the output data format shown is output data (0x70) including visual, auditory and tactile related reproduction data, which is transmitted to 3r devices (0x03), indicating that the display (0x4001) reproduces the visual reproduction data within a limited time, the headphones (0x2001) reproduce the auditory reproduction data within a limited time, and the tactile gloves (0x1001) reproduce the tactile reproduction data within a limited time.
[0105] Figure 7 A data encryption method in a protocol format of a multimodal communication interface according to an embodiment of the present invention is shown.
[0106] As described above, the safety or security issues of the extended reality system are very important depending on the applicable field. It is necessary to limit the extended reality output device so that even if it receives the reproduction data, it does not reproduce it or limits the reproduction of the reproduction data to specific users. In addition, it is also necessary to set permissions differently according to the user, so as to limit the reproduction of the reproduction data to specific users. For example, in the example of a surgical operation, it is necessary to reproduce part of the action to all students, and other parts of the action to specific students only. The extended reality system based on a multimodal interface and the data transmission method according to the embodiment of the present invention can solve the security and authority issues that may occur during use through data encryption.
[0107] Data encryption can be achieved by encrypting the entire area of the protocol format or only a part of it. Figure 7 , the entire area of the protocol format may be encrypted 710 or only a partial area including the device identification and the reproduction data may be encrypted 730.
[0108] Alternatively, data encryption can also be achieved by encrypting the entire area and partial area of the protocol format respectively and making the range of users who can access the encrypted area different from each other. Figure 7 , it can be set to encrypt both the entire area 710 and the partial area 730 of the protocol format, so that the entire area 710 can be accessed through a public key provided to all users, and the partial area 730 can be accessed through a personal key provided only to a specific user. For example, encryption methods such as AES and PKI can be used.
[0109] FIG. 8A to FIG. 8C as well as Fig. 9 An example in which an extended reality system according to an embodiment of the present invention is applied to a surgery training system is shown.
[0110] Reference FIG. 8A to FIG. 8C ,The surgical training system can be composed of a teacher-related device as a host and a learner-related device as a client. FIG. 8A to FIG. 8C 1 teacher and 3 learners are shown in the figure, but the present invention is not limited to this.
[0111] Fig. 8A In the operation, the instructor 100 actually performs the operation, and the sensors 120-1 and 120-2 arranged in the operating room capture the movement of the instructor 100. The learner 200 performs the virtual operation while wearing the HMD 220 and the tactile glove 240 and following the movement of the instructor 100, thereby obtaining the operation training. The operation training system may further include a computer (not shown), which processes the output data including the device identification, the reproduction data and the time-limited output data from the movement of the instructor 100 captured by the sensors 120-1 and 120-2, and broadcasts the output data to the HMD 220 and the tactile glove 240. The HMD 220 and the tactile glove 240 of the learner 200 receive the output data, confirm the device identification and output the reproduction data belonging to each of them within the time limit. For example, the reproduction data of the HMD 220 may be a virtual operation environment, and the reproduction data of the tactile glove 240 may be the position, movement or vibration of the hand, etc.
[0112] Figure 8B In the example, the instructor 100 performs a virtual surgery while wearing the HMD 140 and the haptic glove 160, and the learner 200 performs the same virtual surgery as the instructor 100 while wearing the HMD 220 and the haptic glove 240, thereby obtaining surgical training. The surgical training system may further include a computer (not shown), which receives data related to the subtle movements of the instructor 100 from the HMD 140 and the haptic glove 160 of the instructor 100, processes data to be reproduced on the HMD 220 and the haptic glove 240 of the learner 200 based on the relevant movement data, and then transmits the reproduced data to the HMD 220 and the haptic glove 240. For example, the reproduced data of the HMD 220 may be a virtual surgical environment, and the reproduced data of the haptic glove 240 may be the position, movement or vibration of the hand, etc.
[0113] Figure 8CIn the example, the instructor 100 performs a virtual surgery using the HMD 140 and the controller 180, and the learner 200 performs the same virtual surgery as the instructor 100 while wearing the HMD 220 and the haptic glove 240, thereby obtaining surgical training. The surgical training system may further include a computer (not shown), which receives data related to the subtle movements of the instructor 100 from the HMD 140 and the controller 180 of the instructor 100, processes data to be reproduced on the HMD 220 and the haptic glove 240 of the learner 200 based on the relevant movement data, and then transmits the reproduced data to the HMD 220 and the haptic glove 240. For example, the reproduced data of the HMD 220 may be a virtual surgical environment, and the reproduced data of the haptic glove 240 may be the position, movement or vibration of the hand, etc. Figure 8C The controller 180 Figure 8B The tactile glove 160 can capture the movements of the instructor 100 more accurately.
[0114] Fig. 9 Show FIG. 8A to FIG. 8C The data processing flow of the surgical training system. Fig. 9 In the figure, the devices shown on the left side with the computer 400 as a reference are devices related to the teacher 100, and the devices shown on the right side are devices related to the learner 200.
[0115] Reference Fig. 9 , HMD 401, headset 403 and nine-axis sensor 405 are similar to Figure 8B or Figure 8C The device worn by the instructor 100 when the instructor 100 performs a virtual surgery. The HMD 401 displays the virtual surgery condition generated by the computer 400 after performing 3D processing 414 on the XR object 415 and the 3D audio 413 generated by the computer 400 after performing 3D processing 414 on the XR object 415, and the nine-axis sensor 405 transmits information related to the movement of the instructor 100 to the computer 400 in the virtual surgery condition. It should be noted that Fig. 9 The arrows indicated by dashed lines in FIG. 4 indicate asynchronous data transmission such as broadcasting, and the arrows indicated by solid lines indicate synchronous data transmission. The nine-axis sensor 405 may include Figure 8B Tactile gloves 160 or Figure 8C control device 180. Fig. 9 Although the nine-axis sensor 405 is used to obtain information related to the movement of the instructor 100, the present invention is not limited thereto, and any device that can measure or obtain information related to the movement of the instructor 100 can be used. The 3D processing 414 includes configuring an XR object 415 in the virtual surgical environment.
[0116] Fig.10An example in which an extended reality system according to an embodiment of the present invention is applied to a welding training system is shown.
[0117] Reference Fig.10 , the welding training system may include a computer, an HMD and a controller.
[0118] The computer performs 3D processing on the user's position and orientation data obtained from the nine-axis sensor of the HMD, the spatial recognition data obtained from the optical sensor of the HMD, the position and orientation data of the welding machine obtained from the nine-axis sensor of the controller, the ON / OFF data obtained from the trigger button of the controller, and the augmented reality object data, thereby generating output data and broadcasting it. The augmented reality object data may include material information, and the material information includes 3D models of weldments, arc welding torches, gas welding torches, etc., textures of welding materials (e.g., metal, steel, aluminum, stainless steel, copper, carbon steel, tungsten, nickel, etc.), welding sparks, melting points, and welding effects.
[0119] As a result of the 3D processing, the computer generates reproduction data and broadcasts output data, wherein the reproduction data includes a 3D rendered welding image to be reproduced by the dual displays of the HMD, audio based on the binaural effect to be reproduced by the dual speakers of the HMD, welding heat to be reproduced by the thermoelectric elements of the HMD, welding vibration feedback to be reproduced by the vibration motor of the controller, the force of the welding rod to be reproduced by the feedback motor of the controller, and heat transferred to the welder to be reproduced by the thermoelectric elements of the controller, and the output data adds corresponding device identifiers (i.e., the dual displays, dual speakers and thermoelectric elements of the HMD, the vibration motor, feedback motor and thermoelectric elements of the controller) and limited time to each reproduction data.
[0120] The HMD may include a nine-axis sensor for head tracking of the user and an optical sensor (e.g., a camera and / or a lidar sensor) for spatial recognition as input devices, and may include dual displays for displaying 3D rendered welding images, dual speakers for playing audio based on the binaural effect, and thermoelectric elements for reproducing heat transferred to the user's face during welding as output devices. The dual displays, dual speakers, and thermoelectric elements included in the HMD receive output data broadcast by the computer respectively, and after confirming the device identification from the output data, only reproduce the reproduction data belonging to itself within a limited time.
[0121] The controller plays the role of a welder directly operated by the user in the welding training system, and may include a nine-axis sensor for tracking the movement of the welder and a trigger button for welding as input devices, and may include a vibration motor for reproducing vibration, a feedback motor for reproducing the force of the welding rod, and a thermoelectric element for reproducing the heat transferred to the welder during welding as output devices. The vibration motor, feedback motor and thermoelectric element included in the controller respectively receive the output data broadcast by the computer, and after confirming the device identification from the output data, only reproduce the reproduction data belonging to itself within a limited time.
[0122] Fig.11 An example in which an extended reality system according to an embodiment of the present invention is applied to an excavator training system is shown.
[0123] Reference Fig.11 , the excavator training system may include a computer, an HMD, a steering wheel controller, a joystick controller, a pedal controller and a tactile seat.
[0124] The computer performs 3D processing on the user's position and orientation data obtained from the nine-axis sensor of the HMD, the 3D gesture measurement data obtained from the steering wheel controller, the position data of the excavator's boom, arm and bucket obtained from the joystick controller, the acceleration data of the excavator obtained from the pedal controller, and the augmented reality object data, thereby generating output data and broadcasting it.
[0125] As a result of the 3D processing, the computer generates reproduction data and broadcasts output data, wherein the reproduction data includes 3D rendered images to be reproduced by dual displays, audio based on binaural effect to be reproduced by dual speakers, and 3D pressure and capacitance output to be reproduced by tactile seats, and the output data adds corresponding device identification (i.e., dual displays, dual speakers, and tactile seats) and limited time to each reproduction data.
[0126] The HMD may include a nine-axis sensor for user head tracking and an optical sensor (e.g., a camera and / or a laser radar sensor) for space recognition as input devices, and may include dual displays for displaying 3D rendered images and dual speakers for playing audio based on the binaural effect as output devices. The dual displays and dual speakers receive output data broadcast by the computer, and after confirming the device identification from the output data, only reproduce the reproduction data belonging to themselves within a limited time.
[0127] Fig.12The present invention is a block diagram showing the structure of a wearable temperature transfer device that can be used in an extended reality system based on a multimodal interface according to another embodiment of the present invention. The wearable temperature transfer device can be, for example, a headset, a head mounted display (HMD), a face mounted display (FMD), a helmet mounted display (helmet mounted display) or a garment (garment) form.
[0128] Reference Fig.12 According to another embodiment of the present invention, a wearable temperature transfer device includes a content output device 100, a wearable tool 200, a temperature sensor 300, a thermoelectric element 400 and a processor 500.
[0129] The content output device 100 delivers content to the user. For example, the content output device 100 may be a speaker that plays audio content ( Fig.13 101) or a display showing video content such as VR (Virtual Reality) simulation ( Fig.15 103). The content output device 100 may be attached to a specific position of the wearable tool 200 so as to deliver content to the user.
[0130] The wearing tool 200 can help the user wear the wearable temperature transfer device on his body. For example, when the wearable temperature transfer device is a headset, the wearing tool 200 can be a headband surrounding the user's head ( Fig.13 201), or in the case where the wearable temperature transfer device is an HMD, it may be a main body with a display attached ( Fig.15 203) and a strap (not shown) surrounding the user's head.
[0131] The temperature sensor 300 measures the ambient temperature of the wearable temperature transfer device. Here, "ambient" refers to the surroundings of the temperature transfer device itself, so the "ambient temperature" includes not only the external temperature (e.g., air temperature) or room temperature when the user wears the temperature transfer device, but also the user's body temperature.
[0132] According to an embodiment, the wearable temperature transfer device may include a plurality of temperature sensors 300. For example, a plurality of temperature sensors 300 are attached to the inner side surface (the surface in contact with the user's body) and the outer side surface (the surface not in contact with the user's body) of the wearable temperature transfer device, respectively, so that the temperature of the corresponding position can be measured (hereinafter, the inner side surface of the temperature transfer device refers to the surface in contact with the user's body, and the outer side surface refers to the surface not in contact with the user's body). At this time, the temperature of the position where the temperature sensor 300 is not attached can be estimated using the temperature of the position where the temperature sensor 300 is attached.
[0133] The thermoelectric element 400 transfers temperature to the user. The thermoelectric element 400 can heat the user through an exothermic reaction and cool the user through an endothermic reaction. How much temperature the thermoelectric element 400 transfers to the user (i.e., how much heating or cooling) is determined by the processor 500 taking into account the content and / or the surrounding temperature. The thermoelectric element 400, for example, can be implemented not only by an element having a thermoelectric effect such as the Seebeck effect, the Peltier effect, and the Thompson effect, but can also be implemented by any element using similar technology that can generate a temperature determined by the user or the processor in other ways.
[0134] According to an embodiment, the wearable temperature transfer device may include a plurality of thermoelectric elements 400. For example, a plurality of thermoelectric elements 400 are attached to the inner side of the wearable temperature transfer device, so as to transfer different temperatures to the user according to the parts where the thermoelectric elements 400 are attached.
[0135] Processor 500 determines the temperature to be delivered by thermoelectric element 400 to the user taking into account at least one of the content and the ambient temperature.
[0136] For example, processor 500 may determine the temperature that thermoelectric element 400 is to deliver to the user based on the nature of the content.
[0137] In the case where the content is audio (e.g., music), the processor 500 can determine the temperature that the thermoelectric element 400 is to transmit to the user according to the pitch (i.e., frequency), speed, beat, etc. of the sound. The processor 500 can determine the temperature in the form of causing the thermoelectric element 400 to generate a relatively high temperature at high pitch and a relatively low temperature at low pitch. In addition, the processor 500 can determine the temperature in the form of causing the thermoelectric element 400 to generate a relatively high temperature at a fast speed and a relatively low temperature at a slow speed. In addition, the processor 500 can provide a temperature maintenance function according to the change of the user's body temperature. When listening to audio, the user's body temperature can be changed according to the pitch, speed, beat, etc. For example, if you listen to music or sound with a strong beat, the user's body temperature rises, and the user becomes relatively hot and feels uncomfortable. At this time, the processor 500 determines the temperature of the thermoelectric element 400 according to the user's body temperature measured by the temperature sensor 300, so that the user can hear the audio comfortably.
[0138] In the case where the content is a video (e.g., an XR simulation), the processor 500 can determine the temperature that the thermoelectric element 400 is to transfer to the user based on the XR object. In the XR environment, when there is a hot object near the user, the processor 500 can determine the temperature in the form of causing the thermoelectric element 400 attached in the direction of the hot object to generate a relatively high temperature, and causing the thermoelectric element 400 attached in the opposite direction to generate a relatively low temperature. The temperature transfer device can additionally include a sensor capable of measuring the rotation of the user (e.g., a nine-axis sensor). In the case where the relative position of the hot object changes as the user rotates, the processor 500 can control the temperature of the thermoelectric element 400 based on the changed relative position.
[0139] In addition, the above-mentioned determination of the temperature according to the essence of the content is the same as the above example. The processor 500 can not only determine it by analyzing the essence of the content, but also by the producer or user of the content. For example, the producer of the audio content can decide in advance what kind of temperature the user wants to feel according to the playing time of the audio according to his own creative intention, so that the temperature information can be inserted into the audio content according to the playing time. At this time, the processor 500 can extract the temperature information according to the playing time and control the thermoelectric element 400 to transmit the corresponding temperature to the user. In addition, for example, like the equalizer setting of the audio, the user can set the temperature to be output by the thermoelectric element 400 according to the pitch, speed, beat, playing time, etc. of the sound, and the processor 500 can control the thermoelectric element 400 according to the user's setting.
[0140] In addition, for example, the processor 500 may determine the temperature to be transmitted to the user by the thermoelectric element 400 according to the body temperature of the user. As described above, the temperature sensor 300 is attached to the inner side of the wearable temperature transmission device and can measure the body temperature of the user at the corresponding position. The processor 500 may modify the determined temperature according to the substance of the content and control the thermoelectric element 400 to transmit a relative temperature according to the measured body temperature of the user. In the case where the substance of the content transmits a relatively high temperature, the processor 500 may control the thermoelectric element 400 in the following manner: if the body temperature of the user is 36°C, the thermoelectric element 400 transmits a temperature of 37°C, and if the body temperature of the user is 37°C, the thermoelectric element 400 transmits a temperature of 38°C. Conversely, in the case where the substance of the content transmits a relatively low temperature, the processor 500 may control the thermoelectric element 400 in the following manner: if the body temperature of the user is 36°C, the thermoelectric element 400 transmits a temperature of 35°C, and if the body temperature of the user is 37°C, the thermoelectric element 400 transmits a temperature of 36°C.
[0141] In addition, for example, the processor 500 can determine the temperature that the thermoelectric element 400 is to transmit to the user based on the external temperature (e.g., air temperature or room temperature). As described above, the temperature sensor 300 can be attached to the outer side of the wearable temperature transfer device to measure the external temperature. The processor 500 can control the thermoelectric element 400 to transmit a specific temperature based on the measured external temperature. In the case where the external temperature is low in winter, the processor 500 controls the thermoelectric element 400 to transmit a relatively high temperature, thereby providing a heating function to the user. On the contrary, in the case where the external temperature is high in summer, the processor 500 controls the thermoelectric element 400 to transmit a relatively low temperature, thereby providing a cooling function to the user.
[0142] Fig.13 The structure of a temperature transfer headset according to another embodiment of the present invention is schematically shown.
[0143] Reference Fig.13 , a temperature transfer headset according to another embodiment of the present invention may include speakers 101-1, 101-2, a hair band 201, temperature sensors 300-1 to 300-4, and thermoelectric elements 400-1, 400-2. Fig.13 Although not explicitly shown in FIG. 2 , the temperature transfer headset may include a processor, which may be located inside the headband 201 .
[0144] Speakers 101-1 and 101-2 play audio content such as music. Fig.13Two speakers 101 - 1 and 101 - 2 are shown, but the present invention is not limited thereto. For example, in order to realize multiple channels such as 5.1 channels or 7.1 channels, a larger number of speakers may be included.
[0145] The headband 201 may be configured to surround the user's head so that the user can wear the temperature transfer headset. The headband 201 may be combined with the speakers 101-1, 101-2, the temperature sensors 300-1 to 300-4, and the thermoelectric elements 400-1, 400-2. Fig.13 As shown, the speakers 101 - 1 , 101 - 2 , the temperature sensors 300 - 1 , 300 - 2 , and the thermoelectric elements 400 - 1 , 400 - 2 may be combined on the inner side of the hair band 201 , and the temperature sensors 300 - 3 , 300 - 4 may be combined on the outer side of the hair band 210 .
[0146] The temperature sensors 300-1 to 300-4 measure the temperature of the attached locations. Fig.13 4 temperature sensors 300-1 to 300-4 are shown in the figure, but the present invention is not limited thereto and may include a greater or lesser number of temperature sensors. The temperature sensors 300-1 and 300-2 attached to the inner side of the headband 201 may contact the user's skin when the user wears the temperature transfer device to measure the user's body temperature. The temperature sensors 300-3 and 300-4 attached to the outer side of the headband 201 may measure the external temperature (e.g., air temperature or room temperature) when the user wears the temperature transfer device.
[0147] Thermoelectric elements 400-1, 400-2 transmit the temperature determined by processor 500 to the user. Fig.13 The figure shows two thermoelectric elements 400 - 1 and 400 - 2 that can be connected to and in contact with the inner side surface of the hair band 201 , but the present invention is not limited thereto. A greater number of thermoelectric elements may be included to refine the temperature transfer position.
[0148] In addition, although the temperature transfer headset can be used alone to play audio content, other devices including a display can also be used simultaneously to play combined video and audio content.
[0149] Fig.14 The data processing flow in the temperature transfer headset according to another embodiment of the present invention is shown. Fig.14 The data processing shown may be performed by processor 500 .
[0150] The audio data may include at least one of audio mono data and audio stereo data. In the case of only audio mono data, the VP mixer (MIX) may be used to convert the audio data into audio stereo data. The audio mono data and the audio stereo data may be separated into left output and right output after 3D audio conversion. The separated left and right outputs may be played through respective speakers.
[0151] In addition, temperature leveling can be performed using content tactile data, audio data, and measured temperature. In this case, the audio data separated into left and right outputs can be separated by an equalizer according to each frequency, and thus used for temperature leveling. As described above, temperature leveling can be based on the factors determined by the content producer (i.e., Fig.14 The temperature to be delivered to the user is determined based on the content of the example (tactile data and audio data) and the temperature measured by the temperature sensor.
[0152] Fig.15 The structure of a temperature transfer HMD according to another embodiment of the present invention is schematically shown.
[0153] Reference Fig.15 , a temperature transfer HMD according to another embodiment of the present invention may include a display 103, a main body 203, temperature sensors 301-1 to 301-3, and thermoelectric elements 400-1 to 400-12. Fig.15 Not explicitly shown, but the temperature transfer HMD may include a processor, which may be located inside the body 203 .
[0154] The display 103 plays video contents.
[0155] The display 103, the temperature sensors 301-1 to 301-3, and the thermoelectric elements 400-1 to 400-12 may be combined in the main body 203. Fig.15 As shown, the temperature sensors 301 - 1 to 301 - 3 and the thermoelectric elements 400 - 1 to 400 - 12 may be combined on the inner side of the body 203 .
[0156] The temperature sensors 301-1 to 301-3 measure the temperature of the attached location. Fig.15 3 temperature sensors 300-1 to 300-3 are shown in FIG. 1 , but the present invention is not limited thereto and may include a greater or lesser number of temperature sensors. Fig.15 Although not explicitly shown, a temperature sensor may be attached to the outer side of the main body to measure the external temperature.
[0157] When the user wears the temperature transfer device, the temperature sensors 300-1 to 300-3 can contact the user's skin to measure the user's body temperature. The temperature of the position where the temperature sensors 300-1 to 300-3 are not attached can be estimated using the temperature of the position where the temperature sensors 300-1 to 300-3 are attached. For the estimation of temperature, refer to Fig.16A and Fig. 16B Give a narrative.
[0158] Thermoelectric elements 400 - 1 to 400 - 12 transmit the temperature determined by processor 500 to a user. Fig.15 In, as and Fig.13 In the different examples of the temperature transfer headphones, although 12 thermoelectric elements 400 - 1 to 400 - 12 are attached to the inner side surface of the main body 203 , the present invention is not limited thereto.
[0159] In addition, although the temperature transfer HMD can be used alone to play video content, other devices including speakers can also be used simultaneously to play video and audio combined content.
[0160] Fig.16A and Fig. 16B It is speculation Fig.15 A method of transmitting the temperature of a location in the HMD where no temperature sensor is attached will be described. Fig.16A and Fig. 16B Observed from direction A Fig.15 Plan view of the temperature transfer HMD.
[0161] Reference Fig.16A ,like Fig.15 As shown, three temperature sensors 300-1 to 300-3 and 12 thermoelectric elements 400-1 to 400-12 are attached to the inner side of the temperature transmitting HMD. The temperature sensors 300-1 to 300-3 measure the temperature of the positions where they are attached, and the thermoelectric elements 400-1 to 400-12 transmit the temperature determined by the processor 500 to the user. At this time, the processor 500 should determine the temperature to be transmitted to the user by the thermoelectric elements 400-1 to 400-12 attached to the positions where the temperature sensors 300-1 to 300-3 are not attached based on the temperature measured by the temperature sensors 300-1 to 300-3.
[0162] To this end, the processor 500 may estimate the temperatures of the positions P1, P1-1, P1-2, P2, P2-1, P2-2, P3, P3-1, and P3-3 to which the temperature sensors 300-1 to 300-3 are not attached based on the temperatures measured by the temperature sensors 300-1 to 300-3. For example, the processor 500 may estimate the temperatures of the positions P1, P1-1, P1-2, P2, P2-1, P2-2, P3, P3-1, and P3-3 to which the temperature sensors 300-1 to 300-3 are not attached using the average value of the temperatures measured by the temperature sensors 300-1 to 300-3.
[0163] Reference Fig. 16B For example, when the temperature measured by the temperature sensor 300-1 is 36°C, the temperature measured by the temperature sensor 300-2 is 39°C, and the temperature measured by the temperature sensor 300-3 is 35°C, the processor 500 can estimate the temperatures of the positions P1, P2, and P3 by the average value of the temperatures measured by the temperature sensors 300-1 to 300-3. That is, the processor 500 can estimate the temperature of the position P1 to be 37.5°C, the temperature of the position P2 to be 38.5°C, and the temperature of the position P3 to be 37°C. In addition, the processor 500 can estimate the temperatures of the positions P1-1, P1-2, P2-1, P2-2, P3-1, and P3-2 by the average value of the temperatures estimated at the positions P1, P2, and P3 and the temperatures measured by the temperature sensors 300-1 to 300-3. That is, the processor 500 may infer that the temperature at position P1-1 is 36.75°C, the temperature at position P1-2 is 38.25°C, the temperature at position P2-1 is 38.75°C, the temperature at P2-2 is 38.25°C, the temperature at position P3-1 is 37.5°C, and the temperature at position P3-2 is 36.5°C.
[0164] exist Fig.16A and Fig. 16B In the example, although the processor 500 uses the average value to estimate the temperature of the positions where the temperature sensors 300-1 to 300-3 are not attached, the present invention is not limited to this. A general technician can use other representative values such as the median and the mode other than the average value to estimate the temperature, or can use a weighted average with different weights according to the positions where the temperature sensors are attached or the distances between the temperature sensors to estimate the temperature.
[0165] Fig.17 The structure of a temperature transfer garment according to another embodiment of the present invention is schematically shown. The temperature transfer garment can be used together with the temperature transfer headset or temperature transfer HMD described above.
[0166] Reference Fig.17In order to estimate the temperature of a specific part of the user's body (for example, each joint), a temperature sensor 300 may be attached to a part corresponding to the specific part of the temperature transfer clothing. In this case, the temperature sensor 300 may be attached not only to the front of the user but also to the back.
[0167] The temperature of the portion where the temperature sensor 300 is not attached is the same as the reference Fig.16A and Fig. 16B As described above, the processor 500 can make an inference. In the case of a face, the 3D matrix distribution of the temperature can be predicted and inferred based on the temperature measured at the position where the temperature sensor 300 is attached.
[0168] In addition, temperature transfer clothing can be implemented by using dual materials that can sense the temperature of the entire body and output tactile sensation to perform temperature measurement and temperature reproduction.
[0169] The above description is made with reference to the accompanying drawings and embodiments, but it does not mean that the protection scope of the present invention is limited to the above drawings or embodiments. Practitioners in the relevant technical field can make various modifications and changes to the present invention without departing from the scope of the idea and field of the present invention recorded in the claims.
Claims
1. An extended reality system based on a multimodal interface, characterized in that: include: A computer, which transmits output data including at least one device identification, at least one limited time and at least one reproduction data to at least one extended reality output device; as well as at least one extended reality output device, which receives the output data and reproduces only its own reproduction data within a limited time based on at least one device identification; The output data further includes a command flag including information indicating whether to reproduce the reproduction data. At least one augmented reality output device reproduces its own reproduction data within a limited time based on the command flag.
2. The extended reality system based on a multimodal interface according to claim 1, characterized in that: The computer broadcasts the output data to at least one augmented reality output device.
3. The extended reality system based on a multimodal interface according to claim 1, characterized in that: At least one augmented reality output device does not reproduce its own reproduction data when a limited time has passed.
4. The extended reality system based on a multimodal interface according to claim 1, characterized in that: The output data is in NFXM format with a mirror structure.
5. The multimodal interface-based extended reality system according to claim 4, characterized in that: The NFXM format includes a command, a device number, a device identifier, a data packet size, and a data packet.
6. The extended reality system based on a multimodal interface according to claim 4, characterized in that: The computer selectively encrypts the entire area or a part of the area in the NFXM format and broadcasts the output data.
7. The extended reality system based on a multimodal interface according to claim 4, characterized in that: The computer encrypts the entire area and part of the area in NFXM format. The encrypted whole area and parts of the area can be accessed using different keys.
8. The extended reality system based on a multimodal interface according to claim 1, characterized in that: The reproduction data includes at least one of visual, auditory, tactile, olfactory and taste reproduction data.
9. The extended reality system based on a multimodal interface according to claim 1, characterized in that: The augmented reality output device is any one of a head mounted display, headphones, and a haptic device.
10. The extended reality system based on a multimodal interface according to claim 1, characterized in that: The computer synchronously transmits the output data based on at least one of the reproduction data and the command mark.
11. An extended reality system based on a multimodal interface, characterized in that: include: A computer, which transmits output data including at least one device identification, at least one limited time and at least one reproduction data to at least one extended reality output device; as well as at least one extended reality output device, which receives the output data and reproduces only its own reproduction data within a limited time based on at least one device identification, The computer is located inside any one of the at least one extended reality output device; The output data further includes a command flag including information indicating whether to reproduce the reproduction data. At least one augmented reality output device reproduces its own reproduction data within a limited time based on the command flag.
12. A method for transmitting reproduction data, as a method for transmitting reproduction data from a computer to at least one augmented reality output device, characterized in that: The following steps are involved: synchronizing the timing of actions between the computer and at least one augmented reality output device; as well as The computer transmits output data including at least one device identification, at least one limited time, and at least one reproduction data to at least one extended reality output device; The output data further includes a command flag including information indicating whether to reproduce the reproduction data. At least one augmented reality output device reproduces its own reproduction data within a limited time based on the command flag.
13. A method for receiving reproduction data, as a method for an extended reality output device to receive reproduction data from a computer, characterized in that: The following steps are involved: Synchronizing the timing of actions between a computer and an augmented reality output device; receiving output data transmitted by a computer, the output data comprising at least one device identification, at least one limiting time and at least one reproduction data; and Reproducing the reproduction data belonging to the user within a limited time based on at least one device identification; The output data further includes a command flag including information indicating whether to reproduce the reproduction data. At least one augmented reality output device reproduces its own reproduction data within a limited time based on the command flag.
