Method for carrying time-triggered spatial haptic effect in switching format

Abstract

A method of signaling a haptic JavaScript Object Representation (JSON) Exchange File Format (HJIF) file includes processing the HJIF file into a binary file format for distribution in a code stream, where the HJIF file includes at least two time-triggered spatial haptic effects, where the spatial haptic effects are different from each other, where the spatial haptic effects are different from each other, and where the spatial haptic effects are different from each other. At least one time-triggered spatial haptic effect from the at least two spatial haptic effects varies along a spatial axis, where the at least one time-triggered spatial haptic effect is associated with a first parameter defining a trigger time, and where the at least one time-triggered spatial haptic effect is associated with a second parameter defining a trigger time. A renderer is configured to render the at least one time-triggered spatial haptic effect according to the trigger time.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 525,928 filed on July 10, 2023, U.S. Provisional Application No. 63 / 525,926 filed on July 10, 2023, U.S. Provisional Application No. 63 / 525,924 filed on July 10, 2023, U.S. Provisional Application No. 63 / 525,920 filed on July 10, 2023, and U.S. Application No. 18 / 767,085 filed on July 9, 2024, the disclosure of each of which is incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments of the present application relate to a method for carrying time-triggered spatial haptic effects in an interchange format. Background Art

[0004] The use of haptics has become a part of multimedia presentations. In such applications, haptic signals are delivered to a device or wearable hardware, where the user feels the haptics during the use of the application. Recently, MPEG has started working on compression standards for haptics.

[0005] The Haptic Committee draft includes a JavaScript Object Notation (JSON) format and a binary format. The current JSON format (called the Haptic Interchange Format) carries both temporal and spatial haptic effects. However, spatial haptic effects are constant in time and are therefore time-invariant. In addition, the current JSON format does not include any timed packetization. Summary of the invention

[0006] According to one aspect of the present application, a method for signaling a tactile JavaScript Object Notation (JSON) Interchange File Format (HJIF) file includes: processing the HJIF file into a binary file format for transmission in a bitstream, wherein the HJIF file includes at least two time-triggered spatial tactile effects, wherein at least one of the at least two spatial tactile effects is a time-triggered spatial tactile effect that varies along a spatial axis, wherein the at least one time-triggered spatial tactile effect is associated with a first parameter that defines a trigger time, and configuring a renderer to render the at least one time-triggered spatial tactile effect according to the trigger time.

[0007] According to one aspect of the present application, a method for signaling a tactile JavaScript Object Notation (JSON) Interchange File Format (HJIF) file includes: processing the HJIF file into a binary file format for transmission in a bitstream, wherein the HJIF file includes a data hierarchy, the data hierarchy specifying (i) a tactile perception at a first level of the data hierarchy, (ii) at least one tactile channel corresponding to the tactile perception at a second level of the data hierarchy, and (iii) at least one tactile frequency band corresponding to the at least one tactile channel at a third level of the data hierarchy, wherein the first level in the data hierarchy is higher than the second level in the data hierarchy, and the second level in the data hierarchy is higher than the third level in the data hierarchy, wherein the HJIF file further includes at least two time data packets, wherein each time data packet includes a time parameter and at least one tactile effect, and the renderer is configured to render each of the at least one tactile effect according to the time parameter of each time data packet.

[0008] According to one aspect of the present application, a method for decoding a tactile JavaScript Object Notation JSON Interchange File Format HJIF file includes: receiving a code stream; decoding the code stream to extract the HJIF file; and rendering at least two time-triggered spatial tactile effects included in the HJIF file, wherein at least one time-triggered spatial tactile effect of the at least two spatial tactile effects varies along a spatial axis, the at least one time-triggered spatial tactile effect is associated with a first parameter defining a trigger time, and, based on the trigger time, rendering the at least one time-triggered spatial tactile effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0010] Figure 1 is a schematic diagram of an environment in which the methods, apparatus, and systems described herein may be implemented, according to an embodiment.

[0011] Figure 2 yes Figure 1 A block diagram of example components of at least one device.

[0012] Figure 3 and Figure 4 is a diagram of a haptic codec architecture according to an embodiment.

[0013] Figure 5 The JSON format data hierarchy is shown.

[0014] Figure 6An example data hierarchy is shown according to an embodiment.

[0015] Figure 7 An example data hierarchy is shown according to another embodiment.

[0016] Figure 8 An example data hierarchy is shown according to another embodiment.

[0017] Fig. 9 An example flow diagram of a process for rendering a temporally triggered spatial haptic effect is shown. DETAILED DESCRIPTION

[0018] The following detailed description of example embodiments refers to the accompanying drawings.The same reference numbers in different drawings may identify the same or similar elements.

[0019] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the disclosed precise form. According to the above disclosure, modifications and variations are possible, or modifications and variations can be obtained from the practice of the embodiments. In addition, at least one feature or component of an embodiment can be incorporated into another embodiment (or at least one feature of another embodiment) or combined with another embodiment (or at least one feature of another embodiment). In addition, in the flowcharts and operational descriptions provided below, it should be understood that at least one operation can be omitted, at least one operation can be added, at least one operation can be performed simultaneously (at least in part), and the order of at least one operation can be switched.

[0020] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software codes, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0021] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims, and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim group.

[0022] Unless so clearly described, any element, behavior or instruction used herein should not be interpreted as critical or essential. In addition, as used herein, the articles "a" and "an" are intended to include at least one project and can be used interchangeably with "at least one". When only one project is intended, the term "one" or similar language is used. In addition, as used herein, the terms "has / have / having", "include / including" etc. are intended to be open terms. In addition, unless otherwise clearly stated, the phrase "based on" is intended to mean "based at least in part". In addition, expressions such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B, or both A and B.

[0023] References throughout the specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in connection with the indicated embodiment is included in at least one embodiment of the present solution. Thus, the phrases "in one embodiment," "in an embodiment," and similar language throughout the specification may, but do not necessarily, all refer to the same embodiment.

[0024] In addition, the features, advantages and characteristics described in the present application may be combined in at least one embodiment in any suitable manner. Based on the description herein, those skilled in the relevant art will recognize that the present application may be practiced without at least one of the specific features or advantages of a particular embodiment. In other cases, additional features and advantages that may not exist in all embodiments of the present application may be recognized in certain embodiments.

[0025] Figure 1 1 is a schematic diagram of an environment 100 in which the methods, apparatuses, and systems described herein may be implemented, according to an embodiment. Figure 1 As shown, environment 100 may include user device 110, platform 120, and network 130. The devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0026] The user device 110 includes at least one device that can receive, generate, store, process and / or provide information related to the platform 120. For example, the user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch), or a similar device. In some embodiments, the user device 110 can receive information from the platform 120 and / or send information to the platform 120.

[0027] The platform 120 includes at least one device as described elsewhere herein. In some embodiments, the platform 120 may include a cloud server or a cloud server group. In some embodiments, the platform 120 may be designed to be modular so that software components can be swapped in or out according to specific needs. In this way, the platform 120 can be easily and / or quickly reconfigured to have different uses.

[0028] In some embodiments, as shown, the platform 120 may be hosted in a cloud computing environment 122. It is worth noting that although the embodiments described herein describe the platform 120 as being hosted in a cloud computing environment 122, in some embodiments, the platform 120 is not cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0029] Cloud computing environment 122 includes an environment hosting platform 120. Cloud computing environment 122 may provide computing, software, data access, storage, and other services that do not require end users (e.g., user devices 110) to be aware of the physical location and configuration of the systems and / or devices hosting platform 120. As shown, cloud computing environment 122 may include a set of computing resources 124 (collectively, “computing resources 124” and individually, “computing resource 124”).

[0030] The computing resources 124 include at least one personal computer, workstation computer, server device, or other type of computing and / or communication device. In some embodiments, the computing resources 124 can host the platform 120. Cloud resources can include computing instances executed in the computing resources 124, storage devices provided in the computing resources 124, data transmission devices provided by the computing resources 124, etc. In some embodiments, the computing resources 124 can communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0031] Further Figure 1 As shown, the computing resources 124 include a set of cloud resources, such as at least one application ("APP") 124-1, at least one virtual machine ("VM") 124-2, virtualized storage ("VS") 124-3, at least one hypervisor ("HYP") 124-4, etc.

[0032] The application 124-1 includes at least one software application that can be provided to or accessed by the user device 110 and / or the platform 120. The application 124-1 does not require the software application to be installed and executed on the user device 110. For example, the application 124-1 can include software related to the platform 120, and / or any other software that can be provided through the cloud computing environment 122. In some embodiments, one application 124-1 can send / receive information to or from at least one other application 124-1 through the virtual machine 124-2.

[0033] Virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs, similar to a physical machine. Virtual machine 124-2 can be a system virtual machine or a process virtual machine, depending on the use and correspondence of virtual machine 124-2 to any real machine. System virtual machines can provide a complete system platform that supports the execution of a complete operating system ("OS"). Process virtual machines can execute a single program and can support a single process. In some embodiments, virtual machine 124-2 can execute on behalf of a user (e.g., user device 110) and can manage the infrastructure of cloud computing environment 122, such as data management, synchronization, or long-term data transfer.

[0034] Virtualized storage 124-3 includes at least one storage system and / or at least one device that uses virtualization technology within the storage system or device of computing resource 124. In some embodiments, within the context of a storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage so that the storage system may be accessed without regard to physical storage or heterogeneous structures. Separation may allow administrators of storage systems to flexibly manage storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and the location of the physical storage file. This may optimize the performance of storage usage, server consolidation, and / or non-disruptive file migration.

[0035] Hypervisor 124-4 may provide hardware virtualization technology that allows at least two operating systems (e.g., "guest operating systems") to execute simultaneously on a host computer such as computing resource 124. Hypervisor 124-4 may provide a virtual operating platform to the guest operating systems and may manage the execution of the guest operating systems. At least two instances of the various operating systems may share virtualized hardware resources.

[0036] The network 130 includes at least one wired and / or wireless network. For example, the network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0037] Figure 1 The number and arrangement of devices and networks shown are provided as examples. Figure 1 There may be more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different arrangements of devices and / or networks than those shown. Figure 1 At least two of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as at least two distributed devices. Additionally or alternatively, one set of devices (eg, at least one device) of environment 100 may perform at least one function described as being performed by another set of devices of environment 100.

[0038] Figure 2 yes Figure 1 The device 200 may correspond to the user device 110 and / or the platform 120. Figure 2 As shown, device 200 may include a bus 210 , a processor 220 , a memory 230 , a storage component 240 , an input component 250 , an output component 260 , and a communication interface 270 .

[0039] The bus 210 includes components that allow communication between components of the device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some embodiments, the processor 220 includes at least one processor that can be programmed to perform a function. The memory 230 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory), which stores information and / or instructions for use by the processor 220.

[0040] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-volatile computer-readable medium, and a corresponding drive.

[0041] Input components 250 include components that allow device 200 to receive information, such as through user input, such as a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone. Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., global positioning system (GPS) components, accelerometers, gyroscopes, and / or actuators). Output components 260 include components that provide output information from device 200, such as a display, a speaker, and / or at least one light emitting diode (LED).

[0042] The communication interface 270 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter) that enables the device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 270 can allow the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 270 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0043] Device 200 can perform at least one process described herein. Device 200 can perform these processes in response to processor 220 executing software instructions stored by non-volatile computer readable media (e.g., memory 230 and / or storage component 240). Computer readable media is defined herein as non-volatile memory devices. Memory devices include storage space within a single physical storage device or storage space distributed on at least two physical storage devices.

[0044] The software instructions may be read into the memory 230 and / or storage component 240 from another computer-readable medium or from another device through the communication interface 270. When executed, the software instructions stored in the memory 230 and / or storage component 240 may cause the processor 220 to perform at least one process described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform at least one process described herein. Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0045] Figure 2 The number and arrangement of components shown are provided as examples. Figure 2 The device 200 may include more components, fewer components, different components, or components arranged differently than the components shown. Additionally or alternatively, one set of components (e.g., at least one component) of the device 200 may perform at least one function described as being performed by another set of components of the device 200.

[0046] refer to Figure 3 to Figure 4 , describes an embodiment of the present application for implementing a haptic encoder 300 and a haptic decoder 350 .

[0047] like Figure 3As shown, the haptic encoder 300 can receive descriptive haptic data and waveform haptic data. Therefore, the haptic encoder 300 can process three types of input files: .ohm metadata files (object haptic metadata-text file format for haptic metadata), descriptive haptic files (.ivs, .ahap and .hjif) or waveform PCM files (.wav). Examples of descriptive data may include .ahap (Apple Haptic and Audio Patterns-JSON-like file format that specifies haptic patterns) from Apple (a set of modulated continuous signals and a set of modulated transient parameterizations representing the expected haptic output), .ivs from Immersion (a set of basic effects parameterized by a set of parameters representing the expected haptic output), or the proposed MPEG format-.hjif (Haptic JSON Interchange Format). An example of a waveform Pulse-Code Modulation (PCM) signal may include an .ohm input file that includes metadata information.

[0048] According to an embodiment, the haptic encoder 300 may process the two types of input files in different ways.For descriptive content, the haptic encoder 300 may semantically analyze the input to transcode the data (if necessary) into the proposed encoded representation.

[0049] According to an embodiment, the .ohm metadata input file may include a description of the haptic system and settings. In particular, it may include the name of each associated haptic file (descriptive or PCM) and a description of the signal. It also provides a mapping between each channel of the signal and a target body part on the user's body. For the .ohm metadata input file, the haptic encoder performs metadata extraction by retrieving the associated haptic file from the URI, encoding it based on the haptic file type, extracting metadata from the .ohm file, and mapping the metadata to the metadata information of the data model.

[0050] According to an embodiment, descriptive tactile files (e.g., .ivs, .ahap, and .hjif) can be encoded by a simple process. The tactile encoder 300 first specifically identifies the input format. If the input format is a .hjif file, transcoding is not required, and the file can be further edited, compressed into a binary format, and finally packaged into a MIHS stream. If a .ahap or .ivs input file is used, transcoding is necessary. The tactile encoder 300 first semantically analyzes the input file information and transcodes the input file information into a selected data model format. After transcoding, the data can be exported as a .hjif file, a .hmpg binary file, or a MIHS stream.

[0051] According to an embodiment, the tactile encoder 300 can perform signal analysis to interpret the signal structure of the .wav file and convert it into the proposed coding representation. For waveform PCM content, the signal analysis process can be divided into two sub-processes by the tactile encoder 300. After performing frequency band decomposition on the signal, at the first sub-process, a key frame extraction process can be used to encode the low frequency. At least one low frequency band can then be reconstructed, and the error between the signal and the original low frequency signal can be calculated. The residual signal can then be added to at least one original high frequency band before encoding using a wavelet transform, and encoding using a wavelet transform is a second sub-process. According to an embodiment, when several low frequency bands are used, residual errors from all low frequency bands are added to the high frequency bands before encoding. In an embodiment, when several high frequency bands are used, residual errors from at least one low frequency band are added to the first high frequency band before encoding.

[0052] According to an embodiment, key frame extraction includes obtaining a lower frequency band from the frequency band decomposition and analyzing the content of the lower frequency band in the time domain. According to an embodiment, wavelet processing may include obtaining a high frequency band from the frequency band decomposition and the low frequency residual, and dividing the high frequency band into equal-sized blocks. These equal-sized signal blocks are then analyzed in a psychotactile model. With the help of a psychotactile model, lossy compression can be applied by wavelet transforming the blocks and quantizing them. Finally, each block is saved as a separate effect in a single frequency band, which is done in formatting. Binary compression can apply lossless compression using appropriate coding techniques (e.g., a multi-level tree set splitting (Set Partitioning In Hierarchical Trees, SPIHT) algorithm and arithmetic coding (Arithmetic Coding, AC)).

[0053] like Figure 3 As shown, the tactile encoder 300 can be configured to encode descriptive and quantized tactile data, and can output three types of formats - an interchange format (.hjif), a binary compression format (.hmpg), and a streaming format (e.g., MPEG Immersive Haptic Stream (MIHS)). The .hjif format is a human-readable format based on JSON and can be easily parsed and manually edited, making it an ideal exchange format, especially when designing / creating content. For transmission purposes, the .hjif data can be compressed into a binary .hmpg code stream that is more storage efficient. This compression can be lossy, with different parameters affecting the encoding depth of the amplitude and frequency that make up the code stream. For streaming purposes, the data can be compressed and packaged into an MPEG-I tactile stream (MIHS). The above three formats have complementary purposes, and lossy one-to-one conversion can be performed between them.

[0054] like Figure 4 As shown, the tactile decoder 350 can take a .hmpg compressed binary file format or a MIHS code stream as input. The tactile decoder 350 can output a .hjif interchange format that can be used directly for rendering. Both input formats can be binary decompressed to extract metadata and the data itself from the file and map it to a selected data structure. The data can then be exported to the tactile renderer 380 in .hjif format.

[0055] like Figure 4 As shown, the renderer 380 includes a synthesizer. The synthesizer can render tactile data from a .hjif input file into a PCM output file. Rendering and / or synthesis is informative. According to an embodiment, the synthesizer parses the input file and performs high-level synthesis distribution between vectors, wavelets, etc. The synthesis process is then passed down to the frequency band component of the codec, where the synthesis process is called. All frequency bands of a given channel are then mixed by a simple addition operator to recreate the desired tactile signal.

[0056] Embodiments of the present application relate to a method for carrying time-triggered spatial haptic effects in an exchange format. In at least one example, the spatial haptic effect is defined for a spatial perception modality, for example, the haptic effect changes according to the distance from the origin. The HJIF format supports this effect.

[0057] In at least one example, we define a time-triggered haptic effect as a haptic signal that varies in space. For example, an effect is associated with a spatial axis and may also have a trigger time (e.g., the effect is triggered at a certain time).

[0058] In accordance with at least one embodiment, a time-triggered spatial haptic effect can have two parameters: (i) a position P, which corresponds to the distance from the spatial origin when the haptic effect becomes effective, and (ii) a trigger time T, which corresponds to the time offset relative to time 0 (e.g., the start of playback) when the haptic effect is rendered.

[0059] In at least one example, optional parameters for spatial location are provided in the HJIF file, as illustrated in Table 1.

[0060]

[0061]

[0062]

[0063] Table 1

[0064] In at least one example, trigger time (trigger_time) can be defined as an offset for all effects (eg, temporal and spatial), where location can be used as a spatial offset for spatial effects, as illustrated in Table 2.

[0065]

[0066]

[0067]

[0068] Table 2

[0069] Embodiments of the present application also relate to a method for grouping haptic effects in a haptic exchange format. As understood by those of ordinary skill in the art, the haptic exchange format (.hjif) does not specify the temporal grouping of haptic effects. Instead, according to Figure 5 As shown in the data hierarchy 500, the exchange format specifies a non-timed JSON format. According to the data hierarchy 500, all data structures are in a single time scale.

[0070] In at least one example, haptic avatars can be used as body representations. Haptic avatars define different types of avatars and allow custom 3D meshes to be referenced from companion files. Each haptic perception of the experience is associated with a haptic avatar, which allows for spatialization of haptic effects at the haptic channel level. The same avatar can be used by at least two perceptions. Using a 3D mesh allows for high resolution and accuracy with variable vertex density depending on the application. For example, the density can represent the spatial acuity of a specific perception modality.

[0071] Table 3 shows example properties of a tactile avatar.

[0072]

[0073] Table 3

[0074] In at least one example, a tactile perception is a tactile signal associated with a specific sensory modality. The format supports modalities encoded as a function of time (pressure, acceleration, velocity, position, temperature, vibrotactile, water, wind, force, electrotactile) or space (vibrotactile texture, stiffness, and friction). A list of supported modalities and corresponding units are provided in Table 5. For each tactile perception, metadata information about the modality, the corresponding avatar representation, and the technical characteristics of the target or compatible tactile device is provided. The data associated with the perception can contain at least two channels. Channels are associated with body positions and typically correspond to tactile devices. For example: a vibrotactile suit with 16 channels, corresponding to 16 "vibrotactile" actuators, or a game controller with one "force" feedback trigger.

[0075] A haptic sense can contain an effects library. Error! Reference source not found. Example properties for a haptic sense are shown.

[0076]

[0077]

[0078] Table 4

[0079] In at least one example, a haptic experience can be defined using a reference setup to validate the experience, or using at least two specific target haptic devices (e.g., reference devices). If the experience is played on a different device with different capabilities, the relevant encoded information may have to be presented differently. In order to perform this adaptation, the capabilities of at least one of the original devices (reference or compatible devices) must be known. To this end, a list of reference devices (and their detailed characteristics) is defined for each haptic perception, and each haptic channel can reference a corresponding device. The haptic reference device is described by a series of characteristics, including the type of device, the frequency range of the device, the maximum voltage of the device, and many other properties. An example list of properties is specified in Error! Reference source not found.

[0080]

[0081]

[0082] Table 5

[0083] In at least one example, tactile signals can be encoded on at least two channels. For example, a tactile channel defines a signal presented at a specific body location with a dedicated actuator / device. Metadata stored at the channel level includes information such as the gain associated with the channel, mixing weights, the desired body location of the tactile feedback, and an optional reference device and / or direction. Additional information such as the desired sampling frequency or sample count can also be provided. Finally, the tactile data for the channel is contained in a set of tactile frequency bands defined by their frequency ranges. A list of properties for a tactile channel is detailed in Table 6.

[0084]

[0085]

[0086]

[0087] Table 6

[0088] In at least one example, a haptic frequency band describes a haptic signal for a channel within a given frequency range. A frequency band is defined by a type and an ordered list of haptic effects, each haptic effect containing a set of keyframes. Table 7 is an example list of properties for a haptic frequency band.

[0089]

[0090]

[0091] Table 7

[0092] Table 8 shows example perceptual modalities and corresponding units.

[0093] Modal Perception Unit unit pressure Pa S Acceleration <![CDATA[m / s 2 ]]> s speed m / s s Location M s temperature K s Vibrotactile Normalized to -1 / 1 s water <![CDATA[m 3 ]]> S wind m / s S force N S Electrotactile Normalized to -1 / 1 S Vibration tactile texture Normalized to -1 / 1 M Stiffness N M friction Normalized to -1 / 1 M other Normalized to -1 / 1 s

[0094] Table 8

[0095] In accordance with at least one embodiment, the haptic exchange format is modified to create new levels in the data hierarchy of the temporal data packets. Figure 6 An example data hierarchy is shown with a "time packet" level added.

[0096] like Figure 6 As shown, in at least one example, a band can include at least one time packet. Each time packet can have a parameter T that defines a time offset from an origin (e.g., the playback time of a file). T can be in the time scale of another parameter, so that T / time scale defines the offset from the origin in seconds. In at least one example, a time packet can also have only a duration value. In this case, the start of the packet is at the end of the previous packet. The duration shows the duration of the packet, possibly expressed in a time scale.

[0097] In at least one example, each time packet has at least one effect. The position of the time effect may indicate an offset of the effect from the beginning of the time packet. The position of the effect may be less than the duration of the packet.

[0098] The location of the spatial effect may indicate a spatial offset from the spatial origin. The effect may be rendered at the start time of the packet. In at least one example, a packet without an effect (e.g., the effect field is blank or includes a predetermined value indicating no effect, such as NULL) may indicate a silent time when no effect is required to be rendered.

[0099] Embodiments of the present application also relate to a method for real-time grouping of haptic effects in a haptic exchange format. According to at least one embodiment, a new data hierarchy is provided for time grouping at a higher level. Figure 7 An example data hierarchy 700 is shown with new additions underlined.

[0100] like Figure 7 As shown, a time packet can be an array of avatars and perception levels. For example, an effect may refer to the id number of a perception, channel, and band. Each time packet can have a parameter T that defines the time offset from the origin (e.g., the playback time of a file). T can be in another parameter timescale, so that T / timescale defines the offset from the origin in seconds. A time packet can also have only a duration value. In this case, the start of this packet is at the end of the previous packet. Duration can show the duration of the packet and can be in the scale of the timescale. Each packet can also define whether it is a synchronization packet (e.g., a packet synchronized on a time point or spatial effect).

[0101] In at least one example, each temporal data packet has at least one effect. The position of a temporal effect may indicate an offset of the effect from the start of the temporal data packet. The position of an effect may be less than the duration of the data packet. The position of a spatial effect may indicate a spatial offset from a spatial origin. The effect is rendered at the start time of the data packet.

[0102] In at least one example, a packet without an effect (e.g., the effect field is empty or includes a predetermined value indicating no effect, such as NULL) can indicate a silent time when no effect is required to render. The value of the data hierarchy 700 is that new packets can be added to the file without changing the hierarchy. Therefore, new packets can be added to the packet array. This approach is advantageous for real-time use cases, where packets are gradually added to the end of the file in real time.

[0103] Embodiments of the present application also relate to a method for serializing an HJIF file into a time data packet. According to at least one embodiment, a new level is created in the data hierarchy. Figure 8 An example data hierarchy 800 is shown with new elements underlined.

[0104] like Figure 8 As shown, all information of an HJIF file, except for the high-level information, is serialized as an array of timing packets. Each timing packet can have a parameter T that defines the time offset from the origin (e.g., the playback time of the file). T can be in another parameter timescale, so that T / timescale defines the offset from the origin in seconds. A timing packet may also have only a duration value. In this case, the start of the packet is at the end of the previous packet. The duration can show the duration of the packet and can be in the scale of the timescale. Each packet can be a synchronization packet. This property can be signaled with the synchronization property of the packet.

[0105] Each time packet may have a duration of zero or longer. A packet may have a start time. The start time and duration may be described by a scale of parameter time scale. Zero duration packets may have only metadata such as avatar or perception, reference device, channel and band information.

[0106] In at least one example, a non-zero duration packet has at least one effect. The position of a temporal effect indicates an offset of the effect from the start of the temporal packet. The position of the effect may be less than the duration of the packet. The position of a spatial effect may indicate a spatial offset from a spatial origin. The effect may be rendered at the start time of the packet.

[0107] In at least one example, a packet without an effect (eg, the effect field is empty, or includes a predetermined value indicating no effect, such as NULL) can indicate a silent time when no rendering effect is desired.

[0108] Time-triggered spatial effects have significant advantages, including: (i) spatial effects that occur at a specific time; and (ii) spatial effects can be updated in a timely manner, for example, an effect may overwrite a previous effect at the same location. Thus, spatial effects can be time-varying.

[0109] Timed grouping of effects in HJIF files provides significant advantages, including: (i) dividing the HJIF file into a series of packets, where each packet has multiple effects; (ii) including stream friendly features, as each packet contains a time serialization of the effects in the packet; (iii) efficient rendering, as the renderer starts rendering packet by packet and does not need to read the entire JSON file and process it; and (iv) allowing timed triggering of spatial effects, where a signal can be sent to trigger a spatial effect at a specific time.

[0110] The temporal serialization of HJIF files has significant advantages, including: (i) the file consists of a series of packets, so converting the file to a MIHS stream is simple; (ii) the renderer can read the header of the temporal packet and only decode and render the required packets, so the decoder does not need to read the entire HJIF file; (iii) high-level metadata information, such as perception, reference device, avatar, and channel composition, can be updated in any packet; (iv) real-time generation of HJIF files is possible because each time a temporal packet can be added to the file, including updating the high-level metadata.

[0111] Fig. 9 A flow chart of an example process 900 for rendering a temporally triggered spatial haptic effect is shown.

[0112] The process may start at operation S902, in which a code stream is received. The code stream may be decoded by the decoder 350 ( Figure 4 ) is received. The code stream can be received by the tactile encoder 300 ( Figure 3 )generate.

[0113] The process proceeds to operation S904, in which the code stream is decoded to obtain an HJIF file included in the code stream. The HJIF file may define at least two time-triggered spatial haptic effects according to Table 1 and Table 2. In addition, the spatial haptic effects may be triggered based on the time-triggered spatial haptic effects. Figures 6 to 8 The data hierarchy shown in defines the HJIF file and divides the HJIF file into a series of data packets.

[0114] The process proceeds to operation S906, in which at least two time-triggered spatial haptic effects are rendered. The at least two time-triggered spatial effects may be rendered by the renderer 380 ( Figure 4 ) rendering.

[0115] According to at least one embodiment, a method includes creating at least two time-triggered spatial effects in a haptic exchange format, wherein the time-triggered spatial effects are not static and are dynamically variable over time. Each time-triggered spatial effect is associated with additional parameters that define a corresponding trigger time for each time-triggered spatial effect. A renderer renders the at least two time-triggered spatial effects. By introducing an additional new trigger time, each time-triggered spatial effect can be updated in a timely manner, thereby allowing a second time-triggered spatial effect to overwrite a first time-triggered spatial effect and replace its previous effect.

[0116] According to at least one embodiment, a method includes creating a time-packetized effect in a haptic exchange format, wherein a new level of time packetization is used in a data hierarchy, wherein a frequency band includes at least one time data packet, each data packet including at least one of a start time, a duration, and a timescale. Wherein the data packet information indicates a time interval of the time data packets. Each data packet further includes at least one effect that is valid for the duration of the time data packet, wherein a null data packet corresponds to a silence duration. Spatial effects and / or temporal effects are referenced using the data packet start time. Wherein the file is suitable for streaming. A renderer reads each data packet sequentially and renders each data packet, thereby eliminating the need for the renderer to read the entire file together.

[0117] According to at least one embodiment, a method includes creating a time-packetized tactile exchange format for live use cases. Wherein, a second data layer and a new data layer for time packetization are used separately from the first data layer. Each time data packet includes a start time, a duration and / or a time scale, wherein the data packet information includes a time interval of the time data packets. Each data packet includes at least one effect that starts within the duration of each time data packet, and each effect references a corresponding perception, channel, and band identifier. The empty data packet includes a silence duration. Wherein the data packet start time is used to reference spatial effects and / or temporal effects, wherein the HJIF file is suitable for streaming. Wherein a renderer reads the HJIF file sequentially through each data packet and renders the HJIF file, thereby eliminating the need for the renderer to read the entire HJIF file together.

[0118] According to at least one embodiment, a method includes generating a time serialization of an HJIF file in a haptic interchange format, wherein a series of timed packets include high-level file information, metadata, and effect data. Wherein the HJIF file includes an array of timed packets, wherein metadata and effect data are added and / or updated at corresponding packets. The method further includes creating real-time effects and adding these effects to existing files. Wherein a renderer browses the HJIF file by reading high-level data of the packets, parsing and decoding selected portions of the packets without parsing other packets.

[0119] The methods proposed herein may be implemented by a processing circuit (eg, at least one processor or at least one integrated circuit). In one example, at least one processor executes a program stored in a non-transitory computer-readable medium to perform one or more of the proposed methods.

[0120] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in at least one computer-readable medium.

[0121] The embodiments of the present application may be used alone or in any order. In addition, each embodiment (and method thereof) may be implemented by a processing circuit (e.g., at least one processor or at least one integrated circuit). In one example, at least one processor executes a program stored in a non-volatile computer-readable medium.

[0122] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed.Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the embodiments.

[0123] As used herein, the term component is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0124] The descriptions of various aspects and embodiments have been presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Although combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each of the dependent claims listed below may directly reference only one claim, the disclosure of possible implementations includes each dependent claim in combination with each other claim in the claim group.

[0125] Unless explicitly described to this, the elements, actions or instructions used in this article should not be interpreted as critical or necessary. In addition, as used in this article, the articles "a" and "an" are intended to include at least one item, and can be used interchangeably with "at least one". In addition, as used in this article, the term "set" is intended to include at least one project (for example, related projects, unrelated projects, combinations of related and unrelated projects, etc.), and can be used interchangeably with "at least one". In the case of meaning only one, the term "one" or similar language is used. In addition, as used in this article, the term "has", "have", "containing", etc. are intended to be open terms. In addition, unless explicitly stated otherwise, the phrase "based on" is intended to mean "based at least in part". In addition, statements like "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B, or both A and B.

Claims

1. A method for signaling a Haptic JavaScript Object Notation (JSON) Interchange File Format (HJIF) file, characterized in that: The method comprises: Processing the HJIF file into a binary file format for transmission in a bitstream; wherein the HJIF file includes at least two time-triggered spatial haptic effects, Wherein, at least one of the at least two spatial haptic effects is a time-triggered spatial haptic effect that varies along a spatial axis, wherein the at least one time-triggered spatial haptic effect is associated with a first parameter defining a trigger time, A renderer is configured to render the at least one time-triggered spatial haptic effect according to the triggering time.

2. The method according to claim 1, characterized in that The at least one temporally triggered spatial effect is associated with a second parameter that indicates a distance from a spatially perceived origin of the at least one temporally triggered spatial haptic effect.

3. The method according to claim 1, characterized in that The trigger time is an offset relative to the playback start time.

4. The method according to claim 1, characterized in that The trigger time is a time interval indicating a duration of the at least one time-triggered spatial haptic effect.

5. The method according to claim 1, characterized in that The at least two temporally-triggered spatial haptic effects include another temporally-triggered spatial haptic effect rendered at the same location as the at least one temporally-triggered spatial haptic effect.

6. The method according to claim 5, characterized in that The another time-triggered spatial haptic effect is associated with another trigger time, such that the another time-triggered spatial haptic effect is rendered after the at least one time-triggered spatial haptic effect.

7. A method for signaling a Haptic JavaScript Object Notation (JSON) Interchange File Format (HJIF) file, characterized in that: The method comprises: Processing the HJIF file into a binary file format for transmission in a bitstream, wherein the HJIF file includes a data hierarchy that specifies (i) a tactile perception at a first level of the data hierarchy, (ii) at least one tactile channel corresponding to the tactile perception at a second level of the data hierarchy, and (iii) at least one tactile frequency band corresponding to the at least one tactile channel at a third level of the data hierarchy, wherein the first level in the data hierarchy is higher than the second level in the data hierarchy, and the second level in the data hierarchy is higher than the third level in the data hierarchy, Wherein, the HJIF file further includes at least two time data packets, wherein each time data packet includes a time parameter and at least one haptic effect, and A renderer is configured to render each haptic effect of the at least one haptic effect according to the time parameter of each time data packet.

8. The method according to claim 7, characterized in that The at least two temporal data packets correspond to the at least one haptic frequency band at a fourth level of the data hierarchy.

9. The method according to claim 8, characterized in that The fourth level of the data hierarchy is lower than the third level of the data hierarchy.

10. The method according to claim 8, characterized in that The fourth level of the data hierarchy is higher than the first level of the data hierarchy.

11. The method according to claim 7, characterized in that The at least two time data packets correspond to a first level of the data hierarchy, wherein each of the at least one tactile effect of each time data packet is associated with a first ID corresponding to the tactile perception, a second ID corresponding to the at least one tactile channel, and a third ID corresponding to the at least one tactile frequency band.

12. The method according to claim 7, characterized in that The time parameter for each data packet specifies a time offset relative to an origin.

13. The method according to claim 7, characterized in that The time parameter of each data packet specifies a duration value, wherein a start time of a first time packet of the at least two data packets corresponds to an end time of a second time packet of the at least two data packets.

14. The method according to claim 7, characterized in that Each time data packet comprises a synchronization parameter indicating whether the corresponding time data packet is the start of a new perception or the continuation of a previous perception.

15. The method according to claim 7, characterized in that The at least two time data packets also include at least one time data packet having no effect of indicating a silent time.

16. A method for decoding a tactile JavaScript object notation JSON interchange file format HJIF file, characterized in that: The method comprises: Receive code stream; Decoding the code stream to extract the HJIF file; and rendering at least two time-triggered spatial haptic effects included in the HJIF file, wherein at least one time-triggered spatial haptic effect of the at least two spatial haptic effects varies along a spatial axis, The at least one time-triggered spatial haptic effect is associated with a first parameter defining a trigger time, and the at least one time-triggered spatial haptic effect is rendered according to the trigger time.

17. The method according to claim 16, characterized in that The at least one temporally triggered spatial effect is associated with a second parameter that indicates a distance from a spatially perceived origin of the at least one temporally triggered spatial haptic effect.

18. The method according to claim 16, characterized in that The trigger time is an offset relative to the playback start time.

19. The method according to claim 16, characterized in that The trigger time is a time interval indicating a duration of the at least one time-triggered spatial haptic effect.

20. The method according to claim 16, characterized in that The at least two temporally-triggered spatial haptic effects include another temporally-triggered spatial haptic effect rendered at the same location as the at least one temporally-triggered spatial haptic effect.

Images