Haptic signal processing method and device for performing same

Abstract

The present disclosure relates to a haptic signal processing method and a device for performing same, wherein even when a host device and a client device are interconnected through a legacy communication protocol having only an audio transmission path, a haptic signal is provided from the host device to the client device through the audio transmission path so that the haptic signal can be outputted from the client device. According to the present disclosure, a haptic signal processing method may be provided, the method comprising the steps of: obtaining haptic information about an average energy for each section of a first haptic signal and a signal magnitude of each of N representative low frequencies; arranging the haptic information in a high frequency band of the first haptic signal; generating a mixed signal by mixing the first haptic signal, in which the haptic information is arranged in the high frequency band, with a first audio signal; and transmitting the mixed signal through the audio transmission path.

Description

Method for processing haptic signals and device for performing the same

[0001] The present disclosure relates to a haptic signal processing method for transmitting, receiving, and outputting a haptic signal and a device for performing the same.

[0002] Haptic technology is being widely applied to various multimedia content, including games, movies, and music. Haptic-enabled systems are already widely used in products such as vibrating earphones and headphones, home theater systems, 4D movie theaters, immersive gaming consoles, smartphones, tablets, and wearable devices. Haptic-enabled products provide users with a more realistic and immersive experience with multimedia content.

[0003] Many portable controllers are being introduced as client devices (or slave devices) that connect wirelessly or wired to a host device (or master device), such as a game console, to provide haptic feedback to the user and receive user commands to be inputted and delivered to the host device. Examples of host devices include game consoles, while examples of client devices or portable controllers include dedicated wired / wireless game controllers or smartphones.

[0004] A host device can transmit a haptic signal and an audio signal to a client device. In this case, if each of the haptic signal and the audio signal is a stereo signal, a two-channel stereo transmission path for the haptic signal and a two-channel stereo transmission path for the audio signal, i.e., a four-channel transmission path, may be required. Alternatively, if each of the haptic signal and the audio signal is a mono signal, a one-channel mono transmission path for the haptic signal and a one-channel mono transmission path for the audio signal, i.e., a two-channel transmission path, may be required.

[0005] However, if the host device and the client device are interconnected via a legacy communication protocol that only has an audio transmission path and no separate haptic signal transmission path, there is a problem in that it is difficult to provide haptic signals from the host device to the client device.

[0006] The present disclosure provides a haptic signal processing method and a device for performing the same, which enable a haptic signal to be provided from a host device to a client device through an audio transmission path so that the haptic signal can be output from the client device even when the host device and the client device are interconnected through a legacy communication protocol having only an audio transmission path.

[0007] In order to achieve the above object, the present disclosure may provide a haptic signal processing method including the steps of obtaining haptic information regarding the average energy per section of a first haptic signal and the signal magnitude of each of N representative low frequencies, the step of arranging the haptic information in a high frequency band of the first haptic signal, the step of mixing the first haptic signal in which the haptic information is arranged in the high frequency band with a first audio signal to generate a mixed signal, and the step of transmitting the mixed signal through an audio transmission path.

[0008] The above high frequency band may be an inaudible band.

[0009] The above haptic signal processing method may include a step of buffering a first haptic signal for a first predetermined time length, a step of window function processing the buffered haptic signal while overlapping the first predetermined time length by a second predetermined time length, and a step of calculating the average energy of the window function-processed haptic signal.

[0010] The above haptic signal processing method further includes a step of converting the window function-processed haptic signal into a frequency domain haptic signal, and the haptic information can be arranged on the high frequency band of the frequency domain haptic signal.

[0011] The above haptic signal processing method further includes a step of converting the frequency domain haptic information including the haptic information into a time domain haptic signal, wherein the time domain haptic signal can be mixed with a first audio signal.

[0012] The above audio transmission path may be a two-channel stereo signal transmission path.

[0013] The above average energy and N representative frequency signals can be placed in the high frequency band by being converted into N+1 high frequency signals of the high frequency band, respectively.

[0014] The above haptic signal processing method may further include a step of receiving the mixed signal, a step of extracting the haptic information from the received mixed signal, a step of synthesizing a second haptic signal using the haptic information, and a step of outputting the second haptic signal.

[0015] A second haptic signal can be synthesized from the haptic information using the wave characteristics of the N representative low-frequency stars.

[0016] The above haptic signal processing method may further include a step of extracting a second audio signal from the received mixed signal by performing low-frequency band-pass filtering on the received mixed signal, and a step of outputting the second audio signal.

[0017] In addition, to achieve the above object, the present disclosure may provide a host device including a communication unit for communicating with a client device, a memory for storing multimedia content, a processor for executing the multimedia content to generate a first haptic signal and a first audio signal, obtaining haptic information about an average energy per section of the first haptic signal and a signal magnitude of each of N representative low frequencies, arranging the haptic information in a high frequency band of the first haptic signal, mixing the first haptic signal in which the haptic information is arranged in the high frequency band with a first audio signal to generate a mixed signal, and controlling the transmission of the mixed signal through an audio transmission path with the client device.

[0018] In addition, to achieve the above object, the present disclosure may provide a client device including a haptic module, a communication unit for communicating with a host device, and a processor for receiving a mixed signal through an audio transmission path with the host device, extracting haptic information about an average energy per section of a first haptic signal and a signal magnitude of each of N representative low frequencies from the received mixed signal, synthesizing a second haptic signal using the haptic information, and controlling the second haptic signal to be output through the haptic module.

[0019] The effects of the haptic signal processing method and the device performing the same according to the present disclosure are described as follows.

[0020] According to at least one aspect of the present disclosure, there is an advantage in that a haptic signal can be provided from a host device to a client device via an audio transmission path, such that the haptic signal can be output from the client device even when the host device and the client device are interconnected via a legacy communication protocol that only has an audio transmission path.

[0021] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0022] FIG. 1 illustrates a system capable of transmitting and receiving haptic signals according to one aspect of the present disclosure.

[0023] Figure 2 is a block diagram of the system of Figure 1.

[0024] Figure 3 illustrates an example of a modification of the system of Figure 1.

[0025] FIGS. 4 and 5 illustrate examples of frequency bands of tactile effects that can be output from a client device according to one aspect of the present disclosure.

[0026] FIG. 6 illustrates transmission of audio signals and haptic signals between a host device and a client device according to related technology.

[0027] FIG. 7 illustrates transmission of audio signals and haptic signals between a host device and a client device according to one aspect of the present disclosure.

[0028] Figure 8 is a flowchart regarding the operation of the host device of Figure 7.

[0029] FIG. 9 is a graph of audio signals and haptic signals processed by the host device of FIG. 7.

[0030] Figure 10 is a flowchart regarding the operation of the client device of Figure 7.

[0031] Figure 11 shows a graph of wave characteristics for each representative frequency.

[0032] FIG. 12 is a graph of audio signals and haptic signals processed by the client device of FIG. 7.

[0033] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0034] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0035] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0036] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0037] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0038] Hereinafter, a system for transmitting and receiving haptic signals will be described with reference to FIGS. 1 to 3. FIGS. 1 and 3 illustrate a system for transmitting and receiving haptic signals. FIG. 2 is a block diagram of the system of FIG. 1. FIG. 3 illustrates an example of a modification of the system of FIG. 1.

[0039] As illustrated in FIG. 1, a system for transmitting and receiving a haptic signal includes a host device (100) that generates a haptic signal, and a client device (200) that is connected to the host device (100) by wire or wirelessly and receives and outputs the generated haptic signal. In addition, the system may further include an external multimedia device (300) that is connected to the host device (100) by wire or wirelessly and receives and outputs at least one of a video signal and an audio signal generated by the host device (100). To this end, the external multimedia device (300) may include at least one of an external display (3151) and an external audio output unit (3152).

[0040] In FIG. 1, the host device (100) is illustrated as a game console, the client device (200) is illustrated as a terminal such as a game wireless controller (200A) or a smartphone (200B), and the external multimedia device (300) is illustrated as a TV, but is not limited thereto.

[0041] Below, we will look at the above system in more detail with reference to Fig. 2.

[0042] First, let's look at the host device (100). The host device (100) may include a first wireless communication unit (1110), a first user input unit (1123), a first interface unit (1160), a first processor (1180), and a first power supply unit (1190). The host device (100) may have more or fewer components than the components listed above.

[0043] Let's look at each of these components.

[0044] The first wireless communication unit (1110) is for the host device (100) to communicate with the client device (200) or the external multimedia device (300), and may include at least one of a wireless Internet module and a short-range communication module.

[0045] The above wireless Internet module refers to a module for wireless Internet access, and can be built into or external to the host device (100). The wireless Internet module is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

[0046] Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G, etc., and the wireless Internet module transmits and receives data according to at least one wireless Internet technology, including Internet technologies not listed above.

[0047] The above short-range communication module is for short-range communication, and can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0048] The short-range communication module can detect (or recognize) the client device (200) capable of communicating with the host device (100) around the host device (100). Furthermore, if the detected client device (200) is a device authenticated to communicate with the host device (100) according to the present disclosure, the first processor (1180) can transmit at least a portion of the data processed in the host device (100) to the client device (200) through the short-range communication module. Accordingly, the client device (200) can use the data processed in the host device (100). In addition, the host device (100) can receive at least a portion of the data processed in the client device (200) through the short-range communication module. Accordingly, the host device (100) can use the data processed in the client device (200).

[0049] In the above, it is described that the host device (100) communicates with the client device (200) or the external multimedia device (300) through the first wireless communication unit (1110), but the host device (100) may also communicate with the client device (200) or the external multimedia device (300) through the first interface unit (1160).

[0050] The first user input unit (1123) is for receiving information from a user. When information or a user command is input through the first user input unit (1123), the first processor (1180) can control the operation of the host device (10) to correspond to the input information or user command. The first user input unit (1123) may include at least one of a mechanical input means and a touch input means.

[0051] The first interface unit (1160) serves as a passage between the host device (100) and an external device (e.g., the client device (200) and / or the external multimedia device (300)). The interface unit (1160) receives data from the external device or allows data within the host device (100) to be transmitted to the external device. For example, a wired / wireless data port, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, a USB (Universal Serial Bus) port, etc. may be included in the first interface unit (1160).

[0052] The first wireless communication unit (1110) and the first interface unit (1160) may be collectively referred to as the first communication unit (1100).

[0053] The first processor (1180) controls the overall operation of the host device (100). The first processor (1180) processes signals, data, information, etc. input or output through the components described above, or runs an application program stored in the memory of the host device (100), thereby providing or processing appropriate information or functions to the user.

[0054] The first processor (1180) may control at least some of the components to run an application program stored in the memory. Furthermore, the first processor (1180) may operate at least two or more of the components included in the host device (100) in combination to run the application program.

[0055] The above memory may be configured to be included within the first processor (1180), or may be configured as a separate component independent from the first processor (1180) within the host device (100).

[0056] When the client device (200) is connected to the host device (100), the client device (200) can be recognized by the host device (100). Then, the first processor (1180) can receive an identifier (ID) of the client device (200) from the client device (200) through the first wireless communication unit (1110). The identifier (ID) of the client device (200) is for identifying the type of the client device (200).

[0057] The first processor (1180) can control the overall operation of the host device (100) so that it operates in accordance with the type of the connected client device (200) based on the identifier (ID) of the client device (200).

[0058] The first power supply unit (1190) receives external power and internal power under the control of the first processor (1180) and supplies power to each component included in the host device (100). The first power supply unit (1190) may include a battery, which may be a built-in battery or a replaceable battery. The first power supply unit (1190) may include a PMIC (Power Management IC) for power management of the mobile device.

[0059] Below, we will look at the client device (200).

[0060] The client device (200) may include a second wireless communication unit (2110), a second user input unit (2123), a sensor (2140), a display (2151), an audio output unit (2152), a haptic module (2153), a second interface unit (2160), a second processor (2180), and a second power supply unit (2190). The client device (200) may have more or fewer components than the components listed above. For example, when the client device (200) is configured as the wireless controller for gaming, it may not include at least one of the display (2151) and the audio output unit (2152). Alternatively, when the client device (200) is configured as the wireless controller for gaming, it may not include either the second wireless communication unit (2110) or the second interface unit (2160).

[0061] The second wireless communication unit (2110) is for the client device (200) to communicate with the host device (100), and may include at least one of a wireless Internet module and a short-range communication module. The wireless Internet module and the short-range communication module are as described above.

[0062] In the above, it is described that the client device (200) communicates with the host device (100) through the second wireless communication unit (2110), but the client device (200) may also communicate with the host device (100) through the second interface unit (2160).

[0063] The second user input unit (2123) is for receiving information from a user. When information or a user command is input through the second user input unit (2123), the second processor (2180) can control the operation of the client device (200) to correspond to the input information or user command. The second user input unit (2123) may include at least one of a mechanical input means and a touch input means.

[0064] The sensor (2140) may include at least one of an acceleration sensor, a magnetic sensor, a G-sensor, a gyroscope sensor, and a motion sensor. The second processor (2180) may track real-time 3-DoF (Degrees of Freedom) or 6-DoF movements of the client device (200) through the sensor (2140).

[0065] The display (2151) may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0066] The above display (2151) may be present in one or more forms depending on the implementation form of the client device (200). When composed of multiple displays, the client device (200) may have multiple displays spaced apart from each other or arranged integrally on one surface, or may be arranged on different surfaces.

[0067] The above display (2151) may include a touch sensor that detects a touch on the display (2151) so that a control command can be input by a touch method. Using this, when a touch is made on the display (2151), the touch sensor detects the touch, and the second processor (2180) may be configured to generate a control command corresponding to the touch based on this. The content input by the touch method may be characters or numbers, or instructions or designable menu items in various modes.

[0068] The touch sensor may be formed integrally with the display. For example, the touch sensor may be disposed on the substrate of the display or provided within the display.

[0069] In this way, the display (2151) can form a touch screen together with the touch sensor, and in this case, the touch screen can function as a second user input unit (2123).

[0070] The above-described audio output unit (2152) can output audio data received from the host device (100) or stored in the memory of the client device (200) through the wireless communication unit (2110) or the interface unit (1160). The second audio output unit (2152) also outputs audio signals related to functions performed in the client device (200) (e.g., sound effects, etc.). The audio output unit (2152) may include a receiver, a speaker, a buzzer, etc.

[0071] The haptic module (2153) generates various tactile effects that can be felt by the user. A representative example of the tactile effect generated by the haptic module (2153) may be vibration. The intensity and pattern of the vibration generated by the haptic module (2153) may be controlled by the user's selection or the settings of the second processor (2180). For example, the haptic module (2153) may synthesize and output different vibrations or output them sequentially. Therefore, the haptic module (2153) may also be referred to as a haptic output unit.

[0072] The above haptic module (2153) can generate various tactile effects, such as, in addition to vibration, effects caused by stimulation such as pin arrays moving vertically with respect to the contact skin surface, air injection or suction force through nozzles or suction ports, brushing against the skin surface, contact with electrodes, electrostatic force, and effects caused by reproduction of hot and cold sensations using elements capable of absorbing or exotherming heat.

[0073] The above haptic module (2153) can not only transmit tactile effects through direct contact, but can also be implemented so that a user can feel the tactile effects through the muscle senses of fingers or arms, etc. The above haptic module (2153) can be equipped with at least one actuator to output various vibrations depending on the configuration of the client device (200).

[0074] The second interface unit (2160) acts as a passage between an external device (e.g., the host device (100) or a wired / wireless headset for audio output (not shown)) connected to the client device (200). The second interface unit (2160) receives data from the external device or allows data within the client device (200) to be transmitted to the external device. For example, a wired / wireless data port, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, a USB (Universal Serial Bus) port, etc. may be included in the first interface unit (1160).

[0075] The second wireless communication unit (2110) and the second interface unit (2160) may be collectively referred to as the second communication unit (2100).

[0076] The second processor (2180) controls the overall operation of the client device (200). The second processor (2180) processes signals, data, information, etc. input or output through the components described above, or runs an application program stored in the memory of the client device (200), thereby providing or processing appropriate information or functions to the user.

[0077] The second processor (2180) may control at least some of the components to run the application program stored in the memory. Furthermore, the second processor (2180) may operate at least two or more of the components included in the client device (200) in combination to run the application program.

[0078] The above memory may be configured to be included within the second processor (2180), or may be configured as a separate component independent from the second processor (2180) within the client device (200).

[0079] The second power supply unit (2190) receives external power and internal power under the control of the second processor (2180) and supplies power to each component included in the client device (200). The second power supply unit (2190) may include a battery, which may be a built-in battery or a replaceable battery. The second power supply unit (2190) may include a PMIC (Power Management IC) for power management of the mobile device.

[0080] In the above, it has been described that the host device (100) is connected to the external multimedia device (300) independently or by wire or wirelessly. However, the present disclosure is not limited thereto. The host device (100) may be configured as a single device including the external multimedia device (300). For example, as illustrated in FIG. 3, the host device (100) may be configured as a wearable device and include a display and an audio output unit therein. Examples of the wearable device (100) may include smart glasses, HMD (Head Mounted Display), etc.

[0081] Hereinafter, with reference to FIGS. 4 and 5, a tactile effect (or vibration) that can be output from a client device (200) according to a haptic signal will be described. FIGS. 4 and 5 illustrate examples of frequency bands of tactile effects that can be output from a client device according to one aspect of the present disclosure.

[0082] The haptic module (2153) of the client device (200) may be equipped with one or more actuators to provide vibration.

[0083] In particular, as illustrated in FIG. 4, the haptic module (2153) may include a plurality of actuators having different characteristics so that the client device (200) can provide the user with tactile effects (or vibrations) of various types, i.e., of various vibration frequencies. Accordingly, the user can feel unique tactile effects suitable for various activities, such as shooting, action, and sports, through the client device (200).

[0084] Accordingly, the haptic module (2153) may be configured to provide vibrations of a relatively wide frequency band having multiple resonant frequencies, rather than providing vibrations of a narrow frequency band having a single resonant frequency, as illustrated in FIG. 5. A haptic signal for generating vibrations of such a wide frequency band may also be referred to as a haptic audio signal because it may overlap with the frequency band of an audible audio signal.

[0085] Hereinafter, with reference to FIG. 6, a transmission path of an audio signal and a haptic signal transmitted from the host device (100) to the client device (200) will be described. FIG. 6 illustrates the transmission of an audio signal and a haptic signal between a host device and a client device according to a related technology. Here, the audio signal may refer to a signal generated or intended to be output through the sound output unit (2152), and the haptic signal (2153) may refer to a signal generated or intended to be output through the haptic module.

[0086] The host device (100) can generate the audio signal and the haptic signal. The audio signal and the haptic signal may be generated or reproduced by the host device (100) executing multimedia content stored in its memory or received from another external device.

[0087] The above audio signal and the above haptic signal can be transmitted to the client device (200) through the first communication unit (1100) of the host device (100) and the corresponding second communication unit (2100) of the client device (200).

[0088] At this time, the audio signal and the haptic signal can be transmitted from the host device (100) to the client device (200) through different individual communication paths.

[0089] That is, if the audio signal is a mono signal, a one-channel mono audio signal transmission path is required, and if the audio signal is a stereo signal, a two-channel stereo audio signal transmission path is required. The audio signal transmission path may be formed through the first wireless communication unit (1110) and the corresponding second wireless communication unit (2110), or may be formed through the first interface unit (1160) and the corresponding second interface unit (2160).

[0090] In addition, separate from the audio signal path, if the haptic signal is a mono signal, a one-channel mono haptic signal transmission path is additionally required, and if the haptic signal is a stereo signal, a two-channel stereo haptic signal transmission path is additionally required. The haptic signal transmission path may be formed through the first wireless communication unit (1110) and the corresponding second wireless communication unit (2110), or may be formed through the first interface unit (1160) and the corresponding second interface unit (2160).

[0091] The audio signal transmitted to the client device (200) can be output through the sound output unit (2152) including a speaker amplifier (2152A) and a speaker (2152B).

[0092] The haptic signal transmitted to the client device (200) can be output through the haptic module (2153) including a haptic amplifier (2153A) and an actuator (2153B).

[0093] However, when the host device (100) and the client device (200) are interconnected through a legacy communication protocol that only has an audio transmission path and no separate haptic signal transmission path, there is a problem in that it is difficult for the haptic signal to be provided from the host device (100) to the client device (200).

[0094] Hereinafter, with reference to FIG. 7, transmission of not only the audio signal but also the haptic signal through an audio signal transmission path between the host device (100) and the client device (200) will be described. FIG. 7 illustrates transmission of an audio signal and a haptic signal between a host device and a client device according to one aspect of the present disclosure.

[0095] The first processor (1180) of the host device (100) can encode the haptic signal [S71].

[0096] The first processor (1180) can mix the encoded haptic signal with the audio signal to generate a mixed signal [S72].

[0097] The first processor (1180) can control the first communication unit (1100) of the host device (100) to transmit the generated mixed signal to the client device (200) along the audio signal transmission path (i.e., through one of the first wireless communication unit (1110) and the first interface unit (1160)) [S73].

[0098] The second processor (2180) of the client device (200) can control the second communication unit (2100) of the client device (200) to receive the generated mixed signal from the host device (100) along the audio signal transmission path (i.e., through one of the second wireless communication unit (2110) and the second interface unit (2160)) [S73].

[0099] The second processor (2180) can separate the mixed signal into an audio signal and a haptic signal [S74].

[0100] The second processor (2180) can control the separated audio signal to be output through the sound output unit (2152) of the client device (200) and the separated haptic signal to be output through the haptic module (2153) of the client device (200).

[0101] Hereinafter, with reference to FIGS. 8 and 9, the operation of the host device (100) of FIG. 7 will be examined in more detail. FIG. 8 is a flowchart regarding the operation of the host device of FIG. 7. FIG. 9 is a graph of audio signals and haptic signals processed by the host device of FIG. 7.

[0102] First, referring to FIG. 8, the first processor (1180) of the host device (100) may store the haptic signal in a buffer every first predetermined time length (e.g., 10 ms). The first processor (1180) may perform window function processing while overlapping the stored signals by a second predetermined time length (e.g., 5 ms) shorter than the first predetermined time length so that frequency leakage does not occur [S81]. Examples of the window function include a Hamming window or a Hanning window. If there is no need to worry about frequency leakage, the signals to be stored may not overlap. The first predetermined time length and the second predetermined time length and the information about the window function are pre-selected according to the characteristics of the haptic signal, and may be pre-shared with the client device (200) at the time of factory shipment, or may be shared with the client device (200) when mutual communication is established between the host device (100) and the client device (200).

[0103] Next, the first processor (1180) can calculate the interval average energy (AvE) of each window function-processed haptic signal [S82]. The interval average energy can be obtained, for example, by a root mean square operation for each window function-processed haptic signal. Then, the first processor (1180) can convert each window function-processed haptic signal into a frequency domain haptic signal [S82]. The conversion into the frequency domain haptic signal can be performed, for example, by a fast Fourier transform (FFT) operation for each window function-processed haptic signal.

[0104] Next, the first processor (1180) can obtain the signal size (e.g., 5 AF1, AF2, AF3, AF4, AF5) for each of N representative low frequencies (e.g., 5 low frequencies F1, F2, F3, F4, F5) that have a great influence on vibration or tactile effect by performing frequency analysis on the frequency domain haptic signal [S83]. The N representative low frequencies (F1, F2, F3, F4, F5) belonging to the low frequency band (e.g., less than 500 Hz) are pre-selected according to the characteristics of the haptic signal and may be pre-shared with the client device (200) as factory default values, or when mutual communication is established between the host device (100) and the client device (200), information on the N representative low frequencies may be transmitted to the client device (200).

[0105] The signal size (AF1, AF2, AF3, AF4, AF5) and average energy (AvE) of each representative low frequency section of the above haptic signal are obtained to reduce the data size of the transmitted signal, and may also be referred to as haptic information of the above haptic signal.

[0106] Next, the first processor (1180) can reallocate the average energy per section and the representative low-frequency signal per section to the inaudible high-frequency band (e.g., 18 kHz or higher) of the haptic signal [S84]. For example, the first processor (1180) can assign the average energy per section (AvE) and the magnitude of each representative frequency signal per section (F1, F2, F3, F4, F5) to the inaudible high-frequency signals. That is, while the average energy per section (AvE) is assigned to one high-frequency signal, the magnitudes of N representative low-frequency signals (F1, F2, F3, F4, F5) can be respectively assigned to N high-frequency signals. That is, the average energy per section and the magnitudes of N representative low-frequency signals can be converted into a total of N+1 high-frequency signals. This is just one example, and the average energy per section and the representative low-frequency signals per section may of course be reallocated to the inaudible high-frequency band in other ways.

[0107] In order to enable the client device (100) receiving the N+1 high frequency signals later to smoothly extract the average energy per interval and the magnitude of each low frequency signal from the N+1 high frequency signals, the first processor (1180) may scale the frequency and the signal magnitude. For example, the first processor (1180) may perform signal frequency scaling so that a second frequency interval between neighboring high frequency signals becomes wider than a first frequency interval between neighboring low frequency signals (F1, F2, F3, F4, F5). In addition, the first processor (1180) may perform signal magnitude scaling so that the average energy per interval (AvE) and the first magnitude of each representative low frequency signal per interval (AF1, AF2, AF3, AF4, AF5) become a second magnitude smaller than the first magnitude when they are relocated to the inaudible high frequency band.

[0108] The scaling information and N+1 high frequencies are also pre-selected according to the characteristics of the haptic signal, and are pre-shared with the client device (200) as factory default values, or when mutual communication is established between the host device (100) and the client device (200), the scaling information and N+1 high frequencies can be transmitted to the client device (200).

[0109] Next, the first processor (1180) can convert the haptic signal, in which the average energy (AvE) per interval and the signal magnitude of each representative low frequency (AF1, AF2, AF3, AF4, AF5) are rearranged in the inaudible high frequency band, into a time domain signal [S85]. This can complete the encoding step (S71) of the haptic signal mentioned in FIG. 7.

[0110] The first processor (1180) can mix the encoded haptic signal with the audio signal to generate a mixed signal and transmit it to the client device (200) along the audio signal transmission path in a streaming manner [S72, S73].

[0111] Hereinafter, with reference to FIG. 9, we will look at the audio signal and haptic signal processed in the host device (100).

[0112] (9-1) of Fig. 9 is a graph in the frequency domain of the audio signal (A1) before the S72 step and the haptic signal (H1) before the S71 step.

[0113] Fig. 9 (9-2) is a graph in the frequency domain of the audio signal (A1) before step S72 and the haptic signal (H2) after step S71. Compared to the graph of Fig. 9 (9-1), it can be seen that the haptic information is encoded in the inaudible high-frequency band (HFR) of the haptic signal (H2) after step S71.

[0114] Figure 9 (9-3) is a graph of the frequency domain of the mixed signal (M). It can be seen that the haptic information exists as is in the inaudible high-frequency band (HFR) of the mixed signal.

[0115] Hereinafter, with reference to FIGS. 10 to 12, the operation of the client device (100) of FIG. 7 will be examined in more detail. FIG. 10 is a flowchart regarding the operation of the client device of FIG. 7. FIG. 11 illustrates a graph regarding wave characteristics for each representative frequency. FIG. 12 is a graph of audio signals and haptic signals processed by the client device of FIG. 7.

[0116] First, referring to FIG. 10, the second processor (2180) of the client device (100) can be controlled to receive the mixed signal from the host device (100) through the audio signal transmission path in a streaming manner [S73].

[0117] The second processor (2180) can analyze the inaudible high frequency band of the mixed signal to extract the haptic information [S101]. For example, the second processor (2180) can obtain the second signal magnitude of each of N+1 high frequency signals and perform inverse scaling on the same to obtain the interval average energy (AvE) and the first signal magnitude (AF1, AF2, AF3, AF4, AF5) of each of N representative low frequencies. This is possible because, as described above, the second processor (2180) knows the N low frequency information, the scaling information, and the N+1 high frequency information.

[0118] Next, the client device (200) may store a predefined wave table in its memory (2170). The wave table may include information about wave characteristics for each representative low frequency (F1, F2, F3, F4, F5) as illustrated in FIG. 11. Accordingly, the second processor (2180) may synthesize a haptic signal based on the wave table, the first signal magnitudes (AF1, AF2, AF3, AF4, AF5) of each of the N representative low frequency signals, and the average energy (AvE) of the interval [S102]. In the above, the N is exemplified as being 5, but may be more or less than this. An inverse window function may be processed when synthesizing the haptic signal.

[0119] The second processor (2180) can control the synthesized haptic signal to be output through the haptic module (2153). The synthesized haptic signal is resynthesized to be close to the original haptic signal (i.e., the original haptic signal generated by the host device (100), and can allow the user to experience substantially the same vibration or tactile effect. Hereinafter, in order to distinguish the original haptic signal and the synthesized haptic signal, they may be referred to as a first haptic signal and a second haptic signal, respectively.

[0120] Meanwhile, the second processor (2180) may perform low-frequency bandpass filtering on the mixed signal to extract an audio signal from the mixed signal [S103].

[0121] In addition, the second processor (2180) can control the low-frequency band-passed signal to be output as an audio signal through the audio output unit (2152). Since the haptic signal is mixed in the mixed signal, the low-frequency band-passed signal among the haptic signals can also be output through the audio output unit (2152) together with the audio signal. The audio signal can include a sound effect corresponding to the low-frequency band-passed haptic signal among the haptic signals, and the low-frequency band-passed signal among the haptic signals can have an effect of making the sound effect sound more realistic.

[0122] Hereinafter, with reference to FIG. 12, we will look at the audio signal and haptic signal output from the client device (200).

[0123] Fig. 12 (12-1) shows an example of a mixed signal on the low-pass time domain output through the sound output unit (2152).

[0124] (12-2) of Fig. 12 illustrates an example of the synthesized haptic signal in the time domain output through the haptic module (2153).

[0125] The present invention described above can be implemented as computer-readable code on a medium having a program recorded thereon. Computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also include media implemented in the form of carrier waves (e.g., transmission via the Internet). Therefore, the above detailed description should not be construed as limiting in all respects, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are intended to be included in the scope of the present invention.

Claims

1. A step of obtaining haptic information about the average energy of each section of the first haptic signal and the signal size of each of N representative low frequencies; A step of placing the above haptic information in a high frequency band of the first haptic signal; A step of generating a mixed signal by mixing a first haptic signal in which the haptic information is arranged in the high frequency band with a first audio signal; and A method for processing a haptic signal, comprising: a step of transmitting the mixed signal through an audio transmission path; 2. In paragraph 1, A haptic signal processing method characterized in that the above high frequency band is an inaudible band.

3. In paragraph 1, A step of buffering a first haptic signal for a first predetermined time length; A step of processing the buffered haptic signal with a window function while overlapping the buffered haptic signal by a second predetermined time length more than a first predetermined time length; and A haptic signal processing method, characterized by comprising a step of calculating the average energy of the haptic signal processed by the window function.

4. In paragraph 3, Further comprising a step of converting the above window function processed haptic signal into a frequency domain haptic signal; A haptic signal processing method characterized in that the haptic information is arranged on the high frequency band of the frequency domain haptic signal.

5. In paragraph 4, Further comprising a step of converting the frequency domain haptic information including the haptic information into a time domain haptic signal; A haptic signal processing method characterized in that the time domain haptic signal is mixed with a first audio signal.

6. In paragraph 1, A haptic signal processing method, characterized in that the above audio transmission path is a two-channel stereo signal transmission path.

7. In paragraph 1, A haptic signal processing method characterized in that the above average energy and N representative frequency signals are respectively converted into N+1 high frequency signals of the high frequency band and thus placed in the high frequency band.

8. In paragraph 5, A step of receiving the above mixed signal; A step of extracting the haptic information from the received mixed signal; A step of synthesizing a second haptic signal using the above haptic information; and A haptic signal processing method further comprising a step of outputting a second haptic signal.

9. In paragraph 8, A haptic signal processing method characterized in that a second haptic signal is synthesized from the haptic information using the wave characteristics of the N representative low-frequency stars.

10. In paragraph 8, A step of extracting a second audio signal from the received mixed signal by performing low-frequency band-pass filtering on the received mixed signal; and A method for processing a haptic signal, comprising: a step of outputting a second audio signal; 11. Communication unit for communicating with client devices; Memory for storing multimedia content; Generating a first haptic signal and a first audio signal by executing the above multimedia content, Obtain haptic information about the average energy of each section of the first haptic signal and the magnitude of each signal of N representative low frequencies, The above haptic information is placed in the high frequency band of the first haptic signal, A first haptic signal in which the haptic information is arranged in the high frequency band is mixed with a first audio signal to generate a mixed signal, A host device comprising a processor for controlling transmission of the mixed signal through an audio transmission path with the client device.

12. In paragraph 11, A host device characterized in that the high frequency band is an inaudible band.

13. In the 11th paragraph, the processor, Buffering the first haptic signal for a first predetermined time length, The above buffered haptic signal is processed by a window function while overlapping the first predetermined time length by a second predetermined time length, A host device characterized in that it controls to calculate the average energy of the haptic signal processed by the window function.

14. In the 13th paragraph, the processor, Control to convert the above window function processed haptic signal into a frequency domain haptic signal, A host device characterized in that the haptic information is arranged on the high frequency band of the frequency domain haptic signal.

15. In the 14th paragraph, the processor Control to convert the frequency domain haptic information including the above haptic information into a time domain haptic signal, A host device characterized in that the time domain haptic signal is mixed with a first audio signal.

16. In paragraph 11, A host device characterized in that the above audio transmission path is a two-channel stereo signal transmission path.

17. In paragraph 11, A host device characterized in that the above average energy and N representative frequency signals are respectively converted into N+1 high frequency signals of the high frequency band and thus placed in the high frequency band.

18. Haptic module; A communication unit for communicating with a host device; and Receiving a mixed signal through an audio transmission path with the above host device, Extracting haptic information about the average energy of each section of the first haptic signal and the signal magnitude of each of N representative low frequencies from the received mixed signal, Synthesize a second haptic signal using the above haptic information, A client device comprising a processor for controlling a second haptic signal to be output through the haptic module.

19. In the 18th paragraph, the processor, A client device characterized in that it controls to synthesize a second haptic signal from the haptic information using the wave characteristics of the N representative low frequencies.

20. In paragraph 18, Including further audio output section, The above processor, Extracting a second audio signal from the received mixed signal by performing low-frequency band-pass filtering on the received mixed signal, A client device that controls the output of a second audio signal through the above audio output unit.

Images