Real-time haptics generation method and related devices

Abstract

The embodiment of the present application provides a kind of real-time touch generation method and related equipment, applied to vibration audio integrated device, this method includes the following steps: the way of frame-by-frame reading into audio is obtained to original audio multiple frame audio signals;Determine the opening and closing state of tactile switch: if tactile switch is closed state, audio signal is directly output to vibration audio integrated device after high-pass filtering, for realizing separate sounding effect;If tactile switch is open state, the vibration signal corresponding to the current frame of audio signal is generated, and the current frame of audio signal and the corresponding vibration signal are fused after high-pass filtering and output to vibration audio integrated device, for realizing the real-time touch effect of sound and vibration combination.Compared with related art, the real-time touch feedback experience effect realized by the real-time touch generation method and related equipment provided by the present application is more optimal.

Description

[Technical Field]

[0001] This invention relates to the field of real-time tactile sensation generation methods, and in particular to a real-time tactile sensation generation method and related equipment applied to integrated vibration and audio devices. [Background Technology]

[0002] With the development of electronic products, people have higher requirements for the effects of sound and vibration feedback. Therefore, more and more electronic terminals, such as mobile phones and tablets, are equipped with speakers for audio output and vibration motors for vibration feedback output. However, as the functions of electronic products improve, the number of components is also increasing, leading to increased costs and smaller space occupied by each component. Vibration and audio combined output devices, with their advantages of low overall power consumption, simple signal transmission path hardware, and low manufacturing cost, have shown great potential in applications such as mobile phones.

[0003] The vibration and audio combined output device of related technologies, also known as a vibration-audio integrated device or a sound-vibration integrated device, includes a vibration actuator and a sound actuator connected in series and a capacitor connected in parallel with the vibration actuator. The input terminals of the vibration actuator and the sound actuator are then connected to signal input lines respectively. It drives the vibration actuator and the sound actuator by inputting separate vibration signals and separate audio signals respectively. However, because the vibration and audio trigger signals are input and executed separately, the vibration feedback is pre-programmed according to the scenario design. The resulting vibration feedback does not have a real-time feedback effect, resulting in a poor real-time tactile experience.

[0004] Therefore, it is necessary to provide a new real-time haptic generation method and related equipment to solve the above-mentioned technical problems. [Summary of the Invention]

[0005] The purpose of this invention is to provide a real-time tactile feedback generation method and related equipment, which can be applied to vibration and audio integrated devices to effectively improve their real-time tactile feedback experience.

[0006] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a real-time tactile sensation generation method, applied to a vibration-audio integrated device. The vibration-audio integrated device includes a vibration actuator for generating vibration and a sound actuator for generating sound, connected in series, and a capacitor connected in parallel with the vibration actuator. The real-time tactile sensation generation method includes the following steps:

[0007] The original audio is read in frame by frame to obtain multiple frames of audio signal;

[0008] Determine the on / off state of the tactile switch; wherein, the tactile switch is used to control the working state of the vibration actuator:

[0009] If the tactile switch is in the off state, the audio signal is directly output to the integrated vibration audio device after high-pass filtering to achieve a separate sound effect; wherein, the frequency of the high-pass filter is matched with the capacitance value;

[0010] If the tactile switch is in the on state, a vibration signal corresponding to the audio signal of the current frame is generated, and the audio signal of the current frame after high-pass filtering is fused with the corresponding vibration signal and output to the vibration audio integrated device to realize a real-time tactile effect combining sound and vibration.

[0011] Preferably, when reading in the original audio frame by frame, each frame is 200ms.

[0012] Preferably, the step of obtaining multiple frames of audio signal by reading the original audio frame by frame further includes: when reading the first frame of each original audio, determining the number of channels of the original audio; if it is dual-channel, then using the average value of the dual-channel data when reading each frame.

[0013] Preferably, in the step of determining the on / off state of the tactile switch, if the tactile switch is in the on state, a vibration signal corresponding to the audio signal of the current frame is generated, specifically including the following sub-steps:

[0014] Read the audio signal of the current frame, obtain its instantaneous power, and obtain its power envelope by tracking through a filter;

[0015] The vibration threshold is adaptively generated based on the power envelope. The portion of the power envelope that is greater than the vibration threshold is set to 1, and the portion that is less than the vibration threshold is set to 0, so as to obtain the vibration gain envelope, and the vibration signal is filled into the vibration gain envelope.

[0016] Preferably, the step of adaptively generating a vibration threshold based on the power envelope, setting the portion of the power envelope greater than the vibration threshold to 1, and the portion less than the vibration threshold to 0, to obtain a vibration gain envelope, and filling the vibration gain envelope with a vibration signal, specifically includes:

[0017] Construct an adaptive vibration threshold T(t): T(t) = f(T(t-1), E(t)), such that when E(t) is greater than T(t-1), the threshold T(t) is adjusted upward, and vice versa, the threshold T(t) is adjusted downward, thus obtaining the vibration gain envelope G; where t is the time t of the current frame, f() is the constructed nonlinear function, and E is the power envelope;

[0018] The vibration gain envelope G is convolved with a signal of length L and value x to obtain an updated vibration gain envelope G', and finally a vibration signal H = G' * S is generated; where S is the vibration base signal, which is a single-frequency sine signal of the same length as the audio signal, composed of the resonant frequency f0 and the rated voltage vrms of the vibration actuator.

[0019] Preferably, in the step of determining the on / off state of the tactile switch, the audio signal of the current frame after high-pass filtering is fused with the corresponding vibration signal and then output to the integrated vibration and audio device, specifically including:

[0020] The vibration signal and the high-pass filtered audio signal are added together and normalized to obtain the final fused signal, which is then input to the integrated vibration and audio device.

[0021] Secondly, the present invention also provides a real-time tactile sensing generation system, applied to an integrated vibration and audio device, comprising:

[0022] An audio reading module is used to acquire multiple frames of audio signals from the original audio by reading the audio frame by frame.

[0023] A high-pass filter module is used to filter the audio signal;

[0024] The judgment execution module is used to determine the on / off state of the tactile switch; wherein the tactile switch is used to control the working state of the vibration actuator.

[0025] If the tactile switch is in the off state, the audio signal is filtered by the high-pass filter module and then directly output to the vibration audio integrated device to achieve a separate sound effect;

[0026] If the tactile switch is in the on state, the vibration signal generation module is triggered to work;

[0027] A vibration signal generation module, configured to generate a corresponding vibration signal for real-time tactile feedback based on the audio signal of the current frame when the judgment execution module determines that the tactile switch is in the on state; and,

[0028] The fusion module is used to fuse the audio signal of the current frame after filtering by the high-pass filter module and the corresponding vibration signal when the haptic switch is on, to generate the fused signal, which is used to achieve a real-time haptic effect combining sound and vibration.

[0029] Preferably, the audio reading module is further configured to: determine the number of channels of the original audio when reading the first frame of each original audio; if it is dual-channel, then select the average value of the dual-channel data when reading each frame.

[0030] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a real-time haptic generation program stored in the memory and executable on the processor, wherein when the real-time haptic generation program is executed by the processor, it implements the steps in the real-time haptic generation method described above provided by the present invention.

[0031] Fourthly, the present invention also provides a computer-readable storage medium storing a real-time haptic generation program, wherein the real-time haptic generation program, when executed by a processor, implements the steps in the above-described real-time haptic generation method provided by the present invention.

[0032] Compared with related technologies, the real-time haptic generation method and related equipment of the present invention first acquire multiple frames of audio signals by reading the original audio frame by frame; then, it determines the on / off state of the haptic switch: if the haptic switch is off, the audio signal is directly output to the integrated vibration audio device after high-pass filtering to achieve a standalone sound effect; if the haptic switch is on, a vibration signal corresponding to the audio signal of the current frame is generated, and the high-pass filtered audio signal of the current frame is fused with the corresponding vibration signal before being output to the integrated vibration audio device to achieve a real-time haptic effect combining sound and vibration. Through the above method and related equipment, real-time matching of the generated vibration signal and audio signal is achieved, resulting in a superior real-time haptic experience from the integrated vibration audio device. [Attached Image Description]

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0034] Figure 1 A flowchart illustrating the real-time haptic generation method provided in an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of the internal structure of the vibration and audio integrated device to which the real-time tactile sensation generation method provided in the embodiments of the present invention is applicable;

[0036] Figure 3 This is a sub-flowchart of step S2 in the real-time haptic generation method provided in the embodiments of the present invention;

[0037] Figure 4 This is a schematic diagram of multi-frame vibration signals in the real-time haptic generation method provided in this embodiment of the invention;

[0038] Figure 5 This is a schematic diagram of the multi-frame fused signal when the tactile switch is turned on in the real-time tactile generation method provided in this embodiment of the invention;

[0039] Figure 6 This is a schematic diagram of the multi-frame fused signal when the tactile switch is turned off in the real-time tactile generation method provided in this embodiment of the invention;

[0040] Figure 7 This is a structural block diagram of the real-time haptic generation system provided in an embodiment of the present invention;

[0041] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of the present invention.

Detailed Implementation Methods

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Example 1

[0044] Please refer to Figure 1 This invention provides a real-time tactile sensation generation method, applied to integrated vibration and audio devices. Please refer to... Figure 2 , Figure 2 This is a schematic diagram of the internal structure of the integrated vibration and audio device applicable to the real-time haptic generation method provided in this embodiment of the invention. The integrated vibration and audio device 200 includes a vibration actuator 201 and a sound actuator 202 connected in series, and a capacitor C connected in parallel with the vibration actuator 201. The vibration actuator 201 is used to generate vibration, and the sound actuator 202 is used to generate sound.

[0045] The real-time haptic generation method includes the following steps:

[0046] Step S1: Obtain multiple frames of audio signal by reading in the original audio frame by frame.

[0047] In this embodiment, when reading the original audio frame by frame, the audio signal y is read in 200ms increments. Of course, it is not limited to the above-mentioned 200ms increments; it can be defined according to actual needs.

[0048] Even better, in this step, when reading the first frame of each original audio file, the number of channels of that original audio file is first determined:

[0049] If the number of channels is single, then continue reading each subsequent frame normally;

[0050] If the number of channels is two, the average value of the two channels is used when reading each frame.

[0051] Step S2: Determine the on / off state of the tactile switch; wherein, the tactile switch is used to control the working state of the vibration actuator, and the tactile switch can be implemented by hardware circuitry or software program, both of which are readily apparent. The specific state determination is as follows:

[0052] If the tactile switch is in the off state, the audio signal is directly output to the vibration audio integrated device 200 after high-pass filtering to achieve a separate sound effect. The frequency of the high-pass filter is matched to the capacitance value of the capacitor C.

[0053] If the tactile switch is in the on state, a vibration signal corresponding to the audio signal of the current frame is generated, and the audio signal of the current frame after high-pass filtering is fused with the corresponding vibration signal and output to the vibration audio integrated device 200 to realize a real-time tactile effect combining sound and vibration.

[0054] In this embodiment, in step S2, which determines the on / off state of the tactile switch, if the tactile switch is on, a vibration signal corresponding to the audio signal of the current frame is generated. This specifically includes the following sub-steps, which are detailed below. Figure 3 As shown:

[0055] Step S21: Read the audio signal y of the current frame and obtain its instantaneous power y. 2 And track and obtain its power envelope E; wherein, the power envelope E is tracked and obtained by means of a filter, or other feasible methods in the prior art.

[0056] Step S22: Adaptively generate a vibration threshold T based on the power envelope E, set the portion of the power envelope E that is greater than the vibration threshold T to 1, and the portion that is less than the threshold T to 0, to obtain the vibration gain envelope G, and fill the vibration gain envelope G with the vibration signal H.

[0057] This embodiment further explains step S22, specifically:

[0058] Construct the adaptive vibration threshold T(t):

[0059] T(t) = f(T(t-1), E(t)),

[0060] When E(t) is greater than T(t-1), the threshold T(t) is adjusted upwards, and vice versa, the threshold T(t) is adjusted downwards, thus obtaining the vibration gain envelope G; where t is the time t of the current frame, f() is the constructed nonlinear function, and E is the power envelope.

[0061] The vibration gain envelope G is convolved with a signal of length L and value x (e.g., x = 1 / L) to obtain an updated vibration gain envelope G'. This allows for corresponding fade-in and fade-out effects in each vibration region, preventing abnormal noises from the sound actuator 202 in the integrated vibration audio device 200. The final generated vibration signal is H = G' * S; where S is the fundamental vibration signal, which is a single-frequency sinusoidal signal of equal length to the audio signal y, composed of the resonant frequency f0 and rated voltage vrms of the vibration actuator 201. Figure 4 The diagram shown is a schematic of multi-frame vibration signals in the real-time haptic generation method provided in an embodiment of the present invention.

[0062] The updated vibration gain G' obtained after convolution will be further referenced to the instantaneous power y of the audio signal y. 2 Adjust the size to better match the intensity fluctuations of the audio signal itself.

[0063] In this embodiment, in step S2, which determines the on / off state of the tactile switch, the audio signal of the current frame after high-pass filtering is fused with the corresponding vibration signal and then output to the vibration-audio integrated device 200. Specifically, this includes:

[0064] If the tactile switch is in the ON state, the vibration signal H and the high-pass filtered audio signal y are added together, and then normalized to obtain the final fused signal y'. Figure 5 The diagram shown is a multi-frame fusion signal diagram when the tactile switch is turned on in the real-time tactile generation method provided in this embodiment of the invention. The final integrated signal y' is input to the vibration and audio integrated device 200 to achieve the corresponding sound and tactile sensation combination effect.

[0065] It should be noted that in step S2, if the tactile switch is in the off state, it is equivalent to the vibration signal H being all 0 values, meaning the final fused signal y' only contains the audio signal after high-pass filtering, such as... Figure 6 The diagram shown is a multi-frame fusion signal of the real-time haptic generation method provided in this embodiment of the invention when the haptic switch is turned off. The final fusion signal y' is output to the vibration and audio integrated device 200 to achieve a separate sound effect.

[0066] Compared with related technologies, the real-time haptic generation method of the present invention first acquires multiple frames of audio signals by reading the original audio frame by frame; then it determines the on / off state of the haptic switch: if the haptic switch is off, the audio signal is directly output to the integrated vibration audio device after high-pass filtering to achieve a standalone sound effect; if the haptic switch is on, a vibration signal corresponding to the audio signal of the current frame is generated, and the high-pass filtered audio signal of the current frame is fused with the corresponding vibration signal before being output to the integrated vibration audio device to achieve a real-time haptic effect combining sound and vibration. This method achieves real-time matching between the generated vibration signal and the audio signal, resulting in a superior real-time haptic experience from the integrated vibration audio device.

[0067] Example 2

[0068] Please combine Figure 2 and 7 As shown in the figure, this embodiment of the invention also provides a real-time haptic generation system 700, which is applied to a vibration-audio integrated device 200. The vibration-audio integrated device 200 includes a vibration actuator 201 and a sound actuator 202 connected in series, and a capacitor C connected in parallel with the vibration actuator 201. The vibration actuator 201 is used to generate vibration, and the sound actuator 202 is used to generate sound.

[0069] The real-time haptic generation system 700 includes:

[0070] The audio reading module 701 is used to obtain multiple frames of audio signals from the original audio by reading the audio frame by frame.

[0071] In this embodiment, when the audio reading module 701 reads the original audio frame by frame, it reads the audio signal y in 200ms increments. Of course, it is not limited to the above-mentioned 200ms increments per frame, and can be defined according to actual needs.

[0072] Even better, in this step, when the audio reading module 701 reads the first frame of each original audio, it first determines the number of channels of the original audio: if the number of channels is single channel, it continues to read each subsequent frame normally; if the number of channels is dual channel, it selects the average value of the dual channel data when reading each frame.

[0073] A high-pass filter module 702 is used to filter the audio signal.

[0074] The judgment execution module 703 is used to determine the on / off state of the tactile switch and generate a corresponding vibration signal based on the state. The tactile switch controls the working state of the vibration actuator. The tactile switch can be implemented as hardware circuitry or software program, both readily apparent methods. The judgment execution module 703 determines the state of the tactile switch as either off or on.

[0075] The vibration signal generation module 704 is used to generate a corresponding vibration signal to realize real-time tactile feedback based on the audio signal of the current frame when the judgment execution module 703 determines that the tactile switch is in the on state.

[0076] If the tactile switch is in the off state, and the generated vibration signal is 0, the audio signal is filtered by the high-pass filter module 702 and directly output to the integrated vibration and audio device 200 to achieve a separate sound effect. The frequency of the high-pass filter is matched to the capacitance value of the capacitor C.

[0077] If the tactile switch is in the on state, a vibration signal corresponding to the audio signal of the current frame is generated by the vibration signal generation module 704.

[0078] Specifically, the audio signal y of the current frame is read, and its instantaneous power y is obtained. 2 The power envelope E is tracked and acquired. The power envelope E is acquired through a filter, but other feasible methods in existing technologies are also acceptable. A vibration threshold T is adaptively generated based on the power envelope E. The portion of the power envelope E greater than the vibration threshold T is set to 1, and the portion less than the threshold T is set to 0, to obtain the vibration gain envelope G. The vibration signal H is then filled into the vibration gain envelope G. Specifically:

[0079] Construct the adaptive vibration threshold T(t):

[0080] T(t) = f(T(t-1), E(t)),

[0081] When E(t) is greater than T(t-1), the threshold T(t) is adjusted upwards, and vice versa, the threshold T(t) is adjusted downwards, thus obtaining the vibration gain envelope G; where t is the time t of the current frame, f() is the constructed nonlinear function, and E is the power envelope.

[0082] The vibration gain envelope G is convolved with a signal of length L and value x (e.g., x = 1 / L) to obtain an updated vibration gain envelope G'. This allows for corresponding fade-in and fade-out effects in each vibration region, preventing abnormal noises from the sound actuator 202 in the integrated vibration audio device 200. The final generated vibration signal is H = G' * S; where S is the fundamental vibration signal, which is a single-frequency sinusoidal signal of equal length to the audio signal y, composed of the resonant frequency f0 and rated voltage vrms of the vibration actuator 201. Figure 4 The diagram shown is a schematic of multi-frame vibration signals in the real-time haptic generation method provided in an embodiment of the present invention.

[0083] The updated vibration gain G' obtained after convolution will be further referenced to the instantaneous power y of the audio signal y. 2 Adjust the size to better match the intensity fluctuations of the audio signal itself.

[0084] The fusion module 705 is used to fuse the audio signal filtered by the high-pass filter module 702 and the vibration signal corresponding to the audio signal generated by the vibration signal generation module 704 when the tactile switch is in the on state, to generate the fused signal.

[0085] Specifically, the vibration signal H and the high-pass filtered audio signal y are added together and normalized to obtain the final fused signal y', as shown below. Figure 5 The diagram shown is a multi-frame fusion signal diagram when the tactile switch is turned on in the real-time tactile generation method provided in this embodiment of the invention. The final integrated signal y' is input to the vibration and audio integrated device 200 to achieve the corresponding sound and tactile sensation combination effect.

[0086] It should be noted that if the tactile switch is in the off state, it is equivalent to the vibration signal H being all 0 values, meaning that the final fused signal y' after fusion by the fusion module 705 only contains the audio signal after high-pass filtering, such as... Figure 6 The diagram shown is a multi-frame fusion signal of the real-time haptic generation method provided in this embodiment of the invention when the haptic switch is turned off. The final fusion signal y' is output to the vibration and audio integrated device 200 to achieve a separate sound effect.

[0087] In this embodiment, the technical effects achieved by the real-time haptic generation system are the same as those achieved by the real-time haptic generation method provided above in this invention, and will not be repeated here.

[0088] Example 3

[0089] Please combine Figure 8As shown, this embodiment of the invention also provides an electronic device 800, including a processor 801, a memory 802, and a real-time haptic generation program stored in the memory 802 and executable on the processor 801. When the real-time haptic generation program is executed by the processor 801, it implements the steps in the real-time haptic generation method provided by this invention as described above:

[0090] Step S1: Obtain multiple frames of audio signal by reading in the original audio frame by frame.

[0091] In this embodiment, when reading the original audio frame by frame, the audio signal y is read in 200ms increments. Of course, it is not limited to the above-mentioned 200ms increments; it can be defined according to actual needs.

[0092] Even better, in this step, when reading the first frame of each original audio file, the number of channels of that original audio file is first determined:

[0093] If the number of channels is single, then continue reading each subsequent frame normally;

[0094] If the number of channels is two, the average value of the two channels is used when reading each frame.

[0095] Step S2: Determine the on / off state of the tactile switch; wherein, the tactile switch is used to control the working state of the vibration actuator, and the tactile switch can be implemented by hardware circuitry or software program, both of which are readily apparent. The specific state determination is as follows:

[0096] If the tactile switch is in the off state, the audio signal is directly output to the vibration audio integrated device 200 after high-pass filtering to achieve a separate sound effect. The frequency of the high-pass filter is matched to the capacitance value of the capacitor C.

[0097] If the tactile switch is in the on state, a vibration signal corresponding to the audio signal of the current frame is generated, and the audio signal of the current frame after high-pass filtering is fused with the corresponding vibration signal and output to the vibration audio integrated device 200 to realize a real-time tactile effect combining sound and vibration.

[0098] In this embodiment, in step S2, which determines the on / off state of the tactile switch, if the tactile switch is on, a vibration signal corresponding to the audio signal of the current frame is generated. This specifically includes the following sub-steps, which are detailed below. Figure 3 As shown:

[0099] Step S21: Read the audio signal y of the current frame and obtain its instantaneous power y. 2And track and obtain its power envelope E; wherein, the power envelope E is tracked and obtained by means of a filter, or other feasible methods in the prior art.

[0100] Step S22: Adaptively generate a vibration threshold T based on the power envelope E, set the portion of the power envelope E that is greater than the vibration threshold T to 1, and the portion that is less than the threshold T to 0, to obtain the vibration gain envelope G, and fill the vibration gain envelope G with the vibration signal H.

[0101] This embodiment further explains step S22, specifically:

[0102] Construct the adaptive vibration threshold T(t):

[0103] T(t) = f(T(t-1), E(t)),

[0104] When E(t) is greater than T(t-1), the threshold T(t) is adjusted upwards, and vice versa, the threshold T(t) is adjusted downwards, thus obtaining the vibration gain envelope G; where t is the time t of the current frame, f() is the constructed nonlinear function, and E is the power envelope.

[0105] The vibration gain envelope G is convolved with a signal of length L and value x (e.g., x = 1 / L) to obtain an updated vibration gain envelope G'. This allows for corresponding fade-in and fade-out effects in each vibration region, preventing abnormal noises from the sound actuator 202 in the integrated vibration audio device 200. The final generated vibration signal is H = G' * S; where S is the fundamental vibration signal, which is a single-frequency sinusoidal signal of equal length to the audio signal y, composed of the resonant frequency f0 and rated voltage vrms of the vibration actuator 201. Figure 4 The diagram shown is a schematic of multi-frame vibration signals in the real-time haptic generation method provided in an embodiment of the present invention.

[0106] The updated vibration gain G' obtained after convolution will be further referenced to the instantaneous power y of the audio signal y. 2 Adjust the size to better match the intensity fluctuations of the audio signal itself.

[0107] In this embodiment, in step S2, which determines the on / off state of the tactile switch, the audio signal of the current frame after high-pass filtering is fused with the corresponding vibration signal and then output to the vibration-audio integrated device 200. Specifically, this includes:

[0108] If the tactile switch is in the ON state, the vibration signal H and the high-pass filtered audio signal y are added together, and then normalized to obtain the final fused signal y'. Figure 5The diagram shown is a multi-frame fusion signal diagram when the tactile switch is turned on in the real-time tactile generation method provided in this embodiment of the invention. The final integrated signal y' is input to the vibration and audio integrated device 200 to achieve the corresponding sound and tactile sensation combination effect.

[0109] It should be noted that in step S2, if the tactile switch is in the off state, it is equivalent to the vibration signal H being all 0 values, meaning the final fused signal y' only contains the audio signal after high-pass filtering, such as... Figure 6 The diagram shown is a multi-frame fusion signal of the real-time haptic generation method provided in this embodiment of the invention when the haptic switch is turned off. The final fusion signal y' is output to the vibration and audio integrated device 200 to achieve a separate sound effect.

[0110] The memory 802 can be used to store software programs and various data. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0111] The processor 801 is the control center of the terminal, connecting various parts of the terminal through various interfaces and lines. It executes software programs and / or modules stored in the memory 802, and calls data stored in the memory 802 to perform various functions and process data, thereby providing overall monitoring of the terminal. The processor 801 may include one or more processing units; preferably, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 801.

[0112] It should be noted that when the electronic device 800 is in use, it can achieve the technical effects of the real-time touch generation method described above. For details, please refer to the description of the real-time touch generation method above, which will not be repeated here.

[0113] Example 4

[0114] This invention also provides a computer-readable storage medium storing a real-time haptic generation program. When the real-time haptic generation program is executed by a processor, it implements the steps of the real-time haptic generation method described above, thereby achieving the same technical effect as the real-time haptic generation method described above, which will not be repeated here.

[0115] The computer-readable storage medium mentioned above includes, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0116] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0118] The above description is merely an embodiment of the present invention. It should be noted that those skilled in the art can make improvements without departing from the inventive concept of the present invention, but these improvements all fall within the protection scope of the present invention.

Claims

1. A real-time haptic feedback generation method, applied to a vibration-audio integrated device, the vibration-audio integrated device comprising a vibration actuator for generating vibration and a sound actuator for generating sound, connected in series, and a capacitor connected in parallel with the vibration actuator, characterized in that, The real-time haptic generation method includes the following steps: The original audio is read in frame by frame to obtain multiple frames of audio signal; Determine the on / off state of the tactile switch; wherein, the tactile switch is used to control the working state of the vibration actuator: If the tactile switch is in the off state, the audio signal is directly output to the vibration audio integrated device after high-pass filtering to achieve a separate sound effect; wherein, the frequency of the high-pass filter is matched with the capacitance value of the capacitor; If the tactile switch is in the on state, a vibration signal corresponding to the audio signal of the current frame is generated, and the audio signal of the current frame after high-pass filtering is fused with the corresponding vibration signal and output to the integrated vibration and audio device to realize a real-time tactile effect combining sound and vibration. In the step of determining the on / off state of the tactile switch, if the tactile switch is on, a vibration signal corresponding to the audio signal of the current frame is generated, specifically including the following sub-steps: Read the audio signal of the current frame, obtain its instantaneous power, and obtain its power envelope by tracking through a filter; The vibration threshold is adaptively generated based on the power envelope. The portion of the power envelope that is greater than the vibration threshold is set to 1, and the portion that is less than the vibration threshold is set to 0, so as to obtain the vibration gain envelope, and the vibration signal is filled into the vibration gain envelope. The steps of adaptively generating a vibration threshold based on the power envelope, setting the portion of the power envelope greater than the vibration threshold to 1, and the portion less than the vibration threshold to 0, to obtain a vibration gain envelope, and filling the vibration gain envelope with the vibration signal, specifically include: Construct an adaptive vibration threshold T(t): T(t) = f(T(t-1), E(t)), such that when E(t) is greater than T(t-1), the threshold T(t) is adjusted upward, and vice versa, the threshold T(t) is adjusted downward, thus obtaining the vibration gain envelope G; where t is the time t of the current frame, f() is the constructed nonlinear function, and E is the power envelope; The vibration gain envelope G is convolved with a signal of length L and value x to obtain an updated vibration gain envelope G', ultimately generating the vibration signal H = G'. S; where S is the vibration base signal, which is a single-frequency sine wave signal of the same length as the audio signal, composed of the resonant frequency f0 and the rated voltage vrms of the vibration actuator.

2. The real-time haptic generation method according to claim 1, characterized in that, When reading in the original audio frame by frame, each frame is 200ms.

3. The real-time haptic generation method according to claim 1, characterized in that, The step of obtaining multiple frames of audio signals by reading the original audio frame by frame further includes: when reading the first frame of each original audio, determining the number of channels of the original audio; if it is dual-channel, then using the average value of the dual-channel data when reading each frame.

4. The real-time haptic generation method according to claim 1, characterized in that, In the step of determining the on / off state of the tactile switch, the audio signal of the current frame after high-pass filtering is fused with the corresponding vibration signal and then output to the integrated vibration and audio device, specifically including: The vibration signal and the high-pass filtered audio signal are added together and normalized to obtain the final fused signal, which is then input to the integrated vibration and audio device.

5. A real-time tactile sensing generation system, applied to an integrated vibration and audio device, characterized in that, include: An audio reading module is used to acquire multiple frames of audio signals from the original audio by reading the audio frame by frame. A high-pass filter module is used to filter the audio signal; The judgment execution module is used to determine the on / off state of the tactile switch; wherein the tactile switch is used to control the working state of the vibration actuator. If the tactile switch is in the off state, the audio signal is filtered by the high-pass filter module and then directly output to the vibration audio integrated device to achieve a separate sound effect; If the tactile switch is in the on state, the vibration signal generation module is triggered to work; A vibration signal generation module, configured to generate a corresponding vibration signal for real-time tactile feedback based on the audio signal of the current frame when the judgment execution module determines that the tactile switch is in the on state; and, The fusion module is used to fuse the audio signal of the current frame after it has been filtered by the high-pass filter module and the corresponding vibration signal when the haptic switch is in the on state, to generate a fused signal, which is used to realize a real-time haptic effect combining sound and vibration. The judgment and execution module is also used to read the audio signal of the current frame, obtain its instantaneous power, and obtain its power envelope by tracking through a filter; The vibration threshold is adaptively generated based on the power envelope. The portion of the power envelope that is greater than the vibration threshold is set to 1, and the portion that is less than the vibration threshold is set to 0, so as to obtain the vibration gain envelope, and the vibration signal is filled into the vibration gain envelope. The judgment execution module is further configured to construct an adaptive vibration threshold T(t): T(t) = f(T(t-1), E(t)), such that when E(t) is greater than T(t-1), the threshold T(t) is adjusted upward, and vice versa, the threshold T(t) is adjusted downward, to obtain the vibration gain envelope G; where t is the time t of the current frame, f() is the constructed nonlinear function, and E is the power envelope; The vibration gain envelope G is convolved with a signal of length L and value x to obtain an updated vibration gain envelope G', ultimately generating the vibration signal H = G'. S; where S is the vibration base signal, which is a single-frequency sine wave signal of the same length as the audio signal, composed of the resonant frequency f0 and the rated voltage vrms of the vibration actuator.

6. The real-time haptic generation system according to claim 5, characterized in that, The audio reading module is also used to: determine the number of channels of the original audio when reading the first frame of each original audio; if it is dual-channel, then select the average value of the dual-channel data when reading each frame.

7. An electronic device, characterized in that, The method includes a processor, a memory, and a real-time haptic generation program stored in the memory and executable on the processor, wherein the real-time haptic generation program, when executed by the processor, implements the steps of the real-time haptic generation method as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a real-time haptic generation program, which, when executed by a processor, implements the steps of the real-time haptic generation method as described in any one of claims 1-4.

Images