Method, system and related device for music track matching vibration haptic feedback
Patent Information
- Application Number
- JP2023532299
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-10-20
- Filing Date
- 2022-12-02
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-12-02
AI Technical Summary
【0015】 従来技術と比べて、本発明の触覚フィードバック方法では、予め設定されたディープラーニングモデルによって音楽をトラック分けし、特徴が大きく異なる各トラックを区分し、そして、各トラックのエネルギー占有率によってオリジナルオーディオにおける重要度を決定することで、大きさの異なる重みを設定し、異なるトラックを柔軟に重み付けして組み合わせ、オーディオデータを振動とマッチングし、最後に、オーディオデータのテンポやリズムなどにより精確にマッチングする振動出力を出力して、ユーザーにより良好な触覚フィードバック体験を提供することができる。
Smart Images

Figure 0007590572000001 
Figure 0007590572000002 
Figure 0007590572000003
Abstract
Description
[Technical field]
[0001] The present application relates to the application field of deep learning technology, and in particular to a method, system and related device for music track matching vibration haptic feedback. [Background technology]
[0002] Music can express different emotions such as joy, sadness, anger, and strength by changing the tempo, rhythm, and speed of the music, and vibration haptic feedback technology that matches the tempo and accent of the music can give the audience a more realistic and immersive sensory experience. Since the musical instrument composition included in music varies depending on the genre, different instrument compositions have different roles in analyzing the tempo and rhythm of a piece of music. For example, by hitting percussion regularly, it becomes easier to grasp the tempo and rhythm of the music, and it can be matched with more accurate vibration feedback.
[0003] In the prior art, in methods of generating vibrations using the characteristics of the music itself, corresponding vibrations are often generated based on rhythmic instruments such as drum beats. However, this method is not applicable to music with a slow tempo. At the same time, the prior art is unable to analyze the dynamics of different tempos in the music to generate vibrations that correspond to the vibration levels, which limits the vibration feedback experience provided to the user.
[0004] Therefore, there is a need to provide new haptic feedback methods to obtain vibration outputs that more accurately match the tempo, rhythm, etc. of the music. Summary of the Invention [Problem to be solved by the invention]
[0005] The technical problem that the present invention seeks to solve is to provide a method that can generate a vibration output that more accurately matches the tempo, rhythm, etc. of the music. [Means for solving the problem]
[0006] In order to achieve the above object, in a first aspect, the present invention provides a haptic feedback method for music track-divided matching vibration, the haptic feedback method being based on a deep learning model, the haptic feedback method including: obtaining original audio data; dividing the original audio data into tracks using a preset deep learning model to obtain a plurality of track-divided audio data; calculating a corresponding energy occupancy rate of each of the track-divided audio data in the original audio data; determining a weight of each of the corresponding track-divided audio data according to the energy occupancy rate; performing a weighting calculation for all the track-divided audio data according to a preset weighting rule, obtaining and outputting a time-frequency spectrum; generating a matching vibration signal corresponding to the original audio data according to the time-frequency spectrum; and outputting the matching vibration signal as a driving signal for a driver to realize a haptic feedback effect.
[0007] Preferably, the step of calculating the corresponding energy occupancy rate of each of the track-divided audio data in the original audio data specifically includes: performing a short-time Fourier transform process on each of the track-divided audio data to obtain corresponding transformed track-divided audio data; and calculating the energy occupancy rate of the transformed track-divided audio data in the original audio data.
[0008] Preferably, the step of generating a matching vibration signal corresponding to the original audio data by the time-frequency spectrum specifically includes: normalizing the time-frequency spectrum to obtain a time-frequency curve; setting vibration information corresponding to a part of the time-frequency curve that is greater than a predetermined frequency threshold; and outputting the time-frequency curve including the vibration information as the matching vibration signal.
[0009] Preferably, the track-divided audio data includes at least a first track, a second track, a third track and a fourth track having different track characteristics.
[0010] Preferably, the preset weighting rule specifically determines whether the energy occupancy of the first track is the maximum, and if it is the maximum, determines whether the energy occupancy of the second track is the second, and if the energy occupancy of the second track is the second, weights the time-frequency spectrum of the first track and the time-frequency spectrum of the second track to output the weighted result, and if the energy occupancy of the second track is not the second, outputs only the time-frequency spectrum of the first track, and if it is not the maximum, determines whether the energy occupancy of the second track is the maximum, and if the energy occupancy of the second track is the maximum, outputs the energy occupancy of the first track. if the energy occupancy of the first track is second, weighting the time-frequency spectrum of the first track and the second track to output the weighted spectrum; if the energy occupancy of the first track is not second, outputting only the time-frequency spectrum of the second track; if the energy occupancy of the second track is not maximum, determining whether the energy occupancy of the third track is maximum, and if the energy occupancy of the third track is not maximum, outputting the time-frequency spectrum of the fourth track; if the energy occupancy of the third track is maximum, outputting the time-frequency spectrum of the third track.
[0011] Preferably, the first track is a percussion track, the second track is another instrument track, the third track is a vocal track, and the fourth track is a bass track.
[0012] In a second aspect, the present invention further provides a music track-divided matching vibration haptic feedback system, comprising: an original audio acquisition module for acquiring original audio data; a track division module for dividing the original audio data into tracks using a preset deep learning model to obtain a plurality of track-divided audio data; an occupancy calculation module for calculating corresponding energy occupancy of each of the track-divided audio data in the original audio data; a weight calculation module for determining weights of each of the corresponding track-divided audio data according to the energy occupancy; a weight calculation module for performing weighting calculation on all the track-divided audio data according to a preset weighting rule, and obtaining and outputting a time-frequency spectrum; a matching vibration module for generating a matching vibration signal corresponding to the original audio data according to the time-frequency spectrum; and a haptic feedback module for outputting the matching vibration signal as a driving signal for a driver to realize a haptic feedback effect.
[0013] In a third aspect, the present invention further provides a computer device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, the processor, when executing the computer program, realizing steps in the music track matching vibration haptic feedback method described in any one of the above.
[0014] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored therein, the computer program realizing, when executed by the processor, steps in the music track matching vibration haptic feedback method described in any one of the above. Effect of the Invention
[0015] Compared with the prior art, the haptic feedback method of the present invention divides music into tracks according to a preset deep learning model, distinguishes between tracks with significantly different characteristics, and determines the importance of each track in the original audio according to its energy occupancy rate, sets weights of different magnitudes, flexibly weights and combines different tracks, matches audio data with vibration, and finally outputs a vibration output that more accurately matches the tempo, rhythm, etc. of the audio data, thereby providing the user with a better haptic feedback experience. [Brief description of the drawings]
[0016] In order to more clearly describe the technical solutions in the embodiments of the present invention, the drawings necessary for describing the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can obtain other drawings based on these drawings without any inventive work, among which: [Figure 1] FIG. 1 is a flowchart showing steps of a music track matching vibration haptic feedback method according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating a configuration of a deep learning model according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a schematic diagram of a preset weighting rule according to an embodiment of the present invention. [Figure 4] FIG. 4 is a schematic diagram of tracks after track classification by a deep learning model according to an embodiment of the present invention. [Diagram 5] FIG. 5 is a graph comparing the time-frequency spectrum of each track according to an embodiment of the present invention. [Figure 6] FIG. 6 is a diagram showing a matching vibration signal according to an embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing the configuration of a music track matching vibration haptic feedback system according to an embodiment of the present invention. [Figure 8] FIG. 8 is a diagram showing the configuration of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] Hereinafter, the technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by those skilled in the art without making inventive efforts based on the embodiments of the present invention shall be included in the protection scope of the present invention.
[0018] FIG. 1 is a step flow chart of a music track matching vibration haptic feedback method according to an embodiment of the present invention. As shown in FIG. 1, the haptic feedback method includes:
[0019] S1: Obtain the original audio data.
[0020] Specifically, the original audio data obtained by the embodiments of the present invention is not particularly limited to the musical format expressed, for example, pop, rock, symphony, etc., and the method of obtaining the original audio data includes, but is not limited to, obtaining it from existing audio data, extracting it in real time using a recorder or videographer, and converting it into an independent audio data file.
[0021] S2: Using a pre-configured deep learning model, the original audio data is divided into tracks to obtain a plurality of track-divided audio data.
[0022] Specifically, the deep learning model is a neural network model for separating various audio with different characteristics in the audio data. In an embodiment of the present invention, the configuration of the deep learning model for dividing the original audio data into tracks is shown in FIG. 2, where the deep learning model includes an encoding layer consisting of multiple encoders, a neural network recurrent layer with a LSTM (Long-short-term memory) configuration, and a decoding layer consisting of multiple decoders. In the neural network recurrent layer, different LSTM modules can be provided as necessary to extract audio tracks with different characteristics.
[0023] Preferably, the track-divided audio data includes at least a first track, a second track, a third track and a fourth track having different track characteristics.
[0024] S3: Calculate the corresponding energy occupancy of each of the track-divided audio data in the original audio data.
[0025] Preferably, the step of calculating the corresponding energy occupancy rate of each of the track-divided audio data in the original audio data specifically includes: performing a short-time Fourier transform process on each of the track-divided audio data to obtain corresponding transformed track-divided audio data; and calculating the energy occupancy of the transformed track-divided audio data in the original audio data.
[0026] S4: Determine a weight for each of the corresponding track-divided audio data according to the energy occupancy.
[0027] S5: According to a preset weighting rule, perform weighting calculation for all the track-divided audio data, and obtain and output a time-frequency spectrum.
[0028] Preferably, the pre-set weighting rule specifically determines one track of all the track-divided audio data as the track used for generating the time-frequency spectrum.
[0029] Specifically, in one optional embodiment, the preset weighting rules of the four track divided audio data are as follows:
[0030] It is determined whether the energy occupancy of the first track is maximum.
[0031] If it is maximum, It is determined whether the energy occupancy of the second track is second. If the energy occupancy of the second track is second, the time-frequency spectrum of the first track and the time-frequency spectrum of the second track are weighted and output. If the energy occupancy of the second track is not second, only the time-frequency spectrum of the first track is output.
[0032] If it is not the maximum, It is determined whether the energy occupancy of the second track is maximum.
[0033] If the energy occupancy of the second track is the largest, it is determined whether the energy occupancy of the first track is second. If the energy occupancy of the first track is second, the time-frequency spectrum of the first track and the second track is weighted and output. If the energy occupancy of the first track is not second, only the time-frequency spectrum of the second track is output.
[0034] If the energy occupancy of the second track is not the maximum, determine whether the energy occupancy of the third track is the maximum. If the energy occupancy of the third track is not the maximum, output the time-frequency spectrum of the fourth track. If the energy occupancy of the third track is the maximum, output the time-frequency spectrum of the third track.
[0035] Preferably, the first track is a percussion track, the second track is another instrument track, the third track is a voice track, and the fourth track is a bass track. Figure 3 is a schematic diagram of a preset weighting rule according to an embodiment of the present invention, where the bass track is a low-frequency part of the audio, and the audio also includes mid-range and treble, and the user feels stronger hearing due to the change in bass than the mid-range and treble parts, and the percussion instrument is a part of the audio that mainly expresses the tempo speed, among which percussion expresses regular frequency fluctuations, and instruments other than percussion are often combined with percussion to express musical genres, and the vocal track is special in the audio, and although the voice has no regularity, if the feedback of the expression of the voice in music is vibration, it will have a great impact on the user experience. Note that, in the embodiment of the present invention, the number of tracks to be split can be flexibly changed.
[0036] According to the above-mentioned preset weighting rule, in an embodiment of the present invention, at least one of the track-divided audio data having the largest energy occupancy rate in the audio data can be used as the basic data of the time-frequency spectrum, and the time-frequency spectrum places more emphasis on expressing the characteristics that need to be generated by correspondingly matching the vibration feedback in the audio data.
[0037] S6: Generate a matching vibration signal corresponding to the original audio data according to the time-frequency spectrum.
[0038] Preferably, the step of generating a matching vibration signal corresponding to the original audio data according to the time-frequency spectrum specifically includes: normalizing the time-frequency spectrum to obtain a time-frequency curve; Setting vibration information corresponding to a portion of the time-frequency curve that is greater than a preset frequency threshold; and determining the time-frequency curve containing the vibration information as the matching vibration signal.
[0039] S7: Output the matching vibration signal as a driving signal for a driver to realize a haptic feedback effect.
[0040] In an embodiment of the present invention, the haptic feedback effect must be achieved by a vibration feedback system with a motor-based driver.
[0041] Illustratively, FIG. 4 is a schematic diagram of tracks after track division by a deep learning model according to an embodiment of the present invention, and the tracks in FIG. 4 are, from top to bottom, original audio data, bass track, percussion track, other instrument track, and human voice track. For comparison, referring to the time-frequency spectrum comparison graph of each track shown in FIG. 5, it can be seen that the multiple track-divided audio data obtained by dividing the original audio data into tracks have different basic track characteristics, and therefore the corresponding energy occupancy rates are significantly different. FIG. 6 shows the matching vibration signal generated by weighting according to the preset weighting rule in an embodiment of the present invention according to different energy occupancies, where the first row is a general vibration signal that has not been processed, and the third row is the matching vibration signal generated by weighting in an embodiment of the present invention.
[0042] Compared with the prior art, the haptic feedback method of the present invention divides music into tracks according to a preset deep learning model, distinguishes between tracks with significantly different characteristics, and determines the importance of each track in the original audio according to its energy occupancy rate, sets weights of different magnitudes, flexibly weights and combines different tracks, matches audio data with vibration, and finally outputs a vibration output that more accurately matches the tempo, rhythm, etc. of the audio data, thereby providing the user with a better haptic feedback experience.
[0043] In an embodiment of the present invention, a music track matching vibration haptic feedback system is further provided. As shown in FIG. 7, FIG. 7 is a diagram showing the configuration of a music track matching vibration haptic feedback system 200 according to an embodiment of the present invention. The haptic feedback system 200 includes: an original audio acquisition module 201 for acquiring original audio data; a track dividing module 202 for dividing the original audio data into tracks by using a pre-defined deep learning model to obtain a plurality of divided track audio data; an occupancy calculation module 203 for calculating a corresponding energy occupancy of each of the track-divided audio data in the original audio data; a weight calculation module 204 for determining a weight of each of the corresponding track-divided audio data according to the energy occupancy; a weighting calculation module 205 for performing weighting calculation on all the track-divided audio data according to a preset weighting rule, and obtaining and outputting a time-frequency spectrum; a matching vibration module 206 for generating a matching vibration signal corresponding to the original audio data according to the time-frequency spectrum; and a haptic feedback module 207 for outputting the matching vibration signal as a driving signal for a driver to realize a haptic feedback effect.
[0044] The music track matching vibration haptic feedback system 200 according to the embodiment of the present invention can realize each step of the music track matching vibration haptic feedback method described in the above embodiment, and can achieve the same technical effects. In particular, please refer to the description of the above embodiment, and there is no need to repeat the description here.
[0045] An embodiment of the present invention further provides a computer device. As shown in Fig. 8, Fig. 8 is a diagram showing the configuration of a computer device according to an embodiment of the present invention. The computer device 300 includes a processor 301, a memory 302, and a computer program stored in the memory 302 and executable by the processor 301.
[0046] As shown in FIG. 1, the processor 301 calls a computer program stored in the memory 302. When the processor 301 executes the computer program, the steps of the music track matching vibration haptic feedback method in the above embodiment are realized, specifically: Obtaining original audio data; Dividing the original audio data into tracks using a pre-defined deep learning model to obtain a plurality of track-divided audio data; calculating a corresponding energy occupancy rate of each of the track-divided audio data in the original audio data; determining a weight for each of the corresponding track-divided audio data according to the energy occupancy ratio; Performing weighting calculations on all the track-divided audio data according to a preset weighting rule, and obtaining and outputting a time-frequency spectrum; generating a matching vibration signal corresponding to the original audio data according to the time-frequency spectrum; and outputting the matching vibration signal as a driving signal for a driver to realize a haptic feedback effect.
[0047] Preferably, the step of calculating the corresponding energy occupancy rate of each of the track-divided audio data in the original audio data specifically includes: performing a short-time Fourier transform process on each of the track-divided audio data to obtain corresponding transformed track-divided audio data; and calculating the energy occupancy of the transformed track-divided audio data in the original audio data.
[0048] Preferably, the step of generating a matching vibration signal corresponding to the original audio data according to the time-frequency spectrum specifically includes: normalizing the time-frequency spectrum to obtain a time-frequency curve; Setting vibration information corresponding to a portion of the time-frequency curve that is greater than a preset frequency threshold; and outputting the time-frequency curve containing the vibration information as the matching vibration signal.
[0049] Preferably, the track-divided audio data includes at least a first track, a second track, a third track and a fourth track having different track characteristics.
[0050] Preferably, the preset weighting rule is specifically as follows:
[0051] It is determined whether the energy occupancy of the first track is maximum.
[0052] If it is maximum, It is determined whether the energy occupancy of the second track is second, and if the energy occupancy of the second track is second, the time-frequency spectrum of the first track and the time-frequency spectrum of the second track are weighted and output, and if the energy occupancy of the second track is not second, only the time-frequency spectrum of the first track is output.
[0053] If it is not the maximum, It is determined whether the energy occupancy of the second track is maximum.
[0054] If the energy occupancy of the second track is the largest, it is determined whether the energy occupancy of the first track is second. If the energy occupancy of the first track is second, the time-frequency spectrum of the first track and the second track is weighted and output, and if the energy occupancy of the first track is not second, only the time-frequency spectrum of the second track is output.
[0055] If the energy occupancy of the second track is not maximum, determine whether the energy occupancy of the third track is maximum. If the energy occupancy of the third track is not maximum, output the time-frequency spectrum of the fourth track. If the energy occupancy of the third track is maximum, output the time-frequency spectrum of the third track.
[0056] Preferably, the first track is a percussion track, the second track is another instrument track, the third track is a vocal track, and the fourth track is a bass track.
[0057] The computer device 300 according to the embodiment of the present invention can implement the steps of the music track matching vibration haptic feedback method in the above embodiment, and achieve the same technical effects. In particular, please refer to the description of the above embodiment, and there is no need to repeat the description here.
[0058] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it can realize each process and step of the music track matching vibration haptic feedback method according to the embodiment of the present invention, and achieve the same technical effects, which need not be described again here to avoid duplication.
[0059] It should be noted here that the above described embodiment is merely one embodiment of the present invention, and that those skilled in the art may make improvements without departing from the creative idea of the present invention, all of which are within the scope of protection of the present invention.
Claims
1. A music track matching vibration haptic feedback method, comprising: The haptic feedback method is based on a deep learning model; The haptic feedback method includes: Obtaining original audio data; Dividing the original audio data into tracks using a pre-defined deep learning model to obtain a plurality of track-divided audio data; calculating a corresponding energy occupancy rate of each of the track-divided audio data in the original audio data; determining a weight for each of the corresponding track-divided audio data according to the energy occupancy ratio; Performing weighting calculation on all the track-divided audio data according to a preset weighting rule, and obtaining and outputting a time-frequency spectrum; generating a matching vibration signal corresponding to the original audio data according to the time-frequency spectrum; Outputting the matching vibration signal as a driving signal for a driver to realize a haptic feedback effect. A method for providing haptic feedback of vibration by matching music tracks.
2. The step of calculating the corresponding energy occupancy rate of each of the track-divided audio data in the original audio data is specifically: performing a short-time Fourier transform process on each of the track-divided audio data to obtain corresponding transformed track-divided audio data; calculating the energy occupancy of the converted track-divided audio data in the original audio data; The music track matching vibration tactile feedback method according to claim 1 .
3. The step of generating a matching vibration signal corresponding to the original audio data according to the time-frequency spectrum specifically includes: normalizing the time-frequency spectrum to obtain a time-frequency curve; Setting vibration information corresponding to a portion of the time-frequency curve that is greater than a preset frequency threshold; and outputting the time-frequency curve including the vibration information as the matching vibration signal. The music track matching vibration tactile feedback method according to claim 1 .
4. The track-divided audio data includes at least a first track, a second track, a third track, and a fourth track having different track characteristics. The music track matching vibration tactile feedback method according to claim 1 .
5. The preset weighting rule is specifically as follows: determining whether the energy occupancy of the first track is maximum; If it is maximum, determining whether the energy occupancy of the second track is second, and if the energy occupancy of the second track is second, weighting the time-frequency spectrum of the first track and the time-frequency spectrum of the second track and outputting the weighted spectrum; if the energy occupancy of the second track is not second, outputting only the time-frequency spectrum of the first track; If it is not the maximum, determining whether the energy occupancy of the second track is maximum; If the energy occupancy of the second track is the largest, determine whether the energy occupancy of the first track is second, and if the energy occupancy of the first track is second, weight the time-frequency spectrum of the first track and the time-frequency spectrum of the second track and output the weighted spectrum, and if the energy occupancy of the first track is not second, output only the time-frequency spectrum of the second track; if the energy occupancy of the second track is not maximum, determine whether the energy occupancy of the third track is maximum, and if the energy occupancy of the third track is not maximum, output the time-frequency spectrum of the fourth track, and if the energy occupancy of the third track is maximum, output the time-frequency spectrum of the third track. The music track matching vibration tactile feedback method according to claim 4 .
6. the first track being a percussion track, the second track being another instrument track, the third track being a vocal track, and the fourth track being a bass track; The music track matching vibration tactile feedback method according to claim 4 .
7. an original audio acquisition module for acquiring original audio data; a track dividing module for dividing the original audio data into tracks by using a pre-defined deep learning model to obtain a plurality of divided track audio data; an occupancy calculation module for calculating a corresponding energy occupancy of each of the track-divided audio data in the original audio data; a weight calculation module for determining a weight of each of the track-divided audio data according to the energy occupancy; a weighting calculation module for performing weighting calculation on all the track-divided audio data according to a preset weighting rule, and obtaining and outputting a time-frequency spectrum; a matching vibration module for generating a matching vibration signal corresponding to the original audio data according to the time-frequency spectrum; A haptic feedback module for outputting the matching vibration signal as a driving signal for a driver to realize a haptic feedback effect. A music track matching vibration haptic feedback system.
8. A method for providing haptic feedback of music track matching vibration comprising: a processor; a memory; and a computer program stored in the memory and executable by the processor, the computer program executing the computer program causing the processor to implement steps in the music track matching vibration haptic feedback method according to any one of claims 1 to 6. A computer apparatus comprising:
9. 1. A computer-readable storage medium, comprising: The computer-readable storage medium stores a computer program, which, when executed by a processor, realizes the steps of the music track matching vibration haptic feedback method according to any one of claims 1 to 6. A computer-readable storage medium comprising:
Citation Information
Patent Citations
Audio processing method, server and electronic equipment
CN114115792A
Audio processing method and device, electronic equipment and storage medium
CN114677995A
Systems and methods for generating haptic effects associated with audio signals
JP2015053054A
Electronic apparatus, method and computer program
JP2021076831A
System and method for displaying sound as vibrations
US20110129093A1
