Sound effect vibration control method and system, electronic equipment and storable medium

Abstract

The invention provides a sound effect vibration control method and system, electronic equipment and a storable medium. The method comprises the steps that psychological loudness features of target audio data are extracted, and the psychological loudness features comprise the loudness value, the instantaneous power peak value, the center frequency and the moment of single-frame audio data; calculating intensity information by using the loudness value and the instantaneous power peak value, and mapping the center frequency into a center Mel frequency; obtaining a relative frequency curve according to the center Mel frequency of all the single-frame audio data, and obtaining a relative intensity curve according to the intensity information of all the single-frame audio data; generating a sound effect vibration signal based on the relative frequency curve and the relative intensity curve; and sending the sound effect vibration signal to an actuator so as to drive the actuator to output corresponding vibration feedback. The method for automatically generating the vibration feedback is simple and efficient, the workload of designers can be balanced, and the vibration feedback experience can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of haptic feedback technology, and in particular, to a method, a system, an electronic device, and a storage medium for controlling vibration according to sound effects.

Background Art

[0002] With the development of technology, the application of vibration feedback is no longer limited to the simple vibration prompt function. Currently, common electronic devices such as mobile phones, tablets, and even game controllers, which are the most common carriers for video and game content manufacturers, are gradually adding vibration feedback on the basis of audio and video, thus bringing users a more diverse sensory experience. However, currently, high-quality vibration feedback requires vibration designers to manually add vibration effects that match the scene to the content. As the demand for vibration feedback in audio and video increases, the workload of vibration designers also increases accordingly.

[0003] In addition, according to relevant psychological research, people's subjective auditory perception is not linear, and the loudness value of general audio data cannot directly map to people's subjective auditory perception. The vibration feedback mechanisms provided by related technologies currently are difficult to ensure the consistency of human hearing and touch, so the experience effect of vibration feedback is poor and it is difficult to bring users an immersive sensory experience.

[0004] Therefore, it is necessary to provide a method for controlling vibration according to sound effects that can be automatically generated to balance the workload of relevant designers and effectively improve the experience effect of the human body on the sound effect vibration feedback mechanism.

Summary of the Invention

[0005] The purpose of the present application is to provide a method, a device, an electronic device, and a storage medium for controlling vibration according to sound effects, which can at least solve the problems of large workload of sound effect vibration designers and poor experience effect of the human body on the sound effect vibration feedback mechanism in related technologies.

[0006] To solve the above technical problems, a first aspect of an embodiment of the present application provides a method for controlling vibration according to sound effects, including:

[0007] Extracting the psychological loudness characteristics of target audio data; wherein, the psychological loudness characteristics include: the loudness value, the instantaneous power peak, the center frequency, and the time of a single-frame audio data;

[0008] Calculating intensity information using the loudness value and the instantaneous power peak, and mapping the center frequency to the center mel frequency;

[0009] Obtaining a relative frequency curve according to the center mel frequency of all the single-frame audio data, and obtaining a relative intensity curve according to the intensity information of all the single-frame audio data;

[0010] Generate a vibration signal following the sound effect based on the relative frequency curve and the relative intensity curve;

[0011] Send the vibration signal following the sound effect to an actuator to drive the actuator to output a corresponding vibration feedback.

[0012] The second aspect of the embodiments of the present application provides a vibration control device following the sound effect, including:

[0013] A feature extraction module, configured to extract the psychological loudness features of target audio data; wherein, the psychological loudness features include: the loudness value of single-frame audio data, the instantaneous power peak, the center frequency, and the frame time;

[0014] A feature mapping module, configured to calculate intensity information using the loudness value and the instantaneous power peak, and map the center frequency to the center Mel frequency;

[0015] A parameter determination module, configured to obtain a relative frequency curve according to the center Mel frequencies of all the single-frame audio data, and obtain a relative intensity curve according to the intensity information of all the single-frame audio data;

[0016] A signal generation module, configured to generate a vibration signal following the sound effect based on the relative frequency curve and the relative intensity curve;

[0017] A driving module, configured to send the vibration signal following the sound effect to an actuator to drive the actuator to output a corresponding vibration feedback.

[0018] The third aspect of the present application provides an electronic device, including: a memory and a processor, wherein the processor is configured to execute a computer program stored on the memory, and when the processor executes the computer program, implement the steps in the vibration control method according to the first aspect of the embodiments of the present application.

[0019] The fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, implement the steps in the vibration control method according to the first aspect of the embodiments of the present application.

[0020] As can be seen from the above description, compared with the related art, the beneficial effects of the present application are as follows:

[0021] First, extract the psychological loudness features from the target audio data, calculate the intensity information using the loudness value in the psychological loudness features and the instantaneous power peak, and map the center frequency to the center Mel frequency. Secondly, obtain the relative frequency curve based on the center Mel frequency of all the single-frame audio data, and obtain the relative intensity curve based on the intensity information of all the single-frame audio data. Finally, generate a vibration signal following the sound effect based on the relative frequency curve and the relative intensity curve, and send the vibration signal following the sound effect to the actuator to drive the actuator to output corresponding vibration feedback. As can be seen from the above, in the embodiment of the present application, the psychological loudness features are extracted from the audio data, and in the way of combining psychological evaluation indexes with Mel weighting, the loudness value, the center frequency and the power peak are converted into a relative intensity curve and a relative frequency curve, and a vibration signal following the sound effect is generated according to the relative intensity curve and the relative frequency curve, so as to map the effective information to the vibration parameters of the actuator to the greatest extent, complete the vibration feedback control following the sound effect, effectively improve the richness of the human perception of the vibration feedback, and bring an immersive sensory experience to the user. In addition, the vibration feedback of this method is automatically generated, simple and efficient. On the one hand, it can meet the needs of content manufacturers for independent batch conversion. On the other hand, only a small amount of optimization by designers is required to complete high-quality vibration feedback, which can further balance the workload of designers.

[0022] It should be understood that the content described in this part is not intended to identify the key or important features of the present application, nor to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

Description of the Drawings

[0023] In order to more clearly illustrate the related technologies or the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required to be used in the description of the related technologies or the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, rather than all embodiments. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 It is a schematic flowchart of the vibration control method following the sound effect provided by the embodiment of the present application;

[0025] Figure 2 It is a schematic detailed flowchart of the vibration control method following the sound effect provided by the embodiment of the present application;

[0026] Figure 3 It is a schematic diagram of the vibration signal generated according to the original sound effect in the embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of the program modules of the vibration control device following the sound effect provided by the embodiment of the present application;

[0028] Figure 5 It is a block diagram of a module of an electronic device provided by an embodiment of the present application;

[0029] Figure 6 It is a block diagram of a module of a computer-readable storage medium provided by an embodiment of the present application.

Specific Embodiments

[0030] In order to make the objectives, technical solutions and advantages of the present application more obvious and understandable, the present application will be clearly and completely described below in conjunction with the embodiments of the present application and their accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions from beginning to end. It should be understood that the various embodiments of the present application described below are only used to explain the present application and are not used to limit the present application. That is, based on the various embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0031] Please refer to Figure 1 , Figure 1 It is a schematic flowchart of a method for controlling vibration according to sound effects provided by an embodiment of the present application. The method for controlling vibration according to sound effects includes the following steps 101 to 105.

[0032] Step 101: Extract the psychological loudness feature of the target audio data.

[0033] In an embodiment of the present application, if a final vibration signal according to sound effects is to be generated, it is necessary to first obtain the target audio data of the audio and extract the psychological loudness feature from the obtained target audio data. Among them, the psychological loudness feature includes the loudness value, instantaneous power peak, center frequency and time of a single-frame audio data.

[0034] Among them, the psychological evaluation index in the embodiment of the present application is mainly reflected in the loudness value of the single-frame audio data. Generally, audio data has two main characteristics, sound pressure value and frequency. However, according to relevant research in psychoacoustics, these two physical characteristics cannot directly map people's subjective auditory feelings. Loudness is the subjective sound intensity that people feel. Loudness is usually measured in phons or sones. Loudness usually increases with the increase of sound pressure level, but the relationship between them is not linear, and the human ear's perception of loudness gradually weakens with the increase of sound pressure level. For sounds of different frequencies at the same sound pressure, the human ear's perception of their loudness is also different. Generally, the human ear is more sensitive to medium and high-frequency sounds and less sensitive to low-frequency sounds.

[0035] Thus, in an embodiment, before the step of extracting the psychological loudness feature of the target audio data, it is also necessary to first read in the original audio data with sound effects and perform preprocessing on the audio, including performing intermediate frequency processing using a corresponding filter, so that the vibration generated corresponding to the sound effect is more in line with the human tactile perception threshold, and finally achieving a better vibration feedback effect. Before that, the preprocessing also includes performing downsampling processing on the audio. First, filter through a low-pass filter to remove the high-frequency components in the signal to prevent aliasing during the downsampling process, and then perform downsampling on the filtered signal to reduce the data volume of the audio.

[0036] Step 102: Calculate the intensity information using the loudness value and the instantaneous power peak, and map the center frequency to the center Mel frequency.

[0037] In an embodiment of the present application, after obtaining the psychological loudness feature, it is necessary to convert the feature information into parameters to be considered for specific vibrations, that is, vibration intensity and vibration frequency. Among the above feature information, the loudness value and the instantaneous power peak can characterize the intensity of the audio. Therefore, in this embodiment, corresponding weights are set for the above loudness value and instantaneous power peak, and then the weighted value is obtained, which is the intensity information that can characterize the vibration intensity.

[0038] Similarly, the center frequency in the above feature information can characterize the frequency information of the vibration. However, based on the above psychological research, an audio signal is composed of a series of periodic vibrations with different frequencies. The human ear has different perceptions of sounds with different frequencies and can distinguish different pitches through frequency. However, the human ear's perception of frequency is not linear but exhibits a non-linear characteristic.

[0039] Thus, in an embodiment of the present application, in order to ensure the consistency between hearing and touch, the center frequency is mapped to the center Mel frequency. In an embodiment of the present application, the Mel scale is introduced to convert the center frequency into the center Mel frequency. The Mel scale is a non-linear frequency scale used to convert frequency into Mel values. The conversion between the Mel scale and frequency can be obtained through a formula, and the Mel frequency refers to the frequency value obtained after conversion by the Mel scale. Compared with linear frequency, the Mel frequency is more in line with the human ear's perception of sound. There are various methods to convert general linear frequency into Mel frequency that conforms to the human ear, including using a Mel scale filter bank, Mel-frequency cepstral coefficients (MFCC), etc. The present application does not make a unique limitation here; in this regard, those skilled in the art can make flexible settings according to the actual application scenario based on the embodiment of the present application.

[0040] Step 103: Obtain the relative frequency curve according to the center Mel frequency of all single-frame audio data, and obtain the relative intensity curve according to the intensity information of all single-frame audio data.

[0041] In an embodiment of the present application, after extracting the psychological loudness characteristics of the audio data, it is also necessary to map the characteristics to the vibration parameters of the actuator. Thus, after obtaining the central Mel frequency of the single-frame audio data, it is also necessary to obtain a relative frequency curve according to the central Mel frequency, with the number of frames on the horizontal axis and the relative frequency on the vertical axis, which establishes the corresponding relationship between the relative frequency and the number of frames, that is, characterizes the change of the audio frequency with the number of frames.

[0042] Specifically, first obtain the relative frequency range of the actuator for generating vibration feedback, and then map the central Mel frequency of all the single-frame audio data to the relative frequency range based on a preset first non-linear function to obtain a relative frequency curve. The first non-linear function is a custom-built non-linear function for converting the audio frequency into the relative frequency of the actuator to generate vibration.

[0043] In an embodiment of the present application, it is also necessary to obtain a relative intensity curve according to the intensity information, with the number of frames on the horizontal axis and the relative intensity on the vertical axis, which establishes the corresponding relationship between the relative intensity and the number of frames.

[0044] Specifically, first obtain the relative intensity range of the actuator for generating vibration feedback, and then map the intensity information of all the single-frame data to the relative intensity range based on a preset second non-linear function to obtain an initial relative intensity curve. Further, optimize the initial relative intensity curve based on the vibration intensity threshold perceptible by the human body to obtain the final relative intensity curve. The second non-linear function is a custom-built non-linear function for converting the audio intensity information into the relative intensity of the actuator to generate vibration.

[0045] Step 104: Generate a vibration signal following the sound effect based on the relative frequency curve and the relative intensity curve.

[0046] Specifically, after obtaining the above relative frequency curve and the above relative intensity curve, the relative frequency corresponding to the frame moment can be found on the relative frequency curve, and the relative intensity corresponding to the frame moment can be found on the relative intensity curve. Then, integrate the relative intensity, the relative frequency, and the corresponding frame moment to generate a vibration signal following the sound effect.

[0047] In an alternative embodiment of this embodiment, in order to expand the application range of the vibration signal following the sound effect, the final parameters can be written into a standard vibration file, and then an analysis tool is used to analyze the standard vibration file to generate a corresponding vibration signal following the sound effect. By adjusting the parameters of the analysis tool, the generated vibration feedback can be applied to different vibration devices and different scenarios.

[0048] Step 105: Send the vibration signal following the sound effect to the actuator to drive the actuator to output corresponding vibration feedback.

[0049] In an alternative implementation of this embodiment, the step of obtaining the relative frequency curve based on the center Mel frequency of all the single-frame audio data includes: obtaining the relative frequency range of the actuator for generating vibration feedback; mapping the center Mel frequency of all the single-frame audio data to the relative frequency range based on a preset first non-linear function to obtain the relative frequency curve; the step of obtaining the relative intensity curve based on the intensity information of all the single-frame audio data includes: obtaining the relative intensity range of the actuator for generating vibration feedback; mapping the intensity information of all the single-frame audio data to the relative intensity range based on a preset second non-linear function to obtain the relative intensity curve.

[0050] Specifically, in practical applications, actuators for generating vibration feedback have different operating frequencies and intensities. Therefore, it is necessary to first obtain the relative frequency range and relative intensity range of the actuator for generating vibration feedback, and map the center Mel frequency of the single-frame audio data to the relative frequency range through a preset first non-linear function to obtain the relative frequency curve, and map the intensity information of the single-frame audio data to the relative intensity range through a preset second non-linear function to obtain the relative intensity curve. Further, that is, using the relative frequency curve and relative intensity curve to characterize the characteristics of the audio data, and generating a vibration signal following the sound effect using the relative frequency curve and relative intensity curve in subsequent steps to complete the realization of vibration following the sound effect.

[0051] Further, in an alternative implementation of this embodiment, the method further includes: mapping the relative intensity curve and the relative frequency curve to a vibration perception curve based on a preset vibration perception function; recording the points below the lower limit value of the vibration intensity perceptible by the human body in the vibration perception curve as the first deletion points; obtaining the first corresponding moments of all the first deletion points, and deleting the points in the relative intensity curve and the relative frequency curve at the first corresponding moments to obtain the optimized relative intensity curve and relative frequency curve.

[0052] Specifically, in practical applications, the vibration feedback generated by some actuators is actually difficult to be perceived by the human body. Therefore, the vibration feedback can be optimized according to the lower limit value of the vibration intensity perceptible by the human body. Preset a fitted vibration perception function, map the current relative intensity curve and relative frequency curve according to f(intensity, frequency) to the vibration perception curve, then record the points below the lower limit value of the vibration intensity perceptible by the human body in the vibration perception curve as deletion points, and correspondingly delete the points at the corresponding frame moments in the relative intensity curve and relative frequency curve to obtain the optimized relative intensity curve and relative frequency curve, thereby realizing the optimization of the vibration feedback mechanism following the sound effect.

[0053] Further, in an alternative implementation manner of this embodiment, the step of mapping the relative intensity curve and the relative frequency curve into a vibration feeling curve based on a preset vibration feeling perception function includes: obtaining the relative intensity information included in the relative intensity curve; performing a weighting process on the relative intensity information to obtain a weighted relative intensity curve; and mapping the relative frequency curve and the weighted relative intensity curve into a vibration feeling curve based on a preset vibration feeling perception function.

[0054] Specifically, in practical applications, although the vibration feedback generated by some actuators can be perceived by the human body, it is not obvious enough. Some different vibration feedbacks also cannot produce a large difference in human perception. Therefore, it is necessary to perform a certain weighting on the vibration intensity to increase the intensity contrast, that is, to make the original strong vibration stronger and the original weak vibration weaker, and delete the vibration below the lower limit value of the vibration intensity that can be perceived by the human body in the subsequent steps, ultimately improving the experience effect of the human body on the sound effect vibration feedback mechanism.

[0055] Further, in an alternative implementation manner of this embodiment, the method further includes: performing an adjacent value check on the target points in the vibration feeling curve. If the target points in the vibration feeling curve meet the preset conditions, record the target points as second deletion points; obtain the second corresponding times of all the second deletion points, and delete the points in the optimized relative intensity curve and the optimized relative frequency curve that are at the second corresponding times to obtain a simplified relative intensity curve and a simplified relative frequency curve.

[0056] Specifically, in this embodiment, in addition to optimizing the vibration effect, the long signal can also be simplified. It can be understood that when performing an adjacent value check on the target points in the vibration feeling curve, for any point in the vibration feeling curve, if this point and its predecessor point and successor point can jointly describe the local characteristics of the curve in a preset range, then this point can be recorded as a deletion point, and the points in the optimized relative intensity curve and the optimized relative frequency curve corresponding to the deletion point are omitted, ultimately completing the simplification process of the long signal. The simplification process of the long signal not only reduces the workload of relevant designers but also ensures the continuity of the vibration experience.

[0057] In an alternative implementation manner of this embodiment, the step of generating a vibration signal following the sound effect based on the relative frequency curve and the relative intensity curve includes: generating an initial vibration signal following the sound effect based on the relative frequency curve and the relative intensity curve; adding a short signal to the initial vibration signal following the sound effect according to preset short signal parameters to obtain a final vibration signal following the sound effect; where the short signal parameters include the short signal addition time, the short signal frequency, and the short signal intensity.

[0058] Specifically, in practical applications, in order to further enrich the level of vibration intensity and optimize the human experience of sound effect vibration feedback, short signals can be added at the key positions of the sound effects to enhance the vibration feedback. First, an initial sound effect vibration signal is generated according to the relative frequency curve and the relative intensity curve, and then short signals are added to the initial sound effect vibration signal according to the preset short signal parameters or the short signal parameters generated according to the real-time audio signal to obtain the final sound effect vibration signal.

[0059] Further, in an alternative embodiment of this embodiment, the above method further includes: screening peak points of the relative intensity curve to obtain first peak points; fitting all the instantaneous power peaks into an instantaneous power peak curve, and obtaining target points corresponding to the first peak points in the instantaneous power peak curve based on preset screening conditions; performing peak position verification on the first peak points and the target points to obtain the short signal parameters.

[0060] Further, in an alternative embodiment of this embodiment, the step of screening peak points of the relative intensity curve to obtain first peak points includes: screening peak points of the relative intensity curve to obtain initial peak points; connecting all the initial peak points and performing smoothing processing to obtain a relative intensity envelope curve; screening peak points of the relative intensity envelope curve to obtain first peak points.

[0061] Further, in an alternative embodiment of this embodiment, the step of fitting all the instantaneous power peaks into an instantaneous power peak curve and obtaining target points corresponding to the first peak points in the instantaneous power peak curve based on preset screening conditions includes: obtaining the corresponding moments of all the first peak points, and combining with a preset minimum time difference to obtain all corresponding time intervals; fitting all the instantaneous power peaks into an instantaneous power peak curve, and obtaining all the points in the instantaneous power peak curve that are within the corresponding time intervals; recording all the points within the corresponding time intervals as the target points corresponding to the first peak points.

[0062] Specifically, in this embodiment, first, peak points of the relative intensity curve are screened and the corresponding moments of the peak points are obtained, then a preset minimum time difference is set to obtain corresponding time intervals, and finally, the peak points of the relative intensity curve at the moments and the points within the corresponding time intervals in the instantaneous power peak curve are subjected to peak position verification according to a preset method. If the points at the corresponding moments are determined to be peak points, short signals are added here. Furthermore, by adding short signals at the key positions of the sound effects, the increase in the level of the vibration feedback effect is achieved.

[0063] Further, in an alternative implementation of this embodiment, the step of performing peak position verification on the first peak point and the target point to obtain the short signal parameters includes: screening peak points from the instantaneous power peak curve to obtain second peak points; verifying whether the second peak points exist in the target points; if the second peak points exist in the target points, determining that the first peak point and the target point pass the peak position verification, and recording the verified first peak point as the target first peak point; recording the corresponding moment of the target first peak point as the short signal addition moment; obtaining the relative frequency corresponding to the short signal addition moment on the relative frequency curve to obtain the short signal frequency; obtaining the relative intensity corresponding to the short signal addition moment on the relative intensity curve, and increasing the relative intensity according to a preset relative intensity increase value to obtain the short signal intensity; based on all the short signal addition moments, the short signal frequency, and the short signal intensity, obtaining the short signal parameters.

[0064] Specifically, in this embodiment, if there is also a peak at the corresponding time interval of the peak point at time t on the instantaneous power peak curve, a short signal is added here, the corresponding moment is recorded as the short signal addition moment, and at the same time, the relative frequency corresponding to the short signal addition moment on the relative frequency curve is obtained to obtain the short signal frequency, the relative intensity corresponding to the short signal addition moment on the relative intensity curve is obtained, and the relative intensity is increased according to a preset relative intensity increase value to obtain the short signal intensity. Finally, the short signal parameters can be obtained by combining the short signal addition moment, the short signal frequency, and the short signal intensity.

[0065] As can be seen from the above, the embodiment of the present application extracts the psychological loudness feature from the audio data, adopts the method of combining psychological evaluation indexes with Mel weighting, converts the loudness value, the center frequency, and the power peak into the relative intensity curve and the relative frequency curve, and generates the vibration signal following the sound effect according to the relative intensity curve and the relative frequency curve, so as to map the effective information to the actuator vibration parameters to the greatest extent, complete the vibration feedback control following the sound effect, and can effectively improve the richness of the human perception of the vibration feedback, bringing an immersive sensory experience to the user. In addition, the vibration feedback of this method is automatically generated, simple and efficient. On the one hand, it can meet the needs of content manufacturers for independent batch conversion, and on the other hand, only a small amount of optimization by designers is required to complete high-quality vibration feedback, which can further balance the workload of designers.

[0066] In summary, for the detailed process of the vibration control method following the sound effect involved in the embodiment of the present application, reference can be made to Figure 2 Specifically:

[0067] Step 201, read the original audio data with sound effects, and preprocess the original audio data to obtain the target audio data;

[0068] Step 202, extract the psychological loudness features of the target audio data;

[0069] Step 203, calculate the intensity information using the loudness value and the instantaneous power peak in the psychological loudness features, and map the center frequency to the center Mel frequency;

[0070] Step 204, obtain the relative frequency curve according to the center Mel frequency of all single-frame audio data, and obtain the relative intensity curve according to the intensity information of all single-frame audio data;

[0071] Step 205, optimize the relative intensity curve and the relative frequency curve based on the vibration intensity perceptible by the human body;

[0072] Step 206, simplify the optimized relative intensity curve and relative frequency curve;

[0073] Step 207, generate an initial vibration signal following the sound effect based on the simplified relative intensity curve and relative frequency curve;

[0074] Step 208, add a short signal to the initial vibration signal following the sound effect to obtain the final vibration signal following the sound effect;

[0075] Step 209, send the final vibration signal following the sound effect to the actuator to drive the actuator to output the corresponding vibration feedback.

[0076] For the more detailed processes of each step in Steps 201 to 209, refer to the description of the relevant parts shown above. This embodiment of the present application will not elaborate here.

[0077] It should be understood that the magnitude of the serial numbers of each step in this embodiment does not mean the sequence of step execution. The execution sequence of each step should be determined by its function and internal logic, and should not uniquely limit the implementation process of this embodiment of the present application.

[0078] Please refer to Figure 3 , Figure 3 , which is a schematic diagram of the vibration signal generated by this embodiment of the present application according to the original sound effect. It can be clearly seen that the vibration signal obtained by the vibration control method following the sound effect in this embodiment of the present application can map the effective information to the motor vibration parameters to the maximum extent, simplify the long signal, add short signals to the key positions of the sound effect to enrich the layering of the vibration intensity, be friendly to designers while ensuring the continuity of the vibration, and finally complete the conversion from the sound effect to the vibration.

[0079] Please refer to Figure 4 , Figure 4The vibration control device according to the embodiment of the present application. This device can be used to implement the sound - following vibration control method involved in the embodiment of the present application. The sound - following vibration control device mainly includes:

[0080] A feature extraction module 401, configured to extract the psychological loudness features of target audio data; wherein, the psychological loudness features include: the loudness value, the instantaneous power peak, the center frequency, and the frame time of a single - frame audio data;

[0081] A feature mapping module 402, configured to calculate intensity information using the loudness value and the instantaneous power peak, and map the center frequency to the center Mel frequency;

[0082] A parameter determination module 403, configured to obtain a relative frequency curve according to the center Mel frequencies of all the single - frame audio data, and obtain a relative intensity curve according to the intensity information of all the single - frame audio data;

[0083] A signal generation module 404, configured to generate a sound - following vibration signal based on the relative frequency curve and the relative intensity curve;

[0084] A driving module 405, configured to send the sound - following vibration signal to an actuator to drive the actuator to output a corresponding vibration feedback.

[0085] In some embodiments of this embodiment, the parameter determination module is specifically configured to: obtain the relative frequency range of the actuator for generating vibration feedback; map the center Mel frequencies of all the single - frame audio data to the relative frequency range based on a preset first non - linear function to obtain a relative frequency curve; obtain the relative intensity range of the actuator for generating vibration feedback; map the intensity information of all the single - frame audio data to the relative intensity range based on a preset second non - linear function to obtain a relative intensity curve.

[0086] Further, in some embodiments of this embodiment, the parameter determination module is further configured to: map the relative intensity curve and the relative frequency curve to a vibration perception curve based on a preset vibration perception function; record the points in the vibration perception curve that are lower than the lower limit value of the vibration intensity perceptible by the human body as first deletion points; obtain the first corresponding times of all the first deletion points, and delete the points in the relative intensity curve and the relative frequency curve that are at the first corresponding times to obtain an optimized relative intensity curve and relative frequency curve.

[0087] Further, in some embodiments of the present embodiment, when the parameter determination module executes the function of mapping the relative intensity curve and the relative frequency curve into a vibration feeling curve based on a preset vibration feeling perception function, it specifically is used for: obtaining the relative intensity information included in the relative intensity curve; performing a weighting process on the relative intensity information to obtain a weighted relative intensity curve; and mapping the relative frequency curve and the weighted relative intensity curve into a vibration feeling curve based on a preset vibration feeling perception function.

[0088] Further, in some embodiments of the present embodiment, the parameter determination module is further used for: performing an adjacent value verification on a target point in the vibration feeling curve, and if the target point in the vibration feeling curve can be characterized by the value of its adjacent point, recording the target point as a second deletion point; obtaining the second corresponding moments of all the second deletion points, and deleting the points in the optimized relative intensity curve and the optimized relative frequency curve that are at the second corresponding moments, to obtain a simplified relative intensity curve and a simplified relative frequency curve.

[0089] In some embodiments of the present embodiment, the signal generation module is specifically used for: generating an initial sound effect-vibrating signal based on the relative frequency curve and the relative intensity curve; adding a short signal to the initial sound effect-vibrating signal according to preset short signal parameters to obtain a final sound effect-vibrating signal; wherein the short signal parameters include a short signal addition moment, a short signal frequency, and a short signal intensity.

[0090] Further, in some embodiments of the present embodiment, the signal generation module is further used for: screening peak points of the relative intensity curve to obtain first peak points; connecting all the first peak points and performing a smoothing process to obtain a relative intensity envelope curve; screening peak points of the relative intensity envelope curve to obtain second peak points; obtaining the corresponding moments of all the second peak points, and combining with a preset minimum time difference to obtain all corresponding time intervals; fitting all the instantaneous power peaks into an instantaneous power peak curve, and obtaining the points in the instantaneous power peak curve that are all within the corresponding time intervals; and performing a peak position verification on the second peak points and the points within the corresponding time intervals to obtain the short signal parameters.

[0091] Further, in some embodiments of the present embodiment, when the signal generation module performs the function of verifying the peak position of the second peak point with the points within the corresponding time interval to obtain the short signal parameters, it specifically is used for: screening peak points from the instantaneous power peak curve to obtain a third peak point; if the third corresponding moment of the third peak point is within the corresponding time interval, then recording the third corresponding moment as the short signal addition moment; obtaining the relative frequency corresponding to the short signal addition moment on the relative frequency curve to obtain the short signal frequency; obtaining the relative intensity corresponding to the short signal addition moment on the relative intensity curve, and increasing the relative intensity according to a preset relative intensity increase value to obtain the short signal intensity; based on all the short signal addition moments, the short signal frequency, and the short signal intensity, obtaining the short signal parameters.

[0092] In an alternative embodiment of the present embodiment, the sound effect - following vibration control device further includes: a pre - processing module. The pre - processing module is used for: reading in the original audio data with sound effects and pre - processing the original audio data to obtain target audio data.

[0093] According to the sound effect - following vibration control device provided in this embodiment, extract the psychological loudness feature of the target audio data; calculate the intensity information by using the loudness value and the instantaneous power peak, and map the center frequency to the center Mel frequency; obtain the relative frequency curve according to the center Mel frequency of all the single - frame audio data, and obtain the relative intensity curve according to the intensity information of all the single - frame audio data; generate a sound effect - following vibration signal based on the relative frequency curve and the relative intensity curve; send the sound effect - following vibration signal to the actuator to drive the actuator to output corresponding vibration feedback. Through the implementation of the solution of this application, extract the psychological loudness feature from the audio data, adopt the method of combining psychological evaluation indicators with Mel weighting, convert the loudness value, center frequency, and power peak into a relative intensity curve and a relative frequency curve, and generate a sound effect - following vibration signal according to the relative intensity curve and the relative frequency curve, so as to map the effective information to the actuator vibration parameters to the greatest extent, complete the sound effect - following vibration feedback control, and can effectively improve the richness of the human perception of the vibration feedback, bringing an immersive sensory experience to the user. In addition, the vibration feedback of this method is automatically generated, simple and efficient. On the one hand, it can meet the needs of content manufacturers for independent batch conversion, and on the other hand, only a small amount of optimization by designers is required to complete high - quality vibration feedback, which can further balance the workload of designers.

[0094] Please refer to Figure 5 , Figure 5 which is the module block diagram of the electronic device provided by the embodiment of this application.

[0095] As Figure 5As shown in the figure, an embodiment of the present application further provides an electronic device, which can be used to implement the sound - following vibration control method in the foregoing embodiments. The electronic device includes a memory 501 and at least one processor 502. Among them, the memory 501 is used to store at least one program, and when the at least one program is executed by the at least one processor 502, the at least one processor 502 is caused to execute the sound - following vibration control method provided by the embodiment of the present application.

[0096] Further, the electronic device further includes a vibration feedback device 503, and the vibration feedback device 503 is used to vibrate synchronously according to the sound - following vibration signal output by the processor 502 to achieve a corresponding vibration effect.

[0097] Please refer to Figure 6 , Figure 6 which is a block diagram of a module of the computer - readable storage medium provided by the embodiment of the present application.

[0098] As Figure 6 shown in the figure, an embodiment of the present application further provides a computer - readable storage medium 600. An executable instruction 610 is stored on the computer - readable storage medium 600, and when the executable instruction 610 is executed, the sound - following vibration control method provided by the embodiment of the present application is executed.

[0099] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be implemented directly by hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in a random access memory (RAM), internal memory, read - only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD - ROM, or any other form of storage medium well - known in the technical field.

[0100] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in this application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, fiber optic, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk).

[0101] It should be noted that the various embodiments in the content of this application are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For product-type embodiments, since they are similar to method-type embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method-type embodiments.

[0102] It should also be noted that in the content of this application, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements that are not explicitly listed, or further includes elements that are inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device that includes the element.

[0103] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the content of this application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the content of this application can be implemented in other embodiments without departing from the spirit or scope of the content of this application. Therefore, the content of this application will not be limited to these embodiments shown in the content of this application, but rather to the broadest scope consistent with the principles and novel features disclosed in the content of this application.

Claims

1. A method for controlling vibration with sound effects, characterized in that: Extracting psychological loudness features of target audio data; wherein the psychological loudness features include: loudness value, instantaneous power peak, center frequency and time of single-frame audio data; Calculating intensity information using the loudness value and the instantaneous power peak, and mapping the center frequency to a center Mel frequency; Acquire a relative frequency curve according to the central Mel frequency of all the single-frame audio data, and acquire a relative intensity curve according to the intensity information of all the single-frame audio data; Generate a vibration signal with sound effect based on the relative frequency curve and the relative intensity curve; The vibration signal accompanying the sound effect is sent to the actuator to drive the actuator to output corresponding vibration feedback.

2. The method for controlling vibration with sound effect according to claim 1, characterized in that: Before the step of extracting the psychological loudness feature of the target audio data, the method includes: Read in raw audio data with sound effects; The original audio data is preprocessed to obtain target audio data; wherein the preprocessing includes at least one of the following: filtering processing and downsampling processing.

3. The method for controlling vibration with sound effect according to claim 1, characterized in that: The step of obtaining a relative frequency curve according to the central Mel frequency of all the single-frame audio data comprises: Obtaining a relative frequency interval of an actuator for generating vibration feedback; Mapping the central Mel frequencies of all the single-frame audio data to the relative frequency interval based on a preset first nonlinear function to obtain a relative frequency curve; The step of obtaining a relative intensity curve according to the intensity information of all the single-frame audio data comprises: Obtaining a relative strength range of an actuator for generating vibration feedback; The intensity information of all the single-frame audio data is mapped to the relative intensity interval based on a preset second nonlinear function to obtain a relative intensity curve.

4. The method for controlling vibration with sound effect according to claim 3, characterized in that: The method further comprises: Mapping the relative intensity curve and the relative frequency curve into a vibration curve based on a preset vibration perception function; Based on the lower limit of the vibration intensity perceivable by the human body, recording the point in the vibration curve below the lower limit as the first deletion point; The first corresponding moments of all the first deletion points are obtained, and the points in the relative strength curve and the relative frequency curve at the first corresponding moments are deleted to obtain optimized relative strength curve and relative frequency curve.

5. The method for controlling vibration with sound effect according to claim 4, characterized in that: The step of mapping the relative intensity curve and the relative frequency curve into a vibration curve based on a preset vibration perception function comprises: Obtaining relative strength information contained in the relative strength curve; Performing weighted processing on the relative intensity information to obtain a weighted relative intensity curve; The relative frequency curve and the weighted relative intensity curve are mapped into a vibration curve based on a preset vibration perception function.

6. The method for controlling vibration with sound effect according to claim 4, characterized in that: The method further comprises: Performing an adjacent value check on the target point in the vibration curve, and if the target point in the vibration curve meets a preset condition, recording the target point as a second deletion point; The second corresponding moments of all the second deleted points are obtained, and the points in the optimized relative intensity curve and the optimized relative frequency curve at the second corresponding moments are deleted to obtain a simplified relative intensity curve and a simplified relative frequency curve.

7. The method for controlling vibration with sound effect according to claim 1, characterized in that: The step of generating a vibration signal with a sound effect based on the relative frequency curve and the relative intensity curve comprises: Generate an initial sound effect vibration signal based on the relative frequency curve and the relative intensity curve; According to preset short signal parameters, a short signal is added to the initial vibration signal with sound effect to obtain a final vibration signal with sound effect; wherein the short signal parameters include the short signal adding time, the short signal frequency and the short signal strength.

8. The method for controlling vibration with sound effect according to claim 7, characterized in that: The method further comprises: Performing peak point screening on the relative intensity curve to obtain a first peak point; Fitting all the instantaneous power peaks into an instantaneous power peak curve, and acquiring a target point corresponding to the first peak point in the instantaneous power peak curve based on a preset screening condition; The first peak point and the target point are subjected to a peak position check to obtain the short signal parameter.

9. The method for controlling vibration with sound effect according to claim 8, characterized in that: The step of screening the relative intensity curve for peak points to obtain the first peak point comprises: Performing peak point screening on the relative intensity curve to obtain an initial peak point; Connect all the initial peak points and perform smoothing to obtain a relative intensity envelope curve; The relative intensity envelope curve is subjected to peak point screening to obtain a first peak point.

10. The method for controlling vibration with sound effect according to claim 8, characterized in that: The step of fitting all the instantaneous power peaks into an instantaneous power peak curve, and acquiring a target point corresponding to the first peak point in the instantaneous power peak curve based on a preset screening condition, comprises: Obtaining corresponding moments of all the first peak points, and combining with a preset minimum time difference, obtaining all corresponding time intervals; Fitting all the instantaneous power peaks into an instantaneous power peak curve, and obtaining all points in the instantaneous power peak curve that are within the corresponding time interval; All the points within the corresponding time interval are recorded as target points corresponding to the first peak point.

11. The method for controlling vibration with sound effect according to claim 10, characterized in that: The step of performing peak position verification on the first peak point and the target point to obtain the short signal parameter comprises: Performing peak point screening on the instantaneous power peak curve to obtain a second peak point; Checking whether the second peak point exists in the target point; If the second peak point exists in the target point, determining that the first peak point and the target point pass the peak position check, and recording the first peak point that passes the check as the target first peak point; Recording the time corresponding to the first peak point of the target as the short signal adding time; Obtaining the relative frequency corresponding to the short signal adding moment on the relative frequency curve to obtain the short signal frequency; Obtaining the relative strength corresponding to the short signal adding moment on the relative strength curve, and increasing the relative strength according to a preset relative strength increase value to obtain the short signal strength; Based on all the short signal adding times, the short signal frequencies and the short signal strengths, short signal parameters are obtained.

12. A sound-effect vibration control device, characterized in that: include: A feature extraction module, used to extract the psychological loudness features of the target audio data; wherein the psychological loudness features include: the loudness value, instantaneous power peak, center frequency and frame time of a single frame of audio data; A feature mapping module, configured to calculate intensity information using the loudness value and the instantaneous power peak, and map the center frequency to a center Mel frequency; a parameter determination module, configured to obtain a relative frequency curve according to the central Mel frequency of all the single-frame audio data, and to obtain a relative intensity curve according to the intensity information of all the single-frame audio data; A signal generating module, used for generating a vibration signal with sound effect based on the relative frequency curve and the relative intensity curve; The driving module is used to send the vibration signal accompanying the sound effect to the actuator to drive the actuator to output corresponding vibration feedback.

13. An electronic device, characterized in that: The device comprises a memory and a processor, wherein: The processor is used to execute the computer program stored in the memory; When the processor executes the computer program, the steps of the vibration control method with sound effects described in any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method for controlling vibration with sound effects as described in any one of claims 1 to 11 are implemented.

Images