REAL-TIME AUDIO PROCESSING METHOD, SYSTEM, DEVICE, AND MEDIUM

The real-time audio processing method addresses the lack of player-audio interaction simulation in games by buffering audio data, calculating user displacement, and applying linear interpolation and noise to achieve efficient and realistic scratch effects.

JP7587052B2Active Publication Date: 2024-11-19LILITH TECH (SHANGHAI) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023550173
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-02-22
Filing Date
2021-12-23
Publication Date
2024-11-19
Estimated Expiration
2041-12-23

AI Technical Summary

Technical Problem

Conventional real-time audio processing techniques fail to effectively simulate player-audio interactions, particularly scratch effects, in game environments.

Method used

A method and system for real-time audio processing that involves storing audio data in a buffer divided into blocks, calculating user operation displacement data to determine velocity and acceleration, and reproducing audio based on these parameters to simulate scratch effects, using linear interpolation and noise addition for enhanced realism.

Benefits of technology

The method reduces hardware performance requirements, improves computational efficiency, and achieves smooth and realistic simulation of scratch effects in real-time audio processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007587052000017
    Figure 0007587052000017
  • Figure 0007587052000018
    Figure 0007587052000018
  • Figure 0007587052000019
    Figure 0007587052000019
Patent Text Reader

Abstract

The present invention relates to the technical field of audio processing, and in particular to a real-time audio processing method, system, device and medium. The method includes: receiving real-time audio data, storing the audio data in a buffer, storing the audio data in the buffer in a number of blocks, each block containing a certain number of sampling points, obtaining displacement data of a user operation, and reading and playing a block at a corresponding position in the buffer according to the speed and acceleration calculated by the displacement data. The method realizes the interaction between the user and the real-time audio, including supporting the interaction between the game player and the real-time audio in the game, including accomplishing scratch sound effects, restoring the effect of mechanical scratch, with a delicate and smooth transition of the sound effect, and a small hardware performance occupation rate.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to the field of audio processing technology, and in particular to a real-time audio processing method, system, device and medium. [Background technology]

[0002] Most game audio is achieved by playing all samples (i.e. pre-recorded or created audio files). Further processing may be done on the audio in real time in conjunction with various effects. Examples include room reverberation and low pass filtering of distant sounds. However, none of them involve concrete player interaction with the audio, including scratch effects, although the audio is often triggered and changed with the screen or scene.

[0003] Most of the conventional techniques for real-time audio processing are used in the field of music production software, and there are still no good technical means for real-time audio processing in games, especially for scratch effects. Summary of the Invention

[0004] The objective of the present invention is to provide a real-time audio processing method, system, device and medium for realizing user interaction with real-time audio, including supporting game player interaction with in-game real-time audio, achieving scratch sound effects, restoring the effect of mechanical scratch, and making the transition of sound effects delicate and smooth.

[0005] An embodiment of the present invention provides a real-time audio processing method for a terminal, comprising: Step 1 of receiving real-time audio data, storing the audio data in a buffer, and storing the audio data in the buffer in a divided manner into a plurality of blocks, each block including a certain number of a plurality of sampling points; Step 2 of acquiring displacement data of a user operation; Step 3: reading and playing one block at a corresponding position in the buffer based on the velocity and acceleration calculated by the displacement data, wherein the velocity and acceleration are calculated by the following formula:

[0006]

number

[0007] and acceleration

[0008]

number

[0009] where x denotes the pre-processed displacement data, and x p denotes the displacement data received before playing the preprocessed previous block, s denotes an adjustable coefficient for adjusting the sensitivity of the response to the user operation action, and v p a denotes the calculated speed before playing the previous block, and b denotes the number of sampling point groups in the block, where a sampling point group denotes a group of sampling points that are the same in time but have different audio channels.

[0010] Preferably, step 2 comprises: determining whether the user is not performing an operation or has stopped performing an operation; If yes, play said audio data in real time at normal speed; If "no", then proceeding to step 3.

[0011] Preferably, the step of reading and playing one block at a corresponding position in the buffer based on the velocity and acceleration calculated by the displacement data includes: determining a block to be read based on the velocity and the acceleration; The method further includes performing processing on the sampling points in the block based on the velocity and the acceleration.

[0012] Preferably, performing processing on the sampling points in said block comprises linear interpolation.

[0013] Preferably, the pre-processing of the displacement data includes: This includes scaling the values ​​of the received displacement data such that the range of values ​​of the displacement data that can be received is in the interval [0, number of sampling point groups in the buffer].

[0014] Preferably, reading and reproducing a block from a corresponding position in the buffer based on the velocity and acceleration calculated by the displacement data further comprises adding noise to the block, the magnitude of which is directly proportional to the velocity, before reproducing the block.

[0015] An embodiment of the present invention discloses a real-time audio processing system, the real-time audio processing system including: an audio data receiving module, a displacement data acquiring module and a playing module; The audio data receiving module receives real-time audio data, stores the audio data in a buffer, and stores the audio data in the buffer in a plurality of blocks, each block including a certain number of sampling points; The displacement data acquisition module acquires displacement data of a user operation; The playback module reads and plays one of the blocks at a corresponding position in the buffer according to the velocity and acceleration calculated by the displacement data, where the velocity is calculated according to the following formula:

[0016]

number

[0017] and acceleration

[0018]

number

[0019] where x denotes the pre-processed displacement data, and x p denotes the displacement data received before playing the preprocessed previous block, s denotes an adjustable coefficient for adjusting the sensitivity of the response to the user operation action, and v p denotes the speed calculated before playing the previous block, and b denotes the number of sampling point groups in the block, where a sampling point group indicates a group of sampling points that are the same in time but have different audio channels.

[0020] An embodiment of the present invention discloses a real-time audio processing apparatus comprising a memory in which computer-executable instructions are stored and a processor, the processor being arranged to perform a real-time audio processing method by executing the instructions.

[0021] An embodiment of the present invention discloses a computer storage medium encoded with a computer program, the computer program comprising instructions that, when executed on one or more computers, cause the computer to perform a real-time audio processing method.

[0022] Comparing the embodiments of the present invention with the prior art, the main differences and advantages are as follows:

[0023] In the present invention, real-time audio data is received, and the audio data is stored in a buffer. Since there is no real record, the played audio data does not reside in memory. Therefore, a buffer is provided to buffer one segment of past audio (for example, about 1 to 4 seconds) for use.

[0024] In the present invention, audio data is stored in a buffer in a divided manner into multiple blocks, each block containing a certain number of sampling points. When processing the audio data, one block of data is read out first each time, and the data is pre-processed as a whole, and then each sampling point therein is processed, thereby reducing the requirements for hardware performance, shortening the data read time, and improving the computing performance.

[0025] In the present invention, in step 2, the displacement data of the user operation is obtained, and in step 3, a block at a corresponding position in the buffer is read and played based on the velocity and acceleration calculated by the displacement data, and the velocity is calculated by the following formula:

[0026]

number

[0027] and acceleration

[0028]

number

[0029] where x denotes the pre-processed displacement data, and x p denotes the preprocessed displacement data received before playing the previous block, s denotes an adjustable coefficient for adjusting the sensitivity of the response to the user operation operation, and v pindicates the speed calculated before playing the previous block, and b indicates the number of sampling point groups in the block, where a sampling point group indicates a group of sampling points with the same time but different audio channels. The user's operation information is obtained, and audio data within a certain period of time stored in the buffer memory is played back based on the operation speed, acceleration, etc., so that scratches can be simulated in real-time audio. If the finger displacement information is obtained in real time when processing each sampling point, it is accurate, but the overhead is too large. Moreover, general hardware does not have such a high refresh rate. Data is read in units of blocks, and the displacement data of the user's operation is obtained before each read, the speed is calculated according to the displacement data, and the block data is played back according to the speed, so that the amount of calculations can be reduced, the requirements for hardware performance can be reduced, and the processing speed of audio data can be increased. In addition, the acceleration is calculated according to the displacement data and superimposed on the speed, and the block data is played back based on the calculated speed and acceleration, so that the smoothness of audio playback can be further improved.

[0030] In the present invention, step 2 further includes judging whether the user is not operating or has stopped operating, and if it is "yes", playing real-time audio data at normal speed, and if it is "no", proceeding to step 3. If the user is not operating, real-time audio can be played normally, and if the user operates, for example, moves the record after scratching, and then releases his / her hand and resumes normal playback speed at the position where he / she released his / her hand, there will be a certain time difference, that is, delay, between the audio to be played next and the real-time audio. Therefore, in the present invention, when the user releases his / her hand, this is equivalent to fast-forwarding the record to the position to be played at the current speed if the record had been played at normal speed all along, and then resuming normal playback, thereby solving the problem of delay.

[0031] In the present invention, reading and playing one block at a corresponding position in the buffer based on the speed and acceleration calculated by the displacement data further includes determining the block to be read based on the speed and acceleration, and processing the sampling points in the block based on the speed and acceleration. By addressing the buffer according to the user's operation and processing the sampling points in the block at the corresponding position, it is possible to easily play the audio according to the speed and acceleration of the user's operation after outputting to hardware such as a sound card, and since the essence of scratching is the result of "alternating high-speed playback and slow playback" of audio, the pitch becomes higher when the playback speed is fast and the pitch becomes lower when the playback speed is slow, the scratch effect can be simulated by playing the audio according to the speed and acceleration of the user's operation. In addition, other processing may be performed simultaneously.

[0032] In the present invention, the process performed on the sampling points in the block includes linear interpolation, and when the user drags too slowly during operation, due to the limited number of sampling points in the buffer data, when generating a few new sampling points, the same points in the buffer data are accessed within a certain period of time, and the output waveform resembles sawtooth, and a large amount of high frequency components are drawn in, which may cause the audio to sound harsh. Interpolation can greatly reduce this problem, and linear interpolation has low performance requirements and is fast.

[0033] In the present invention, the pre-processing of the displacement data includes scaling the numerical value of the received displacement data so that the range of the numerical value of the receivable displacement data is in the interval [0, the number of sampling point groups in the buffer]. Since only certain positions in the buffer can be addressed, the numerical value of the displacement data is scaled and mapped into the addressing range of the corresponding buffer, which facilitates addressing based on the displacement data.

[0034] In the present invention, reading and playing a block from a corresponding position in the buffer based on the velocity and acceleration calculated by the displacement data further includes adding a noise to the block before playing the block, the magnitude of which is directly proportional to the velocity, to simulate the sound generated by the fast friction of the stylus against the record surface during scratching, and the magnitude of the friction sound is related to the speed of the scratch. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 is a flow chart illustrating a real-time audio processing method according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a flow chart illustrating a real-time audio processing method according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a block diagram illustrating a real-time audio processing system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The present application will be further described below with reference to specific examples and drawings. It will be understood that the specific examples described herein are merely for the purpose of interpreting the present application, and do not limit the present application. For the sake of convenience, the drawings show only parts related to the present application, and do not show all configurations or processes. It should be noted that in this specification, similar symbols and letters represent similar items in the following drawings.

[0037] It should be understood that although features may be described herein using terms such as "first," "second," etc., these features should not be limited by these terms. These terms are merely for distinguishing purposes and should not be understood as indicating or implying relative importance. By way of example, a first feature may be referred to as a second feature and similarly a second feature may be referred to as a first feature without departing from the scope of the embodiments.

[0038] In the description of this application, unless otherwise clearly specified or limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, they may be fixedly connected, detachably connected, or integrally connected. They may be mechanically connected or electrically connected. They may be directly connected, indirectly connected via an intermediate medium, or internally connected between two elements. Those skilled in the art can specifically understand the specific meaning of the above terms in this embodiment.

[0039] Exemplary embodiments of the present application include, but are not limited to, real-time audio processing methods, systems, devices, and mediums.

[0040] The various aspects of the exemplary embodiments are described using terms commonly employed by those skilled in the art to convey the substance of their work to those skilled in the art. However, it will be apparent to those skilled in the art that some alternative embodiments may be implemented using some of the described features. For purposes of interpretation, specific numbers and configurations are detailed to provide a more complete understanding of the exemplary embodiments. However, it will be apparent to those skilled in the art that alternative embodiments may be implemented in the absence of the specific details. In some other instances, the specification omits or simplifies some well-known features to avoid obscuring the exemplary embodiments of the present application.

[0041] Also, various operations are described as multiple independent operations in a manner that is most helpful in understanding the exemplary embodiments. However, the order of description should not be construed as implying that the operations must be in the order described, and many of the operations may occur in parallel or simultaneously. Also, the order of the operations may be permuted. The process may end when the described operations are completed, but may include additional steps that are not shown. The process may correspond to a method, a function, a regulation, a subroutine, a subprogram, etc.

[0042] References in the specification to "one embodiment," "an embodiment," "an exemplary embodiment," and the like, indicate that the described embodiment may include a particular feature, structure, or characteristic, but each embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, these phrases do not necessarily refer to the same embodiment. Furthermore, when particular features are described with reference to specific embodiments, the knowledge of one of ordinary skill in the art may affect the combination of those features with other embodiments, whether or not those embodiments are explicitly described.

[0043] Unless otherwise indicated by context, the terms "comprise," "have," and "include" are synonymous. The phrase "A and / or B" means "(A), (B), or (A and B)."

[0044] The term "module" as used herein may include, as part thereof, memory (shared, dedicated or grouped) for executing one or more software or firmware programs, application specific integrated circuits (ASICs), electronic circuits and / or processors (shared, dedicated or grouped), combinatorial logic circuits, and / or other suitable components that provide the functionality.

[0045] In the drawings, certain structure or method features are shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order is not required. In some embodiments, these features may be described in a different form and / or order than that shown in the example drawings. Also, structure or method features included in a particular drawing are not meant to be included in all embodiments, and some embodiments may not include these features or may combine these features with other features.

[0046] In order to make the objectives, technical means and advantages of the present application clearer, the embodiments of the present application will be described in more detail below with reference to the drawings.

[0047] The objective of the present invention is to realize user interaction with real-time audio, including supporting game player interaction with in-game real-time audio, including accomplishing scratch sound effects, restoring the effect of mechanical scratching, and making the transition of sound effects delicate and smooth.

[0048] The scratch effect is a general term for artificial pitch changes and other additive distortion effects that are caused by moving a record on a record player with your fingers, artificially affecting the rotation speed of the record and changing that speed drastically. The above principles can be modeled and described as follows:

[0049] The essence of a scratch is the result of "alternating between high-speed and slow-speed playback" of sound. If the speed direction in which a record is normally played is the positive direction, then a scratch in the negative direction will generate a negative acceleration. Due to this acceleration, the original positive speed will become smaller, and the pitch will drop. Then, this acceleration will completely cancel the positive speed, and the record will start to reverse, resulting in a negative speed, which will become increasingly larger due to the presence of acceleration. As the record is reversed, the pitch will gradually rise again until the negative acceleration disappears. When the negative acceleration disappears, the record itself will continue to play forward, so the record player itself will generate a positive acceleration until the playback speed returns to normal, but at this time the playback position is smaller than (the initial position + the displacement that should have elapsed over time). In this process, the speed of the record's reverse will decrease, and the pitch will again decrease until the speed reverses to the positive speed, and finally return to normal speed. Of course, the user may also apply a positive acceleration to the record, which will be superimposed on the acceleration produced by the record player itself.

[0050] In reality, scratching can be done by back scratching or forward scratching because all audio information is recorded on the record, but if real-time processing is performed on the sound, it is impossible to know what kind of sound will appear in the future. Therefore, only the process of "moving the record later" and "moving the record later, releasing the hand, and returning it to the original position" can be simulated. Moreover, if the user releases the hand directly after moving the record later, and the sound is returned to the normal playback speed at the position where the hand was released, there is actually a certain delay between the playback process and the actual sound. Therefore, in the present invention, when the user releases the hand, it is equivalent to instantaneously fast-forwarding the record to the position where it should be currently played if it has been played at normal speed all this time, and then resuming normal playback.

[0051] FIG. 1 is a flow chart illustrating a real-time audio processing method according to an embodiment of the present invention.

[0052] As shown in FIG. 1, the method includes: Step 102 of receiving real-time audio data, storing the audio data in a buffer, and storing the audio data in the buffer in a plurality of blocks, each block including a certain number of a plurality of sampling points; A step 104 of acquiring displacement data of a user operation; Step 106: reading and playing one block at a corresponding position in the buffer based on the velocity and acceleration calculated by the displacement data, where the velocity is calculated by the following formula:

[0053]

number

[0054] and acceleration

[0055]

number

[0056] where x denotes the pre-processed displacement data, and x p denotes the displacement data received before playing the preprocessed previous block, s denotes an adjustable coefficient for adjusting the sensitivity of the response to the user operation action, and v p and step 106, where b denotes the calculated speed before playing the previous block, and b denotes the number of sampling point groups in the block, where a sampling point group denotes a group of sampling points that are identical in time but have different audio channels.

[0057] For example, in some scenes (e.g., during a game), some real-time sounds (e.g., sounds generated in the game in real time, voices input from outside) are generated or input, and scratch effects can be simulated in the real-time sounds. A buffer is provided to receive real-time audio data of the sounds, and the audio data is stored in the buffer, and the audio data is divided into a plurality of blocks and stored in the buffer, and each block includes a certain number of a plurality of sampling points. When a user (e.g., a game player) clicks on the screen, displacement data of the user's operation (e.g., the position of the click point on the screen of the user's finger or mouse pointer is obtained) is obtained at a certain interval (e.g., a certain time interval, a screen update interval, or an interval when a certain amount of data is read). The motion parameters of the user's operation, such as speed and acceleration, are calculated based on the obtained displacement data, and the audio data is read and played based on these motion parameters (e.g., these motion parameters are mapped to the addressing speed and acceleration in the buffer and / or the speed and acceleration at which the audio data is played). Specifically, for example, in a game, the game player interacts with sounds generated in real time and voices input from outside during the game progress, and in a scene simulating a scratch effect in these sounds, the received real-time audio data is prevented from being sent to the sound card, and the received real-time audio data is stored in a buffer, which may be a circular buffer, and a write pointer advances in one direction as data is stored in the buffer, and when the write pointer reaches the edge of the buffer, the write pointer returns to the beginning of the buffer again, thereby making the buffer a circular buffer. When the player is not performing a scratch operation, the real-time audio is played back normally.When the game player starts to move the interaction control unit (such as a virtual disk) displayed on the screen in the forward or reverse direction, the position of the click point on the screen of the player's finger or mouse pointer is obtained, and immediately after starting to move in the reverse direction, the position is obtained three times at a certain time interval, and the difference between each two adjacent click point positions among the three click point positions is calculated as the displacement, and the corresponding speed and acceleration are calculated based on these displacements, and then the read pointer is addressed by turning back from the position currently located in the buffer, and one block at the corresponding position in the buffer is read and played. During the player's scratch process, each time a block is read and played, the player's click point position is obtained and preprocessed, and a new click point position is obtained and preprocessed each time, and the preprocessed click point position x and the preprocessed click point position x obtained previously are obtained. p Based on the formula

[0058]

number

[0059] Calculate the scratch speed v of the corresponding player by p Based on the formula

[0060]

number

[0061] Calculate the corresponding player scratch acceleration a by, where s represents an adjustable coefficient, which is for adjusting the sensitivity of the response to the user operation action, and b represents the number of sampling point groups in the block, where a sampling point group represents a group of sampling points with the same time but different audio channels. Then, read and play the next block according to the speed v and acceleration a.

[0062] In the present invention, one block of data is read first each time, and the data is preprocessed as a whole before processing each sampling point therein, so that the hardware performance requirements can be reduced, the data read time can be shortened, and the calculation performance can be improved. The user's operation information is obtained, and audio data within a certain period of time stored in the buffer memory is played back based on the operation speed, acceleration, etc., so that scratches can be simulated in real-time audio. If the finger displacement information is obtained in real time when processing each sampling point, it is accurate, but the overhead is too large. Moreover, general hardware does not have such a high refresh rate. Data is read in blocks, the displacement data of the user's operation is obtained before each read, the speed is calculated according to the displacement data, and the block data is played back according to the speed, so that the calculation amount can be reduced, the hardware performance requirements can be reduced, and the processing speed of audio data can be increased. The acceleration is calculated according to the displacement data and superimposed on the speed, and the block data is played back based on the calculated speed and acceleration, so that the smoothness of audio playback can be further improved.

[0063] According to some embodiments of the present application, step 2 comprises: determining whether the user is not performing an operation or has stopped performing an operation; If yes, play the real-time audio data at normal speed, If "no", then proceeding to step 3.

[0064] If the user does not perform any operation, real-time audio can be played normally, but if the user performs an operation, for example, moves the finger backwards when scratching, then releases the finger directly and resumes normal playback speed at the position where the finger was released, there will be a certain time difference, i.e., delay, between the next audio to be played and the real-time audio. Therefore, in the present invention, when the user releases his / her finger, this is equivalent to instantly fast-forwarding to the position to be currently played if the record had been played at normal speed all along, and then normal playback is resumed, thereby solving the problem of delay.

[0065] According to some embodiments of the present application, the pre-processing of the displacement data includes: Scaling the received displacement data values ​​includes scaling the range of receivable displacement data values ​​to be in the interval [0, number of sampling point groups in the buffer].

[0066] For example, the operation numerical value (drag displacement or speed data) input using scra_pos is preprocessed and the preprocessed numerical value is stored. The preprocessing method is as follows. scra_pos = Clamp(Abs(mouse_pos), 0, max_mouse_pos) * factor According to this method, the data generated by the operation is mapped into the range [0, less than or equal to the number of sampling point groups buffered in p_buffer], and the subsequent operations are based on it. It is mapped into this range because it can only be addressed within this range. Among them, max_mouse_pos needs to be set at initialization according to the actual situation. For example, if the operation is on the screen, the range can be limited to several hundred pixel points. After the range is exceeded, the representation in the DSP will have the position fixed at the maximum value. factor is also calculated at initialization. Its role is to scale the numerical value within the range of max_mouse_pos to the corresponding range of p_buffer[], and the algorithm is as follows: factor = (total_samps - buffer_size) / max_mouse_pos / channels Since the present invention can only address certain locations in the buffer, the displacement data is numerically scaled and mapped into a corresponding buffer address range to facilitate addressing based on the displacement data.

[0067] According to some embodiments of the present application, reading and playing one block at a corresponding position in the buffer based on the velocity and acceleration calculated by the displacement data includes: determining a block to be read based on the velocity and the acceleration; The method further includes performing processing on the sampling points in the block based on the velocity and the acceleration.

[0068] For example, reading and playing the next block based on the speed v and acceleration a includes determining the next block to be read based on the speed v and acceleration a (for example, determining the direction of the player's scratching action based on the positive or negative of the speed v and acceleration a to determine the addressing direction in the pointer buffer), performing block processing @block on the block, performing sampling point processing @sample on the sampling points in the block, and playing this next block based on the speed v and acceleration a (for example, sending the processed block to the sound card buffer, and since it is a block processed based on the corresponding speed and acceleration data, the sound card can read the sampling point data in the block, convert it into voltage at regular time intervals and output it, and then play this block based on the corresponding speed and acceleration data).

[0069] For example, the present invention is realized based on DSP (Digital Signal Processing). It is considered that any program that processes audio and video is executed using DSP, but generally, software for music production is realized by VST (a set of development tools developed by Steinberg). A runtime library generated by this technology cannot be directly applied to a game engine. It may be realized by a DSP interface provided by FMOD. FMOD is an audio middleware that integrates functions such as audio playback control, mixing, and effects into a game engine to facilitate audio manipulation and processing.

[0070] Typically 40,000+ sample points are sent to a sound card per second, and the operations performed by @sample are generally relatively computationally intensive, since processing every single point would necessarily put much higher demands on performance.

[0071] A common approach is to process one batch of points at a time and send it to the sound card, and this batch of sample points is called a block. A block is a memory space that stores a number of sample points. The audio program sends one block at a time to the sound card's buffer. The sound card reads the sample point data in the block one batch at a time, then converts it to a voltage at regular intervals and outputs it. @block is generally used to process variables that do not need to change with each sample point. Or, for variables that do change with each sample point, some pre-processing is done to reduce the amount of calculations in @sample.

[0072] When processing the memory that records the sampling points, it is also necessary to pay attention to the number of audio channels. The audio we normally hear is stereo, with two channels, meaning that the sound card outputs two sampling points for the left and right channels at the same time. For example, one segment of audio, which is assumed to have a time length of 1024 points, actually contains information for 2048 points. Audio engines generally support standards such as 5.1 channels and 7.1 channels, in which case there are more sampling points per time. In the case of multi-channel, the sampling point information of different channels is stored alternately in the memory, and sampling points of the same time are adjacent. For example, the audio of the left and right channels is represented in the memory as LRLRLRLR...

[0073] In the following, we use channels to represent the number of channels. The size of a block generally does not change during program execution. Therefore, it can be treated as a constant and can be obtained at initialization. In the following, we use block_size to represent the size of one block, that is, how many groups of sampling points it contains. The number of sample points actually contained in one block is represented by buffer_size = block_size * channels.

[0074] The above block processing @block and sampling point processing @sample may be as follows:

[0075] 1) Block processing @block This part is a calculation performed for each block. In fact, the process of processing the sampling points is also performed in @block. Traversing each sampling point in block by loop is the process of processing the sampling points.

[0076] a) Buffering the audio Since there is no real record, the played audio data does not reside in memory. Therefore, a buffer must be set up to buffer one segment of audio from the past (approximately 1-4 seconds is sufficient) for future use. At initialization, a value is given to the variable buffer_blocks to specify how many blocks to buffer. buffer_blocks = (int)Floor(buffer_time * samplerate / block_size) At initialization time, we request memory of fixed size total_samps to be used as a circular buffer, where: total_samps = buffer_size * buffer_blocks Below, we point to the buffer in question with p_buffer[], and use curr_block to record which block in the circular buffer we are currently in, then record the current input buffer (its pointer address is in_buffer), copy it to the corresponding position in p_buffer[], and mark the memory start position of the current buffer block using curr_buff_pos = curr_block * buffer_size. Copy(in_buffer, p_buffer, current_buff_pos, buffer_size) As each block is sent to the buffer, the value of curr_block is incremented by 1, and when it reaches a certain value it becomes zero, making p_buffer[] a circular buffer.

[0077] b) Receive and calculate operation information The input operation values ​​(drag displacement and speed data) are preprocessed and the preprocessed values ​​are stored by scra_pos. The preprocessing method is as follows: scra_pos = Clamp(Abs(mouse_pos), 0, max_mouse_pos) * factor) This method maps the numbers generated by the operation into the range [0, less than or equal to the number of sample point groups buffered in p_buffer], and the subsequent operations are based on it. It is mapped to this range because it can only be addressed within this range. Among them, max_mouse_pos needs to be set at initialization according to the actual situation. For example, if the operation is on the screen, the range can be limited to several hundred pixel points. After the range is exceeded, the representation in the DSP will have the position fixed at the maximum value. factor is also calculated at initialization. Its role is to scale the numbers within the range of max_mouse_pos to the corresponding range of p_buffer[], and the algorithm is as follows: factor = (total_samps - buffer_size) / max_mouse_pos / channels If we have the calculated scra_pos, it would be logical to output the sampling point at the position (sampling point position - rounded scra_pos) in the buffer as is, but doing so would lead to many problems, such as the following, and further processing such as interpolation and smoothing is required:

[0078] c) Smoothing - Difference and Second Difference With only first-order differentials, the addressing speed does not change for each block, and the addressing speed only differs between blocks. This means that the degree to which the waveform is dense, i.e. the change in pitch, is discontinuous. To the ear, there is a sense of many grains. With second-order differentials, the "rate of change in speed," or acceleration, is calculated. By superimposing it on the addressing position of each sampling point, smoothing can be introduced to each sampling point. This results in a waveform in which the speed changes continuously without sudden changes. The differential and second-order differential processes are as follows:

[0079] At initialization, the declaration variable scra_pos_prev is used to record the value of scra_pos in the previous @block. Its initial value is 0. The declaration variable scra_pos_prev_delta is used to record the value of scra_pos_delta in the previous @block. Its initial value is 0. These two values ​​are used in @sample, which will be described later.

[0080] In @block, first order differences are calculated. scra_pos_delta = (scr_pos - scra_pos_prev) / smooth Here, smooth is an adjustable coefficient whose value ranges from hundreds to tens of thousands. It is used to adjust the response sensitivity of the scratch.

[0081] For second-order differences, delta_delta = (scra_pos_delta - scra_pos_prev_delta) / block_size scra_pos_delta calculates the difference between the position information received by the two @blocks before and after, and differentiates the displacement to obtain the velocity. delta_delta calculates the difference between the two deltas before and after, and differentiates the velocity to obtain the acceleration.

[0082] Here, when delta decays to a certain threshold, it goes directly to zero, preventing the generation of explosive noise. The algorithm is as follows: if(scra_pos_delta< 0.001 &&scra_pos_delta> -0.001) { scra_pos_delta = 0; }

[0083] 2) Sampling point processing @sample Once the calculation in the @block is completed, we move on to the @sample part. This part is completed by two layers of loops in @block. The inner loop scans the number of audio channels and operates on each sampling point, and the outer loop scans the number of sample point groups in the block. The algorithm is as follows: for(int samp = 0; samp <block_size; samp++){ @sample for(int chan = 0; chan< channels; chan++){ @channel buffer[chan + samp * channels] } } where @sample and @channel are as follows:

[0084] a) @sample If no operations are performed, calculate which position in p_buffer[] is currently being used: curr_pos = current_buff_pos + (samp * channels) The two delta values ​​are updated. scra_pos_prev_delta += delta_delta; scra_pos_prev += scra_pos_prev_delta; Calculate core_pos, which uses curr_pos to calculate the buffer position to be read after the scratch drag displacement or velocity is introduced. core_pos = curr_pos - 2 * (int)Floor(scra_pos_prev); if (core_pos< 0) { core_pos = total_samps + curr_pos - 2 * (int)Floor(scra_pos_prev); } Since core_pos is rounded, if the current addressing position is between two sampling points, we connect them and take the corresponding value on the connection. We need to calculate the itp_factor variable for the subsequent linear interpolation operation. itp_factor = scra_pos_prev - Floor(scra_pos_prev);

[0085] b) @channel Specifically, when processing sampling points, a small space is also required to store the data currently being processed. This is requested at initialization and its size only needs to be one block. We denote it as o_buffer[] and use the variable samp_clamp below to mark the sampling point addressing range starting from 0 and give it a value at initialization. The algorithm is as follows: samp_clamp = total_samps - channels - 1; The addressing and difference operations are performed below and the result is stored in o_buffer[]. if(scra_pos_prev_delta == 0) { o_buffer[chan + samp * channels] = p_buffer[chan + core_pos]; } else if(scra_pos_prev_delta> 0) { if (core_pos > channels) o_buffer[chan + samp * channels] = p_buffer[chan + core_pos] * (1 - itp_factor) + p_buffer[chan + core_pos - channels] * itp_factor; else o_buffer[chan + samp * channels] = p_buffer[chan + core_pos] * (1 - itp_factor) + p_buffer[chan] * itp_factor; } else { if(core_pos <samp_clamp) o_buffer[chan + samp * channels] = p_buffer[chan + core_pos] * itp_factor + p_buffer[chan + core_pos + channels] * (1- itp_factor); else o_buffer[chan + samp * channels] = p_buffer[chan + core_pos] * itp_factor + p_buffer[chan + samp_clamp] * (1 - itp_factor); } First, scra_pos_prev_delta is used to determine whether or not a drag is currently occurring and the direction of the drag. If no dragging is occurring, the original data is sent directly to o_buffer[]. If a drag is occurring, the target point and the point just before or after the target point are searched for and interpolated depending on the direction of the drag.

[0086] 3) Returning data and updating markings After exiting the two for loops, it returns to the @block section, and immediately after completing the processing, it sends the data stored in o_buffer[] to the output. Copy(o_buffer, out_buffer, 0, buffer_size); Regarding updating variables, scra_pos_prev_delta = scra_pos_delta; current_block += 1; if(current_block>= buffer_blocks) current_block = 0; FIG. 2 shows a flow chart of this embodiment.

[0087] In the present invention, by addressing the buffer according to the user's operation and processing the sampling points in the block at the corresponding position, it is possible to easily play back the audio according to the speed and acceleration of the user's operation after outputting to the hardware such as a sound card, and the essence of scratching is the result of audio "alternating high-speed playback and slow playback", where the faster the playback speed, the higher the pitch, and the slower the playback speed, the lower the pitch, so that the audio can be played back according to the speed and acceleration of the user's operation to simulate the scratch effect. In addition, other processing may be performed at the same time.

[0088] According to some embodiments of the present application, the processing for the sampling points in the block includes linear interpolation.

[0089] For example, processing the sampling points in the block according to the speed and acceleration includes interpolating the sampling points. Generally, a Sinc interpolation algorithm may be adopted, or a low-pass filter may be performed using an IIR filter. However, considering that the performance is consumed significantly and that the sound itself is significantly distorted by the scratch algorithm, highly accurate interpolation is not required, and linear interpolation is preferably used. Of course, other suitable interpolation may be used.

[0090] In the present invention, if the user drags too slowly during operation, due to the limited number of sampling points in the buffer data, when generating a few new sampling points, the same points in the buffer data accessed within a certain period of time will be output, and the output waveform will resemble sawtooth, and a large amount of high-frequency components will be drawn in, causing the sound to have harsh parts. Interpolation can greatly reduce this problem, and linear interpolation has low performance requirements and is fast.

[0091] According to some embodiments of the present application, reading and playing a block from a corresponding position in the buffer based on the velocity and acceleration calculated by the displacement data further includes adding noise to the block before playing the block, the noise having a magnitude directly proportional to the velocity.

[0092] For example, before playing the audio data (e.g., before playing a block), we may add a small amount of noise to the processed audio, which makes the scratch more realistic. The noise modulation formula may be as follows: out_sample *= random() * scratch_speed_factor + const;

[0093] The noise source can essentially be a random number generator, and the random numbers that fit the average distribution have a frequency domain representation of white noise. One random number is generated for each sampling point, which is mapped to the desired range (corresponding to the magnitude of the noise percentage) and added to one constant const. It is necessary to ensure that the maximum value of the noise is less than or equal to 1 after adding this constant. That is, if the noise value is assumed to be a maximum of 0.2, the maximum value of the constant is 0.8. This variable is then related to the scratch speed so that it exhibits a positive correlation. In this way, the magnitude of the noise changes according to the scratch speed, completing the process of superimposing the noise. The reason for using multiplication instead of addition is that in the case of noise superimposed by addition, the volume only changes according to the scratch speed. On the other hand, when multiplication is used, this is called "modulation" in the field of signal processing, and the modulated signal is related to the original signal. The modulation method used here is amplitude modulation, or AM. Its specific expression is that the original signal is retained, and the frequency distribution of the noise is correlated with the frequency of the audio before modulation. Note that with noise modulation, the volume should be consistent when the audio is processed and when it is unprocessed.

[0094] In the present invention, by adding noise, it is possible to simulate the sound that is generated when a record needle rubs quickly against the surface of a record during scratching.

[0095] FIG. 3 is a diagram illustrating a real-time audio processing system according to an embodiment of the present invention.

[0096] As shown in FIG. 3, the system 300 includes an audio data receiving module 302, a displacement data acquiring module 304, and a playback module 306. The audio data receiving module 302 receives real-time audio data, stores the audio data in a buffer, divides the audio data into a number of blocks and stores them in the buffer, each block including a certain number of sampling points; The displacement data acquisition module 304 acquires displacement data of a user operation, The playback module 306 reads and plays one block at a corresponding position in the buffer according to the velocity and acceleration calculated by the displacement data, where the velocity is calculated according to the following formula:

[0097]

number

[0098] and acceleration

[0099]

number

[0100] where x denotes the pre-processed displacement data, and x p denotes the preprocessed displacement data received before playing the previous block, s denotes an adjustable coefficient for adjusting the sensitivity of the response to the user operation operation, and v p where a denotes the speed calculated before playing the previous block, and b denotes the number of sampling point groups in the block, where a sampling point group denotes a group of sampling points with the same time but different audio channels. The first embodiment is a method embodiment corresponding to the present embodiment, and the present embodiment can be implemented together with the first embodiment. The details of the related technology described in the first embodiment are still valid in the present embodiment, and will not be described here in detail to reduce redundancy. Correspondingly, the details of the related technology described in the present embodiment can also be used in the first embodiment.

[0101] According to some embodiments of the present application, a real-time audio processing apparatus is disclosed, the apparatus comprising a memory in which computer-executable instructions are stored and a processor, the processor being arranged to perform a real-time audio processing method by executing the instructions.

[0102] The first embodiment is a method embodiment corresponding to the present embodiment, and the present embodiment can be implemented together with the first embodiment. The details of the related technology described in the first embodiment are still valid in the present embodiment, and will not be described here in detail to reduce duplication. Correspondingly, the details of the related technology described in the present embodiment can also be used in the first embodiment.

[0103] According to some embodiments of the present application, a computer storage medium is disclosed that is encoded with a computer program, the computer program including instructions that are executable by one or more computers to perform a real-time audio processing method.

[0104] The first embodiment is a method embodiment corresponding to the present embodiment, and the present embodiment can be implemented together with the first embodiment. The details of the related technology described in the first embodiment are still valid in the present embodiment, and will not be described here in detail to reduce duplication. Correspondingly, the details of the related technology described in the present embodiment can also be used in the first embodiment.

[0105] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may be implemented in the form of instructions or programs stored or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors, etc. When the instructions or programs are executed by a machine, the machine may perform the various methods described above. For example, the instructions may be distributed over a network or other computer-readable media. As such, a machine-readable medium may include, but is not limited to, any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, such as, but not limited to, a floppy disk, an optical disk, optical disk read-only memories (CD-ROMs), magnetic disk, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), magnetic or optical cards, or flash memory or tangible machine-readable memory for transmitting network information in electrical, optical, acoustic or other forms of signals (e.g., carrier waves, infrared signals, digital signals, etc.). As such, a machine-readable medium includes any machine-readable medium suitable for storing or transmitting any form of electronic instructions or machine (e.g., computer) readable information.

[0106] Although the embodiments of the present application have been described in detail above with reference to the accompanying drawings, the use of the technical means of the present application is not limited to the various applications mentioned in the embodiments of the present invention, and various structures and modifications can be easily implemented with reference to the technical means of the present application, and various beneficial effects mentioned in the present specification can be achieved. Any changes made within the scope of knowledge possessed by a person skilled in the art without departing from the spirit of the present application should be included in the scope of coverage of the present patent.

Claims

1. A real-time audio processing method for simulating scratch in a game, for use in a terminal, comprising: A step 1 of receiving real-time audio data, storing the audio data in a buffer, storing the audio data in the buffer in a plurality of blocks, each block including a certain number of sampling points; A step 2 of acquiring displacement data of a user operation; The first difference of the displacement data is a velocity, and the second difference of the displacement data is an acceleration, and one of the blocks at a corresponding position in the buffer is read and played according to the velocity and the acceleration, where the first difference is expressed by the following formula: [0010] and the second difference [0025] where x represents the pre-processed displacement data, and x p denotes the preprocessed displacement data received before playing the previous block, s denotes an adjustable coefficient for adjusting the sensitivity of the response to a user manipulation operation, and d p and 3) where b denotes the first difference calculated before reproducing the previous block, and b denotes the number of sampling point groups in the block, where the sampling point group denotes a group of sampling points having the same time but different audio channels.

2. Step 2 is to determine whether the user is not performing an operation or has stopped performing an operation; If yes, play said audio data in real time at normal speed; 2. The method of claim 1, further comprising: if "no," proceeding to step 3.

3. Reading and playing one of the blocks at a corresponding position in the buffer based on the speed and the acceleration, determining the block to be read based on the velocity and acceleration; 3. The method of claim 2, further comprising: performing processing on the sampling points in the block based on the velocity and acceleration.

4. 4. The method of claim 3, wherein performing the processing on the sampling points in the block comprises linear interpolation.

5. The pre-processing of the displacement data includes: The range of the receivable values ​​of the displacement data is within the interval [0, the number of sampling point groups in the buffer] by scaling the values ​​of the received displacement data.

2. The method of claim 1, wherein

6. A method according to any one of claims 1 to 5, characterized in that reading and playing one of the blocks from a corresponding position in the buffer based on the velocity and the acceleration further comprises adding noise to the block, the magnitude of which is directly proportional to the velocity, before playing the block.

7. A real-time audio processing system for simulating scratches in a game, comprising: an audio data receiving module, a displacement data acquiring module, and a playback module; The audio data receiving module receives real-time audio data, stores the audio data in a buffer, and stores the audio data in the buffer in a plurality of blocks, each of the blocks including a certain number of sampling points; The displacement data acquisition module acquires displacement data of a user operation; The playback module is configured to read and play one of the blocks at a corresponding position in the buffer according to the velocity and the acceleration, and to calculate the first difference of the displacement data as a velocity and a second difference of the displacement data as an acceleration, and to calculate the first difference according to the following formula: [0030] and the second difference [0045] where x represents the pre-processed displacement data, and x p denotes the preprocessed displacement data received before playing the previous block, s denotes an adjustable coefficient for adjusting the sensitivity of the response to a user manipulation operation, and d p denotes the first difference calculated before reproducing the previous block, and b denotes the number of sampling point groups in the block, where the sampling point groups represent a group of sampling points having the same time but different audio channels.

8. 7. A real-time audio processing device for simulating scratches in games, comprising: a memory having computer executable instructions stored therein; and a processor arranged to execute said instructions to implement a real-time audio processing method for simulating scratches in games as claimed in any one of claims 1 to 6.

9. 10. A computer storage medium encoded with a computer program, the computer program comprising one or more computer executable instructions for implementing a real-time audio processing method for simulating a scratch in a game as claimed in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Control method and device of music playing

    CN102842319A

  • Audio information processing method and device, and storage medium

    CN111782865A

  • Music direction game machine, direction operation instruction system for music direction game and computer readable storage medium recording program for game

    JP2000350861A

  • Musical sound control device, musical sound control method, program, and electronic musical instrument

    JP2016065903A