Audio generation method, device and equipment and computer readable storage medium

Abstract

The invention discloses an audio generation method, device and equipment and a computer readable storage medium, and is applied to the technical field of computers, and the method comprises the steps: obtaining a plurality of preset tone parameters; each tone parameter comprises oscillator control information, filter control information and effector control information; according to the sound production information and the timbre parameters, performing audio stream generation by combining an oscillator, a filter and an effector to obtain a plurality of audio clips; wherein the sounding information comprises pitch information and duration information; and superposing the plurality of audio clips to obtain a target audio. According to the invention, the tone parameters can be set according to requirements to render the audio, so that only the tone parameters need to be updated for expansion of new tone and modification of the tone, and therefore, the capability of expanding the tone and the experience feeling of a user are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an audio generation method, device, equipment and computer-readable storage medium. Background Art

[0002] MIDI (Musical Instrument Digital Interface) is a digital music communication protocol used for communication between electronic musical instruments. The MIDI protocol controls various parameters of musical instruments, such as pitch, volume, timbre, sound effects, etc., by sending digital instructions, thereby realizing the creation, performance and recording of music. MIDI, a digital music description standard, can generate corresponding audio waveforms according to the instructions in the MIDI file through MIDI chips, etc. The method of rendering and generating audio based on MIDI chip simulators relies on MIDI format communication and has poor interactive compatibility with other data structures. Secondly, the rendering of MIDI chips uses waveform rendering algorithms to generate specific instrument sounds, but users cannot control these sounds or add the sounds they want to play.

[0003] Therefore, the current MIDI-based audio generation method has the technical problem of single generated timbre. Summary of the invention

[0004] In view of this, an object of the present invention is to provide an audio generation method, device, equipment and computer-readable storage medium, which solve the technical problem of single timbre of audio generation in the prior art.

[0005] In order to solve the above technical problems, the present invention provides an audio generation method, comprising:

[0006] Acquire a plurality of preset timbre parameters; wherein each of the timbre parameters includes oscillator control information, filter control information and effector control information;

[0007] Perform audio streaming generation according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain a plurality of audio clips;

[0008] A plurality of the audio clips are superimposed to obtain a target audio.

[0009] Optionally, the timbre parameters also include waveform information; wherein the waveform information includes at least one of self-made wavetable information, low-frequency oscillator oscillation information and sequencer information; the waveform information is the timbre parameters required by the oscillator.

[0010] Optionally, the audio stream generation is performed according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain multiple audio clips, including:

[0011] Processing the oscillator control information using the oscillator to obtain audio sampling point data;

[0012] Processing the audio sampling point data using the filter based on the filter control information to obtain waveform data;

[0013] The waveform data is processed by the effector based on the effector control information to obtain the audio segment.

[0014] Optionally, after the plurality of audio clips are superimposed to obtain the target audio, the method further includes:

[0015] Input the target audio into an audio buffer;

[0016] Determining the interface parameter duration according to the downstream interface type; wherein the downstream interface type is the type of the interface used to export the audio;

[0017] The target audio in the audio buffer is spliced ​​according to the interface parameter duration in data frames to obtain a long time domain; wherein the long time domain is undivided smooth audio.

[0018] Optionally, the method further includes:

[0019] Determine whether the duration of each pitch information corresponding to the target audio is greater than the interface parameter duration;

[0020] When the duration of the current pitch information is greater than the interface parameter duration, the cross-domain information corresponding to the long time domain is stored in an array according to the pitch information, so as to control the end time of the target audio according to the cross-domain information in the array; wherein the cross-domain information is the pitch information when two interface parameter durations in the long time domain are spliced.

[0021] Optionally, the audio stream generation is performed according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain multiple audio clips, including:

[0022] Encapsulating the plurality of timbre parameters with the oscillator, the filter and the effector respectively to obtain a plurality of single-tone sound processing models; wherein each single-tone sound processing model is a model for generating a single timbre;

[0023] Encapsulating the three-stage envelope information into the single-tone sound processing model; wherein the three-stage envelope information is a parameter describing the signal change process over time;

[0024] Accordingly, the audio stream is generated according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain a plurality of audio clips, including:

[0025] Audio streaming is generated according to the timbre parameters using a single-tone sound processing model that encapsulates the three-stage envelope information to obtain the multiple audio clips.

[0026] Optionally, in the process of using the oscillator for processing, the method further includes:

[0027] The wavetable index of the oscillator is set by the number of beats per minute, and the generated waveform data is updated based on the wavetable index; wherein the wavetable index is a file that determines the order of waveform samples.

[0028] Optionally, before generating audio streams according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain a plurality of audio clips, the method further includes:

[0029] According to different requirements, some or all of the oscillators are selected from the oscillators, and some or all of the filters are selected as oscillators for audio streaming generation; wherein, when there are multiple oscillators, each oscillator supports a different oscillation method, and when there are multiple filters, each filter supports a different filtering algorithm.

[0030] The present application also provides an audio generation device, comprising:

[0031] A timbre parameter acquisition module, used to acquire a plurality of preset timbre parameters; wherein each of the timbre parameters includes oscillator control information, filter control information and effector control information;

[0032] An audio clip generation module, used for generating audio streams according to the timbre parameters in combination with an oscillator, a filter and an effector, to obtain a plurality of audio clips;

[0033] The audio determination module is used to superimpose multiple audio clips to obtain target audio.

[0034] The present application also provides an audio generation device, comprising:

[0035] Memory for storing computer programs;

[0036] A processor is used to implement the steps of the above-mentioned audio generation method when executing the computer program.

[0037] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned audio generation method are implemented.

[0038] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned audio generation method when executed by a processor.

[0039] It can be seen that the present invention obtains multiple preset timbre parameters; wherein the timbre parameters include oscillator control information, filter control information, and effector control information; audio streaming is generated according to the timbre parameters in combination with the oscillator, filter, and effector to obtain multiple audio clips; multiple audio clips are superimposed to obtain target audio. The beneficial effects of the present invention are: the present invention can design the timbre parameters by itself, so as to generate audio according to the timbre parameters in combination with the oscillator, filter, and effector. Since the timbre parameters can be designed by itself, the expansion of new timbres and the modification of existing timbres only need to update the timbre parameter data, and the timbre has great plasticity potential and has the ability to simulate a large number of timbres.

[0040] In addition, the present invention also provides an audio generating device, equipment and computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 A flowchart of an audio generation method provided by an embodiment of the present invention;

[0043] Figure 2 An example flow chart of an audio generation method provided by an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of an oscillator processing provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of a Clamp method provided in an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of a single tone sound processing module provided in an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of processing a long time domain provided by an embodiment of the present invention;

[0048] Figure 7 A schematic diagram of the structure of an audio generating device provided by an embodiment of the present invention;

[0049] Figure 8 A schematic diagram of the structure of an audio generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Please refer to Figure 1 , Figure 1 A flowchart of an audio generation method provided by an embodiment of the present invention. The method may include:

[0052] S101, obtaining a plurality of preset timbre parameters; wherein each timbre parameter includes oscillator control information, filter control information and effector control information.

[0053] The execution subject of this embodiment is an electronic device, for example, the electronic device can be a computer; or the electronic device in this embodiment can also be a device composed of an oscillator, a filter, etc. The timbre parameters in this embodiment are set by the user according to the needs. This embodiment does not limit the specific timbre parameters. For example, the timbre parameters in this embodiment may include oscillator control information (harmonic richness, envelope information, etc.), filter control information, effector control information, waveform modulation information; or the timbre parameters in this embodiment may include oscillator control information, filter control information, effector control information and waveform modulation information and self-made wavetable information; or the timbre parameters in this embodiment may include oscillator control information, filter control information, effector control information. The oscillator control information in this embodiment mainly includes frequency control information, phase control information, amplitude control information and waveform control information, etc.; the filter control information in this embodiment refers to the parameters and instructions used to control the behavior and performance of the filter; the effector control information in this embodiment refers to the various parameters used to adjust and control the effector to produce specific sound effects; the timbre parameters in this embodiment can also include waveform modulation information, and the waveform modulation information in this embodiment refers to the various parameters and settings used to control and adjust the waveform modulation process so as to produce specific signal characteristics according to application requirements.

[0054] It should be further explained that in order to improve the effect of audio rendering, the above-mentioned timbre parameters may also include waveform information; wherein the waveform information includes at least one of self-made wavetable information, low-frequency oscillator oscillation information and sequencer information; the waveform information is the timbre parameter required by the oscillator. The self-made wavetable information, low-frequency oscillator oscillation information and sequencer information in this embodiment are set according to user needs. The self-made wavetable information in this embodiment refers to digitized sound data sampled using real musical instrument timbre, which is stored in a wavetable file for calling synthesis during audio playback. The low-frequency oscillator oscillation information in this embodiment refers to a low-frequency signal or waveform generated by an electronic circuit. The sequencer information in this embodiment is information for producing arpeggio effects. The waveform information in this embodiment can be input to the oscillator so that the oscillator performs processing according to the waveform information.

[0055] S102, generating audio streams according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain a plurality of audio clips.

[0056] The oscillator in this embodiment is the core of generating audio signals, and it generates corresponding waveforms according to pitch information. In complex synthesis systems, different types of oscillators (such as sine waves, square waves, sawtooth waves, etc.) may be used to enrich the color of the sound. The filter in this embodiment is used to shape the spectrum of the audio signal, remove or enhance specific frequency components, and thus change the timbre. For example, a low-pass filter can remove high-frequency components to make the sound softer; a high-pass filter can remove low-frequency components to make the sound clearer. The effector in this embodiment is used to add specific sound effects, such as reverberation, delay, distortion, etc., to increase the spatial sense and dynamic range of the sound. The streaming generation in this embodiment means that the audio is continuously generated according to a preset time period, and the generated audio stream needs to be cut or segmented, that is, the start and end points of each segment are determined according to the duration information and other music parameters. In this embodiment, the oscillator control information in the timbre parameters is input into the oscillator, the filter control information is input into the filter, and the effector control information is input into the effector. After the code angle is input, the direct parameters are defined. In the generation process, this embodiment can be generated in combination with the sound information including pitch information and duration information. The pitch information in this embodiment is usually directly related to the frequency of the oscillator. The higher the frequency, the higher the generated tone. In digital music synthesis, this is usually achieved by changing the frequency of the oscillator to achieve different pitches. The duration information in this embodiment determines the duration of the note, which in audio synthesis usually affects the output time of the oscillator by controlling the envelope or modulation depth. The duration information determines when the oscillator starts and ends generating a signal of a specific frequency. This solution can achieve real-time rendering output through streaming generation and can be deployed on the end side. Unlike MIDI, this solution does not need to maintain a persistent audio stream. It only allocates resources when a call signal occurs, and the termination process can be ended freely.

[0057] It should be further explained that, in order to improve the accuracy of processing using oscillators, filters and effectors, the above-mentioned audio streaming generation based on the sound information and timbre parameters in combination with oscillators, filters and effectors to obtain multiple audio clips can include: using an oscillator to process the oscillator control information to obtain audio sampling point data; using a filter to process the audio sampling point data based on the filter control information to obtain waveform data; using an effector to process the waveform data based on the effector control information to obtain an audio clip. The oscillator in this embodiment is an electronic component or device that converts direct current power into alternating current power with a certain frequency. The filter in this embodiment is a device or circuit that has a processing function on a signal. The effector in this embodiment is an electronic instrument dedicated to producing various sound effects, which is used to change the waveform of the original sound, modulate or delay the phase of the sound wave, enhance the harmonic component of the sound wave, etc., to produce various special sound effects. The number of oscillators is not limited in this embodiment. For example, the oscillator in this embodiment can be single; or the oscillator in this embodiment can be multiple. After the oscillator control information (mainly harmonic richness, envelope information, etc.) is input into the oscillator, audio sampling point data will be obtained, and this waveform data is the original waveform data. The filter in this embodiment can modify the filter control information in the audio sampling point data by changing different frequency bands to obtain waveform data. In this embodiment, waveform data refers to digital information representing sound waves or other vibration waveforms, which is usually composed of a series of sampling points, and each sampling point contains information such as amplitude (sometimes also phase). In this embodiment, the waveform data is processed by the effector, which can include distortion, delay, reverberation, flanging and other effector chains. This embodiment improves the accuracy of processing using oscillators, filters and effectors by providing a method for continuous processing using oscillators, filters and effectors.

[0058] It should be further explained that in order to ensure that the beginning and end of each sound change regularly, audio streaming is generated according to the timbre parameters in combination with oscillators, filters and effects to obtain multiple audio clips, which may include: encapsulating multiple timbre parameters with oscillators, filters and effects respectively to obtain multiple single-tone sound processing models; wherein each single-tone sound processing model is a model for generating a single timbre; encapsulating three-stage envelope information into the single-tone sound processing model; wherein the three-stage envelope information is a parameter describing the process of signal change over time; accordingly, audio streaming is generated according to the timbre parameters in combination with oscillators, filters and effects to obtain multiple audio clips, which may include: audio streaming is generated according to the timbre parameters using a single-tone sound processing model encapsulating the three-stage envelope information to obtain multiple audio clips. In this embodiment, using the timbre parameters, oscillators, filters and effects as a single-tone sound processing model means that each timbre parameter, oscillator, filter and effector is used as a model to generate audio as a whole. The three-stage envelope in this embodiment acts on the entire audio. The three-stage envelope state in this embodiment is a commonly used method in audio processing and music production, which is used to describe and control the four stages of attack, decay, sustain, and release of sound, usually referred to as the ADSR model. It can be understood that when the sound processing module starts receiving parameters, it will automatically enter a three-stage envelope state process, which mainly simulates the attack of the sound. When the sound processing module ends, it will automatically enter the decay process. The three-stage envelope state in this embodiment is processed in the oscillator and filter, and the data after the three-stage envelope is sent to the effector for processing.

[0059] It should be further explained that, in order to ensure the continuity of the basic waveform, the process of using the oscillator for processing may also include: setting the wavetable index of the oscillator by the beats per minute, and updating the generated waveform data based on the wavetable index; wherein the wavetable index is a file that determines the order of the waveform samples. The wavetable index in this embodiment needs to be used every time the frame level is generated, and is used as a parameter of the oscillator at the beginning of the generation. The beats per minute (BPM) in this embodiment is used to describe the rate or rhythm speed of the music. It can be understood that the wavetable is a file containing multiple waveform samples, each sample corresponding to a specific index value. Based on the current wavetable index, the corresponding waveform sample is read from the wavetable and used to generate an audio signal. As the rhythm of the music changes, the wavetable index will be dynamically updated, thereby generating different waveform data. This embodiment sets the index update of the oscillator wavetable, so that the continuity of the basic waveform can be ensured.

[0060] It should be further explained that, in order to improve the timbre rendering effect, before the audio stream is generated according to the timbre parameters combined with the oscillator, filter and effector to obtain multiple audio clips, the following steps are also included:

[0061] According to different needs, select some or all of the oscillators from the oscillator, and select some or all of the filters as oscillators for audio streaming generation; wherein, when there are multiple oscillators, each oscillator supports different oscillation methods, and when there are multiple filters, each filter supports different filtering algorithms. It can be understood that in order to ensure its ability and diversity in generating timbre, this method uses a total of three oscillator modules for waveform generation, and some or all of the oscillators can be selected for use according to different needs. Each module supports a variety of different oscillation methods, including analog oscillators (which can generate basic sine waves, square waves, etc.), frequency modulation oscillators (which can control waveform information through modulation signals), wavetable oscillators (which can accept personalized and customized waveform wavetables), and other digital signal processing algorithms. The algorithm will have some built-in wavetable files of basic waveforms, which is convenient for basic waveform superposition, and the waveform can be deformed and modulated according to the information when oscillating. In this embodiment, three filters can be used to correct the waveform, and each filter also supports different filtering algorithms. The filter can modify the target audio by changing different frequency bands. This solution can use a biquad filter to achieve basic high-pass, low-pass and band-pass functions. At the same time, this method integrates unique filtering methods such as comb filter and formant filter to quickly shape a more unique timbre. Like the oscillator, the parameters of the filter are all controllable.

[0062] S103: superimpose multiple audio clips to obtain target audio.

[0063] In this embodiment, superimposing multiple audio clips means accurately aligning all audio clips in time according to pitch to ensure that they belong to the same audio. After obtaining the target audio, this embodiment can input the target audio into the audio buffer. In this embodiment, the audio data stored in the audio buffer can be called and transmitted through various downstream interfaces (such as I2S (Inter-IC Sound, serial bus interface standard), USB (Universal Serial Bus, widely used serial bus standard), HDMI (High-Definition Multimedia Interface, digital interface supporting high-resolution video transmission) to meet the needs of different devices and applications. These interfaces should have the characteristics of high data transmission rate and low latency to ensure the real-time and accuracy of audio data.

[0064] It should be further explained that in order to achieve streaming generation, after superimposing multiple audio clips to obtain the target audio, it can also include: inputting the target audio into the audio buffer; determining the interface parameter duration according to the downstream interface type; wherein the downstream interface type is the type of the interface used to export the audio; according to the interface parameter duration, the target audio in the audio buffer is spliced ​​according to data frames to obtain a long time domain; wherein the long time domain is undivided smooth audio.

[0065] This embodiment requires inputting the target audio into the audio buffer. The long time domain in this embodiment is the concept of time length without any segmentation. The long time domain is smooth audio without segmentation. The user can not perceive that it is output by frame, and the user perceives smooth audio. This embodiment can determine the interface parameter duration according to the downstream interface type. For example, the current downstream interface type is a short time domain of 5 seconds, and the interface parameter duration corresponding to (5 seconds x 441000 sampling points) divided by 1024 (the number of sampling points in each frame) is 214. The common types of interfaces for exporting audio in this embodiment include: PCM (Pulse Code Modulation): an uncompressed audio format commonly used to transmit audio data between professional audio equipment and software. WAV: lossless audio format, widely used to store uncompressed audio data. MP3 / AAC: lossy compressed audio format, suitable for music distribution and streaming services. USB / Bluetooth: an interface for transmitting audio data to external devices. This embodiment uses a long time domain related to the interface parameter duration, so that it can be applied to various downstream interface types for exporting audio, so as to facilitate the call of the downstream interface.

[0066] It should be further explained that in order to improve the efficiency of audio generation, the above method may also include: determining whether the duration of each pitch information corresponding to the target audio is greater than the interface parameter duration; when the duration of the current pitch information is greater than the interface parameter duration, storing the cross-domain information corresponding to the long time domain in the array according to the pitch information, so as to control the end time of the target audio according to the cross-domain information in the array; wherein the cross-domain information is the pitch information when the two interface parameter durations are spliced ​​in the long time domain. In this embodiment, the duration (duration information) of the pitch corresponding to the pitch information is the time length for which the pitch information remains stable within a certain period of time, or the time span for a single note or sound to keep its pitch unchanged. For ease of understanding, if the length of the data frame is 512 sampling points, the receiving interface parameter duration is 441000 sampling points, and when a certain interface receives a sound signal with a pitch of 43 that is greater than an interface duration, the data will be recorded in the format of (pitch, and the number of frames that need to be continued) in a map array that is updated every interface duration. The map array is a data structure that maps unique keys to values. In this embodiment, recording multiple audio clips in an array according to pitch information means that data will be recorded in the format of pitch in an array that is updated every interface duration. It can be understood that in the field of audio processing, pitch information is an important parameter that determines the fundamental frequency of the note, that is, the frequency of the note. By extracting the pitch information of the audio clips and recording them in an array, these data can be conveniently sorted, analyzed or processed. This embodiment can store the audio clips in the array according to the pitch information when the duration of the pitch information is greater than the interface parameter duration. When it is not greater than the interface parameter duration, the action of storing in the array is not performed, and the generated audio content is read normally, thereby reducing the frequency of storage, and controlling the end time of the target audio according to the cross-domain information in the array, thereby improving the efficiency and accuracy of audio generation.

[0067] The audio generation method provided by the embodiment of the present invention may include: S101, obtaining a plurality of preset timbre parameters; wherein each timbre parameter includes oscillator control information, filter control information and effector control information; S102, generating audio streams according to the timbre parameters in combination with the oscillator, filter and effector to obtain a plurality of audio clips; S103, superimposing the plurality of audio clips to obtain the target audio. Compared with the current method of controlling timbre parameters based on the MIDI protocol, which makes it impossible to change the timbre parameters, the present invention can design the timbre parameters by itself, thereby generating audio according to the timbre parameters. Since the timbre parameters can be set by itself according to the needs, the expansion of new timbres and the modification of existing timbres only need to update the text parameter data, and the plasticity potential of the timbre is huge, and it has the ability to simulate a large number of timbres.

[0068] In order to make the present invention easier to understand, please refer to Figure 2 , Figure 2 An example flow chart of an audio generation method provided by an embodiment of the present invention may specifically include:

[0069] S201, construct multiple timbre parameters, and use the multiple timbre parameters, oscillators, filters and effects to construct multiple single-tone sound processing units; the timbre emitted by each single-tone sound processing unit is different.

[0070] In this embodiment, a set of parameters will be constructed for each timbre and saved in XML (eXtensible Markup Language) format. Specifically, a complete timbre parameter should contain information in the following formats: oscillator control information, filter control information, effect control information, and waveform modulation information. Some timbres may require waveform information such as homemade wavetable information, low-frequency oscillator (Lfo) oscillation information, sequencer information, etc. The storage method uses "id" as the parameter name identifier and "value" as the numerical information. The storage format of the numerical information is text, but it will be converted into the corresponding data format after reading. A basic example of timbre parameter representation: id="osc1-freq" value="0.99999341114"; where id="osc1-freq" represents the wave length of the first envelope, and value="0.99999341114" represents the numerical form corresponding to the timbre parameter text.

[0071] In this embodiment, in order to ensure the ability and diversity of generating timbre, three oscillator modules are selected for waveform generation. Some or all of the oscillators can be selected for use according to different needs. Each module supports a variety of different oscillation methods, including analog oscillators (which can generate basic sine waves, square waves, etc.), frequency modulation oscillators (which can control waveform information through modulation signals), wavetable oscillators (which can accept personalized customized waveform wavetables) and other digital signal processing algorithms. The algorithm will have some built-in wavetable files of basic waveforms to facilitate basic waveform superposition. When oscillating, the waveform can be deformed and modulated according to the information. After determining the wavetable, the audio sampling point data can be extracted according to the index value. All oscillator parameters (mainly base frequency control, modulation ratio, etc.) and the type of oscillator can be controlled and changed. After the timbre parameters are input into the oscillator, the audio sampling point data will be obtained. This waveform data is the original waveform data. For ease of understanding, please refer to Figure 3 , Figure 3 A schematic diagram of an oscillator processing provided by an embodiment of the present invention, wherein the parameter data is parameter data corresponding to the input timbre parameters.

[0072] In this embodiment, the output of the oscillator will be passed to the filter group. In this process, three filters are also reserved for waveform correction, and each filter also supports different filtering algorithms. The filter can modify the target audio by changing different frequency bands. This solution uses a biquad filter to implement basic high-pass, low-pass and band-pass functions. At the same time, this method integrates unique filtering methods such as comb filter and formant filter to quickly shape a more unique timbre. Like the oscillator, the parameters of the filter are all controllable. After the audio data point output of the oscillator is processed by the filter group, waveform data with harmonic rules that are relatively similar to the target timbre can be obtained.

[0073] This embodiment also provides a series of effect processing solutions to adjust the generated sound, including distortion, delay, reverb, flanger and other effect chains. The essence of the distortion effect is to selectively destroy the waveform at a certain threshold, and this destruction method includes clamping, folding, zeroing and other methods. Figure 4 A schematic diagram of a clamping method provided by an embodiment of the present invention. In the figure, Threshold represents a threshold value, wherein a horizontal line represents a volume threshold value to be clamped, that is, Threshold, and the entire curve represents a waveform diagram, with the horizontal axis being time and the vertical axis being volume. Delay, flanger, and reverberation algorithms are essentially the final signal obtained by fusing the signal (wet signal) generated after modifying the original signal with the original signal (dry signal). Delay is the buffering and delay processing of audio in the time domain, which is the basic sound effect implementation. Specifically, the user can control the delay time, delay energy, and delay duration to regulate the value of the feedback signal after energy conversion and filtering. The implementation of reverberation is very similar to delay, both of which are obtained by processing dry signals to obtain wet signals and superimpose them, and their function is also to simulate the formation of reflected sound to construct a sound field environment. The flanger algorithm adds the signal that generates the time difference to the original signal, but the difference is that the time difference is a value that changes, and the resulting phase cancellation effect will also change accordingly. The waveform data generated by the filter will be processed by the effector in turn and finally sent to an audio buffer for streaming operations. Users can also control the switches and relative positions of each effector through parameters.

[0074] S202, activating multiple single-tone sound processing units in the single-tone sound processing module according to a preset time period, and cycling through multiple activated single-tone sound processing units in sequence to obtain multiple audio clips.

[0075] This application only requires timbre parameters, pitch information and duration information for streaming generation, and can be relatively well ported to different upstream tasks. This solution can achieve real-time rendering output and can be deployed on the end side. Unlike MIDI, this solution does not need to maintain a persistent audio stream. It only allocates resources when a call signal occurs, and the termination process can be ended freely. In this solution, the smallest waveform generation unit is a single-tone sound processing model (unit sound processing module). A single-tone sound processing model can emit a single tone with a single timbre. Each sounding single tone encapsulates the above-mentioned timbre parameter reading, oscillator, filter, and effector methods, and independently receives external information parameters. In the overall sound processing module, multiple activated single-tone sound processing models (up to 24) will be cycled in sequence and the audio will be superimposed. The final result will be input into the audio buffer for downstream interface calls. For ease of understanding, please refer to Figure 5 , Figure 5 A schematic diagram of a single-tone sound processing module provided for an embodiment of the present invention. It can be seen from the figure that multiple single-tone sound processing modules constitute a sound processing module, wherein each single-tone sound processing module includes timbre parameters, oscillators, filters, and effects. Users can customize the input timbre parameter information (including oscillator control information, filter control information, effector control information, waveform modulation information, self-made wavetable information, low-frequency oscillator oscillation information, sequencer information) as well as duration information and pitch information. The input parameters are passed through the oscillator, filter, and effector to generate a frame-level audio stream file that enters the audio buffer. The streaming output is adaptively performed according to the time domain size of the interface.

[0076] In this embodiment, the index update of the oscillator wave table can be set first through external BPM information, so as to ensure the continuity of the basic waveform. Secondly, the ADSR envelope design is used to ensure that the beginning and end of each sound change regularly. When the sound processing module starts receiving parameters, it will automatically enter a three-stage envelope state process, which mainly simulates the onset of the sound. When the sound processing module ends, it will automatically enter the attenuation process. In the effect chain composed of the effector, since it is necessary to mix signals with time differences, there is also a buffer for dry audio inside the effect chain and a parameter smoothing that is automatically updated to process the change law of the effector based on the time domain.

[0077] S203, superimposing the multiple audio clips to obtain target audio, and inputting the target audio into the audio buffer.

[0078] S204: Determine the interface parameter duration according to the downstream interface type.

[0079] S205, splicing the target audio in the audio buffer according to the interface parameter duration according to the data frames to obtain a long time domain, and playing the long time domain as audio.

[0080] It is understandable that, since the input of the interface information may not be a streaming input at the frame level, this solution also designs a two-layer timing logic in the time domain generation logic of the data frame to facilitate the transmission of interface parameter information. The overall time domain logic is: the interface releases a generation signal to the generation module according to a controllable (integer multiple of the frame) interface parameter time interval, and the sound processing module (multiple single-tone sound processing modules) decomposes the generation task into the output of the data frame and responds to the upper layer's fetching command through the audio buffer. It should be noted that this embodiment can be combined with the macro definition of basic parameters (some specific parameters) (binding multiple basic parameters as a group for collective modification) as an interactive interface (interactive interface refers to the interface with the user), making it more controllable. This solution proposes a streaming timbre rendering method from parameters to audio, which can become a downstream interface for symbolic audio generation to make the audio generation process more perfect.

[0081] In the time domain processing at the unit interface parameter level in this embodiment, in addition to processing the relevant information received by the interface, the unfinished audio end requirements inherited from the past are also retrieved to control their end time. For easier understanding, please refer to Figure 6 , Figure 6 A schematic diagram of processing a long time domain provided by an embodiment of the present invention. Figure 6 The interface cross-domain information in refers to information that does not exist independently in a single interface time domain. The interface parameter duration is definable. The interface cross-domain information will continuously affect the audio of each interface segment. In this embodiment, the interface parameter duration is determined according to demand. For example, 5 seconds corresponds to 210 frames. If a more common interface is used, the short time domain of an interface is 5 seconds, and 5 seconds x 441000 sampling points divided by 1024 (the number of sampling points in each frame) corresponds to an interface parameter duration of 214.

[0082] This solution mainly includes four aspects: data reading, waveform generation, effect processing, and streaming logic. The embodiment of the present invention only needs pitch information and duration information for streaming generation, and can be relatively well transplanted to different upstream tasks. At the same time, this method achieves lightweight through the characteristics of parameter synthesis, and does not require waveform sampling. This method only needs to update text parameter data for the expansion of new timbres and the modification of existing timbres, and the plasticity potential of the timbre is huge, and it has the ability to simulate a large number of timbres. This method can become a downstream interface for symbol generation tasks, and can realize real-time timbre rendering and audio generation at the frame level.

[0083] The audio generating device provided in an embodiment of the present invention is introduced below. The audio generating device described below and the audio generating method described above can refer to each other.

[0084] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of an audio generation device provided by an embodiment of the present invention may include:

[0085] The timbre parameter acquisition module 100 is used to acquire a plurality of preset timbre parameters; wherein each of the timbre parameters includes oscillator control information, filter control information and effector control information;

[0086] An audio clip generation module 200, configured to generate audio streams according to the timbre parameters in combination with an oscillator, a filter and an effector, to obtain a plurality of audio clips;

[0087] The audio determination module 300 is used to superimpose multiple audio clips to obtain target audio.

[0088] Further, based on the above embodiment, the timbre parameters also include waveform information; wherein the waveform information includes at least one of self-made wavetable information, low-frequency oscillator oscillation information and sequencer information; the waveform information is the timbre parameters required by the oscillator.

[0089] Further, based on the above embodiment, the above audio segment generating module 200 may include:

[0090] An oscillator processing unit, used to process the oscillator control information using the oscillator to obtain audio sampling point data;

[0091] A filter processing unit, configured to process the audio sampling point data using the filter based on the filter control information to obtain waveform data;

[0092] An effector processing unit, configured to process the waveform data using the effector based on the effector control information to obtain the audio clip

[0093] Further, based on any of the above embodiments, the above audio generating device may further include:

[0094] An input to audio buffer module, used for inputting the target audio into the audio buffer;

[0095] An interface parameter duration determination module, used to determine the interface parameter duration according to a downstream interface type; wherein the downstream interface type is the type of the interface used to export audio;

[0096] The long time domain determination module is used to splice the target audio in the audio buffer into data frames according to the interface parameter duration to obtain a long time domain; wherein the long time domain is undivided smooth audio.

[0097] Further, based on the above embodiment, the above audio generating device may further include:

[0098] A judgment module, used to determine whether the duration of each pitch information corresponding to the target audio is greater than the interface parameter duration;

[0099] A storage module for an array is used to store the cross-domain information corresponding to the long time domain into an array according to the pitch information when the duration of the current pitch information is greater than the duration of the interface parameter, so as to control the end time of the target audio according to the cross-domain information in the array; wherein the cross-domain information is the pitch information when two interface parameter durations in the long time domain are spliced.

[0100] Further, based on the above embodiment, the above audio segment generating module 200 may include:

[0101] A packaging module, used for packaging the plurality of timbre parameters with the oscillator, the filter and the effector respectively, to obtain a plurality of single-tone sound processing models; wherein each single-tone sound processing model is a model for generating a single timbre;

[0102] A three-stage envelope information encapsulation module, used to encapsulate the three-stage envelope information into the single-tone sound processing model; wherein the three-stage envelope information is a parameter describing the signal change process over time;

[0103] Accordingly, the audio segment generating module 200 includes:

[0104] The audio clip generation unit based on the three-stage envelope information is used to generate audio streams according to the timbre parameters using a single-tone sound processing model that encapsulates the three-stage envelope information to obtain the multiple audio clips.

[0105] Further, based on any of the above embodiments, the above audio generating device may further include:

[0106] The wavetable index determination module is used to set the wavetable index of the oscillator according to the number of beats per minute, and update the generated waveform data based on the wavetable index; wherein the wavetable index is a file that determines the order of waveform samples.

[0107] Further, based on any of the above embodiments, the above audio generating device may further include:

[0108] The oscillator and filter determination module is used to select part or all of the oscillators from the oscillators according to different requirements, and select part or all of the filters as oscillators for audio streaming generation; wherein, when there are multiple oscillators, each oscillator supports a different oscillation method, and when there are multiple filters, each filter supports a different filtering algorithm.

[0109] It should be noted that the order of the modules and units in the above-mentioned audio generating device can be changed without affecting the logic.

[0110] The audio generation device provided by the embodiment of the present invention may include: a timbre parameter acquisition module 100, used to acquire a plurality of preset timbre parameters; wherein each of the timbre parameters includes oscillator control information, filter control information and effector control information; an audio segment generation module 200, used to generate audio streams according to the timbre parameters in combination with oscillators, filters and effects to obtain a plurality of audio segments; and an audio determination module 300, used to superimpose a plurality of the audio segments to obtain target audio. Compared with the current method of controlling timbre parameters based on the MIDI protocol, which makes it impossible to change the timbre parameters, the present invention can design the timbre parameters by itself, thereby generating audio according to the timbre parameters. Since the timbre parameters can be set by itself according to the needs, the expansion of new timbres and the modification of existing timbres only need to update the text parameter data, and the plasticity potential of the timbre is huge, and it has the ability to simulate a large number of timbres.

[0111] An audio generating device provided by an embodiment of the present invention is introduced below. The audio generating device described below and the audio generating method described above can refer to each other.

[0112] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of an audio generating device provided by an embodiment of the present invention may include:

[0113] A memory 10, used for storing computer programs;

[0114] The processor 20 is used to execute a computer program to implement the above-mentioned audio generation method.

[0115] The memory 10 , the processor 20 , and the communication interface 30 all communicate with each other via a communication bus 40 .

[0116] In the embodiment of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:

[0117] Acquire a plurality of preset timbre parameters, wherein each timbre parameter includes oscillator control information, filter control information and effector control information;

[0118] Audio streaming is performed according to the timbre parameters combined with oscillators, filters and effects to obtain multiple audio clips;

[0119] Multiple audio clips are superimposed to obtain the target audio.

[0120] In a possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0121] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include an NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0122] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10 .

[0123] The communication interface 30 may be an interface of a communication module, and is used to connect to other devices or systems.

[0124] Of course, it should be noted that Figure 8 The structure shown does not constitute a limitation on the audio generating device in the embodiment of the present invention. In actual applications, the audio generating device may include Figure 8 More or fewer components than shown, or combinations of certain components.

[0125] The computer-readable storage medium provided in an embodiment of the present invention is introduced below. The computer-readable storage medium described below and the audio generation method described above can be referenced to each other.

[0126] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned audio generation method are implemented.

[0127] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0128] The present invention also provides a computer program product, comprising a computer program / instruction, wherein the computer program / instruction implements the steps of the above-mentioned audio generation method when executed by a processor.

[0129] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0130] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0131] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0132] The above is a detailed introduction to an audio generation method, device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An audio generation method, characterized in that: include: Acquire a plurality of preset timbre parameters; wherein each of the timbre parameters includes oscillator control information, filter control information and effector control information; Perform audio streaming generation according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain a plurality of audio clips; A plurality of the audio clips are superimposed to obtain a target audio.

2. The audio generation method according to claim 1, characterized in that: The timbre parameters also include waveform information; wherein the waveform information includes at least one of self-made wave table information, low frequency oscillator oscillation information and sequencer information; the waveform information is the timbre parameters required by the oscillator.

3. The audio generation method according to claim 1, characterized in that: The audio stream generation is performed according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain multiple audio clips, including: Processing the oscillator control information using the oscillator to obtain audio sampling point data; Processing the audio sampling point data using the filter based on the filter control information to obtain waveform data; The waveform data is processed by the effector based on the effector control information to obtain the audio segment.

4. The audio generation method according to any one of claims 1 to 3, characterized in that: After the plurality of audio clips are superimposed to obtain the target audio, the method further includes: Input the target audio into an audio buffer; Determining the interface parameter duration according to the downstream interface type; wherein the downstream interface type is the type of the interface used to export the audio; The target audio in the audio buffer is spliced ​​according to the interface parameter duration in data frames to obtain a long time domain; wherein the long time domain is undivided smooth audio.

5. The audio generation method according to claim 4, characterized in that: The method further comprises: Determine whether the duration of each pitch information corresponding to the target audio is greater than the interface parameter duration; When the duration of the current pitch information is greater than the interface parameter duration, the cross-domain information corresponding to the long time domain is stored in an array according to the pitch information, so as to control the end time of the target audio according to the cross-domain information in the array; wherein the cross-domain information is the pitch information when two interface parameter durations in the long time domain are spliced.

6. The audio generation method according to claim 4, characterized in that: The audio stream generation is performed according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain multiple audio clips, including: Encapsulating the plurality of timbre parameters with the oscillator, the filter and the effector respectively to obtain a plurality of single-tone sound processing models; wherein each single-tone sound processing model is a model for generating a single timbre; Encapsulating the three-stage envelope information into the single-tone sound processing model; wherein the three-stage envelope information is a parameter describing the signal change process over time; Accordingly, the audio stream is generated according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain a plurality of audio clips, including: Audio streaming is generated according to the timbre parameters using a single-tone sound processing model that encapsulates the three-stage envelope information to obtain the multiple audio clips.

7. The audio generation method according to claim 1, characterized in that: In the process of processing using the oscillator, the method further includes: The wavetable index of the oscillator is set by the number of beats per minute, and the generated waveform data is updated based on the wavetable index; wherein the wavetable index is a file that determines the order of waveform samples.

8. The audio generation method according to claim 1, characterized in that: Before generating audio streams according to the timbre parameters in combination with an oscillator, a filter and an effector to obtain a plurality of audio clips, the method further includes: According to different requirements, some or all of the oscillators are selected from the oscillators, and some or all of the filters are selected as oscillators for audio streaming generation; wherein, when there are multiple oscillators, each oscillator supports a different oscillation method, and when there are multiple filters, each filter supports a different filtering algorithm.

9. An audio generating device, characterized in that: include: A timbre parameter acquisition module, used to acquire a plurality of preset timbre parameters; wherein each of the timbre parameters includes oscillator control information, filter control information and effector control information; An audio clip generation module, used for generating audio streams according to the timbre parameters in combination with an oscillator, a filter and an effector, to obtain a plurality of audio clips; The audio determination module is used to superimpose multiple audio clips to obtain target audio.

10. An audio generating device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the audio generation method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the audio generation method according to any one of claims 1 to 8 are implemented.

Images