Audio processing method and device

Abstract

The present application discloses an audio processing method and device, which relate to the field of audio processing technology. The method comprises obtaining an audio file to be processed and extracting key parameters from the audio file to be processed, wherein the key parameters include at least a beat list, a beat type and a beat number speed parameter. According to the beat type and the beat number speed parameter, the note duration range corresponding to the interval between each adjacent beat is calculated. When matching the note sample, the method searches for each candidate note in the note duration range that meets preset rules and matches the recording rhythm from the note sample library. The preset rules are determined by the beat list, the preset music parameters and the music theory constraints. The method searches for the optimal note sequence from each candidate note, and renders the optimal note sequence by a preset rendering method to obtain an audio signal, thereby completing the process of creating a musical work.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and more specifically, to an audio processing method and device. Background Art

[0002] With the development of audio signal processing technology, more and more tools and audio signal processing technologies are being used for audio analysis and processing. Audio signal processing technology can be used to extract rich audio features such as frequency, amplitude, rhythm, and timbre from audio files such as recordings. Music creation tools can then use these audio features to create music.

[0003] However, existing music creation tools have high technical barriers for non-professionals, requiring users to have certain music theory knowledge and music production skills, as well as the arrangement of rhythm and melody during the music creation process. Therefore, the traditional music creation process requires a lot of time and energy, resulting in low efficiency in music creation.

[0004] Therefore, how to improve the efficiency of music creation is a problem that needs to be solved urgently in this application. Summary of the Invention

[0005] In view of this, the present application discloses an audio processing method and device, which aim to quickly generate music signals that conform to the recording rhythm, improve the efficiency of music creation, and make the generated audio works have high restoration and good auditory effects.

[0006] In order to achieve the above purpose, the disclosed technical solutions are as follows:

[0007] In a first aspect, the present application discloses an audio processing method, the method comprising:

[0008] Get the audio file to be processed;

[0009] Extracting key parameters from the audio file to be processed; wherein the key parameters include at least a beat list, a beat type, and a beat number speed parameter;

[0010] Calculating the note duration range corresponding to the interval between each adjacent beat according to the beat type and the beat speed parameter;

[0011] When matching the note samples, searching the note sample library for candidate notes that match the recording rhythm within the note duration range and conform to preset rules; wherein the preset rules are determined by the beat list, preset music parameters, and music theory constraints;

[0012] Searching for the optimal note sequence from each candidate note;

[0013] The optimal note sequence is rendered using a preset rendering method to obtain an audio signal to complete the music creation process.

[0014] Preferably, extracting key parameters from the audio file to be processed includes:

[0015] Preprocessing the audio file to be processed; the preprocessing at least includes audio decoding, channel separation and amplitude normalization;

[0016] The pre-processed audio files are subjected to feature extraction through digital signal processing technology to obtain a time series reflecting the changes in music energy;

[0017] Performing threshold detection on the time series reflecting the change of music energy to obtain a time position sequence of beats;

[0018] Analyzing the distribution pattern of adjacent beat intervals in the temporal position sequence of the beats to obtain a beat interval sequence, and determining the beat type of the music based on a preset clustering algorithm and the beat interval sequence;

[0019] The average beat interval of the beat interval sequence is calculated to obtain a beat speed parameter.

[0020] Preferably, the analyzing the distribution pattern of adjacent beat intervals in the temporal position sequence of the beats to obtain a beat interval sequence, and determining the beat type of the music based on a preset clustering algorithm and the beat interval sequence, includes:

[0021] performing difference calculation on adjacent elements in the temporal position sequence of the beats to obtain a beat interval sequence;

[0022] Clustering the beat interval sequence using a K-Means clustering algorithm to obtain multiple cluster centers;

[0023] Obtaining ratios corresponding to a plurality of cluster centers, and performing similarity comparison between the ratios corresponding to the plurality of cluster centers and a preset ratio to obtain a plurality of comparison results;

[0024] The ratio with the greatest similarity among the multiple comparison results is determined as the beat type of the music.

[0025] Preferably, when matching the note samples, searching the note sample library for candidate notes that match the recording rhythm and conform to preset rules within the note duration range includes:

[0026] When matching note samples, the note samples that meet the preset rules within the note duration range are retrieved from the note sample library through the query conditions;

[0027] According to the file storage path corresponding to the note sample, each candidate note matching the recording rhythm is obtained.

[0028] Preferably, searching for the optimal note sequence from each candidate note includes:

[0029] Scoring each candidate note using a scoring function in a dynamic programming algorithm; wherein the scoring function is used to score each candidate note based on its rhythmic matching, melodic fluency, and harmonic rationality to ensure that the rhythm of the searched note sequence is consistent with the rhythm of the audio file to be processed and that the musicality of the searched note sequence is optimal;

[0030] By backtracking the state, the optimal note sequence is obtained from each candidate note after scoring.

[0031] Preferably, the rendering of the optimal note sequence by a preset rendering method to obtain an audio signal to complete the music composition process includes:

[0032] Performing a processing operation on the optimal note sequence according to a musical form template; the processing operation at least includes a repetition operation of a musical element, a transformation operation of a musical element, and a combination operation of a musical element;

[0033] Performing an operation of adding musical elements to the optimal note sequence after the processing operation to obtain a target music segment; the adding operation at least includes adding chords and ornaments;

[0034] Rendering the target music clip into pulse code modulation audio data using a preset software music library;

[0035] Grouping the pulse code modulation audio data according to instrument type to obtain a grouping result; wherein each group of notes in the grouping result corresponds to a musical instrument digital interface track;

[0036] For each MIDI track, converting a note event into a MIDI message in chronological order; wherein the note event is used to indicate a note start and a note end;

[0037] Transmitting the Musical Instrument Digital Interface message into a software synthesizer for rendering to obtain an audio digital signal generated by each Musical Instrument Digital Interface track;

[0038] The audio digital signals are mixed to obtain audio signals to complete the music creation process.

[0039] Preferably, mixing the audio digital signal to obtain an audio signal includes:

[0040] Aligning the audio digital signals generated by each MIDI track in time dimension;

[0041] Perform weighted summation on the audio digital signals aligned in the time dimension to obtain mixed audio data;

[0042] The mixed audio data is subjected to sound and image balance processing and relative volume adjustment to obtain an audio signal; wherein the sound and image balance processing is used to adjust the relative strength of the left and right channels and simulate the position of the sound in space; the relative volume adjustment is used to ensure that the volume of the mixed audio data is within a preset volume range.

[0043] Preferably, it also includes:

[0044] The audio signal is written into an audio buffer, and a real-time playback control interface is provided.

[0045] A second aspect of the present application discloses an audio processing device, the device comprising:

[0046] An acquisition unit, used to acquire the audio file to be processed;

[0047] An extraction unit, configured to extract key parameters from the audio file to be processed; wherein the key parameters include at least a beat list, a beat type, and a beat number and speed parameter;

[0048] a calculation unit, configured to calculate a note duration range corresponding to an interval between adjacent beats according to the beat type and the beat velocity parameter;

[0049] a search unit configured to search, when matching a note sample, from the note sample library for candidate notes that match the recording rhythm and conform to a preset rule within the note duration range; wherein the preset rule is determined by the beat list, preset music parameters, and music theory constraints;

[0050] A search unit, used to search for an optimal note sequence from various candidate notes;

[0051] The rendering unit is used to render the optimal note sequence in a preset rendering method to obtain an audio signal to complete the music creation process.

[0052] Preferably, the extraction unit comprises:

[0053] A preprocessing module, configured to preprocess the audio file to be processed; the preprocessing includes at least audio decoding, channel separation, and amplitude normalization;

[0054] The extraction module is used to extract features from the pre-processed audio files using digital signal processing technology to obtain a time series reflecting the changes in music energy;

[0055] A threshold detection module is used to perform threshold detection on the time series reflecting the change of music energy to obtain a time position sequence of beats;

[0056] an analysis and determination module, configured to analyze the distribution pattern of adjacent beat intervals in the temporal position sequence of the beats to obtain a beat interval sequence, and determine the beat type of the music based on a preset clustering algorithm and the beat interval sequence;

[0057] The calculation module is used to calculate the average beat interval of the beat interval sequence to obtain a beat speed parameter.

[0058] It can be seen from the above technical solution that the present application discloses an audio processing method and device, which obtains an audio file to be processed and extracts key parameters from the audio file to be processed, wherein the key parameters include at least a beat list, a beat type and a beat number speed parameter, and calculates the note duration range corresponding to the interval between each adjacent beat based on the beat type and the beat number speed parameter. When matching the note sample, search the note sample library for each candidate note that meets the preset rules and matches the recording rhythm within the note duration range, wherein the preset rules are determined by the beat list, preset music parameters and music theory constraints, search for the optimal note sequence from each candidate note, render the optimal note sequence by a preset rendering method, and obtain an audio signal to complete the music creation process. Through the above solution, users do not need to have music theory knowledge and music production skills in the process of creating music works. They only need to intelligently analyze and convert the rhythm in the audio file to be processed, automatically extract information such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, search for the optimal note sequence from each candidate note, and render the optimal note sequence through a preset rendering method to obtain an audio signal. Therefore, there is no need to spend a lot of time and energy in the process of creating music works, and a music signal that conforms to the recording rhythm can be quickly generated, thereby improving the efficiency of music creation. In addition, the use of advanced audio signal processing technology can accurately identify parameters such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, ensuring that the generated music work is highly consistent with the rhythm of the audio file to be processed, so that the generated audio work has a high degree of restoration and good listening effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0060] Figure 1A schematic structural diagram of an automatic music composition system disclosed in an embodiment of the present application;

[0061] Figure 2 A flowchart of an audio processing method disclosed in an embodiment of the present application;

[0062] Figure 3 A schematic structural diagram of an audio processing device disclosed in an embodiment of the present application;

[0063] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0065] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0066] As can be seen from the background technology, existing music creation tools have a high technical threshold for non-professionals. Users are required to have certain music theory knowledge and music production skills, as well as the arrangement of rhythm and melody during the music creation process. Therefore, the traditional music creation process requires a lot of time and energy, resulting in low efficiency in music creation. Therefore, how to improve the efficiency of music creation is an urgent problem to be solved in this application.

[0067] In order to solve the above problems, the present application discloses an audio processing method and device. In the process of creating a musical work, it is not necessary to require the user to have music theory knowledge and music production skills. It only needs to intelligently analyze and convert the rhythm in the audio file to be processed, automatically extract the rhythm, beat type, beat number and speed parameters and other information in the audio file to be processed, search for the optimal note sequence from each candidate note, and render the optimal note sequence through a preset rendering method to obtain an audio signal. Therefore, there is no need to spend a lot of time and energy in the process of creating a musical work, and a music signal that conforms to the recording rhythm is quickly generated, thereby improving the efficiency of musical work creation. In addition, the use of advanced audio signal processing technology can accurately identify parameters such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, ensuring that the generated musical work is highly consistent with the rhythm of the audio file to be processed, so that the generated audio work has a high degree of restoration and a good auditory effect. The specific implementation method is specifically described through the following embodiments.

[0068] refer to Figure 1 As shown, it is a structural diagram of an automatic composition system disclosed in an embodiment of the present application. The automatic composition system includes a rhythm analysis module, a rhythm generation module, a pitch sequence generation module, an orchestration module, a music element generation module and an audio synthesis and output module.

[0069] The data interaction process between the rhythm analysis module, rhythm generation module, pitch sequence generation module, instrumentation module, music element generation module and audio synthesis and output module is as follows:

[0070] Rhythm analysis module:

[0071] When the rhythm parsing module receives a user-input audio file (either uploaded or recorded), it parses the file to extract key parameters and passes them to the rhythm generation module. These parameters include the beat list, beat type, beat count (BPM), tempo, user-input number of notes, and mode. These key parameters can be used to determine the style of the generated music file.

[0072] The process of the rhythm analysis module extracting key parameters from the audio file to be processed is as follows:

[0073] First, the audio file to be processed is preprocessed, including audio decoding, channel separation, amplitude normalization, etc.

[0074] The automatic composition system uses a sixth-order Butterworth low-pass filter to smooth and reduce noise on the original audio signal (i.e., the audio file to be processed). The energy envelope curve E(t) is obtained by taking the absolute value and smoothing the filtered signal, as shown in formula (1).

[0075] E(t)=Smooth(|x(t)*h(t)|) (1)

[0076] Where x(t) is the original audio signal; h(t) is the unit impulse response of the filter; * represents the convolution operation; || represents taking the absolute value; the smoothing function Smooth can select common algorithms such as moving average and Gaussian smoothing.

[0077] It should be noted that the Butterworth filter has a flat passband response and linear phase characteristics, effectively removing high-frequency noise while preserving rhythmic information. The filter's cutoff frequency can be set to 150Hz, covering the fundamental frequency range of most percussion instruments.

[0078] Secondly, digital signal processing technology is used to extract features from the audio to obtain a time series reflecting the changes in music energy. Threshold detection is performed on the time series reflecting the changes in music energy to obtain the time position sequence of the beats.

[0079] It should be noted that music energy variation refers to the pattern of how the amplitude of an audio signal changes over time. By analyzing audio features such as short-term energy or Mel-Frequency Cepstral Coefficients (MFCCs), we can obtain a time series that reflects the rhythm and intensity of the music.

[0080] Then, the distribution pattern of the intervals between adjacent beats in the temporal position sequence of beats is analyzed to obtain the beat interval sequence;

[0081] Finally, according to a preset clustering algorithm (such as K-Means clustering algorithm) and the beat interval sequence, the beat type of the music is determined, the average beat interval of the beat interval sequence is calculated, and the beat number speed parameter is obtained.

[0082] Specifically, according to the preset clustering algorithm and the beat interval sequence, the beat type of the music is determined, the average beat interval of the beat interval sequence is calculated, and the process of obtaining the beat number and speed parameters is shown in A1-A3:

[0083] A1: Calculate the difference between adjacent elements in the beat time position sequence to obtain a beat interval sequence. The beat interval sequence is used to represent the distribution pattern of the intervals between adjacent beats in the beat time position sequence.

[0084] A2: Cluster the beat interval sequence using the K-Means clustering algorithm to obtain multiple cluster centers and obtain the corresponding ratios of the multiple cluster centers;

[0085] A3: Compare the ratios corresponding to the multiple cluster centers with a preset ratio to obtain multiple comparison results. The ratio with the greatest similarity among the multiple comparison results is determined as the beat type of the music.

[0086] The process of parsing an audio file involves beat detection. The key to beat detection is to identify the peak points in the energy envelope curve, which represent the beat locations. Traditional methods use a fixed threshold to compare energy values, but this approach struggles to adapt to music of varying styles and rhythmic intensities. This system innovatively proposes an adaptive threshold algorithm that dynamically adjusts the threshold function to detect beats of varying intensities by performing a sliding window analysis on the energy envelope.

[0087] The adaptive threshold algorithm first divides E(t) into window segments of length L, and overlapping is allowed between adjacent windows. For the i-th window segment, its expression is shown in formula (2).

[0088] Ei=[E(t)fort inrange(i*H, (i+1)*H)](2)

[0089] Where i is the window number; H is the step size of the window, satisfying L>H; Ei is the window segment.

[0090] Formula (2) indicates segmenting the energy envelope curve E(t) and extracting the subsequence corresponding to the i-th window.

[0091] For each window segment, the following feature parameters are extracted:

[0092] Ei_avg: average energy of the fragment;

[0093] Ei_max: maximum energy of the fragment;

[0094] Ei_std: energy standard deviation of the fragment;

[0095] Ei_ratio: ratio of maximum energy to average energy, Ei_ratio = Ei_max / Ei_avg.

[0096] Then, the automatic composition system uses a trained logistic regression model to dynamically calculate the threshold coefficient wi based on the above feature parameters, and its calculation formula is shown in formula (3).

[0097] wi=Logistic(a0+a1Ei_avg+a2Ei_max+a3Ei_std+a4Ei_ratio) (3)

[0098] Among them, a0, a1, a2, a3, and a4 are all model parameters, which require offline training using labeled beat data; Logistic is the Logistic function.

[0099] The calculation between the threshold value (THi) of the i-th segment and wi is as shown in formula (4).

[0100] THi=wi*Ei_max(4)

[0101] Here, wi ranges from 0 to 1, measuring the relative magnitude of the beat intensity. When Ei_max is much greater than the average energy, wi approaches 1, and the threshold is equal to the peak value. When Ei_max is close to the average energy, wi approaches 0, and the threshold is less than the peak value. This allows for adaptive detection of varying beat types.

[0102] Within each segment, any beat whose energy exceeds THi is marked as a beat. For beats with an interval less than 100ms, the one with the highest energy is retained. The algorithm outputs a temporal sequence of beat positions. This temporal sequence of beat positions is obtained by applying a threshold check to the temporal sequence of musical energy changes. The temporal sequence of beat positions corresponds to the original temporal sequence.

[0103] It should be noted that energy refers to the square of the instantaneous amplitude of an audio signal, reflecting the intensity changes of the musical sound. The music energy analyzed here is the music energy of audio files uploaded or recorded by users.

[0104] The original time series refers to the time series of changes in music energy.

[0105] The input of the algorithm is the time domain waveform of the music signal, and the temporal position sequence of the beats is obtained through the following steps:

[0106] Divide the music signal into several short segments of fixed length (e.g. 10 milliseconds);

[0107] Calculate the energy value of each short segment to form a time series of energy changes E = [E1, E2, ..., Em], where Ei represents the energy value of the i-th segment and m is the total number of segments. This time series of energy changes E is the original time series;

[0108] In each segment, beats are detected based on an adaptive threshold THi. The calculation of THi takes into account the maximum energy Ei_max and the average energy Ei_avg within the segment, as well as an adjustable weight parameter wi. When the energy within the segment exceeds THi, it is marked as a beat;

[0109] Post-process the detected beats to remove beats with too small an interval (e.g., less than 100 milliseconds) and retain the one with larger energy;

[0110] The temporal position sequence of the output beats B = [b1, b2, ..., bn], where bi represents the corresponding position of the i-th beat in the original time sequence E.

[0111] The beat position sequence B is obtained by performing threshold detection on the original energy variation time sequence E. Each beat position bi corresponds to a segment position in the original sequence E. Therefore, the original time sequence refers to the energy variation sequence E of the music signal, which describes the energy distribution characteristics of the music signal in the temporal dimension. By analyzing this original sequence, the beat detection algorithm obtains a higher-level representation of the rhythmic structure, namely the beat position sequence B.

[0112] Based on the temporal position sequence of beats, the beat type of music is determined by analyzing the distribution of adjacent beat intervals using a preset clustering algorithm (such as the K-Means clustering algorithm). Assume that the beat interval sequence is as shown in formula (5).

[0113] D=[d1,d2,...,dn](5)

[0114] Where D is the beat interval sequence, which refers to the time interval sequence between two adjacent beat time positions, and can be obtained by taking the difference between adjacent elements in the beat sequence, such as [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, ...]; di represents the time interval between the i-th beat and the i+1-th beat.

[0115] The preset clustering algorithm includes but is not limited to the K-Means clustering algorithm. The preset clustering algorithm of this solution is preferably the K-Means clustering algorithm.

[0116] The K-Means algorithm is used to cluster D, and the number of categories K is 3 (corresponding to 2 beats, 3 beats, and 4 beats). The expressions of the three cluster centers obtained by clustering are shown in formula (6).

[0117] C=[c1,c2,c3](6)

[0118] Among them, C is the clustering result; c1 <c2<c3。

[0119] Compare the similarity of c1:c2:c3 with 1:2:3, 1:2:4, and 1:3:4. The ratio with the highest similarity is determined as the beat type of the music. For example, c1:c2:c3 = 1:1.98:3.96, which is closest to 1:2:4, so the beat type of the music is considered to be 4 / 4.

[0120] By analyzing the distribution of intervals between beats, a K-Means clustering algorithm is used to determine the beat type of the music. The intervals between beats are essentially the difference between two adjacent time positions in the beat sequence. In the example above, the interval sequence is [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, ...], indicating that the interval between each beat is 0.5 seconds, corresponding to the duration of a quarter note. By clustering these intervals, it can be inferred that the beat type of the music is 4 / 4.

[0121] Finally, the BPM speed parameter is obtained by calculating the average beat interval. The specific calculation process is shown in formula (7).

[0122] BPM=60 / Mean(D) (7)

[0123] Where Mean represents the average value; D is the beat interval sequence; Mean(D) calculates the average value of D in seconds.

[0124] Rhythm generation module:

[0125] The rhythm generation module adopts a rule-based rhythm pattern generation algorithm to randomly select rhythm patterns that meet the beat number and speed requirements from a predefined rhythm pattern library.

[0126] A rhythmic pattern that meets the beat number and tempo requirements is a rhythmic sequence that satisfies the following conditions:

[0127] 1. The total number of beats of the rhythm pattern is consistent with the number of beats specified by the user;

[0128] 2. The note duration distribution in the rhythm pattern matches the tempo parameter, for example, 120 BPM corresponds to 0.5 seconds per beat;

[0129] 3. A rhythmic pattern typically consists of several bars, each measuring a specific beat type. This pattern plays a key role in the subsequent generation of musical elements. The note sequence must strictly adhere to the tempo of the pattern to ensure that the generated music rhythmically aligns with the user's input. The pattern also influences the distribution of note dynamics.

[0130] The rhythm generation module receives the key parameters and calculates the note duration range corresponding to the interval between each adjacent beat based on the beat type and beat speed parameters in the key parameters.

[0131] When matching note samples, the rhythm generation module searches the note sample library for candidate notes that match the recording's rhythm within a range of note durations and conform to preset rules determined by a beat list, preset musical parameters, and music theory constraints. The music element generation module uses the beat list and user-specified musical parameters (such as mode and number of notes) to generate a note sequence that matches the processed audio file's rhythm.

[0132] The preset rules include but are not limited to the following examples:

[0133] Matching rules between note duration and beat: For example, in a 4 / 4 measure, note duration can only be 1 / 4, 1 / 8, 1 / 16, etc.

[0134] Constraints on interval jumps: For example, within a musical passage, the pitch difference between two adjacent notes should not exceed one octave;

[0135] Chord and melody matching rules: For example, within a chord, the melody notes need to match the constituent notes of the chord;

[0136] Constraints of musical style: For example, when composing a jazz-style piece of music, the choice of notes needs to conform to the scales, rhythm patterns and other characteristics of jazz.

[0137] The above rules can be summarized and refined through music theory knowledge and adjusted appropriately according to specific needs.

[0138] The rhythm generation module uses a dynamic programming algorithm to search for the optimal note sequence from each candidate. This algorithm considers multiple music theory constraints, including the range of note durations corresponding to beat intervals, pitch variation patterns, and dynamics patterns. It then searches for the highest-scoring note combination, which is considered the optimal note sequence.

[0139] Specifically, when using the dynamic programming algorithm to search for the optimal note sequence, you can follow the following ideas:

[0140] State definition: Let dp[i][j] represent the optimal score when matching the i-th beat and using the j-th candidate note;

[0141] Initial condition: dp[0][j] = 0, indicating that in beat 0, no candidate note is used and the score is 0;

[0142] The state transition equation: dp[i][j] = max(dp[i-1][k] + score(k, j)) indicates that the optimal score for using the jth candidate note at the i-th beat can be obtained by using the kth candidate note at the i-1th beat and then transferring to the jth note.

[0143] The scoring function score(k, j) takes into account factors such as the rhythm matching, melodic fluency, and harmonic rationality of the transition from the kth note to the jth note, and is calculated using a weighted summation method;

[0144] Final result: After the dynamic programming algorithm completes, dp[n][j] represents the optimal score for the jth candidate note after matching all n beats. By backtracking through the dp array, we can find the optimal note sequence.

[0145] The rhythm generation module is based on rhythm information and automatically generates expressive and musical melody fragments through a series of algorithms such as music sample matching, dynamic planning path search, and ADSR envelope synthesis.

[0146] The construction of a music sample library is the foundation of the rhythm generation module. This solution collects a large number of single-note samples played by various instruments, including but not limited to piano, guitar, violin, flute, etc., covering different combinations of pitch (C2-C7, with a range of more than 5 octaves), duration (from 32nd notes to whole notes), and velocity (pp to ff, corresponding to MIDI velocity values ​​20-120).

[0147] Each note sample is extracted for features such as fundamental frequency, overtones, and envelope, and a multi-level index is established based on instrument type, pitch, and duration. Given a query, the automatic composition system can return the matching sample file path in sub-second time.

[0148] It should be noted that the query conditions refer to a series of parameters used when retrieving and matching note samples in the music sample library, which are automatically generated by the system based on user input and music generation requirements.

[0149] The query conditions include but are not limited to the following aspects: instrument type, pitch range, duration range, velocity range, etc.

[0150] Returning the file path of the hit sample refers to retrieving the matching note samples from the music sample library based on the query criteria. This involves using a rule-based rhythm pattern generation algorithm to randomly select note samples from a predefined rhythm pattern library that meet the beat count and tempo requirements and return the file storage path for these note samples. The goal of this step is to identify all possible candidate notes that could form a melody.

[0151] Assume that the expression of the temporal position sequence of the beats output by the rhythm analysis module is as shown in formula (8):

[0152] B=[b1,b2,...,bn](8)

[0153] Where B is the temporal position sequence of beats; bi represents the temporal position of the i-th beat.

[0154] It's important to note that in music, a beat is a regular unit of time division, with each beat corresponding to a specific time position. By analyzing the audio signal, we can obtain a series of beat time positions, forming a beat sequence. For example, a piece of music in 4 / 4 time might have a beat sequence of [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, ...], where each number represents the time position (in seconds) at which the corresponding beat occurred.

[0155] The time interval between adjacent beats is shown in formula (9) and formula (10).

[0156] I=[i1,i2,...,i(n-1)](9)

[0157] ik=b(k+1)-bk(10)

[0158] Where I is the time interval between adjacent beats; i1, i2, ..., i(n-1) is the sequence of time intervals between adjacent beats; ik is the duration of the k-th beat interval; b(k+1) is the time position of the k+1-th beat; and bk is the time position of the k-th beat.

[0159] It should be noted that I represents an interval sequence, which includes the time interval between each pair of adjacent beats.

[0160] i1, i2, ..., i(n-1) represent the time intervals between the 1st to (n-1th)th pairs of adjacent beats respectively.

[0161] n is the length of the beat sequence B, that is, the total number of beats.

[0162] ik=b(k+1)-bk further illustrates how to calculate the value of each interval, where ik represents the kth interval, which is equal to the time position of the (k+1)th beat minus the time position of the kth beat.

[0163] Assume that the beat sequence is B = [0.0, 0.5, 1.0, 1.5, 2.0], then:

[0164] i1=b2-b1=0.5-0.0=0.5, i2=b3-b2=1.0-0.5=0.5, i3=b4-b3=1.5-1.0=0.5, i4=b5-b4=2.0-1.5=0.5, then the interval sequence I=[0.5, 0.5, 0.5, 0.5].

[0165] Based on the beat type and BPM parameters, calculate the note duration range corresponding to the interval between each adjacent beat. For example, for music with a 4 / 4 time signature and 120 BPM, a beat interval of 0.5 seconds corresponds to a note duration range of [0.25, 0.5, 1] ​​seconds, which includes a quarter note, a half note, and a whole note.

[0166] After obtaining the note duration range, the pre-collected note sample library is searched for candidate notes that match the recording rhythm according to preset rules. These are then used for subsequent musical element generation. The preset rules are determined by the beat list, preset musical parameters (such as mode and number of notes), and music theory constraints.

[0167] When matching note samples, in addition to the duration conditions, the preset rules also need to consider whether the pitch changes and dynamic changes of the note combination meet the requirements of music theory, and whether the duration of the notes is within the range corresponding to the beat interval to ensure that the generated music rhythm matches the original beat sequence. The automatic composition system defines some heuristic rules, such as:

[0168] The pitch changes of note combinations should conform to the rules of the scale. Each scale corresponds to a fixed set of pitches, and the notes should mainly develop within the scale, with occasional variations. For example, in various scales, the pitch combination of notes needs to follow certain rules. For example, in C major, the commonly used scale is CDEFGAB, and the notes mainly develop on these seven pitches. Occasionally, variations such as F sharp and B flat can be used to add richness to the melody. Each scale has a different commonly used scale, which constitutes the characteristics of the scale.

[0169] The pitch changes of a note combination should conform to the rules of melodic motion. Common melodic motions include steps (pitch rises or falls sequentially), leaps (pitch rises or falls in leaps), and retracements (pitch rises then falls or falls then rises).

[0170] Continuous large leaps (greater than a fifth) should be avoided as they are detrimental to the flow of the melody.

[0171] Dissonant intervals (such as minor second, augmented fourth, etc.) should be avoided as much as possible, especially on the strong beat;

[0172] The dynamics of the notes should be consistent with the rhythmic accents, with stronger dynamics generally corresponding to stronger beats. Dynamics should also conform to the overall musical structure, such as crescendo and diminuendo.

[0173] Based on these rules, the automatic composition system uses a dynamic programming algorithm to search for the optimal note combination path. The state is defined as dp[i][p][d] = the optimal score corresponding to the i-th beat, pitch p, and velocity d.

[0174] Here, dp[i][p][d] represents the score of the optimal subproblem corresponding to the i-th beat, pitch p, and velocity d. By dynamically programming the fill table, we can obtain the optimal note combination path for the entire music clip.

[0175] The state transition equation is dp[i][p][d]=max(dp[i-1][p'][d']+score(p, p', d, d', i)).

[0176] Here, p' and d' traverse all possible pitches and dynamics of the previous beat. The score function calculates the transition score between two notes based on a scoring rule, taking into account factors such as pitch variation, dynamic variation, and mode stability. The scoring function can be trained offline using a machine learning algorithm.

[0177] The optimal note combination path involves using a dynamic programming algorithm to search and evaluate different note combinations after obtaining a set of candidate notes, ultimately finding the highest-scoring note sequence as the optimal path for generating the melody. The goal of this step is to select the best possible combination from the candidate notes, ensuring that it not only meets constraints such as rhythm and mode, but also achieves optimal melodic fluidity and harmonic plausibility.

[0178] The pitch sequence generation module traverses all possible pitches of the previous beat based on a data-driven pitch sequence generation algorithm.

[0179] The final note combination path is dp[n][p][d]=max(dp[n][p][d])forallpossiblep andd, where for allpossiblep andd represents all possible pitches p and velocities d.

[0180] The music element generation module scores each candidate note using a scoring function in a dynamic programming algorithm. This scoring function is used to score each candidate note's rhythmic matching, melodic fluency, and harmonic rationality to ensure that the rhythm of the searched note sequence is consistent with the rhythm of the audio file to be processed, and that the musicality of the searched note sequence is optimal.

[0181] The scoring function needs to comprehensively consider multiple factors such as rhythm matching, melodic fluency, and harmonic rationality to ensure that the generated music not only matches the rhythm of the audio file to be processed, but is also the best in musicality.

[0182] The music element generation module can obtain the optimal note sequence by reverse tracing the state.

[0183] The process of obtaining the optimal note sequence by backtracking the state is as follows:

[0184] 1. Find the optimal state (pitch and dynamics) combination for the last beat, recorded as (p_n, d_n);

[0185] 2. Starting from (p_n, d_n), search in reverse order for the optimal state of the previous beat (p_n-1, d_n-1), satisfying dp[n-1][p_n-1][d_n-1] + trans(p_n-1, d_n-1, p_n, d_n) = dp[n][p_n][d_n]. Where trans is the state transition function;

[0186] 3. Repeat the above steps until you reach the first beat and obtain a complete optimal state sequence;

[0187] 4. According to the optimal state sequence, take out the corresponding notes to form the final note sequence.

[0188] Adapter module:

[0189] The orchestration module repeats, transforms, and combines musical elements based on the template-based orchestration algorithm and predefined musical form templates. The specific process is as follows:

[0190] 1. The orchestration module selects a predefined musical form template, such as two-part form, three-part form, sonata form, etc. Each template contains several sections, each of which has a specific number of repetitions and variation rules.

[0191] 2. The orchestration module divides the note sequence output by the music element generation module into several segments according to the length of the music segment.

[0192] 3. The instrumentation module repeats or transforms each musical section according to the rules of the musical form template. Transformations include pitch modulation, rhythmic value variation, and melody variation.

[0193] The orchestration module repeats, transforms, and combines musical elements based on the musical template, adding chords, ornaments, and other musical elements to enrich the texture and form of the music. Finally, the complete musical segment is rendered as an audio signal.

[0194] 4. The orchestration module adds transitional musical materials between musical sections, such as cadences and connecting parts, making the entire musical structure smooth and natural.

[0195] 5. The orchestration module splices the repeated and transformed music segments to form a complete musical structure.

[0196] 6. The orchestration module inserts chord notes at key positions in the music according to the definition of the harmonic direction in the musical form template to enrich the musical texture.

[0197] 7. The orchestration module evaluates the preliminarily assembled music clips and fine-tunes the notes according to predefined music theory rules (such as avoiding parallel fifths, the tendency of interval jumps, etc.).

[0198] 8. Output a music clip that conforms to the target musical form.

[0199] Audio synthesis and output module:

[0200] The audio synthesis and output module renders the optimal note sequence using a preset rendering method, and the specific process of obtaining the audio signal is shown in B1-B8.

[0201] B1: The audio synthesis and output module processes the optimal note sequence according to the musical form template; the processing operation at least includes the repetition operation of the music elements, the transformation operation of the music elements and the combination operation of the music elements.

[0202] B2: The audio synthesis and output module performs an operation of adding musical elements to the optimal note sequence after the processing operation to obtain a target music clip (a complete music clip); the adding operation at least includes adding chords and ornaments.

[0203] The audio synthesis and output module in the automatic composition system receives a complete note sequence from the music element generation module, which is the target music clip. The target music clip contains the pitch, duration, dynamics, timbre and other attribute parameters of each note. These attribute parameters are generated based on the time and speed characteristics extracted by the rhythm analysis module, as well as predefined musical rules such as mode and melodic pattern.

[0204] B3: The audio synthesis and output module renders the target music clip into pulse code modulation (PCM) audio data through a preset software music library.

[0205] The preset software music library includes but is not limited to a software sound source library of a software synthesizer (FluidSynth) via a Musical Instrument Digital Interface (MIDI) message input.

[0206] The format of MIDI data includes event information such as track, time, and note. When converting the target music clip into MIDI data, you need to follow the following rules:

[0207] The timestamps of note events are calculated and aligned based on the beat list information;

[0208] The pitch of a note is encoded based on the note name and octave information in the note sequence;

[0209] Note parameters such as velocity and duration are mapped according to musical notation symbols;

[0210] Musical ornaments such as chords and grace notes also need to be converted into corresponding MIDI events.

[0211] When rendering and synthesizing MIDI data, the wavetable synthesis method is mainly used. The steps are as follows:

[0212] According to the instrument type specified by the MIDI track, load the corresponding sound library and obtain the waveform samples of each note;

[0213] According to the timestamps of MIDI note events, the waveform samples are sorted and spliced ​​to obtain the audio digital signals of each part;

[0214] Mix the audio digital signals of each part, set the relative volume and delay of the left and right channels, and simulate the position of the sound in space;

[0215] Normalize the volume of the mixed stereo signal so that its peak volume is within -3dB to prevent signal distortion.

[0216] After the above MIDI audio rendering process, realistic and natural music works can be obtained.

[0217] Use the FluidSynth software sound library to render note sequences into PCM audio data. FluidSynth is a MIDI wavetable synthesis engine based on SoundFont, with built-in sound libraries for various common instruments, which can synthesize realistic sounds such as piano, guitar, strings, and wind instruments.

[0218] B4: The audio synthesis and output module groups the pulse code modulation audio data according to the instrument type to obtain a grouping result; each group of notes in the grouping result corresponds to a musical instrument digital interface (MIDI) track.

[0219] B5: The audio synthesis and output module converts note events into MIDI messages in chronological order for each Musical Instrument Digital Interface (MIDI) track; wherein the note events are used to indicate the start of a note (NOTE_ON) and the end of a note (NOTE_OFF).

[0220] Note events are a type of message in the MIDI protocol that indicates the start and end of a note. A complete note consists of two events: NOTE_ON and NOTE_OFF, marking the note's start time and duration, respectively. Note events are converted from the note sequence output by the Musical Element Generation module.

[0221] For each MIDI track, the process of converting note events into MIDI messages in time sequence is shown in C1-C4:

[0222] C1: traverse all notes of the track;

[0223] C2: For each note, a NOTE_ON message is generated, which contains the note's pitch, velocity, start time and other parameters;

[0224] C3: Generates the corresponding NOTE_OFF message based on the duration of the note to mark the end of the note;

[0225] C4: Arranges NOTE_ON and NOTE_OFF messages in chronological order to form an ordered MIDI message sequence.

[0226] B6: The audio synthesis and output module transmits the Musical Instrument Digital Interface message to the software synthesizer (FluidSynth synthesizer) for rendering, and obtains the audio digital signal generated by each Musical Instrument Digital Interface (MIDI) track.

[0227] The audio synthesis and output module transmits the MIDI message sequence to FluidSynth's synthesizer, triggering the wavetable synthesis process. FluidSynth retrieves pre-sampled instrument waveforms based on parameters such as pitch and velocity in the MIDI message, and applies effects such as ADSR envelopes and filters to generate the audio digital signal in real time.

[0228] B7: The audio synthesis and output module mixes the audio digital signals to obtain audio signals to complete the music creation process.

[0229] In B7, the audio digital signals synthesized by multiple MIDI tracks are mixed through a software synthesizer, and the relative volume and sound image balance are adjusted to obtain a complete multi-part music audio.

[0230] The specific process of mixing the audio digital signals synthesized by multiple MIDI tracks and adjusting the relative volume and sound image balance is shown in D1-D3.

[0231] D1: Align the audio digital signals generated by each MIDI track in time dimension.

[0232] D2: Perform weighted summation on the time-aligned audio digital signals to obtain mixed audio data.

[0233] At each time point, the audio data of all tracks are weighted and summed to obtain mixed audio data.

[0234] The weight coefficients of the weighted sum reflect the relative volume of each audio track.

[0235] D3: Performs image balancing and relative volume adjustment on the mixed audio data to obtain an audio signal. Image balancing adjusts the relative strength of the left and right channels and simulates the position of the sound in space. Relative volume adjustment ensures that the volume of the mixed audio data is within a preset range to achieve a volume level suitable for playback.

[0236] The preset volume range is set according to actual conditions and is not specifically limited in this application.

[0237] Perform sound image balance processing on the mixed audio data, adjust the relative strength of the left and right channels, and simulate the position of the sound in space.

[0238] Adjust the overall volume of the mixed audio data to reach a volume level suitable for playback.

[0239] B8: Audio Synthesis and Output Module The audio synthesis and output module converts MIDI time sequences into audible audio signals, outputs the processed audio signals, and transmits the processed audio signals to the audio playback device in real time for playback.

[0240] Finally, a complete piece of polyphonic music audio data is obtained, which can be further encoded into WAV, MP3 and other formats and output.

[0241] The audio synthesis and output module is the final step in the automatic composition system. Its task is to mix and render the optimal note sequence generated by the music element generation module into a final audio signal within various generation constraints and provide an interface for real-time playback control. This module utilizes industry-standard digital audio processing and software sound source technologies to generate high-fidelity, low-latency music audio and supports a variety of common audio file formats.

[0242] Finally, the system writes the synthesized PCM audio data to an audio buffer and provides a real-time playback control interface. Users can control audio playback, pause, stop, fast forward, rewind, and other operations through a graphical interface or programming interface. The system also provides audio file output, supporting common audio formats such as WAV, MP3, and OGG. Users can save the generated music to a local file for offline listening or secondary creation.

[0243] This solution significantly improves the efficiency and convenience of audio creation. The traditional composition process requires a solid foundation in music theory and extensive creative experience, which is time-consuming and laborious. This system uses intelligent algorithms to automatically generate musical elements such as melody, harmony, and rhythm. Users only need to provide simple parameters such as mode and tempo to quickly generate an expressive and musical original music clip, greatly lowering the threshold for composition and allowing ordinary users to experience the joy of creation.

[0244] The audio processing method of this solution can be integrated into various music production software as a plug-in or standalone tool, providing users with creative assistance. After a user uploads an audio sample, the present invention can automatically extract its rhythm, melody, and other characteristics, and automatically generate a matching music clip based on user-defined parameters such as instrument and style. The user can then further edit and modify the generated music clip according to their needs, ultimately completing the creation of the musical work.

[0245] This solution significantly reduces the barrier to entry and the difficulty of music creation. Even users without professional music theory knowledge can quickly create professional-level music with the help of artificial intelligence. Furthermore, this invention can also be applied to games, film, television, animation, and other fields, automatically generating background music that matches the images and plot, thereby improving the efficiency and quality of multimedia content production.

[0246] Compared with the prior art, the present invention has the following advantages and innovations:

[0247] Fully automated: The system can automatically extract music features, match appropriate notes, and generate matching music clips based on audio samples and user settings, without requiring users to possess professional music knowledge or creative skills.

[0248] Accurate rhythm: The present invention adopts beat extraction, note duration constraint and other technologies to ensure that the generated music rhythm is highly consistent with the original audio, avoiding rhythm deviation and distortion problems;

[0249] Strong musicality: This invention takes into account various musical rules such as melodic fluency and harmonic rationality when matching notes, and the generated music clips are close to the level of professional creation in terms of musicality;

[0250] High efficiency: The present invention uses a dynamic programming algorithm to search for note sequences, which can quickly lock the optimal solution among a large number of candidate notes, greatly improving the efficiency of music creation;

[0251] High-fidelity output: This invention adopts a high-quality sound library and advanced audio rendering technology. The generated audio signal has the characteristics of high fidelity and low distortion, and the listening experience is full and natural.

[0252] This solution enriches the audio library and provides more options for music production. Currently, commonly used music libraries rely primarily on manually created content, with limited categories and styles. This system can generate a wide variety of musical clips based on different parameter combinations. These can be used directly as source material or as inspiration for further creative work by composers. Furthermore, machine-generated music is not subject to copyright restrictions and can be freely used and distributed, reducing the cost of music production.

[0253] This solution provides new research ideas and experimental platforms for the field of audio artificial intelligence (AI). This automatic composition system integrates many cutting-edge technologies in the current field of music artificial intelligence, including music generation based on deep learning, audio signal processing, intelligent music editing, etc., and has important reference value for related theoretical research. The various modules of the system adopt a loosely coupled and replaceable design. Researchers can easily experiment with new algorithm models in a certain module. For example, using a Transformer-based sequence generation model to replace the original Markov chain model to quickly verify ideas and evaluate the effects. At the same time, the large-scale music data generated by the system can also be used for various experimental studies in the field of music artificial intelligence, such as model training and effect evaluation for tasks such as music style transfer and music emotion recognition.

[0254] This solution solves the following problems:

[0255] High threshold issues for non-professionals

[0256] Traditional music creation tools and methods are technically challenging for non-professionals, requiring users to possess a certain level of music theory and production skills. This invention automatically analyzes rhythmic information in recordings and generates corresponding music, enabling non-professionals to easily create high-quality music, thus lowering the barrier to entry for music creation.

[0257] The problem of low efficiency in music creation

[0258] The traditional music creation process requires a lot of time and effort, especially for the arrangement of rhythm and melody. This invention uses intelligent analysis and conversion technology to quickly generate musical elements that match the recording rhythm, greatly improving the efficiency of music creation and reducing creation time.

[0259] The problem of accurately extracting recording rhythm information

[0260] Accurately extracting rhythm information from recordings is a key technical challenge in achieving automatic music generation. This invention uses advanced audio signal processing technology to accurately identify parameters such as rhythm, beat, and tempo in recordings, ensuring that the generated music is highly consistent with the recorded rhythm, preserving the auditory effect and user experience.

[0261] Adaptation issues for multi-format recording

[0262] Audio files come in a variety of formats, such as WAV, MP3, and AAC. Parsing and processing audio files of different formats may lead to compatibility issues. This invention supports the input of multiple recording formats and, through a unified parsing and conversion mechanism, achieves seamless adaptation to recordings of different formats, ensuring the system's wide applicability.

[0263] This solution aims to provide a rhythm recording analysis and conversion technology that can automatically analyze the rhythm information in the recording and generate corresponding audio, so that non-professionals can quickly create audio works, greatly reducing the difficulty and labor cost of audio creation, and improving the efficiency of audio generation. This solution automatically extracts rhythm, beat, speed and other information from the recording through intelligent analysis and conversion of the rhythm in the recording, generates corresponding audio elements, and supports user-defined editing of music content. The technical solution of the present invention can be adapted to a variety of recording formats, and the generated audio works have a high degree of restoration and a good listening experience.

[0264] In the embodiment of the present application, in the process of creating a musical work, it is not necessary for the user to have music theory knowledge and music production skills. Instead, the user only needs to intelligently analyze and convert the rhythm in the audio file to be processed, automatically extract information such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, search for the optimal note sequence from each candidate note, and render the optimal note sequence using a preset rendering method to obtain an audio signal. Therefore, the process of creating a musical work does not require a lot of time and energy, and a music signal that conforms to the recording rhythm is quickly generated, thereby improving the efficiency of musical work creation. In addition, the use of advanced audio signal processing technology can accurately identify parameters such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, ensuring that the generated musical work is highly consistent with the rhythm of the audio file to be processed, so that the generated audio work has a high degree of restoration and a good auditory effect.

[0265] refer to Figure 2 As shown, an audio processing method disclosed in an embodiment of the present application is applicable to the above embodiment. Figure 1 The automatic composition system of the present invention mainly includes the following steps:

[0266] S201: Obtain the audio file to be processed;

[0267] S202: Extract key parameters from the audio file to be processed; wherein the key parameters include at least a beat list, a beat type, and a BPM speed parameter.

[0268] Key parameters. Key parameters are used to determine the style and characteristics of the generated music file.

[0269] The specific process of extracting key parameters from the audio file to be processed is shown in E1-E5.

[0270] E1: Preprocess the audio file to be processed; the preprocessing at least includes audio decoding, channel separation and amplitude normalization.

[0271] E2: Use digital signal processing technology to extract features from the pre-processed audio files and obtain a time series that reflects the changes in music energy.

[0272] E3: Perform threshold detection on the time series reflecting the change of music energy to obtain the time position sequence of the beat.

[0273] E4: Analyze the distribution pattern of adjacent beat intervals in the temporal position sequence of beats to obtain a beat interval sequence, and determine the beat type of the music based on a preset clustering algorithm and the beat interval sequence.

[0274] First, adjacent elements in the temporal position sequence of beats are subjected to difference calculation to obtain a beat interval sequence; wherein the beat interval sequence is used to represent the distribution pattern of adjacent beat intervals in the temporal position sequence of beats.

[0275] Secondly, the beat interval sequence is clustered using the K-Means clustering algorithm to obtain multiple cluster centers;

[0276] Then, the ratios corresponding to the multiple cluster centers are obtained, and the ratios corresponding to the multiple cluster centers are compared with the preset ratios for similarity to obtain multiple comparison results;

[0277] Finally, the ratio with the greatest similarity among multiple comparison results is determined as the beat type of the music.

[0278] E5: Calculate the average beat interval of the beat interval sequence to obtain the beat speed parameter.

[0279] The execution process and execution principle of E1-E5 are the same as those in the above embodiment. Figure 1 The execution process and execution principle of the rhythm analysis module are consistent, which can be referred to and will not be repeated here.

[0280] S203: Calculate the note duration range corresponding to the interval between each adjacent beat according to the beat type and beat speed parameters.

[0281] S204: When matching the note samples, search the note sample library for candidate notes that match the recording rhythm within the note duration range and conform to preset rules; wherein the preset rules are determined by a beat list, preset music parameters, and music theory constraints.

[0282] In S204, when matching note samples, note samples that meet preset rules within the note duration range are retrieved from the note sample library through query conditions, and candidate notes that match the recording rhythm are obtained according to the file storage path corresponding to the note samples.

[0283] S205: Searching for an optimal note sequence from each candidate note.

[0284] The optimal note sequence is searched for from each candidate note using a dynamic programming algorithm, which takes into account multiple constraints, including the range of note durations corresponding to beat intervals, pitch variation patterns, and dynamic variation patterns.

[0285] In S205, each candidate note is scored using a scoring function in a dynamic programming algorithm, and an optimal note sequence is obtained from each scored candidate note by backtracking the state.

[0286] Among them, the scoring function is used to score the rhythm matching, melodic fluency and harmonic rationality of each candidate note to ensure that the rhythm of the searched note sequence is consistent with the rhythm of the audio file to be processed, and to ensure that the musicality of the searched note sequence is optimal.

[0287] S206: Rendering the optimal note sequence using a preset rendering method to obtain an audio signal to complete the music creation process.

[0288] Writes audio signals to the audio buffer and provides a real-time playback control interface.

[0289] The execution process and execution principle of S201-S206 are the same as those in the above embodiment. Figure 1 The execution process and execution principle of the automatic composition system are consistent with those of the , which can be referred to and will not be described in detail here.

[0290] Specifically, the optimal note sequence is rendered through a preset rendering method to obtain an audio signal to complete the music creation process, as shown in F1-F7.

[0291] F1: performing a processing operation on the optimal note sequence according to the musical form template; the processing operation at least includes a repetition operation of the musical elements, a transformation operation of the musical elements, and a combination operation of the musical elements;

[0292] F2: Adding musical elements to the optimal note sequence after the processing operation to obtain a target music fragment; the adding operation at least includes adding chords and ornaments;

[0293] F3: Renders the target music clip into pulse code modulation (PCM) audio data using the preset software music library;

[0294] F4: Group the pulse code modulation audio data according to the instrument type to obtain the grouping results; each group of notes in the grouping results corresponds to a MIDI track;

[0295] F5: For each MIDI track, converts note events into Musical Instrument Digital Interface messages in chronological order; note events are used to indicate note start and note end;

[0296] F6: Transmits the Musical Instrument Digital Interface message to the software synthesizer (FluidSynth's synthesizer) for rendering, and obtains the audio digital signal generated by each MIDI track;

[0297] F7: Mix the audio digital signal to obtain the audio signal to complete the music creation process.

[0298] The execution process and execution principle of F1-F7 are the same as those in the above embodiment. Figure 1 The execution process and execution principle of B1-B8 are the same, which can be used as a reference and will not be repeated here.

[0299] The process of mixing the audio digital signals to obtain the audio signals is shown in G1-G3.

[0300] G1: Align the audio digital signals generated by each MIDI track in the time dimension;

[0301] G2: Perform weighted summation on the time-aligned audio digital signals to obtain mixed audio data;

[0302] G3: Performs image balancing and relative volume adjustment on the mixed audio data to obtain an audio signal. Image balancing is used to adjust the relative strength of the left and right channels and simulate the position of the sound in space. Relative volume adjustment is used to keep the volume of the mixed audio data within a preset volume range.

[0303] The execution process and execution principle of G1-G3 are the same as those in the above embodiment. Figure 1 The execution process and execution principle of D1-D3 are the same, which can be referred to and will not be repeated here.

[0304] In the embodiment of the present application, in the process of creating a musical work, it is not necessary for the user to have music theory knowledge and music production skills. Instead, the user only needs to intelligently analyze and convert the rhythm in the audio file to be processed, automatically extract information such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, search for the optimal note sequence from each candidate note, and render the optimal note sequence using a preset rendering method to obtain an audio signal. Therefore, the process of creating a musical work does not require a lot of time and energy, and a music signal that conforms to the recording rhythm is quickly generated, thereby improving the efficiency of musical work creation. In addition, the use of advanced audio signal processing technology can accurately identify parameters such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, ensuring that the generated musical work is highly consistent with the rhythm of the audio file to be processed, so that the generated audio work has a high degree of restoration and a good auditory effect.

[0305] Based on the above embodiment Figure 2 The disclosed audio processing method, the embodiment of the present application also discloses an audio processing device, such as Figure 3 As shown, the audio processing device includes:

[0306] An acquisition unit 301 is used to acquire an audio file to be processed;

[0307] The extraction unit 302 is used to extract key parameters from the audio file to be processed; wherein the key parameters include at least a beat list, a beat type, and a beat number and speed parameter;

[0308] The calculation unit 303 is used to calculate the note duration range corresponding to the interval between each adjacent beat according to the beat type and beat speed parameters;

[0309] A search unit 304 is configured to search the note sample library for candidate notes that match the recording rhythm within a note duration range and conform to a preset rule when matching the note sample; wherein the preset rule is determined by the beat list, preset music parameters, and music theory constraints;

[0310] Search unit 305, for searching for an optimal note sequence from each candidate note;

[0311] The rendering unit 306 is used to render the optimal note sequence using a preset rendering method to obtain an audio signal to complete the music creation process.

[0312] Furthermore, the extraction unit 302 includes:

[0313] A preprocessing module is used to preprocess the audio file to be processed; the preprocessing includes at least audio decoding, channel separation and amplitude normalization;

[0314] An extraction module is used to extract features from the pre-processed audio files using digital signal processing technology to obtain a time series reflecting the changes in music energy;

[0315] A threshold detection module is used to perform threshold detection on the time series reflecting the change of music energy to obtain the time position sequence of the beat;

[0316] An analysis and determination module is used to analyze the distribution pattern of adjacent beat intervals in the temporal position sequence of beats, obtain a beat interval sequence, and determine the beat type of the music based on a preset clustering algorithm and the beat interval sequence;

[0317] The calculation module is used to calculate the average beat interval of the beat interval sequence to obtain the beat speed parameter.

[0318] Furthermore, the analysis and determination modules include:

[0319] A calculation submodule is used to perform difference calculation on adjacent elements in the time position sequence of beats to obtain a beat interval sequence;

[0320] The clustering submodule is used to cluster the beat interval sequence using the K-Means clustering algorithm to obtain multiple cluster centers;

[0321] An acquisition submodule is used to obtain the ratios corresponding to the multiple cluster centers, and compare the ratios corresponding to the multiple cluster centers with the preset ratios to obtain multiple comparison results;

[0322] The determination submodule is used to determine the ratio with the greatest similarity among the multiple comparison results as the beat type of the music.

[0323] Furthermore, the search unit 304 includes:

[0324] A retrieval module is used to retrieve note samples that meet preset rules within the note duration range from the note sample library through query conditions when matching note samples;

[0325] The first acquisition module is used to acquire each candidate note that matches the recording rhythm according to the file storage path corresponding to the note sample.

[0326] Furthermore, the search unit 305 includes:

[0327] A scoring module is used to score each candidate note using a scoring function in a dynamic programming algorithm. The scoring function is used to score each candidate note based on its rhythmic matching, melodic fluency, and harmonic rationality to ensure that the rhythm of the searched note sequence is consistent with the rhythm of the audio file to be processed and that the musicality of the searched note sequence is optimal.

[0328] The second acquisition module is used to obtain the optimal note sequence from each scored candidate note through reverse backtracking state.

[0329] Furthermore, the rendering unit 306 includes:

[0330] A processing operation module, configured to perform processing operations on the optimal note sequence according to the musical form template; the processing operations at least include repetition operations of musical elements, transformation operations of musical elements, and combination operations of musical elements;

[0331] An adding operation module is used to perform an adding operation of musical elements on the optimal note sequence after the processing operation to obtain a target music segment; the adding operation includes at least adding chords and ornaments;

[0332] A first rendering module is used to render the target music clip into pulse code modulation audio data using a preset software music library;

[0333] a grouping module for grouping the pulse code modulation audio data according to the instrument type to obtain a grouping result; each group of notes in the grouping result corresponds to a musical instrument digital interface track;

[0334] a conversion module, configured to convert, for each MIDI track, a note event into a MIDI message in chronological order; wherein the note event is used to indicate a note start and a note end;

[0335] a second rendering module, configured to transmit the MIDI message to the software synthesizer for rendering, and obtain an audio digital signal generated by each MIDI track;

[0336] The mixing module is used to mix the audio digital signals to obtain audio signals to complete the music creation process.

[0337] Furthermore, the mixing module includes:

[0338] An alignment submodule, for aligning the audio digital signals generated by each MIDI track in a time dimension;

[0339] A weighted summation submodule is used to perform weighted summation on the audio digital signals aligned in the time dimension to obtain mixed audio data;

[0340] The processing and adjustment submodule is used to perform sound and image balance processing and relative volume adjustment on the mixed audio data to obtain an audio signal; among them, the sound and image balance processing is used to adjust the relative strength of the left and right channels and simulate the position of the sound in space; the relative volume adjustment is used to keep the volume of the mixed audio data within a preset volume range.

[0341] Furthermore, the audio processing device further includes:

[0342] The writing unit is used to write the audio signal into the audio buffer and provide a real-time playback control interface.

[0343] In the embodiment of the present application, in the process of creating a musical work, it is not necessary for the user to have music theory knowledge and music production skills. Instead, the user only needs to intelligently analyze and convert the rhythm in the audio file to be processed, automatically extract information such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, search for the optimal note sequence from each candidate note, and render the optimal note sequence using a preset rendering method to obtain an audio signal. Therefore, the process of creating a musical work does not require a lot of time and energy, and a music signal that conforms to the recording rhythm is quickly generated, thereby improving the efficiency of musical work creation. In addition, the use of advanced audio signal processing technology can accurately identify parameters such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, ensuring that the generated musical work is highly consistent with the rhythm of the audio file to be processed, so that the generated audio work has a high degree of restoration and a good auditory effect.

[0344] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned audio processing method.

[0345] The present application also provides an electronic device, the structure of which is shown in FIG. Figure 4 As shown, it specifically includes a memory 401 and one or more instructions 402, wherein the one or more instructions 402 are stored in the memory 401 and are configured to be executed by one or more processors 403 to perform the above-mentioned audio processing method.

[0346] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0347] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For system-related embodiments, since they are generally similar to method-related embodiments, their description is relatively simple. For relevant details, refer to the description of the method-related embodiments.

[0348] The steps in the methods of the various embodiments of the present application can be adjusted in sequence, combined, and deleted according to actual needs.

[0349] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0350] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

[0351] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An audio processing method, characterized in that: The method comprises: Get the audio file to be processed; Extracting key parameters from the audio file to be processed; wherein the key parameters include at least a beat list, a beat type, and a beat number speed parameter; Calculating the note duration range corresponding to the interval between each adjacent beat according to the beat type and the beat speed parameter; When matching the note samples, searching the note sample library for candidate notes that match the recording rhythm within the note duration range and conform to preset rules; wherein the preset rules are determined by the beat list, preset music parameters, and music theory constraints; Searching for the optimal note sequence from each candidate note; The optimal note sequence is rendered using a preset rendering method to obtain an audio signal to complete the music creation process.

2. The method according to claim 1, characterized in that The step of extracting key parameters from the audio file to be processed includes: Preprocessing the audio file to be processed; the preprocessing at least includes audio decoding, channel separation and amplitude normalization; The pre-processed audio files are subjected to feature extraction through digital signal processing technology to obtain a time series reflecting the changes in music energy; Performing threshold detection on the time series reflecting the change of music energy to obtain a time position sequence of beats; Analyzing the distribution pattern of adjacent beat intervals in the temporal position sequence of the beats to obtain a beat interval sequence, and determining the beat type of the music based on a preset clustering algorithm and the beat interval sequence; The average beat interval of the beat interval sequence is calculated to obtain a beat speed parameter.

3. The method according to claim 2, characterized in that The analyzing the distribution pattern of adjacent beat intervals in the temporal position sequence of the beats to obtain a beat interval sequence, and determining the beat type of the music based on a preset clustering algorithm and the beat interval sequence, includes: performing difference calculation on adjacent elements in the temporal position sequence of the beats to obtain a beat interval sequence; Clustering the beat interval sequence using a K-Means clustering algorithm to obtain multiple cluster centers; Obtaining ratios corresponding to a plurality of cluster centers, and performing similarity comparison between the ratios corresponding to the plurality of cluster centers and a preset ratio to obtain a plurality of comparison results; The ratio with the greatest similarity among the multiple comparison results is determined as the beat type of the music.

4. The method according to claim 1, wherein When matching the note samples, searching the note sample library for candidate notes that match the recording rhythm and conform to preset rules within the note duration range includes: When matching note samples, the note samples that meet the preset rules within the note duration range are retrieved from the note sample library through the query conditions; According to the file storage path corresponding to the note sample, each candidate note matching the recording rhythm is obtained.

5. The method according to claim 1, wherein The step of searching for an optimal note sequence from each candidate note comprises: Scoring each candidate note using a scoring function in a dynamic programming algorithm; wherein the scoring function is used to score each candidate note based on its rhythmic matching, melodic fluency, and harmonic rationality to ensure that the rhythm of the searched note sequence is consistent with the rhythm of the audio file to be processed and that the musicality of the searched note sequence is optimal; By backtracking the state, the optimal note sequence is obtained from each candidate note after scoring.

6. The method according to claim 1, characterized in that The step of rendering the optimal note sequence in a preset rendering manner to obtain an audio signal to complete the music composition process includes: Performing a processing operation on the optimal note sequence according to a musical form template; the processing operation at least includes a repetition operation of a musical element, a transformation operation of a musical element, and a combination operation of a musical element; Performing an operation of adding musical elements to the optimal note sequence after the processing operation to obtain a target music segment; the adding operation at least includes adding chords and ornaments; Rendering the target music clip into pulse code modulation audio data using a preset software music library; Grouping the pulse code modulation audio data according to instrument type to obtain a grouping result; wherein each group of notes in the grouping result corresponds to a musical instrument digital interface track; For each MIDI track, converting a note event into a MIDI message in chronological order; wherein the note event is used to indicate a note start and a note end; Transmitting the Musical Instrument Digital Interface message into a software synthesizer for rendering to obtain an audio digital signal generated by each Musical Instrument Digital Interface track; The audio digital signals are mixed to obtain audio signals to complete the music creation process.

7. The method according to claim 6, characterized in that The step of mixing the audio digital signal to obtain an audio signal comprises: Aligning the audio digital signals generated by each MIDI track in time dimension; Perform weighted summation on the audio digital signals aligned in the time dimension to obtain mixed audio data; The mixed audio data is subjected to sound and image balance processing and relative volume adjustment to obtain an audio signal; wherein the sound and image balance processing is used to adjust the relative strength of the left and right channels and simulate the position of the sound in space; the relative volume adjustment is used to ensure that the volume of the mixed audio data is within a preset volume range.

8. The method according to claim 1, characterized in that Also includes: The audio signal is written into an audio buffer, and a real-time playback control interface is provided.

9. An audio processing device, characterized in that: The device comprises: An acquisition unit, used to acquire the audio file to be processed; An extraction unit, configured to extract key parameters from the audio file to be processed; wherein the key parameters include at least a beat list, a beat type, and a beat number and speed parameter; a calculation unit, configured to calculate a note duration range corresponding to an interval between adjacent beats according to the beat type and the beat velocity parameter; a search unit configured to search, when matching a note sample, from the note sample library for candidate notes that match the recording rhythm and conform to a preset rule within the note duration range; wherein the preset rule is determined by the beat list, preset music parameters, and music theory constraints; A search unit, used to search for an optimal note sequence from various candidate notes; The rendering unit is used to render the optimal note sequence in a preset rendering method to obtain an audio signal to complete the music creation process.

10. The device according to claim 9, characterized in that The extraction unit comprises: A preprocessing module, configured to preprocess the audio file to be processed; the preprocessing includes at least audio decoding, channel separation, and amplitude normalization; The extraction module is used to extract features from the pre-processed audio files using digital signal processing technology to obtain a time series reflecting the changes in music energy; A threshold detection module is used to perform threshold detection on the time series reflecting the change of music energy to obtain a time position sequence of beats; an analysis and determination module, configured to analyze the distribution pattern of adjacent beat intervals in the temporal position sequence of the beats to obtain a beat interval sequence, and determine the beat type of the music based on a preset clustering algorithm and the beat interval sequence; The calculation module is used to calculate the average beat interval of the beat interval sequence to obtain a beat speed parameter.

Images