Audio processing method and device

By extracting beat and beat parameters in audio files, automatically matching the note sequence and rendering, the problem of existing music creation tools being inefficient for non-professionals is solved, and efficient and low-threshold music generation is achieved.

CN118942481AActive Publication Date: 2024-11-12小芒电子商务有限责任公司

Patent Information

Application Number
CN202411009542.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-12
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The existing musical work creation tools have a high technical threshold for non-professionals, and require music theory knowledge and music production skills, resulting in low creative efficiency.

Method used

By extracting beat list, beat type and beat count speed parameters in the audio file, automatically match and render the note sequence to generate music signals that match the recording rhythm, and use digital signal processing technology and dynamic programming algorithms to improve music creation efficiency.

Benefits of technology

It is possible to quickly generate musical works with high restoration and good auditory effects without having music theory knowledge and music production skills, which significantly improves the efficiency of music creation and lowers the creative threshold.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942481A_ABST
    Figure CN118942481A_ABST
Patent Text Reader

Abstract

The invention discloses an audio processing method and device, and relates to the technical field of audio processing, a to-be-processed audio file is obtained, key parameters are extracted from the to-be-processed audio file, the key parameters at least comprise a beat list, a beat type and a beat number speed parameter, and according to the beat type and the beat number speed parameter, the beat number speed parameter is calculated; a note duration range corresponding to the interval between every two adjacent rhythms is calculated, when a note sample is matched, candidate notes matched with the recording rhythm and meeting a preset rule in the note duration range are searched for from a note sample library, the preset rule is determined by a rhythm list, preset music parameters and music theory constraint conditions, and the preset music parameters are determined by the rhythm list; and searching an optimal note sequence from the candidate notes, and rendering the optimal note sequence in a preset rendering mode to obtain an audio signal so as to complete a musical work creation process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and more specifically, to an audio processing method and device. Background Art

[0002] With the development of audio signal processing technology, more and more tools and audio signal processing technologies are used for audio analysis and processing. Through audio signal processing technology, rich audio features such as frequency, amplitude, rhythm and timbre can be extracted from audio files such as recordings, and music creation tools can be used to create music works based on audio features.

[0003] However, the existing music creation tools have high technical barriers for non-professionals, requiring users to have certain music theory knowledge and music production skills, as well as the arrangement of rhythm and melody in the process of music creation. Therefore, the traditional music creation process requires a lot of time and energy, resulting in low efficiency in music creation.

[0004] Therefore, how to improve the efficiency of music composition is a problem that needs to be solved urgently in this application. Summary of the invention

[0005] In view of this, the present application discloses an audio processing method and device, which aims to quickly generate a music signal that conforms to the recording rhythm, improve the efficiency of music creation, and make the generated audio work have a high degree of restoration and good auditory effect.

[0006] In order to achieve the above purpose, the disclosed technical solution is as follows:

[0007] In a first aspect, the present application discloses an audio processing method, the method comprising:

[0008] Get the audio file to be processed;

[0009] Extracting key parameters from the audio file to be processed; wherein the key parameters at least include a beat list, a beat type and a beat number speed parameter;

[0010] Calculating the note duration range corresponding to the interval between each adjacent beat according to the beat type and the beat speed parameter;

[0011] When matching the note samples, searching the note sample library for candidate notes that match the recording rhythm and meet preset rules within the note duration range; wherein the preset rules are determined by the beat list, preset music parameters and music theory constraints;

[0012] Searching for the best note sequence from each candidate note;

[0013] The optimal note sequence is rendered in a preset rendering manner to obtain an audio signal to complete the music work creation process.

[0014] Preferably, extracting key parameters from the audio file to be processed includes:

[0015] Preprocessing the audio file to be processed; the preprocessing at least includes audio decoding, channel separation and amplitude normalization;

[0016] The pre-processed audio files are subjected to feature extraction through digital signal processing technology to obtain a time series reflecting the energy changes of the music;

[0017] Performing threshold detection on the time series reflecting the change of music energy to obtain a time position sequence of beats;

[0018] Analyzing the distribution pattern of adjacent beat intervals in the temporal position sequence of the beats to obtain a beat interval sequence, and determining the beat type of the music according to a preset clustering algorithm and the beat interval sequence;

[0019] The average beat interval of the beat interval sequence is calculated to obtain a beat speed parameter.

[0020] Preferably, the step of analyzing the distribution pattern of the intervals between adjacent beats in the temporal position sequence of the beats to obtain a beat interval sequence, and determining the beat type of the music according to a preset clustering algorithm and the beat interval sequence, comprises:

[0021] Performing difference calculation on adjacent elements in the temporal position sequence of the beats to obtain a beat interval sequence;

[0022] Clustering the beat interval sequence using a K-Means clustering algorithm to obtain multiple cluster centers;

[0023] Obtaining ratios corresponding to a plurality of cluster centers, and performing similarity comparison between the ratios corresponding to the plurality of cluster centers and a preset ratio to obtain a plurality of comparison results;

[0024] The ratio with the greatest similarity among the multiple comparison results is determined as the beat type of the music.

[0025] Preferably, when matching the note samples, searching the note sample library for candidate notes that match the recording rhythm and conform to preset rules within the note duration range includes:

[0026] When matching the note samples, the note samples that meet the preset rules within the note duration range are retrieved from the note sample library through the query conditions;

[0027] According to the file storage path corresponding to the note sample, each candidate note matching the recording rhythm is obtained.

[0028] Preferably, searching for the optimal note sequence from each candidate note includes:

[0029] Scoring each candidate note through a scoring function in a dynamic programming algorithm; wherein the scoring function is used to score the rhythm matching, melodic fluency, and harmonic rationality of each candidate note, so as to ensure that the rhythm of the searched note sequence is consistent with the rhythm of the audio file to be processed, and to ensure that the musicality of the searched note sequence is optimal;

[0030] By backtracking the state, the optimal note sequence is obtained from each scored candidate note.

[0031] Preferably, the rendering of the optimal note sequence by a preset rendering method to obtain an audio signal to complete the music composition process includes:

[0032] Processing the optimal note sequence according to the musical form template; the processing operation at least includes a repetition operation of the music element, a transformation operation of the music element and a combination operation of the music element;

[0033] Performing an operation of adding music elements to the optimal note sequence after the processing operation to obtain a target music clip; the adding operation at least includes adding chords and ornaments;

[0034] Rendering the target music clip into pulse code modulated audio data through a preset software music library;

[0035] The pulse code modulation audio data is grouped according to the instrument type to obtain a grouping result; each group of notes in the grouping result corresponds to a musical instrument digital interface track;

[0036] For each MIDI track, converting the note event into a MIDI message in time sequence; wherein the note event is used to indicate the start and end of a note;

[0037] The MIDI message is transmitted to a software synthesizer for rendering to obtain an audio digital signal generated by each MIDI track;

[0038] The audio digital signals are mixed to obtain audio signals to complete the music creation process.

[0039] Preferably, mixing the audio digital signal to obtain an audio signal comprises:

[0040] Aligning the audio digital signals generated by each MIDI track in time dimension;

[0041] Performing weighted summation on the audio digital signals aligned in the time dimension to obtain mixed audio data;

[0042] The mixed audio data is subjected to sound and image balance processing and relative volume adjustment to obtain an audio signal; wherein the sound and image balance processing is used to adjust the relative strength of the left and right channels and simulate the position of the sound in space; and the relative volume adjustment is used to make the volume of the mixed audio data within a preset volume range.

[0043] Preferably, it also includes:

[0044] The audio signal is written into an audio buffer, and a real-time playback control interface is provided.

[0045] A second aspect of the present application discloses an audio processing device, the device comprising:

[0046] An acquisition unit, used for acquiring an audio file to be processed;

[0047] An extraction unit, used for extracting key parameters from the audio file to be processed; wherein the key parameters at least include a beat list, a beat type and a beat number speed parameter;

[0048] A calculation unit, used to calculate the note duration range corresponding to the interval between each adjacent beat according to the beat type and the beat speed parameter;

[0049] A search unit, used for searching, when matching the note samples, each candidate note that matches the recording rhythm and conforms to a preset rule within the note duration range from the note sample library; wherein the preset rule is determined by the beat list, preset music parameters and music theory constraints;

[0050] A search unit, used for searching for an optimal note sequence from various candidate notes;

[0051] The rendering unit is used to render the optimal note sequence in a preset rendering manner to obtain an audio signal to complete the music work creation process.

[0052] Preferably, the extraction unit comprises:

[0053] A preprocessing module, used for preprocessing the audio file to be processed; the preprocessing at least includes audio decoding, channel separation and amplitude normalization;

[0054] An extraction module is used to extract features from the pre-processed audio file through digital signal processing technology to obtain a time series reflecting the change of music energy;

[0055] A threshold detection module, used to perform threshold detection on the time series reflecting the change of music energy to obtain a time position sequence of beats;

[0056] An analysis and determination module, used for analyzing the distribution law of adjacent beat intervals in the temporal position sequence of the beats, obtaining a beat interval sequence, and determining the beat type of the music according to a preset clustering algorithm and the beat interval sequence;

[0057] The calculation module is used to calculate the average beat interval of the beat interval sequence to obtain the beat number speed parameter.

[0058] It can be seen from the above technical scheme that the present application discloses an audio processing method and device, which obtains an audio file to be processed, extracts key parameters from the audio file to be processed, wherein the key parameters include at least a beat list, a beat type and a beat number speed parameter, and calculates the note duration range corresponding to the interval between each adjacent beat according to the beat type and the beat number speed parameter. When matching the note sample, search the note sample library for each candidate note that meets the preset rules and matches the recording rhythm within the note duration range, wherein the preset rules are determined by the beat list, preset music parameters and music theory constraints, search for the optimal note sequence from each candidate note, render the optimal note sequence by a preset rendering method, and obtain an audio signal to complete the music creation process. Through the above scheme, in the process of creating music works, users do not need to have music theory knowledge and music production skills. They only need to intelligently analyze and convert the rhythm in the audio file to be processed, automatically extract the rhythm, beat type, beat number and speed parameters in the audio file to be processed, search for the optimal note sequence from each candidate note, and render the optimal note sequence through a preset rendering method to obtain an audio signal. Therefore, there is no need to spend a lot of time and energy in the process of creating music works, and quickly generate music signals that meet the recording rhythm, thereby improving the efficiency of music work creation. In addition, the use of advanced audio signal processing technology can accurately identify parameters such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, ensuring that the generated music work is highly consistent with the rhythm of the audio file to be processed, so that the generated audio work has a high degree of restoration and good auditory effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0060] Figure 1A schematic diagram of the structure of an automatic composition system disclosed in an embodiment of the present application;

[0061] Figure 2 A flowchart of an audio processing method disclosed in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of the structure of an audio processing device disclosed in an embodiment of the present application;

[0063] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0066] As can be seen from the background technology, the existing music creation tools have a high technical threshold for non-professionals, requiring users to have certain music theory knowledge and music production skills, as well as the arrangement of rhythm and melody in the process of music creation. Therefore, the traditional music creation process requires a lot of time and energy, resulting in low efficiency in music creation. Therefore, how to improve the efficiency of music creation is a problem that this application urgently needs to solve.

[0067] In order to solve the above problems, the present application discloses an audio processing method and device. In the process of creating a musical work, it is not necessary to require the user to have music theory knowledge and music production skills. It only needs to intelligently analyze and convert the rhythm in the audio file to be processed, automatically extract the rhythm, beat type, beat number and speed parameters in the audio file to be processed, search for the optimal note sequence from each candidate note, and render the optimal note sequence through a preset rendering method to obtain an audio signal. Therefore, there is no need to spend a lot of time and energy in the process of creating a musical work, and quickly generate a music signal that conforms to the recording rhythm, thereby improving the efficiency of creating a musical work. In addition, by using advanced audio signal processing technology, it is possible to accurately identify parameters such as the rhythm, beat type, beat number and speed parameters in the audio file to be processed, ensuring that the generated musical work is highly consistent with the rhythm of the audio file to be processed, so that the generated audio work has a high degree of restoration and a good auditory effect. The specific implementation method is specifically described by the following embodiments.

[0068] refer to Figure 1 As shown, it is a structural schematic diagram of an automatic composition system disclosed in an embodiment of the present application, and the automatic composition system includes a rhythm analysis module, a rhythm generation module, a pitch sequence generation module, an orchestration module, a music element generation module and an audio synthesis and output module.

[0069] The data interaction process between the rhythm analysis module, the rhythm generation module, the pitch sequence generation module, the instrumentation module, the music element generation module and the audio synthesis and output module is as follows:

[0070] Rhythm analysis module:

[0071] When the rhythm parsing module receives the audio file to be processed (the audio file uploaded by the user or the recorded audio file), it parses the audio file to be processed to extract key parameters and passes them to the rhythm generation module. Among them, the key parameters include the beat list, beat type, beat number (BPM) speed parameters, the number of notes entered by the user, the mode and other parameters. The key parameters can be used to determine the style of the generated music file.

[0072] The process of the rhythm analysis module extracting key parameters from the audio file to be processed is as follows:

[0073] First, the audio file to be processed is preprocessed, including audio decoding, channel separation, amplitude normalization, etc.

[0074] The automatic composition system uses a sixth-order Butterworth low-pass filter to smooth and reduce noise on the original audio signal (i.e., the audio file to be processed). The energy envelope curve E(t) is obtained by taking the absolute value and smoothing the filtered signal, as shown in formula (1).

[0075] E(t)=Smooth(|x(t)*h(t)|) (1)

[0076] Among them, x(t) is the original audio signal; h(t) is the unit impulse response of the filter; * represents the convolution operation; || represents taking the absolute value; the smoothing function Smooth can select common algorithms such as moving average and Gaussian smoothing.

[0077] It should be noted that the Butterworth filter has a flat passband response and linear phase characteristics, which can effectively remove high-frequency noise while retaining rhythm information. The cutoff frequency of the filter can be set to 150Hz, which can cover the fundamental frequency range of most percussion instruments.

[0078] Secondly, digital signal processing technology is used to extract features from audio to obtain a time series reflecting the energy changes of music. Threshold detection is performed on the time series reflecting the energy changes of music to obtain the time position sequence of beats.

[0079] It should be noted that music energy change refers to the law of how the amplitude of an audio signal changes over time. By analyzing features such as the short-time energy or Mel-Frequency Cepstral Coefficients (MFCC) of audio, a time series reflecting the rhythm and strength of the music can be obtained.

[0080] Then, the distribution law of the intervals between adjacent beats in the temporal position sequence of the beats is analyzed to obtain the beat interval sequence;

[0081] Finally, according to a preset clustering algorithm (such as K-Means clustering algorithm) and the beat interval sequence, the beat type of the music is determined, the average beat interval of the beat interval sequence is calculated, and the beat number speed parameter is obtained.

[0082] Specifically, according to the preset clustering algorithm and the beat interval sequence, the beat type of the music is determined, the average beat interval of the beat interval sequence is calculated, and the process of obtaining the beat number speed parameter is shown in A1-A3:

[0083] A1: Perform difference calculation on adjacent elements in the beat time position sequence to obtain a beat interval sequence; the beat interval sequence is used to represent the distribution law of adjacent beat intervals in the beat time position sequence;

[0084] A2: Cluster the beat interval sequence using the K-Means clustering algorithm to obtain multiple cluster centers and obtain the corresponding ratios of the multiple cluster centers;

[0085] A3: Compare the ratios corresponding to the multiple cluster centers with the preset ratio to obtain multiple comparison results, and determine the ratio with the greatest similarity among the multiple comparison results as the beat type of the music.

[0086] In the process of parsing the audio file to be processed, beat detection is involved. The key to beat detection is to find the peak point in the energy envelope curve, that is, the location of the beat. The traditional method uses a fixed threshold to compare the energy value, but it is difficult to adapt to music of different styles and rhythm density. This system innovatively proposes an adaptive threshold algorithm, which dynamically adjusts the threshold function to detect beats of different intensities by performing a sliding window analysis on the energy envelope.

[0087] The adaptive threshold algorithm first divides E(t) into window segments of length L, and overlapping is allowed between adjacent windows. For the i-th window segment, its expression is shown in formula (2).

[0088] Ei=[E(t)fort inrange(i*H, (i+1)*H)](2)

[0089] Where i is the window number; H is the step size of the window, satisfying L>H; Ei is the window segment.

[0090] Formula (2) indicates segmenting the energy envelope curve E(t) and extracting the subsequence corresponding to the i-th window.

[0091] For each window segment, the following feature parameters are extracted:

[0092] Ei_avg: average energy of the fragment;

[0093] Ei_max: maximum energy of the fragment;

[0094] Ei_std: energy standard deviation of the fragment;

[0095] Ei_ratio: the ratio of maximum energy to average energy, Ei_ratio = Ei_max / Ei_avg.

[0096] Then, the automatic composition system uses a trained logistic regression model to dynamically calculate the threshold coefficient wi according to the above feature parameters, and its calculation formula is shown in formula (3).

[0097] wi=Logistic(a0+a1Ei_avg+a2Ei_max+a3Ei_std+a4Ei_ratio) (3)

[0098] Among them, a0, a1, a2, a3, and a4 are all model parameters, which require the use of labeled beat data for offline training; Logistic is the Logistic function.

[0099] For the threshold value (THi) of the i-th segment, the calculation between wi is as shown in formula (4).

[0100] THi=wi*Ei_max(4)

[0101] Among them, wi is between 0 and 1, which measures the relative size of the beat intensity. When Ei_max is much larger than the average energy, wi approaches 1, and the threshold is equal to the peak value; when Ei_max is close to the average energy, wi approaches 0, and the threshold is less than the peak value. In this way, the beat type with strong and weak changes can be detected adaptively.

[0102] In each segment, as long as the energy exceeds THi, it is marked as a beat. If the interval between adjacent beats is less than 100ms, the one with the largest energy is retained. The algorithm outputs the time position sequence of the beats. The time position sequence of the beats is obtained by threshold detection of the time sequence of music energy changes. There is a corresponding relationship between the time position sequence of the beats and the original time sequence.

[0103] It should be noted that energy refers to the instantaneous amplitude square of the audio signal, which reflects the intensity change of the music sound. What is analyzed here is the music energy of the audio files uploaded or recorded by the user.

[0104] The original time series refers to the time series of changes in music energy.

[0105] The input of the algorithm is the time domain waveform of the music signal, and the time position sequence of the beats is obtained through the following steps:

[0106] Divide the music signal into several short segments of fixed length (e.g. 10 milliseconds);

[0107] Calculate the energy value of each short-time segment to form a time series of energy changes E = [E1, E2, ..., Em], where Ei represents the energy value of the i-th segment and m is the total number of segments. This time series of energy changes E is the original time series;

[0108] In each segment, the beat is detected according to the adaptive threshold THi. The calculation of THi takes into account the maximum value Ei_max and the average value Ei_avg of the energy in the segment, as well as an adjustable weight parameter wi. When the energy in the segment exceeds THi, it is marked as a beat;

[0109] Post-process the detected beats to remove the beats with too small intervals (e.g., less than 100 milliseconds) and retain the one with larger energy;

[0110] The time position sequence of the output beats B = [b1, b2, ..., bn], where bi represents the corresponding position of the i-th beat in the original time sequence E.

[0111] Among them, the beat time position sequence B is obtained by threshold detection of the original energy change time series E. Each beat position bi corresponds to a segment position in the original sequence E. Therefore, the original time series refers to the energy change sequence E of the music signal, which describes the energy distribution characteristics of the music signal in the time dimension. The beat detection algorithm obtains a higher-level rhythm structure representation, namely the beat time position sequence B, by analyzing this original sequence.

[0112] Based on the temporal position sequence of the beats, the beat type of the music is determined by analyzing the distribution law of the intervals between adjacent beats using a preset clustering algorithm (such as the K-Means clustering algorithm). Assume that the beat interval sequence is as shown in formula (5).

[0113] D=[d1,d2,...,dn](5)

[0114] Wherein, D is the beat interval sequence, which refers to the time interval sequence between two adjacent beat time positions, and can be obtained by taking the difference between adjacent elements in the beat sequence, such as [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, ...]; di represents the time interval between the i-th beat and the i+1-th beat.

[0115] The preset clustering algorithm includes but is not limited to the K-Means clustering algorithm. The preset clustering algorithm of this solution is preferably the K-Means clustering algorithm.

[0116] The K-Means algorithm is used to cluster D, and the number of categories K is 3 (corresponding to 2 beats, 3 beats, and 4 beats). The expressions of the three cluster centers obtained by clustering are shown in formula (6).

[0117] C = [c1, c2, c3] (6)

[0118] Among them, C is the clustering result; c1 <c2<c3。

[0119] Compare the similarities of c1:c2:c3 with 1:2:3, 1:2:4, and 1:3:4, and the ratio with the greatest similarity is determined as the beat type of the music. For example, c1:c2:c3=1:1.98:3.96 is closest to 1:2:4, so the beat type of the music is considered to be 4 / 4.

[0120] By analyzing the distribution of adjacent beat intervals, the K-Means clustering algorithm is used to determine the beat type of the music. The adjacent beat interval here is actually the difference between two adjacent time positions in the beat sequence. In the above example, the adjacent beat interval sequence is [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, ...], indicating that the time interval between each beat is 0.5 seconds, corresponding to the duration of a quarter note. By clustering and analyzing these interval values, it can be inferred that the beat type of the music is 4 / 4.

[0121] Finally, the BPM speed parameter is obtained by calculating the average beat interval. The specific calculation process is shown in formula (7).

[0122] BPM=60 / Mean(D) (7)

[0123] Where Mean means taking the average value; D is the beat interval sequence; Mean(D) calculates the average value of D in seconds.

[0124] Rhythm generation module:

[0125] The rhythm generation module adopts a rule-based rhythm pattern generation algorithm to randomly select rhythm patterns that meet the beat number and speed requirements from a predefined rhythm pattern library.

[0126] A rhythmic pattern that meets the requirements of beat number and speed refers to a rhythmic sequence that meets the following conditions:

[0127] 1. The total number of beats of the rhythm pattern is consistent with the number of beats specified by the user;

[0128] 2. The note value distribution in the rhythm pattern matches the speed parameter, for example, 120 BPM corresponds to 0.5 seconds per beat;

[0129] 3. Rhythm patterns are usually composed of several bars, each of which has a specified beat type. Rhythm patterns play a key role in the subsequent generation of musical elements. The note sequence needs to strictly follow the time structure of the rhythm pattern to ensure that the generated music is consistent with the user's input requirements in terms of rhythm. At the same time, the rhythm pattern will also affect the distribution of note intensity changes.

[0130] The rhythm generation module receives the key parameters and calculates the note duration range corresponding to the interval between each adjacent beat according to the beat type and beat speed parameters in the key parameters.

[0131] When matching note samples, the rhythm generation module searches the note sample library for candidate notes that match the recording rhythm within the note duration range and meet preset rules; the preset rules are determined by the beat list, preset music parameters, and music theory constraints. The music element generation module generates a note sequence that matches the rhythm of the processed audio file based on the beat list and the music parameters specified by the user (such as mode, number of notes, etc.).

[0132] The preset rules include but are not limited to the following examples:

[0133] Matching rules between note duration and beat: For example, in a 4 / 4 measure, the note duration can only be 1 / 4, 1 / 8, 1 / 16, etc.

[0134] Constraints on interval jumps: For example, within a musical passage, the pitch difference between two adjacent notes should not exceed one octave;

[0135] Chord and melody matching rules: within a chord, the melody notes need to match the constituent notes of the chord;

[0136] Constraints of musical style: For example, when composing a jazz-style piece of music, the choice of notes needs to conform to the scales, rhythm patterns and other characteristics of jazz music.

[0137] The above rules can be summarized and refined through music theory knowledge and adjusted appropriately according to specific needs.

[0138] The rhythm generation module searches for the optimal note sequence from each candidate note through a dynamic programming algorithm. The dynamic programming algorithm comprehensively considers multiple music theory constraints, including the note duration range corresponding to the beat interval, the pitch change law, the dynamic change law, etc., to search for the note combination sequence with the highest score, which is regarded as the optimal note sequence.

[0139] Specifically, when using the dynamic programming algorithm to search for the optimal note sequence, you can follow the following ideas:

[0140] State definition: Let dp[i][j] represent the optimal score when matching the i-th beat and using the j-th candidate note;

[0141] Initial condition: dp[0][j] = 0, which means that in the 0th beat, no candidate note is used and the score is 0;

[0142] State transfer equation: dp[i][j] = max(dp[i-1][k] + score(k, j)), which means the optimal score of using the jth candidate note in the i-th beat can be obtained by using the kth candidate note in the i-1th beat and then transferring to the jth note;

[0143] The scoring function score(k, j) takes into account the rhythm matching degree, melodic fluency and harmonic rationality of the transition from the kth note to the jth note, and uses the weighted summation method for calculation;

[0144] Final result: After the dynamic programming algorithm is completed, dp[n][j] means the best score of the jth candidate note after matching all n beats. By backtracking the dp array, the optimal note sequence can be obtained.

[0145] The rhythm generation module is based on rhythm information and automatically generates expressive and musical melody fragments through a series of algorithms such as music sample matching, dynamic planning path search, and ADSR envelope synthesis.

[0146] The construction of the music sample library is the basis of the rhythm generation module. This solution collects a large number of single-note samples played by various instruments, including but not limited to piano, guitar, violin, flute, etc., covering different pitches (C2-C7, with a range of more than 5 octaves), durations (from 32nd notes to whole notes), and dynamics (pp to ff, corresponding to MIDI dynamic values ​​20-120).

[0147] Each note sample has its fundamental frequency, overtones, envelope and other features extracted, and a multi-level index is established based on instrument type, pitch and duration. Given a query condition, the automatic composition system can return the path of the matching sample file in sub-second time.

[0148] It should be noted that the query condition refers to a series of parameters used when retrieving and matching note samples in the music sample library, which is automatically generated by the system based on user input and music generation requirements.

[0149] The query conditions include but are not limited to the following aspects: instrument type, pitch range, duration range, velocity range, etc.

[0150] Returning the hit sample file path means retrieving the note samples that meet the conditions in the music sample library according to the query conditions, that is, using the rule-based rhythm pattern generation algorithm, randomly selecting the note samples of the rhythm pattern that meet the beat number and speed requirements from the predefined rhythm pattern library, and returning the file storage path of these note samples. The purpose of this step is to find all the candidate notes that may be used to form a melody.

[0151] Assume that the expression of the temporal position sequence of the beats output by the rhythm analysis module is as shown in formula (8):

[0152] B=[b1,b2,...,bn](8)

[0153] Among them, B is the time position sequence of beats; bi represents the time position of the i-th beat.

[0154] It should be noted that in music, beats are a regular unit of time division. Each beat corresponds to a specific time position. By analyzing the audio signal, a series of beat time positions can be obtained to form a beat sequence. For example, a piece of music with a 4 / 4 beat may have a beat sequence of [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, ...], where each number represents the time position (in seconds) at which the corresponding beat occurs.

[0155] The time interval between adjacent beats is shown in formula (9) and formula (10).

[0156] I=[i1,i2,...,i(n-1)](9)

[0157] ik=b(k+1)-bk(10)

[0158] Among them, I is the time interval between adjacent beats; i1, i2, ..., i(n-1) is the sequence of time intervals between adjacent beats; ik is the duration of the k-th beat interval; b(k+1) is the time position of the k+1-th beat; bk is the time position of the k-th beat.

[0159] It should be noted that I represents an interval sequence, which includes the time interval between each pair of adjacent beats.

[0160] i1, i2, ..., i(n-1) respectively represent the time intervals between the 1st to the (n-1)th pairs of adjacent beats.

[0161] n is the length of the beat sequence B, that is, the total number of beats.

[0162] ik=b(k+1)-bk further illustrates how to calculate the value of each interval, where ik represents the kth interval, which is equal to the time position of the (k+1)th beat minus the time position of the kth beat.

[0163] Assuming the beat sequence is B = [0.0, 0.5, 1.0, 1.5, 2.0], then:

[0164] i1=b2-b1=0.5-0.0=0.5, i2=b3-b2=1.0-0.5=0.5, i3=b4-b3=1.5-1.0=0.5, i4=b5-b4=2.0-1.5=0.5, then the interval sequence I=[0.5, 0.5, 0.5, 0.5].

[0165] According to the beat type and BPM speed parameters, the note duration range corresponding to the time interval between each adjacent beat is calculated. For example, for music with 4 / 4 beat and 120 BPM, a beat interval is 0.5 seconds, and the corresponding note duration range is [0.25, 0.5, 1] ​​seconds, that is, a quarter note, a half note, and a whole note.

[0166] After obtaining the note duration range, the candidate notes that match the recording rhythm and meet the preset rules are searched in the pre-collected note sample library for subsequent music element generation. The preset rules are determined by the beat list, preset music parameters (such as mode, number of notes, etc.) and music theory constraints.

[0167] When matching note samples, in addition to the duration conditions, the preset rules also need to consider whether the pitch changes and dynamic changes of the note combination meet the music theory conditions, and whether the duration of the notes is within the range corresponding to the beat interval to ensure that the generated music rhythm matches the original beat sequence. The automatic composition system defines some heuristic rules, such as:

[0168] The pitch changes of note combinations should conform to the rules of the scale. Each scale corresponds to a fixed set of pitches, and the notes should mainly develop within the scale, and occasionally variable notes can be used; for example, in various scales, the pitch combination of notes needs to follow certain rules. For example, in C major, the commonly used scale is CDEFGAB, and the notes are mainly developed on these 7 pitches. Occasionally, variable notes such as F sharp and B flat can be used to increase the richness of the melody. Each scale has different commonly used scales, which constitute the characteristics of the scale;

[0169] The pitch changes of note combinations should conform to the rules of melodic motion. Common melodic motions include stepwise (pitch rises or falls in sequence), leaps (pitch rises or falls in leaps), and retracements (pitch rises first and then falls or falls first and then rises).

[0170] Continuous large leaps (greater than the fifth) should be avoided as they are detrimental to the fluency of the melody.

[0171] Dissonant intervals (such as minor second, augmented fourth, etc.) should be avoided as much as possible, especially on the strong beat;

[0172] The dynamics of the notes should be consistent with the rhythmic accents, and generally the strong beats correspond to the stronger dynamics. At the same time, the dynamics should conform to the overall structure of the music, such as crescendo and diminuendo.

[0173] Based on the above rules, the automatic composition system uses a dynamic programming algorithm to search for the optimal note combination path. The state is defined as dp[i][p][d] = the optimal score corresponding to the i-th beat, pitch p, and intensity d.

[0174] Among them, dp[i][p][d] represents the score of the optimal sub-problem corresponding to the ith beat, pitch p, and intensity d. By dynamically programming the fill table, the optimal note combination path of the entire music clip can be obtained.

[0175] The state transfer equation is dp[i][p][d]=max(dp[i-1][p'][d']+score(p, p', d, d', i)).

[0176] Among them, p' and d' traverse all possible pitches and dynamics of the previous beat, and the score function calculates the transfer score between two notes according to the scoring rules, taking into account factors such as pitch change, dynamic change, and mode stability. The scoring function can be obtained through offline training of machine learning algorithms.

[0177] The optimal note combination path means that after obtaining the candidate note set, the dynamic programming algorithm is used to search and evaluate different note combinations, and finally a note sequence with the highest score is found as the optimal path for generating the melody. The purpose of this step is to select the best combination from the candidate notes so that it not only meets the constraints of rhythm, mode, etc., but also achieves the best in terms of melody fluency and harmony rationality.

[0178] The pitch sequence generation module traverses all possible pitches of the previous beat based on a data-driven pitch sequence generation algorithm.

[0179] The final note combination path is dp[n][p][d]=max(dp[n][p][d])forallpossiblepandd, where forallpossiblepandd represents all possible pitches p and dynamics d.

[0180] The music element generation module scores each candidate note through the scoring function in the dynamic programming algorithm; the scoring function is used to score the rhythm matching, melodic fluency and harmonic rationality of each candidate note to ensure that the rhythm of the searched note sequence is consistent with the rhythm of the audio file to be processed, and to ensure that the musicality of the searched note sequence is optimal.

[0181] The scoring function needs to comprehensively consider multiple factors such as rhythm matching, melody fluency, and harmony rationality to ensure that the generated music not only matches the rhythm of the audio file to be processed in terms of rhythm, but is also the best in terms of musicality.

[0182] The music element generation module can obtain the optimal note sequence by backtracking the state.

[0183] The process of obtaining the optimal note sequence by backtracking the state is as follows:

[0184] 1. Find the optimal state (pitch and intensity) combination of the last beat, recorded as (p_n, d_n);

[0185] 2. Starting from (p_n, d_n), search in reverse order for the optimal state of the previous beat (p_n-1, d_n-1), satisfying dp[n-1][p_n-1][d_n-1]+trans(p_n-1, d_n-1, p_n, d_n)=dp[n][p_n][d_n]. Where trans represents the state transition function;

[0186] 3. Repeat the above steps until you trace back to the first beat and get a complete optimal state sequence;

[0187] 4. According to the optimal state sequence, take out the corresponding notes to form the final note sequence.

[0188] Distributor module:

[0189] The orchestration module repeats, transforms and combines musical elements based on the template-based orchestration algorithm and predefined musical form templates. The specific process is as follows:

[0190] 1. The orchestration module selects a predefined music form template, such as two-part form, three-part form, sonata form, etc. Each template contains several sections, each of which has a specific number of repetitions and variation rules.

[0191] 2. The orchestration module divides the note sequence output by the music element generation module into several segments according to the length of the music segment.

[0192] 3. The instrumentation module repeats or changes each section according to the rules of the music form template. The change techniques include pitch change, rhythm change, melody variation, etc.

[0193] The instrumentation module repeats, transforms and combines musical elements according to the musical form template, and adds other musical elements such as chords and ornaments to enrich the texture and form of the music. Finally, the complete music clip is rendered as an audio signal.

[0194] 4. The orchestration module adds transitional music materials between music sections, such as cadences and connecting parts, making the entire music structure smooth and natural.

[0195] 5. The orchestration module splices the repeated and transformed music segments to form a complete music structure.

[0196] 6. The orchestration module inserts chord notes at key positions in the music according to the definition of the harmonic direction in the form template to enrich the texture of the music.

[0197] 7. The orchestration module evaluates the initially composed music clips and fine-tunes the notes according to predefined music theory rules (such as avoiding parallel fifths, interval jump tendency, etc.).

[0198] 8. Output a music clip that conforms to the target form.

[0199] Audio synthesis and output module:

[0200] The audio synthesis and output module renders the optimal note sequence through a preset rendering method, and the specific process of obtaining the audio signal is shown in B1-B8.

[0201] B1: The audio synthesis and output module processes the optimal note sequence according to the music form template; the processing operation at least includes the repetition operation of the music elements, the transformation operation of the music elements and the combination operation of the music elements.

[0202] B2: The audio synthesis and output module performs an operation of adding music elements to the optimal note sequence after the processing operation to obtain a target music clip (a complete music clip); the adding operation at least includes adding chords and ornaments.

[0203] The audio synthesis and output module in the automatic composition system obtains a complete note sequence data from the music element generation module, namely the target music clip, which contains the pitch, duration, intensity, timbre and other attribute parameters of each note. These attribute parameters are generated based on the time and speed features extracted by the rhythm analysis module, as well as predefined music rules such as mode and melody mode.

[0204] B3: The audio synthesis and output module renders the target music clip into pulse code modulation (PCM) audio data through a preset software music library.

[0205] The preset software music library includes but is not limited to a software sound source library of a software synthesizer (FluidSynth) that receives musical instrument digital interface (MIDI) messages.

[0206] The format of MIDI data includes event information such as track, time, and notes. When converting the target music clip into MIDI data, you need to follow the following rules:

[0207] The timestamps of note events are calculated and aligned based on the beat list information;

[0208] The pitch of a note is encoded based on the note name and octave information in the note sequence;

[0209] The note's intensity, duration and other parameters are mapped according to the musical notation symbols;

[0210] Chords, ornaments and other musical ornaments also need to be converted into corresponding MIDI events.

[0211] When rendering and synthesizing MIDI data, the wavetable synthesis method is mainly used. The steps are as follows:

[0212] According to the instrument type specified by the MIDI track, load the corresponding timbre library and obtain the waveform samples of each note;

[0213] According to the timestamps of MIDI note events, the waveform samples are sorted and spliced ​​to obtain the audio digital signals of each part;

[0214] Mix the audio digital signals of each part, set the relative volume and delay of the left and right channels, and simulate the position of the sound in space;

[0215] Normalize the volume of the mixed stereo signal so that its peak volume is within -3dB to prevent signal distortion.

[0216] After the above MIDI audio rendering process, a realistic and natural music work can be obtained.

[0217] Use FluidSynth software sound source library to render the note sequence into PCM audio data. FluidSynth is a MIDI wavetable synthesis engine based on SoundFont, with built-in sound libraries of various common instruments, which can synthesize realistic piano, guitar, string, wind and other sounds.

[0218] B4: The audio synthesis and output module groups the pulse code modulation audio data according to the instrument type to obtain a grouping result; each group of notes in the grouping result corresponds to a musical instrument digital interface (MIDI) track.

[0219] B5: The audio synthesis and output module converts note events into MIDI messages in chronological order for each musical instrument digital interface (MIDI) track; wherein the note events are used to indicate the start of a note (NOTE_ON) and the end of a note (NOTE_OFF).

[0220] It should be noted that the note event is a message type in the MIDI protocol, which is used to indicate the start and end of a note. A complete note contains two events, NOTE_ON and NOTE_OFF, which mark the start time and duration of the note respectively. The note event is converted from the note sequence output by the music element generation module.

[0221] For each MIDI track, the process of converting note events into MIDI messages in time order is shown in C1-C4:

[0222] C1: traverse all the notes of this track;

[0223] C2: For each note, a NOTE_ON message is generated, which contains the note's pitch, velocity, start time and other parameters;

[0224] C3: Generates the corresponding NOTE_OFF message according to the duration of the note to mark the end of the note;

[0225] C4: Arrange NOTE_ON and NOTE_OFF messages in chronological order to form an ordered sequence of MIDI messages.

[0226] B6: The audio synthesis and output module transmits the MIDI message to the software synthesizer (FluidSynth synthesizer) for rendering, and obtains the audio digital signal generated by each MIDI track.

[0227] The audio synthesis and output module transmits the MIDI message sequence to the FluidSynth synthesizer to trigger the wavetable synthesis process. FluidSynth will retrieve the pre-sampled instrument waveform according to the pitch, velocity and other parameters of the MIDI message, and apply effects such as ADSR envelope and filter to generate audio digital signals in real time.

[0228] B7: The audio synthesis and output module mixes the audio digital signal to obtain the audio signal to complete the music creation process.

[0229] In B7, the audio digital signals synthesized from multiple MIDI tracks are mixed by a software synthesizer, and the relative volume and image balance are adjusted to obtain a complete multi-part music audio.

[0230] The specific process of mixing the audio digital signals synthesized from multiple MIDI tracks and adjusting the relative volume and sound image balance is shown in D1-D3.

[0231] D1: Align the audio digital signals generated by each MIDI track in time dimension.

[0232] D2: Perform weighted summation on the audio digital signals aligned in the time dimension to obtain mixed audio data.

[0233] At each time point, the audio data of all tracks are weighted and summed to obtain mixed audio data.

[0234] The weight coefficients of the weighted sum reflect the relative volume of each audio track.

[0235] D3: Performing sound and image balance processing and relative volume adjustment on the mixed audio data to obtain an audio signal; wherein the sound and image balance processing is used to adjust the relative strength of the left and right channels and simulate the position of the sound in space; the relative volume adjustment is used to make the volume of the mixed audio data within a preset volume range to achieve a volume level suitable for playback.

[0236] The preset volume range is set according to actual conditions and is not specifically limited in this application.

[0237] Perform image balance processing on the mixed audio data, adjust the relative strength of the left and right channels, and simulate the position of the sound in space.

[0238] The overall volume of the mixed audio data is adjusted to reach a volume level suitable for playback.

[0239] B8: Audio Synthesis and Output Module The audio synthesis and output module converts the MIDI time sequence into an audible audio signal, outputs the processed audio signal, and transmits the processed audio signal to the audio playback device in real time for playback.

[0240] Finally, a complete piece of polyphonic music audio data is obtained, which can be further encoded into WAV, MP3 and other formats and output.

[0241] The audio synthesis and output module is the last link of the entire automatic composition system. Its task is to mix and render the optimal note sequence generated by the music element generation module into the final audio signal under various generation constraints, and provide an interface for real-time playback control. This module uses industry-standard digital audio processing and software sound source technology to generate high-fidelity, low-latency music audio and supports a variety of common audio file formats.

[0242] Finally, the system writes the synthesized PCM audio data into the audio buffer and provides a real-time playback control interface. Users can control audio playback, pause, stop, fast forward and rewind through a graphical interface or programming interface. At the same time, the system also provides an audio file output function, supporting common audio formats such as WAV, MP3, and OGG. Users can save the generated music to a local file for offline listening or secondary creation.

[0243] This solution significantly improves the efficiency and convenience of audio creation. The traditional composition process requires the composer to have a solid foundation in music theory and rich creative experience, which is time-consuming and laborious. This system uses intelligent algorithms to automatically generate musical elements such as melody, harmony, and rhythm. Users only need to provide simple parameters such as mode and speed to quickly obtain an expressive and musical original music clip, which greatly reduces the threshold for composition and allows ordinary users to experience the fun of creation.

[0244] The audio processing method of this solution can be integrated into various music production software in the form of a plug-in or an independent tool to provide users with auxiliary creation functions. After the user uploads an audio sample, the present invention can automatically extract its rhythm, melody and other characteristics, and automatically generate a music clip that matches it according to the parameters such as instruments and styles set by the user. The user can further edit and modify the generated music clip according to the needs, and finally complete the creation of the music work.

[0245] The use of this solution can greatly reduce the threshold and difficulty of music creation. Even users without professional music theory knowledge can quickly create professional-level music works with the help of artificial intelligence. At the same time, the present invention can also be applied to games, movies, animations and other fields to automatically generate background music that matches the picture and plot, thereby improving the production efficiency and quality of multimedia content.

[0246] Compared with the prior art, the present invention has the following advantages and innovations:

[0247] Fully automated: The present invention can automatically extract music features, match appropriate notes, and generate matching music clips based on audio samples and user settings, without requiring users to have professional music knowledge and creative skills;

[0248] Accurate rhythm: The present invention adopts technologies such as beat extraction and note duration constraint to ensure that the generated music rhythm is highly consistent with the original audio, avoiding the problems of rhythm deviation and distortion;

[0249] Strong musicality: When matching notes, the present invention comprehensively considers various musical rules such as melody fluency and harmony rationality, and the generated music clips are close to the level of professional creation in terms of musicality;

[0250] High efficiency: The present invention uses a dynamic programming algorithm to search for note sequences, which can quickly lock the optimal solution among a large number of candidate notes, greatly improving the efficiency of music creation;

[0251] High-fidelity output: The present invention adopts a high-quality sound library and advanced audio rendering technology. The generated audio signal has the characteristics of high fidelity and low distortion, and the listening experience is full and natural.

[0252] This solution enriches the content of the audio material library and provides more options for music production. The current commonly used music material library mainly relies on manually created content, with limited categories and styles. This system can generate music clips with diverse styles and rich categories based on different parameter combinations. It can be used directly as material or as a source of inspiration for composers to further create and process. At the same time, machine-generated music is not subject to copyright restrictions and can be used and disseminated freely, reducing the cost of music production.

[0253] This solution provides a new research idea and experimental platform for the field of audio artificial intelligence (AI). This automatic composition system integrates many cutting-edge technologies in the current field of music artificial intelligence, including music generation based on deep learning, audio signal processing, intelligent music editing, etc., and has important reference value for related theoretical research. The various modules of the system adopt a loosely coupled and replaceable design. Researchers can easily experiment with new algorithm models in a certain module, such as using a Transformer-based sequence generation model to replace the original Markov chain model, quickly verify ideas and evaluate the effects. At the same time, the large-scale music data generated by the system can also be used for various experimental studies in the field of music artificial intelligence, such as model training and effect evaluation for tasks such as music style transfer and music emotion recognition.

[0254] This solution solves the following problems:

[0255] High threshold issues for non-professionals

[0256] Traditional music creation tools and methods have high technical barriers for non-professionals, requiring users to have certain music theory knowledge and music production skills. The present invention automatically analyzes the rhythm information in the recording and generates corresponding music, so that non-professionals can easily create high-quality music works, thus lowering the threshold for music creation.

[0257] The problem of low efficiency in music creation

[0258] The traditional music creation process requires a lot of time and energy, especially for the arrangement of rhythm and melody. The present invention can quickly generate music elements that match the recording rhythm through intelligent analysis and conversion technology, greatly improving the efficiency of music creation and reducing the creation time.

[0259] The problem of accurately extracting recording rhythm information

[0260] Accurately extracting rhythm information from recordings is a key technical difficulty in achieving automatic music generation. The present invention uses advanced audio signal processing technology to accurately identify parameters such as rhythm, beat, and speed in recordings, ensuring that the generated music is highly consistent with the recording rhythm, thus ensuring the auditory effect of the music and user experience.

[0261] Adaptation issues for multi-format recording

[0262] There are many formats of audio files, such as WAV, MP3, AAC, etc. Audio files of different formats may encounter compatibility issues when parsing and processing. The present invention supports the input of multiple audio formats, and through a unified parsing and conversion mechanism, achieves seamless adaptation of recordings of different formats, ensuring the wide applicability of the system.

[0263] This solution aims to provide a technology based on rhythm recording analysis and conversion, which can automatically analyze the rhythm information in the recording and generate corresponding audio, so that non-professionals can also quickly create audio works, greatly reducing the difficulty and labor cost of audio creation, and improving the efficiency of audio generation. This solution automatically extracts the rhythm, beat, speed and other information in the recording through intelligent analysis and conversion of the rhythm in the recording, generates corresponding audio elements, and supports users to customize and edit music content. The technical solution of the present invention can be adapted to a variety of recording formats, and the generated audio works have a high degree of restoration and a good listening experience.

[0264] In the embodiment of the present application, in the process of creating a musical work, it is not necessary for the user to have music theory knowledge and music production skills. It is only necessary to intelligently analyze and convert the rhythm in the audio file to be processed, automatically extract the rhythm, beat type, beat number and speed parameters in the audio file to be processed, search for the optimal note sequence from each candidate note, and render the optimal note sequence through a preset rendering method to obtain an audio signal. Therefore, it is not necessary to spend a lot of time and energy in the process of creating a musical work, and quickly generate a music signal that conforms to the recording rhythm, thereby improving the efficiency of creating a musical work. In addition, the use of advanced audio signal processing technology can accurately identify the rhythm, beat type, beat number and speed parameters in the audio file to be processed, ensuring that the generated musical work is highly consistent with the rhythm of the audio file to be processed, so that the generated audio work has a high degree of restoration and good auditory effect.

[0265] refer to Figure 2 As shown, an audio processing method disclosed in an embodiment of the present application is applicable to the above-mentioned embodiment. Figure 1 The automatic composition system of the invention mainly comprises the following steps:

[0266] S201: Obtain the audio file to be processed;

[0267] S202: extracting key parameters from the audio file to be processed; wherein the key parameters at least include a beat list, a beat type and a BPM speed parameter.

[0268] Key parameters. Key parameters are used to determine the style and characteristics of the generated music file.

[0269] The specific process of extracting key parameters from the audio file to be processed is shown in E1-E5.

[0270] E1: Preprocess the audio file to be processed; the preprocessing at least includes audio decoding, channel separation and amplitude normalization.

[0271] E2: Use digital signal processing technology to extract features from the pre-processed audio files and obtain a time series that reflects the changes in music energy.

[0272] E3: Perform threshold detection on the time series reflecting the change of music energy to obtain the time position sequence of the beats.

[0273] E4: Analyze the distribution pattern of adjacent beat intervals in the temporal position sequence of beats to obtain a beat interval sequence, and determine the beat type of the music according to a preset clustering algorithm and the beat interval sequence.

[0274] Firstly, adjacent elements in the temporal position sequence of the beats are calculated for difference, so as to obtain a beat interval sequence; wherein the beat interval sequence is used to represent the distribution law of adjacent beat intervals in the temporal position sequence of the beats.

[0275] Secondly, the beat interval sequence is clustered using the K-Means clustering algorithm to obtain multiple cluster centers;

[0276] Then, the ratios corresponding to the multiple cluster centers are obtained, and the ratios corresponding to the multiple cluster centers are compared with the preset ratios for similarity to obtain multiple comparison results;

[0277] Finally, the ratio with the greatest similarity among multiple comparison results is determined as the beat type of the music.

[0278] E5: Calculate the average beat interval of the beat interval sequence and obtain the beat speed parameter.

[0279] The execution process and execution principle of E1-E5 are the same as those of the above embodiments. Figure 1 The execution process and execution principle of the rhythm analysis module are consistent, which can be referred to and will not be repeated here.

[0280] S203: Calculate the note duration range corresponding to the interval between each adjacent beat according to the beat type and the beat speed parameters.

[0281] S204: When matching the note samples, search the note sample library for candidate notes that match the recording rhythm within the note duration range and conform to preset rules; wherein the preset rules are determined by a beat list, preset music parameters, and music theory constraints.

[0282] In S204, when matching note samples, note samples that meet preset rules within the note duration range are retrieved from the note sample library through query conditions, and candidate notes that match the recording rhythm are obtained according to the file storage path corresponding to the note samples.

[0283] S205: Searching for the optimal note sequence from each candidate note.

[0284] The optimal note sequence is searched from each candidate note through a dynamic programming algorithm, which takes into account multiple constraints, including the note duration range corresponding to the beat interval, the pitch change law, the dynamic change law, etc.

[0285] In S205, each candidate note is scored by using a scoring function in a dynamic programming algorithm, and an optimal note sequence is obtained from each scored candidate note by backtracking the state.

[0286] Among them, the scoring function is used to score the rhythm matching, melodic fluency and harmonic rationality of each candidate note to ensure that the rhythm of the searched note sequence is consistent with the rhythm of the audio file to be processed, and to ensure that the musicality of the searched note sequence is optimal.

[0287] S206: Rendering the optimal note sequence by a preset rendering method to obtain an audio signal to complete the music composition process.

[0288] Write the audio signal to the audio buffer and provide a real-time playback control interface.

[0289] The execution process and execution principle of S201-S206 are the same as those of the above embodiment. Figure 1 The execution process and execution principle of the automatic composition system are consistent with each other, which can be referred to and will not be described in detail here.

[0290] Specifically, the optimal note sequence is rendered through a preset rendering method to obtain an audio signal to complete the music creation process, as shown in F1-F7.

[0291] F1: performing a processing operation on the optimal note sequence according to the musical form template; the processing operation at least includes a repetition operation of the music elements, a transformation operation of the music elements and a combination operation of the music elements;

[0292] F2: performing an operation of adding music elements to the optimal note sequence after the processing operation to obtain a target music clip; the adding operation at least includes adding chords and ornaments;

[0293] F3: Render the target music clip into pulse code modulation (PCM) audio data through the preset software music library;

[0294] F4: Group the pulse code modulation audio data according to the instrument type to obtain a grouping result; each group of notes in the grouping result corresponds to a MIDI track;

[0295] F5: For each MIDI track, convert note events into MIDI messages in time sequence; note events are used to indicate note start and note end;

[0296] F6: Transmit the MIDI message to the software synthesizer (FluidSynth's synthesizer) for rendering, and obtain the audio digital signal generated by each MIDI track;

[0297] F7: Mix the audio digital signal to obtain the audio signal to complete the music creation process.

[0298] The execution process and execution principle of F1-F7 are the same as those of the above embodiment. Figure 1 The execution process and execution principle of B1-B8 are the same, which can be referred to and will not be repeated here.

[0299] The process of mixing the audio digital signals to obtain the audio signals is shown in G1-G3.

[0300] G1: Align the audio digital signals generated by each MIDI track in the time dimension;

[0301] G2: Perform weighted summation on the audio digital signals aligned in the time dimension to obtain mixed audio data;

[0302] G3: Performing sound and image balance processing and relative volume adjustment on the mixed audio data to obtain an audio signal; wherein, the sound and image balance processing is used to adjust the relative strength of the left and right channels and simulate the position of the sound in space; the relative volume adjustment is used to make the volume of the mixed audio data within a preset volume range.

[0303] The execution process and execution principle of G1-G3 are the same as those of the above embodiments. Figure 1 The execution process and execution principle of D1-D3 are consistent, which can be referred to and will not be repeated here.

[0304] In the embodiment of the present application, in the process of creating a musical work, it is not necessary for the user to have music theory knowledge and music production skills. It is only necessary to intelligently analyze and convert the rhythm in the audio file to be processed, automatically extract the rhythm, beat type, beat number and speed parameters in the audio file to be processed, search for the optimal note sequence from each candidate note, and render the optimal note sequence through a preset rendering method to obtain an audio signal. Therefore, it is not necessary to spend a lot of time and energy in the process of creating a musical work, and quickly generate a music signal that conforms to the recording rhythm, thereby improving the efficiency of creating a musical work. In addition, the use of advanced audio signal processing technology can accurately identify the rhythm, beat type, beat number and speed parameters in the audio file to be processed, ensuring that the generated musical work is highly consistent with the rhythm of the audio file to be processed, so that the generated audio work has a high degree of restoration and good auditory effect.

[0305] Based on the above embodiments Figure 2 The disclosed audio processing method, the embodiment of the present application also discloses an audio processing device, such as Figure 3 As shown, the audio processing device comprises:

[0306] An acquisition unit 301 is used to acquire an audio file to be processed;

[0307] The extraction unit 302 is used to extract key parameters from the audio file to be processed; wherein the key parameters at least include a beat list, a beat type and a beat number speed parameter;

[0308] The calculation unit 303 is used to calculate the note duration range corresponding to the interval between each adjacent beat according to the beat type and the beat number speed parameter;

[0309] A search unit 304 is used to search for candidate notes that match the recording rhythm and meet preset rules within the note duration range from the note sample library when matching the note samples; wherein the preset rules are determined by the beat list, preset music parameters and music theory constraints;

[0310] A search unit 305, used to search for an optimal note sequence from each candidate note;

[0311] The rendering unit 306 is used to render the optimal note sequence by a preset rendering method to obtain an audio signal to complete the music composition process.

[0312] Furthermore, the extraction unit 302 includes:

[0313] A preprocessing module, used for preprocessing the audio file to be processed; the preprocessing includes at least audio decoding, channel separation and amplitude normalization;

[0314] An extraction module is used to extract features from the pre-processed audio file through digital signal processing technology to obtain a time series reflecting the change of music energy;

[0315] A threshold detection module is used to perform threshold detection on a time series reflecting changes in music energy to obtain a time position sequence of beats;

[0316] An analysis and determination module is used to analyze the distribution law of adjacent beat intervals in the temporal position sequence of beats, obtain a beat interval sequence, and determine the beat type of the music according to a preset clustering algorithm and the beat interval sequence;

[0317] The calculation module is used to calculate the average beat interval of the beat interval sequence to obtain the beat number speed parameter.

[0318] Further, the analysis and determination modules include:

[0319] A calculation submodule is used to perform difference calculation on adjacent elements in the time position sequence of beats to obtain a beat interval sequence;

[0320] The clustering submodule is used to cluster the beat interval sequence by using the K-Means clustering algorithm to obtain multiple cluster centers;

[0321] An acquisition submodule is used to acquire the ratios corresponding to the multiple cluster centers, and compare the ratios corresponding to the multiple cluster centers with the preset ratios for similarity to obtain multiple comparison results;

[0322] The determination submodule is used to determine the ratio with the greatest similarity among the multiple comparison results as the beat type of the music.

[0323] Furthermore, the searching unit 304 includes:

[0324] A retrieval module is used to retrieve note samples that meet preset rules within the note duration range from the note sample library through query conditions when matching note samples;

[0325] The first acquisition module is used to acquire each candidate note matching the recording rhythm according to the file storage path corresponding to the note sample.

[0326] Furthermore, the search unit 305 includes:

[0327] A scoring module is used to score each candidate note by using a scoring function in a dynamic programming algorithm; wherein the scoring function is used to score the rhythm matching, melodic fluency and harmonic rationality of each candidate note, so as to ensure that the rhythm of the searched note sequence is consistent with the rhythm of the audio file to be processed, and to ensure that the musicality of the searched note sequence is optimal;

[0328] The second acquisition module is used to obtain the optimal note sequence from each scored candidate note through a reverse backtracking state.

[0329] Furthermore, the rendering unit 306 includes:

[0330] A processing operation module, used for performing processing operations on the optimal note sequence according to the musical form template; the processing operations at least include repetition operations of music elements, transformation operations of music elements and combination operations of music elements;

[0331] An adding operation module is used to perform an adding operation of music elements on the optimal note sequence after the processing operation to obtain a target music clip; the adding operation at least includes adding chords and ornaments;

[0332] A first rendering module, used for rendering the target music clip into pulse code modulation audio data through a preset software music library;

[0333] A grouping module, used for grouping the pulse code modulation audio data according to the instrument type to obtain a grouping result; each group of notes in the grouping result corresponds to a musical instrument digital interface track;

[0334] A conversion module, for converting the note events into MIDI messages in time sequence for each MIDI track; wherein the note events are used to indicate the start and end of a note;

[0335] A second rendering module is used to transmit the MIDI message to the software synthesizer for rendering to obtain the audio digital signal generated by each MIDI track;

[0336] The mixing module is used to mix the audio digital signal to obtain the audio signal to complete the music creation process.

[0337] Furthermore, the mixing module includes:

[0338] An alignment submodule, for aligning the audio digital signals generated by each MIDI track in a time dimension;

[0339] A weighted summation submodule, used for performing weighted summation on the audio digital signals aligned in the time dimension to obtain mixed audio data;

[0340] The processing and adjustment submodule is used to perform sound and image balance processing and relative volume adjustment on the mixed audio data to obtain an audio signal; wherein the sound and image balance processing is used to adjust the relative strength of the left and right channels and simulate the position of the sound in space; the relative volume adjustment is used to make the volume of the mixed audio data within a preset volume range.

[0341] Furthermore, the audio processing device further includes:

[0342] The writing unit is used to write the audio signal into the audio buffer and provide a real-time playback control interface.

[0343] In the embodiment of the present application, in the process of creating a musical work, it is not necessary for the user to have music theory knowledge and music production skills. It is only necessary to intelligently analyze and convert the rhythm in the audio file to be processed, automatically extract the rhythm, beat type, beat number and speed parameters in the audio file to be processed, search for the optimal note sequence from each candidate note, and render the optimal note sequence through a preset rendering method to obtain an audio signal. Therefore, it is not necessary to spend a lot of time and energy in the process of creating a musical work, and quickly generate a music signal that conforms to the recording rhythm, thereby improving the efficiency of creating a musical work. In addition, the use of advanced audio signal processing technology can accurately identify the rhythm, beat type, beat number and speed parameters in the audio file to be processed, ensuring that the generated musical work is highly consistent with the rhythm of the audio file to be processed, so that the generated audio work has a high degree of restoration and good auditory effect.

[0344] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned audio processing method.

[0345] The present application also provides an electronic device, whose structural diagram is as follows: Figure 4 As shown, it specifically includes a memory 401 and one or more instructions 402, wherein the one or more instructions 402 are stored in the memory 401 and are configured to be executed by one or more processors 403 to perform the above-mentioned audio processing method.

[0346] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0347] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0348] The steps in the methods of the embodiments of the present application can be adjusted in order, combined, and deleted according to actual needs.

[0349] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0350] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0351] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An audio processing method, characterized in that: The method comprises: Get the audio file to be processed; Extracting key parameters from the audio file to be processed; wherein the key parameters at least include a beat list, a beat type and a beat number speed parameter; Calculating the note duration range corresponding to the interval between each adjacent beat according to the beat type and the beat speed parameter; When matching the note samples, searching the note sample library for candidate notes that match the recording rhythm and meet preset rules within the note duration range; wherein the preset rules are determined by the beat list, preset music parameters and music theory constraints; Searching for the best note sequence from each candidate note; The optimal note sequence is rendered in a preset rendering manner to obtain an audio signal to complete the music work creation process.

2. The method according to claim 1, characterized in that The step of extracting key parameters from the audio file to be processed includes: Preprocessing the audio file to be processed; the preprocessing at least includes audio decoding, channel separation and amplitude normalization; The pre-processed audio files are subjected to feature extraction through digital signal processing technology to obtain a time series reflecting the energy changes of the music; Performing threshold detection on the time series reflecting the change of music energy to obtain a time position sequence of beats; Analyzing the distribution pattern of adjacent beat intervals in the temporal position sequence of the beats to obtain a beat interval sequence, and determining the beat type of the music according to a preset clustering algorithm and the beat interval sequence; The average beat interval of the beat interval sequence is calculated to obtain a beat speed parameter.

3. The method according to claim 2, characterized in that The step of analyzing the distribution law of adjacent beat intervals in the temporal position sequence of the beats to obtain a beat interval sequence, and determining the beat type of the music according to a preset clustering algorithm and the beat interval sequence, comprises: Performing difference calculation on adjacent elements in the temporal position sequence of the beats to obtain a beat interval sequence; Clustering the beat interval sequence using a K-Means clustering algorithm to obtain multiple cluster centers; Obtaining ratios corresponding to a plurality of cluster centers, and performing similarity comparison between the ratios corresponding to the plurality of cluster centers and a preset ratio to obtain a plurality of comparison results; The ratio with the greatest similarity among the multiple comparison results is determined as the beat type of the music.

4. The method according to claim 1, characterized in that When matching the note samples, searching the note sample library for candidate notes that match the recording rhythm and conform to preset rules within the note duration range includes: When matching the note samples, the note samples that meet the preset rules within the note duration range are retrieved from the note sample library through the query conditions; According to the file storage path corresponding to the note sample, each candidate note matching the recording rhythm is obtained.

5. The method according to claim 1, characterized in that The step of searching for an optimal note sequence from each candidate note comprises: Scoring each candidate note through a scoring function in a dynamic programming algorithm; wherein the scoring function is used to score the rhythm matching, melodic fluency, and harmonic rationality of each candidate note, so as to ensure that the rhythm of the searched note sequence is consistent with the rhythm of the audio file to be processed, and to ensure that the musicality of the searched note sequence is optimal; By backtracking the state, the optimal note sequence is obtained from each scored candidate note.

6. The method according to claim 1, characterized in that The optimal note sequence is rendered in a preset rendering manner to obtain an audio signal to complete the music composition process, including: Processing the optimal note sequence according to the musical form template; the processing operation at least includes a repetition operation of the music element, a transformation operation of the music element and a combination operation of the music element; Performing an operation of adding music elements to the optimal note sequence after the processing operation to obtain a target music clip; the adding operation at least includes adding chords and ornaments; Rendering the target music clip into pulse code modulated audio data through a preset software music library; The pulse code modulation audio data is grouped according to the instrument type to obtain a grouping result; each group of notes in the grouping result corresponds to a musical instrument digital interface track; For each MIDI track, converting the note event into a MIDI message in time sequence; wherein the note event is used to indicate the start and end of a note; The MIDI message is transmitted to a software synthesizer for rendering to obtain an audio digital signal generated by each MIDI track; The audio digital signals are mixed to obtain audio signals to complete the music creation process.

7. The method according to claim 6, characterized in that The step of mixing the audio digital signal to obtain an audio signal comprises: Aligning the audio digital signals generated by each MIDI track in time dimension; Performing weighted summation on the audio digital signals aligned in the time dimension to obtain mixed audio data; The mixed audio data is subjected to sound and image balance processing and relative volume adjustment to obtain an audio signal; wherein the sound and image balance processing is used to adjust the relative strength of the left and right channels and simulate the position of the sound in space; and the relative volume adjustment is used to make the volume of the mixed audio data within a preset volume range.

8. The method according to claim 1, characterized in that Also includes: The audio signal is written into an audio buffer, and a real-time playback control interface is provided.

9. An audio processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring an audio file to be processed; An extraction unit, used for extracting key parameters from the audio file to be processed; wherein the key parameters at least include a beat list, a beat type and a beat number speed parameter; A calculation unit, used to calculate the note duration range corresponding to the interval between each adjacent beat according to the beat type and the beat speed parameter; A search unit, used for searching, when matching the note samples, each candidate note that matches the recording rhythm and conforms to a preset rule within the note duration range from the note sample library; wherein the preset rule is determined by the beat list, preset music parameters and music theory constraints; A search unit, used for searching for an optimal note sequence from various candidate notes; The rendering unit is used to render the optimal note sequence in a preset rendering manner to obtain an audio signal to complete the music work creation process.

10. The device according to claim 9, characterized in that The extraction unit comprises: A preprocessing module, used for preprocessing the audio file to be processed; the preprocessing at least includes audio decoding, channel separation and amplitude normalization; An extraction module is used to extract features from the pre-processed audio file through digital signal processing technology to obtain a time series reflecting the change of music energy; A threshold detection module, used to perform threshold detection on the time series reflecting the change of music energy to obtain a time position sequence of beats; An analysis and determination module, used for analyzing the distribution law of adjacent beat intervals in the temporal position sequence of the beats, obtaining a beat interval sequence, and determining the beat type of the music according to a preset clustering algorithm and the beat interval sequence; The calculation module is used to calculate the average beat interval of the beat interval sequence to obtain the beat number speed parameter.

Citation Information

Patent Citations

  • Note segmentation method based on music score

    CN105976803A

  • Audio processing method and device, equipment and medium

    CN112382257A

  • Music work generation method and device thereof, music work synthesis method and device, equipment, medium and product

    CN113611268A

  • Music score generation method, electronic equipment and readable storage medium

    CN113763913A

  • Music accompaniment generation method and device, equipment, storage medium and program product

    CN117765902A

Cited By

  • AI music generation method and system based on BPM rhythm matching

    CN120375787A

  • An AI music generation method and system based on BPM rhythm matching

    CN120375787B

  • Audio synthesis method, audio synthesis system and storage medium

    CN120599996A

  • Real-time music generation and sound and picture linkage method and system based on voice feature analysis

    CN122290628A

  • Real-time audio special effect processing method suitable for electronic music

    CN122575314A