Automatic mixing device
By using a music feature calculator and a mixing point calculator, mixing points are automatically calculated, solving the problems of low efficiency and harsh mixing results in manual mixing, and achieving efficient and accurate automatic mixing of multi-track music.
Patent Information
- Application Number
- CN201910800888.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-28
- Filing Date
- 2019-08-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-08-27
AI Technical Summary
In existing technologies, manual mixing is inefficient, costly, and produces harsh results with a high error rate. It is particularly ineffective in multi-track music processing, especially in the processing of vocal tracks, which is not precise enough.
Using a music feature calculator, which calculates music beat points, chord features, chroma vectors and tonality through deep learning and convolutional neural networks, and combined with a mixing point calculator, it automatically calculates mixing points to achieve automated mixing of multi-track music.
It automates the mixing process, improves mixing efficiency, reduces error rates, and provides a more natural mixing effect.
Smart Images

Figure CN110867174B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of music mixing, in particular to an automatic mixing device. Background Art
[0002] Music mixing generally refers to the operation in which a disc jockey (DJ) selects and plays pre-recorded music (such as pop songs) and mixes it on the spot with a computer to create unique music that is different from the original song. Software that assists DJ mixing includes Traktor, Serato, Mixed in Key, etc. These software are based on the similarity of music rhythm and tonality. They can assist disc jockeys to manually adjust the speed and tonality of the music. This type of DJ mixing is to connect multiple songs in series. At the mixing point, a song will replace the previous song and continue to play.
[0003] However, this manual mixing method is too inefficient, costly, and has few applicable scenarios. To improve efficiency, some commercial solutions have appeared on the market to assist users in selecting medley songs. Most of these solutions are based on the similarity of music rhythm and music tonality, replacing one song with another song as a whole. Although such a design provides some prompts to assist users in operation, users still need to manually select the song to be replaced and specify the time point for the song replacement, and the replacement time point (mixing point) cannot be calculated automatically. In addition, multi-track music is not taken into account. The replacement part of a song will be replaced as a whole by part of another song, resulting in a too blunt replacement result. In addition, some solutions add chord comparisons, but no special processing is performed on the vocal track, and the chord detection error rate is also high. Summary of the invention
[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide an automatic mixing device, which is used to take a song selected by a user as the main song, select several other similar songs from the calculated database, and find the mixing points of the main song and the similar songs that can replace the parts. The purpose of the present invention is to provide an automatic mixing device that solves the problems that the prior art cannot automatically calculate the mixing points, and the mixing results are stiff and have a high error rate.
[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides an automatic mixing device, characterized in that it includes: a music feature calculator, the input music of the music feature calculator includes melody, bass, percussion and vocal tracks; the music feature calculator selects one or more of the melody, bass, percussion and vocal tracks, and calculates one or more of the characteristics of the input music, including the beat point time, chord at the beat, chroma vector at the beat, sound energy at the beat point, tonality, and speed of the music.
[0006] The automatic mixing device of the present invention can calculate the music features in the music according to different audio tracks, and automatically calculate the mixing points according to the music features, thereby realizing the automation of mixing and solving the problems of low mixing efficiency and rough mixing effects in the prior art. Therefore, it has extremely high industrial application value. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 It is a music feature calculator workflow diagram of the present invention;
[0008] Figure 2 It is a schematic diagram of a music passage;
[0009] Figure 3 is a flow chart for calculating the mixing point. DETAILED DESCRIPTION
[0010] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.
[0011] Please refer to the attached drawings. It should be noted that the illustrations provided in this embodiment are only used to illustrate the basic concept of the present invention in a schematic manner, and the drawings only show the components related to the present invention rather than the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0012] The automatic mixing device of the present invention comprises a music feature calculator and a mixing point calculator. The music feature calculator and the mixing point calculator are respectively introduced below with reference to the accompanying drawings.
[0013] See first Figure 1 , Figure 1 This is a working flow chart of the music feature calculator of the present invention. The music features defined by the music feature calculator of the present invention include the beat point time of the music re-beat, the chord and chromaticity vector at the music re-beat, the sound energy at the music re-beat, the rhythm and tonality of the music. The calculation result of the music features is an important reference for finding the mixing point.
[0014] The input of the music feature calculator includes 4 tracks: melody, bass, percussion and vocals. Different track combinations are required for different feature calculations. The following is an introduction to the preferred implementation methods for calculating each music feature:
[0015] Music beat time and beat time: Music beat refers to the first beat of each measure. Common music has 4 beats per measure, and a beat is taken every 4 beats. The time of the first beat needs to be calculated, and after the first beat point is obtained, a beat is taken every 4 beats. For example, traditional methods such as the method of calculating the correlation of music appearance time in signal processing can be used to find the music beat point. In this example, multiple recursive neural networks based on deep learning are used to calculate the beat point time of music, and the time of the first beat is calculated from the calculated beat time through the hidden Markov model. There are many implementation tools for this type of method, such as the madmom software package, in which DBNDownBeatTrackingProcessor can be used to calculate the time of the music beat point. The input is melody + bass + percussion tracks. The vocal track input is not used to calculate the music beat point to avoid the interference of vocals on beat search.
[0016] Chords at the beat: After getting the beat time, use the convolutional neural network to calculate the music chord features, input the actual melody and bass track. After getting the music chord features, use the conditional random field method to identify the chord at the beat.
[0017] Chroma vector at the music re-beat: Chroma vector refers to a multi-element vector that represents the energy of each pitch within a period of time (such as 1 frame) (the energy of the pitch is proportional to the amplitude of the sound, and the calculation method can refer to the calculation of mechanical wave energy, which will not be repeated here). In this example, the chroma vector uses 12 elements, which represent the energy of 12 pitches within a period of time (such as 1 frame), and the energy of the same pitch in different octaves is accumulated. For vocal tracks, melody tracks, and bass tracks, the harmonic spectrum can be calculated based on the deep neural network method to extract the chroma vector.
[0018] Sound energy at the music beat: In this example, the energy is calculated by taking the square root mean of the amplitude of the sound wave at the beat.
[0019] Music tonality: In this example, a convolutional neural network is used to calculate the tonality of the entire piece of music, with the input being the melody + bass track.
[0020] The speed of the music: The speed of the music can be calculated by the beat. The formula for calculating the speed is
[0021]
[0022] Beat refers to the beat in the phrase, and i is the beat number. Although the most intuitive way to calculate the speed of a piece of music is to calculate it by the duration of the entire piece of music and the total number of beats, this method of calculation is relatively time-consuming. According to experimental data, the speed of a piece of music usually tends to be stable after a period of time. That is, if sampling is performed at the appropriate position in the middle of the piece of music, the speed of the music calculated at the sampling point will be very close to the speed value calculated by the duration of the entire piece of music and the total number of beats. Calculation by sampling points is obviously much more time-saving. According to a large amount of experimental data, the 20th to 90th beats of a piece of music are usually relatively stable. In this example, i is taken as 70.
[0023] After obtaining the music feature value, the mixing point can be calculated based on the music feature value. However, in this example, a music segmenter is preferably included to segment the music before calculating the mixing point. The structure of the music can be divided into a prelude, a chorus, a verse, a bridge, and an ending. There are already some toolkits on the market that implement the calculation of music segments, such as the msaf software package. This software package can set a variety of different algorithms to find music segments. In this example, a method based on structural features is used. Figure 2 It is a schematic diagram of a music segment. The segments of the music are prelude, verse, chorus, bridge, etc. In order to find more mixing points, the length of the music segment is cut into phrases of integer multiples of 4 bars, and then the 4-bar, 8-bar, and 16-bar phrases are compared with each other to find the mixing points of the music. Experiments show that the probability of hitting the mixing point is highest when the phrases are cut into integer multiples of 4 bars.
[0024] The following combination Figure 2 The steps for calculating the mixing point are described in detail. Each phrase of the verse is compared with the phrases of the same length in other songs to determine whether the two phrases have the same structure. For example, the phrase of the verse is only compared with the phrase of the verse of other songs. Before the comparison, it is necessary to make sure that both phrases have enough energy. The energy of each beat calculated previously is used to calculate the energy of the phrase. If both phrases have enough energy, the following comparison is performed.
[0025] Mixing point calculation for percussion: The comparison of percussion does not need to consider the harmony and other properties of the music. It only needs to consider whether the rhythm of the two songs is too different. The rhythm ratio can be used to measure the degree of difference in the rhythm of the music. The rhythm ratio refers to the ratio of the beats per minute (bpm) of the two songs. When the rhythm ratio is too large, changing the rhythm of a phrase will be abrupt and not suitable for replacement. When the rhythm ratio is between 0.7-1.3, if the energy of the two phrases is greater than the preset value, they can be replaced. The preset value here. The time point is the start time of the phrase. The duration is the phrase time. Record the rhythm ratio for subsequent mixing.
[0026] Calculation of melody and bass mixing points: Here, a harmony-based comparison is used. Harmonic comparison includes two parts, one is the comparison of chords, and the other is the comparison of chromaticity vector features. Chord comparison is the comparison of the chord sequence of each beat of a phrase with the chord of each beat of other phrases. If only the root note of the chord is considered here, there are 12 types of chords. Each chord is represented by a letter, namely C, C#, D, D#, E, F, F#, G, G#, A, A#, B. If a beat chord is empty, it is represented by N. The comparison of chords is equivalent to the comparison of phrase chord strings. The local alignment method in bioinformatics is applied here to compare two chord strings. Local alignment uses the character differences between two sequences to determine the similarity between sequences. If the characters in the corresponding positions of the two sequences are different, the similarity of the sequences is low, otherwise, the similarity of the sequences is high. The difference between the two chords is the difference between the corresponding strings, and the similarity of the two phrases can be calculated using a score based on musical harmony. When performing sequence alignment, there are two aspects that directly affect the similarity score: the substitution matrix and the gap penalty. The substitution matrix uses the replacement scores of the chords shown in the following table:
[0027] Chord difference (number of semitones) Fraction 0 2.85 1 -2.85 2 -2.475 3 -0.825 4 -0.825 5 0 6 -1.8
[0028] The gap penalty is 0. If N is compared with any chord, the score is 0. The sum of the scores of each phrase comparison is the chord score of this phrase. For example, if CGFF is compared with AGEF, the score is -0.825+2.85-2.85+2.85=2.025.
[0029] The chromaticity vector feature calculates the cosine similarity of the chromaticity vectors of two phrases. The two scores are added with different weights as needed. If the score is low, the phrase to be compared is transposed to the tonality of the main song phrase and re-compared. If the result score is high enough, the time when the phrase starts is the mixing point time. It is also necessary to record the phrase length, the rhythm ratio of the phrase, and the number of semitones of transposition to facilitate mixing. In this example, the weights of the two scores are both 0.5.
[0030] Calculation of the mixing point of vocals: The mixing point of vocals is similar to that of melody and bass, but also different. If the energy of the phrase music (melody + bass) where the vocals appear is strong enough, the mixing point of the corresponding phrase of melody and bass is used directly. If the energy of melody and bass is insufficient, the cosine similarity of the chromaticity vectors of the two vocal phrases is directly compared. Also record the start time of the phrase, the length of the phrase, the rhythm ratio of the phrase, and the number of semitones of the transposition.
[0031] When the automatic mixing device of the present invention is used, all songs in the user's music library are pre-processed first, and the music feature calculation method and mixing point calculation method are used to calculate the mixing points of any song in the music library as the main song and other songs respectively and store them in the database. If the mixing points found with other songs when this song is the main song are sufficient, and the rhythm ratio of other songs to the main song is 0.7-1.3, and the tonality difference is within 3, then other songs that meet the conditions are regarded as similar songs of this song, and these songs are directly called when mixing.
[0032] In summary, the automatic mixing device of the present invention calculates music features for multiple tracks respectively, and calculates mixing points based on the calculated features, thereby realizing automatic mixing and solving the problems of low mixing efficiency, stiff mixing results and high error rate in the prior art.
[0033] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. An automatic mixing device, It is characterized in that include: a music feature calculator, wherein the input music of the music feature calculator includes melody, bass, percussion and vocal tracks; The music feature calculator selects the melody, bass, and percussion to calculate the beat time of the input music; selects the melody and the bass to calculate the chord at the beat; selects the human voice, the melody, and the bass to calculate the chromaticity vector at the beat; and calculating one or more of the following characteristics: the energy of the sound at the beat, the tonality, and the tempo of the music; A mixing point calculator is used to calculate the mixing points of the vocal part, the melody bass part and the percussion part of the music according to the calculation results of the music feature calculator.
2. The automatic mixing device according to claim 1, It is characterized in that When the rhythm ratio of the two phrases is between 0.7 and 1.3, the starting points of the two phrases are used as the mixing points of the percussion part; wherein the rhythm ratio refers to the ratio of the beats per minute of the two phrases.
3. The automatic mixing device according to claim 1, It is characterized in that The mixing point calculation of the melody bass part is based on the harmonic comparison of the music; the harmonic comparison includes the comparison of chords and the comparison of chromaticity vectors.
4. The automatic mixing device according to claim 3, It is characterized in that The chord comparison method comprises: Use characters to represent chord roots and convert phrases into strings; Compare strings and calculate the difference between each character in the strings; The chord similarity is calculated based on the difference.
5. The automatic mixing device according to claim 4, It is characterized in that Computes the difference of each character in a string using a substitution matrix and gap penalties.
6. The automatic mixing device according to claim 3, It is characterized in that The comparison of the chroma vectors includes calculating the cosine similarity of the chroma vectors of two phrases.
7. The automatic mixing device according to claim 1, It is characterized in that The calculation of the mixing point of the vocal part includes: Determine whether the vocal part includes melody and bass. If so, directly use the mixing points of the corresponding phrases of the melody and bass; if not, compare the cosine similarity of the chromaticity vectors of the vocal phrases.
8. The automatic mixing device according to claim 1, It is characterized in that When calculating the beat time of the music, multiple recursive neural networks based on deep learning are used to calculate the beat time of the music, or the beat time of the music is found based on a method based on the correlation of the time when the music appears.
9. The automatic mixing device according to claim 8, It is characterized in that The time of the first re-beat is calculated from the calculated beat time of the music through a hidden Markov model.
10. The automatic mixing device according to claim 1, It is characterized in that The formula for calculating the speed of the music is: , Beat refers to the beat in the phrase. i The sequence number of the beat.
11. The automatic sound mixing device according to claim 10, It is characterized in that Said i The value range is 20~90.
12. The automatic mixing device according to claim 1, It is characterized in that Also included is a song splitter for splitting up the music before calculating mix points.
13. The automatic mixing device according to claim 12, It is characterized in that The music segmenter segments music using a method based on music structure features.
14. The automatic sound mixing device according to claim 12, It is characterized in that The music divider cuts the music into phrases of an integral multiple of 4 measures.
