Digital music multi-track intelligent sound mixing method based on perceptual coding

Abstract

The invention discloses a digital music multi-track intelligent sound mixing method based on perceptual coding, and belongs to the field of digital music. The method comprises four steps of music content understanding and analysis, adaptive track grouping mixing, perception importance dynamic adjustment and optimization, and final sound mixing signal generation and output. According to the method, the music content understanding layer based on the deep learning model is introduced, so that structured semantic analysis of multi-track music is realized, and the sound mixing process is improved from the traditional signal processing level to the music semantic understanding level. The system can automatically identify key features such as musical instruments, harmony and rhythm, and understand music function association between tracks, so that accurate context information is provided for subsequent intelligent processing, a sound mixing decision is more in line with the artistic law of music creation, and the music expressive force and specialty of a sound mixing result are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to digital music, and in particular to a method for multi-track intelligent mixing of digital music based on perceptual coding. BACKGROUND

[0002] One of the core steps in digital music production is multi-track mixing, which aims to fuse multiple independently recorded instrument and vocal tracks into a stereo or surround sound file, achieving clear, pleasant and expressive artistic effects in terms of volume balance, frequency distribution, dynamic control and spatial positioning. Traditionally, this work heavily relies on the experience and subjective listening of professional sound engineers, requiring tedious manual parameter adjustment of each track in a digital audio workstation, which has the problems of steep learning curve, time-consuming and inconsistent results. To improve efficiency, some automated or intelligent mixing tools have appeared in the industry, which are usually based on simple rules or static presets, such as applying fixed equalization or compression settings according to track type, but such methods lack understanding of the context of musical content and are difficult to deal with works of different styles and different complexity of arrangement, and the quality and professionalism of mixing are often unsatisfactory.

[0003] On the other hand, perceptual audio coding technology (such as MP3, AAC format) has been a mature technology in the field of audio compression, which is to use psychoacoustic models to remove signal components that are not sensitive to human ears to achieve data compression. However, the current technology is mainly applied to encoding optimization in the audio transmission and storage link, aiming to ensure the subjective listening quality of single-channel signals at low bit rates, and has not been creatively applied to the intelligent mixing of multi-track source signals in the creative process. Existing related technologies are mostly focused on improving coding efficiency or multi-channel encoding and decoding itself, such as optimizing quantizer or filter bank design, and do not involve how to use perceptual principles to intelligently guide mixing decisions. Therefore, there is an urgent need in the field for a systematic method that deeply integrates music content understanding, intelligent decision-making mechanism and perceptual coding principles to achieve truly intelligent and high-quality automatic mixing based on understanding of musical semantics. SUMMARY

[0004] The purpose of the present application is to provide a method for multi-track intelligent mixing of digital music based on perceptual coding, aiming to solve the problem that current digital music production methods are difficult to deal with works of different styles and different complexity of arrangement.

[0005] Technical solution: A method for multi-track intelligent mixing of digital music based on perceptual coding, comprising the following steps: S1, music content understanding and analysis: obtain a multi-track digital music signal to be mixed, use a deep learning model based on a Transformer architecture to analyze the structured music content of the multi-track signal, extract music features including instrument type, harmonic progression, rhythmic movement, and melody contour, and build a semantic correlation matrix representing the correlation between tracks based on the music features to form an overall semantic understanding of the music work; S2, adaptive track grouping and mixing: based on the semantic correlation matrix and music features obtained in step S1, use a dynamic clustering algorithm to intelligently divide the multi-track signal into multiple logical mixing groups, each logical mixing group containing one or more tracks with similar or closely related music functions; assign an independent mixing strategy to each logical mixing group, the mixing strategy including at least group-level level balance, frequency equalization, and dynamic range compression parameters; based on the mixing strategy, preliminarily mix the tracks in each logical mixing group to generate a group mixed signal and process potential spectral conflicts between groups; S3, dynamic adjustment and optimization of perceptual importance: take each group mixed signal obtained in step S2 as input, combine a psychoacoustic perception model and a deep learning model to calculate the perceptual importance score of each track and each logical mixing group in the current music segment in real time, the perceptual importance score is determined based on the masking effect of human hearing and the auditory saliency model; dynamically adjust the global mixing parameters according to the perceptual importance score, the global mixing parameters include the total output level, the stereo image position of each logical mixing group, the fine balance of the overall frequency response, and the amount of reverb effect, so that in the final mixing result, elements with high perceptual importance are highlighted, while elements with low importance are suppressed, thus achieving optimal mixing results consistent with human perception characteristics; S4, final mixed signal generation and output: high-precision perceptual encoding is performed on the mixed signal after dynamic adjustment and optimization in step S3, and an encoding algorithm based on an improved psychoacoustic model is used to perform moderate data compression while ensuring audio quality, finally generating and outputting a digital audio file with completed intelligent mixing.

[0006] Further, in step S1, the specific process of the deep learning model based on the Transformer architecture analyzing the multi-track signal includes: taking the time-frequency spectrogram or audio feature sequence of each track as input, calculating the correlation weight between different track features through the multi-head self-attention mechanism of the model, and generating the semantic correlation matrix; at the same time, the model identifies and labels the lead instrument track, rhythm base track, harmonic padding track, and decorative track through pre-trained music knowledge, providing classification basis based on music semantics for subsequent adaptive grouping.

[0007] Further, in step S2, the dynamic clustering algorithm is an online clustering algorithm based on music context, which can dynamically adjust the grouping result according to the paragraph change of music structure; the process of assigning an independent mixing strategy to each logical mix group is: an expert mixing rule library is established in advance for different music styles and instrument combinations, the system matches the closest reference strategy from the rule library according to the current analyzed music characteristics, and the parameters are fine-tuned based on the specific audio characteristics of the current track group.

[0008] Further, in step S3, the method of calculating the perceptual importance score in real time includes: constructing a comprehensive evaluation function that integrates classic psychoacoustic parameters and neural network output values, the psychoacoustic parameters include critical band energy, instantaneous loudness and timbre distinctness, and the neural network output values are calculated by a specially trained importance evaluation network according to the current music context; the process of dynamically adjusting the global mixing parameters is based on the score results, and a closed-loop feedback control system is used for real-time iterative optimization until the preset perceptual target is reached.

[0009] Further, in step S4, when the high-precision perceptual encoding process is performed, the bit allocation strategy is associated with the final perceptual importance score obtained in step S3; more encoding bits are allocated to the frequency region with high perceptual importance to retain details, and fewer encoding bits are allocated to the frequency region with low perceptual importance, so that perceptual optimization encoding based on mixing results can be achieved.

[0010] Further, the method further includes a user interaction module that allows professional users to fine-tune the automatic mixing results; the user interaction module provides a visual interface based on the perceptual importance score, highlights the key points of system decision, and allows users to guide the system to generate a final mix that meets individual preferences by adjusting the importance score or directly modifying the mixing parameters.

[0011] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the above-mentioned perceptual encoding-based digital music multi-track intelligent mixing method.

[0012] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the above-mentioned perceptual encoding-based digital music multi-track intelligent mixing method.

[0013] Beneficial effects: The application realizes the structured semantic analysis of multi-track music by introducing a music content understanding layer based on a deep learning model, which upgrades the mixing process from the traditional signal processing level to the music semantic understanding level. The system can automatically identify key features such as instruments, harmonies, rhythms, etc., and understand the functional correlation between tracks, thereby providing accurate context information for subsequent intelligent processing, making the mixing decision more in line with the artistic rules of music creation, and significantly improving the musical expressiveness and professionalism of the mixing result. The application proposes an adaptive track grouping and mixing mechanism. Based on the results of music semantic analysis, this method dynamically divides functionally similar tracks into logical groups for collaborative processing and assigns targeted mixing strategies. This not only effectively solves the complex spectral masking and conflict problems in multi-track mixing, achieving a clearer sense of sound field hierarchy, but also simulates the working logic of professional mixers, breaking down complex global optimization problems into more manageable intra-group and inter-group optimization, greatly improving processing efficiency and controllability of the results. The application introduces the core idea of perceptual coding model, the perceptual importance dynamic adjustment algorithm, into the mixing link, which is another major advantage of the application. This algorithm can assess the auditory contribution of each track or track group in the overall mixing in real time, and dynamically optimize the level, sound image and effect parameters accordingly to ensure that the final output is balanced and harmonious in terms of auditory perception. This method makes the mixing output naturally adapt to the perceptual characteristics of the human ear, ensuring excellent subjective listening quality while achieving automated processing. In summary, the application first combines deep music content understanding, adaptive grouping processing and perceptual importance dynamic adjustment, providing a complete and innovative solution for intelligent mixing with high quality, high efficiency and high adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a method flowchart of the application. DETAILED DESCRIPTION

[0015] To make the technical solutions of the application clearer, the application will be further described in detail below in combination with the drawings and specific embodiments.

[0016] Embodiment 1 A digital music multi-track intelligent mixing method based on perceptual coding, comprising: Step one: music content understanding and analysis Firstly, the system loads multi-track digital music signals (e.g. in WAV format, sampling rate 44.1 kHz, bit depth 24 bit). A deep learning model based on Transformer architecture (pre-trained on a large annotated multi-track music dataset, e.g. MUSDB18) is used for structured analysis. Specifically, the time-frequency spectrogram (obtained by short-time Fourier transform, window length 2048 points, hop size 512 points) of each track is taken as the input feature sequence. The multi-head self-attention mechanism of the model (8 heads, hidden layer dimension 512) calculates the correlation weights of the features between tracks, for example, identifying the synchrony of the vocal track and the drum group track in rhythm, and the complementarity of the guitar and keyboard in harmony. The model output includes: instrument type (e.g. vocal as the leading instrument, drum group as the rhythm base), harmonic progression (e.g. the progression of the C major chord to the minor chord), rhythmic movement (e.g. 4 / 4 beat at BPM 120), and melody contour (e.g. the pitch change of the vocal melody line). Based on these features, a semantic correlation matrix (size N x N, N is the number of tracks) is constructed, and the matrix element value represents the correlation strength between the track pairs (e.g. the correlation degree between the vocal and bass tracks is 0.8, while the correlation degree between the drum group and string instrument tracks is 0.3). This matrix forms the overall semantic understanding of the music work, providing the basis for subsequent grouping.

[0017] Step two: adaptive track grouping mixing Based on the semantic correlation matrix of step one, a dynamic clustering algorithm (e.g. an online K-means variant based on musical context, with dynamic adjustment of the number of clusters K, ranging from 3 to 5 groups) is used for intelligent grouping. For example, the vocal and main chord instruments (guitar, keyboard) are divided into the "main melody group", the drum group and bass are divided into the "rhythm base group", and the string instruments and background pads are divided into the "padding group". The grouping result is dynamically adjusted with the change of the music section (e.g. the importance of the vocal track is enhanced in the chorus section, which may be grouped separately). Each logical mix group is assigned an independent mixing strategy: matching the reference strategy from the expert mixing rule library (preset rules for popular, rock, etc. styles), and fine-tuning the parameters based on the specific features of the tracks. For example, the "main melody group" is assigned an equalization strategy (boosting mid-frequency 2-5 kHz to enhance clarity), compression ratio (2:1), and level (-6 dBFS); the "rhythm base group" is assigned a dynamic control strategy (side-chain compression, triggered by the main melody group). Subsequently, the tracks within each group are preliminarily mixed: using digital equalizers, compressors, and other effectors to process, generating a group mixed signal. At the same time, spectral conflicts between groups are handled, such as suppressing the masking of "rhythm base group" low frequency and "main melody group" mid-frequency through multi-band compression.

[0018] Step three: dynamic adjustment and optimization of perceptual importance The group mixed signal obtained in step two is input, combined with a psychoacoustic model (such as MPEG Psychoacoustic Model 2) and a deep learning model (importance evaluation network, CNN structure, input is the Mel spectrogram of the mixed signal). The perceptual importance score of each track and group is calculated in real time: the psychoacoustic model extracts critical band energy (24 bands), instantaneous loudness (based on ITU-R BS.1770) and timbre distinctness (spectral centroid); the importance evaluation network outputs weights based on the musical context. The comprehensive evaluation function (weighted sum, psychoacoustic parameter weight 0.6, network output weight 0.4) generates a score (0-1 range). For example, in the chorus section, the score of the vocal track rises to 0.9, while the score of the background string instrument drops to 0.2. Based on the score, the global mixing parameters are dynamically adjusted through a closed-loop feedback control system (PID controller): adjust the total output level (vocal group is raised by 3 dB), stereo image (vocal group is centered, drum group is slightly widened), overall equalization (sensitive frequency band of high importance track is raised) and reverb send (background group increases reverb to enhance the sense of space). The optimization process is iterated (100 updates per second) until the mixing result is balanced in terms of listening.

[0019] Step four: final mixing signal generation and output The optimized mixed signal is subjected to high-precision perceptual encoding. The encoding algorithm uses an AAC variant based on an improved psychoacoustic model, in which the bit allocation strategy is associated with the perceptual importance score of step three: for frequency regions with a score higher than 0.7 (such as 2-5 kHz for vocals), allocate 256 kbps bit rate; for regions with a score lower than 0.3 (such as low frequencies in the background), allocate 64 kbps bit rate. After encoding, the final MP4 format audio file is generated and output saved. At the same time, the user interaction module provides a visual interface (such as a DAW plugin GUI) to display the perceptual importance score curve of each track and the mixing parameter adjustment history, allowing engineers to manually fine-tune the score or parameters (such as dragging the score slider to reset the importance of vocals), and the system re-optimizes in real time.

[0020] The above steps are implemented to realize automatic and intelligent mixing of multi-track music, and the output quality meets professional standards. Based on the description, a skilled person can configure a corresponding software environment.

[0021] Embodiment 2 A digital music multi-track intelligent mixing method based on perceptual encoding, comprising: Step one: music content understanding and analysis The device receives multiple audio signals in real time (sampling rate 48 kHz, bit depth 16 bit). A lightweight Transformer model (pruned and quantized, parameter size <1M) is used for real-time analysis. The input features are MFCC coefficients (13 dimensions, frame length 25 ms) for each signal, and the model quickly calculates the relevance through a self-attention mechanism (inference delay <10 ms). For example, in a live scene analysis: identify the microphone track as vocals (lead), the background music track as accompaniment (harmonic padding), and the system prompt as sound effects (decorative). Output music features (such as the matching degree of vocal rhythm and background music BPM) and construct a semantic correlation matrix (update frequency 50 Hz). This matrix succinctly represents track relationships, such as high correlation between vocals and background music (0.7) and low correlation between prompt sound (0.1).

[0022] Step two: adaptive track grouping and mixing Based on the real-time updated semantic correlation matrix, a lightweight dynamic clustering algorithm (such as online DBSCAN, parameter ε=0.5) is used for grouping. For example, vocals and background music are divided into the "main content group", and the prompt sound is separately grouped. The grouping strategy is adaptive to scene changes: for example, when the prompt sound appears frequently, it is temporarily classified into the "effect group". Each group is assigned a mixing strategy: select the reference strategy from the built-in rule library (preset for speech, music, etc.) and fine-tune it. For example, the "main content group" is assigned a speech optimization strategy (high-pass filter 100 Hz, compression ratio 3:1), and the "effect group" is assigned a short-time decay strategy (fast fade-in and fade-out). Then, use an embedded digital signal processor to mix each group, such as applying FIR filters for equalization and solving inter-group conflicts (such as side-chain compression of background music to avoid vocals).

[0023] Step three: dynamic adjustment and optimization of perceptual importance The group mixed signal is sent to the perceptual optimization module. The psychoacoustic model uses a simplified version (based on ERB band calculation of masking threshold), and the importance evaluation network is a miniature CNN (input is a spectral frame, output score). Real-time calculation of perceptual importance score: for example, the score of vocals is 0.9 when speaking and 0.1 when silent; the score of background music is constant at 0.5. The comprehensive evaluation function (linear combination) outputs the score to drive dynamic adjustment: adjust parameters through a software PID controller, such as increasing the volume of vocals (+6 dB), adjusting the sound image (vocals centered), and suppressing background music low frequencies. The adjustment process is iterated every frame (20 ms) to ensure real-time performance.

[0024] Step four: final mixing signal generation and output The optimized signal is encoded in real time using a low-delay Opus encoder, with bit allocation linked to importance score: high bit rate (80 kbps) for high frequency region of human voice, low bit rate (20 kbps) for low frequency region of background music. The output encoded stream is sent to a speaker or network. The user can view the score curve through the device touch screen interaction module, manually override the automatic settings (such as locking the background music level), and the system responds immediately.

[0025] The embodiment shows the applicability of the present application in a resource-limited environment, and realizes low-delay real-time processing through an optimized model and algorithm. Based on the description, a skilled person can develop an embedded audio firmware.

[0026] The above-described embodiments only express several embodiments of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be noted that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A perceptually coded based digital music multi-track intelligent mixing method, characterized in that, The method comprises the following steps: S1, music content understanding and analysis: obtaining a multi-track digital music signal to be mixed, using a deep learning model based on a Transformer architecture to analyze the structure of the multi-track signal, extracting music features including instrument type, harmonic progression, rhythmic movement, and melody contour, and constructing a semantic correlation matrix representing the correlation between tracks based on the music features to form an overall semantic understanding of the music work; S2, adaptive track grouping and mixing: based on the semantic correlation matrix and music features obtained in step S1, using a dynamic clustering algorithm to intelligently divide the multi-track signal into multiple logical mixing groups, each logical mixing group containing one or more tracks with similar or closely related music functions; assigning an independent mixing strategy to each logical mixing group, the mixing strategy including at least group-level level balance, frequency equalization, and dynamic range compression parameters; Based on the mixing strategy, the tracks in each logical mixing group are preliminarily mixed to generate a group mixed signal, and potential spectral conflicts between groups are processed; S3, dynamic adjustment and optimization of perceptual importance: taking the group mixed signals obtained in step S2 as input, combining a psychoacoustic perception model and a deep learning model to calculate the perceptual importance score of each track and each logical mixing group in the current music segment in real time, the perceptual importance score being determined based on the masking effect of human hearing and the auditory saliency model; dynamically adjusting global mixing parameters according to the perceptual importance score, the global mixing parameters including total output level, stereo image position of each logical mixing group, subtle equalization of overall frequency response, and amount of reverb effect, so that in the final mixing result, elements with high perceptual importance are highlighted, while elements with low importance are appropriately suppressed, thus achieving optimal mixing results consistent with human auditory perception characteristics; S4, final mixed signal generation and output: high-precision perceptual encoding is performed on the mixed signal after dynamic adjustment and optimization in step S3, and an encoding algorithm based on an improved psychoacoustic model is used to perform moderate data compression while ensuring audio quality, finally generating and outputting a digital audio file after intelligent mixing.

2. The perceptually coded based digital music multi-track intelligent mixing method according to claim 1, characterized in that, In step S1, the specific process of the deep learning model based on the Transformer architecture for analyzing the multi-track signal includes: taking the time-frequency spectrum or audio feature sequence of each track as input, calculating the correlation weight between different track features through the multi-head self-attention mechanism of the model, and generating the semantic correlation matrix; at the same time, the model identifies and labels the lead instrument track, rhythm base track, harmonic padding track, and decorative track based on the music knowledge learned through pre-training, providing classification basis based on music semantics for subsequent adaptive grouping.

3. The perceptually coded based digital music multi-track intelligent mixing method of claim 1, wherein, In step S2, the dynamic clustering algorithm is an online clustering algorithm based on music context, which can dynamically adjust the grouping result according to the paragraph change of the music structure; the process of assigning an independent mixing strategy to each logical mix group is: an expert mixing rule library is established in advance for different music styles and instrument combinations, the system matches the closest reference strategy from the rule library according to the current analyzed music characteristics, and the parameters are fine-tuned based on the specific audio characteristics of the current track group.

4. The perceptually coded based digital music multi-track intelligent mixing method of claim 1, wherein, In step S3, the method of calculating the perceptual importance score in real time includes: constructing a comprehensive evaluation function that integrates classic psychoacoustic parameters and neural network output values, the psychoacoustic parameters include critical band energy, instantaneous loudness and timbre distinctness, and the neural network output values are calculated by a specially trained importance evaluation network according to the current music context; the process of dynamically adjusting the global mixing parameters is based on the score results, and a closed-loop feedback control system is used for real-time iterative optimization until the preset perceptual target is reached.

5. The perceptually coded based digital music multi-track intelligent mixing method of claim 1, wherein, In step S4, the high-precision perceptual encoding process is associated with the final perceptual importance score obtained in step S3 when performing data compression; more encoding bits are allocated to the frequency area with high perceptual importance to retain details, and fewer encoding bits are allocated to the frequency area with low perceptual importance, which can realize perceptual optimization encoding based on the mixing results.

6. The perceptually coded based digital music multi-track intelligent mixing method of claim 1, wherein, The method further includes a user interaction module that allows professional users to fine-tune the automatic mixing results; the user interaction module provides a visual interface based on the perceptual importance score, highlights the key points of system decision, and allows users to guide the system to generate a final mix that better meets individual preferences by adjusting the importance score or directly modifying the mixing parameters.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the perceptual coding-based digital music multi-track intelligent mixing method according to any one of claims 1-6.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the perceptual coding-based digital music multi-track intelligent mixing method according to any one of claims 1-6.

Images