Karaoke all-in-one machine intelligent tuning method based on AI sound effect optimization

Abstract

The invention relates to the technical field of artificial intelligence and digital audio signal processing, and particularly discloses a Karaoke all-in-one machine intelligent tuning method based on AI sound effect optimization, and the method comprises the steps: obtaining an original mixed audio signal sung by a user, and carrying out the preprocessing; processing the preprocessed audio by using a multi-source audio separation model based on a deep neural network, and extracting a human voice track and accompaniment components; extracting time-frequency domain features from the human voice track, and performing quantitative evaluation on the voice tension degree of the user in combination with the trained voice state recognition model; dynamically adjusting an equalizer and a dynamic range compression parameter according to an evaluation result to realize adaptive sound effect optimization; synthesizing the optimized voice and accompaniment and outputting the synthesized voice and accompaniment to monitoring equipment; and finally, establishing a personalized user behavior feature vector by collecting user singing performance and historical tuning data, and performing online fine tuning on a system strategy based on a deep Q network reinforcement learning mechanism to continuously optimize a tuning effect.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and digital audio signal processing, and in particular to an intelligent tuning method for a karaoke all-in-one machine based on AI sound effect optimization. Background Art

[0002] With the rapid development of artificial intelligence and digital audio processing technologies, smart entertainment devices such as all-in-one karaoke machines have been widely used in both home and commercial settings. Traditional karaoke systems mainly rely on fixed-parameter equalizers and compressors for sound processing, which makes it difficult to dynamically adjust according to the user's real-time vocal state. This leads to problems such as distortion, increased harshness, and vocal fatigue during high-pitched singing, prolonged use, or singing by non-professional users. In recent years, deep learning-based speech separation, voice analysis, and adaptive tuning technologies have gradually been applied to the field of vocal enhancement. Multi-source audio separation models can achieve high-quality extraction of vocals and accompaniment, and combined with physiological acoustic characteristics, the user's vocal state can be identified, thereby enhancing the singing experience. However, how to achieve accurate vocal perception in a complex musical context and optimize personalized sound effects based on individual differences remains a core challenge facing current intelligent tuning systems.

[0003] The existing technology has the following deficiencies: In scenarios with strong background noise or a wide dynamic range in musical accompaniment, the system may misclassify non-vocal signals (such as high-frequency overtones of electric guitars and percussive drum beats) as signs of tension in the user's voice during real-time analysis, triggering unnecessary EQ curve adjustments or gain reduction. This misidentification not only weakens the expressiveness of the vocals but can also lead to unbalanced audio output, severely impacting the performer's auditory feedback and performance experience. This exposes the limitations of current speech feature extraction algorithms in terms of interference resistance and semantic understanding, necessitating improvements through enhanced sound source separation capabilities and context-aware logic. Summary of the Invention

[0004] The purpose of the present invention is to provide an intelligent tuning method for a karaoke all-in-one machine based on AI sound effect optimization to solve the problems in the above background.

[0005] The purpose of the present invention can be achieved through the following technical solutions: An intelligent tuning method for a karaoke machine based on AI sound effect optimization includes the following steps: S1: Obtaining an original mixed audio signal sung by a user, and preprocessing the mixed audio signal to eliminate environmental noise and gain fluctuations; S2: Use a deep neural network model to perform multi-source audio separation on the pre-processed mixed audio signal to obtain multiple independent audio components including vocal tracks, drum tracks, melodic instrument tracks, and low-frequency tracks; S3: extracting time-frequency domain speech feature data from the vocal track, and quantitatively evaluating the tension level of the user's voice based on the trained voice state recognition model; S4: dynamically adjusting equalizer parameters and dynamic range compression parameters in the audio processing module according to the vocal tension evaluation result to achieve adaptive sound effect optimization of the human voice signal; S5: Resynthesizes the optimized vocal signal with the accompaniment track and outputs it to the monitoring device for real-time user feedback; S6: Based on the user's real-time singing performance and historical tuning data, the voice state recognition model and sound effect optimization strategy are fine-tuned online through a reinforcement learning mechanism to improve personalized tuning capabilities.

[0006] As a further solution of the present invention: the deep neural network model is a multi-source audio separation model built based on D3Net, which includes the following processing steps: Perform short-time Fourier transform on the preprocessed mixed audio signal to obtain time-frequency domain representation; Inputting the time-frequency domain representation into a deep network structure comprising a multi-scale convolutional module and a gated recurrent unit; The mask estimation mechanism is used to generate spectral masks corresponding to human voice, drum beats, melodic instruments and low-frequency components respectively; The original mixed spectrum is reconstructed based on the spectrum mask, and multiple independent audio tracks are output.

[0007] As a further solution of the present invention, the process of performing multi-source audio separation further includes the following enhancement steps: Introducing a multi-head self-attention mechanism in deep neural networks to model long-range dependencies between different audio sources; Dynamically allocate attention weights based on audio content to strengthen focus on the human voice frequency band; Use attention-weighted feature vectors to perform post-processing optimization on each audio track; The optimized vocal track is output for subsequent voice status identification and tuning parameter adjustment.

[0008] As a further solution of the present invention: the multi-source audio separation process further includes the following semantic assistance steps: During the training phase, each audio component is labeled with corresponding semantic label information, including vocal type (capella or harmony), percussion type (bass drum or snare drum), and melodic instrument type (guitar or piano). Build a multi-task learning framework with a semantic embedding layer to simultaneously predict the audio component categories during the separation process; In the inference stage, the target separation channel is dynamically selected according to the prediction results, and the energy output of non-target components is suppressed; The final output is a semantically interpretable separated audio track, which improves the system's adaptability to complex accompaniment scenarios.

[0009] As a further solution of the present invention: the process of extracting speech features and evaluating the degree of vocal tension includes the following steps: Performing short-time Fourier transform and wavelet packet decomposition on the vocal track to obtain linear spectrum and nonlinear time-frequency features respectively; Extract Mel-frequency cepstral coefficients, fundamental frequency trajectory and high-frequency energy ratio as basic physiological acoustic features; Inputting multiple types of features into a fusion network, wherein the fusion network includes parallel convolution branches and an attention gating mechanism; The vocal tension score is output based on the fusion features, and the score result is used to drive the subsequent adaptive tuning module.

[0010] As a further solution of the present invention: the voice state recognition model adopts the following enhancement strategy during the training process: Construct a dataset containing various voice states; Using transfer learning methods, we conduct cross-group generalization training on singing samples of professional singers, ordinary users, and users of different genders; A weighted cross-entropy loss function is introduced in the model output layer to enhance the sensitivity of identifying nervousness and broken voice categories; The trained model is used to classify and score real-time human voices, and a continuous tension value is output for tuning control.

[0011] As a further solution of the present invention, the quantitative evaluation of the user's vocal tension further includes the following dynamic analysis steps: Real-time detection of the target pitch curve of the current singing segment and comparison with the user's historical singing ability; The singing difficulty coefficient is calculated based on the deviation rate between the target pitch and the actual singing pitch; Combining the current pitch position with the vocal fold tension model to predict potential phonation burden; The singing difficulty coefficient and vocal burden are input into the timing neural network, the vocal tension assessment weight is dynamically adjusted, and personalized assessment results are output.

[0012] As a further solution of the present invention: the adaptive sound optimization of the human voice signal specifically includes: A nonlinear mapping function is established between vocal tension and audio parameters, and the tension score is converted into an equalizer band gain adjustment value and a compressor threshold offset. When it detects that the user is in a state of high tension, it automatically reduces the high-frequency gain and moderately improves the clarity of the mid-frequency band to alleviate the harshness; Dynamically adjust the attack and recovery time of the dynamic range compressor to enhance the control of transient energy in the high-pitched range; The adjusted parameters are loaded into the digital signal processing module in real time to complete the adaptive sound optimization of the vocal signal, improving auditory comfort and singing stability.

[0013] As a further solution of the present invention, the optimized vocal signal is resynthesized with the accompaniment track and outputted to a monitoring device for real-time user feedback, specifically including: Time-align the optimized vocal signal with the accompaniment track to compensate for system delays introduced by audio separation and processing; Apply virtual sound field rendering algorithm to perform spatial processing on the synthesized audio signal to simulate the auditory perception effect in KTV box or stage environment; Dynamically adjust the reverberation feedback intensity and stereo image distribution based on the type of monitoring device the user is wearing, whether it is headphones or speakers; The processed synthesized audio is output to the monitoring device in real time to provide a low-latency, highly immersive singing feedback experience.

[0014] As a further solution of the present invention: the online fine-tuning of the voice state recognition model and the sound effect optimization strategy by the reinforcement learning mechanism specifically includes: Real-time collection of user pitch deviation, rhythm stability, and vocal state change information, combined with historical tuning parameters to build a personalized user behavior feature vector; Inputting the feature vector into a reinforcement learning controller based on a deep Q network to generate an optimal tuning action strategy for the current singing state; Dynamically update the policy network parameters of the reinforcement learning model based on the user's subjective feedback on the tuning results or the reward signal provided by the system scoring mechanism; Based on the updated strategy, lightweight online fine-tuning of the multi-source audio separation model and sound effect optimization module is performed to achieve continuous personalized evolution of the tuning system.

[0015] Beneficial effects of the present invention: (1) This invention introduces a multi-source audio separation technology based on a deep neural network, combines a multi-head self-attention mechanism with a semantic auxiliary label modeling method, and constructs a human voice extraction system with strong generalization and context perception capabilities. After performing a short-time Fourier transform on the mixed audio signal, the system uses a deep network structure containing a multi-scale convolution module and a gated recurrent unit (GRU) to achieve high-precision separation of human voice, drum beats, melodic instruments and low-frequency components; and dynamically enhances the focus on the key frequency bands of human voice through the attention mechanism, and can still maintain stable and pure human voice extraction capabilities in complex accompaniment, high dynamic range or strong rhythm background. At the same time, the introduction of semantic label information during the training process enables the model to have the ability to understand and predict the type of audio components. In the inference stage, the separation strategy can be dynamically optimized according to the song style, thereby significantly improving the adaptability and robustness of the system in various music scenes. This technological breakthrough effectively avoids the problem of misjudgment of voice status caused by background music interference, provides a high-quality and reliable human voice signal foundation for subsequent tension assessment and adaptive tuning based on physiological acoustic characteristics, and greatly improves the stability and user experience of the overall intelligent tuning system.

[0016] (2) The present invention achieves accurate perception of the user's vocal state and intelligent generation of adaptive tuning strategies by constructing a vocal tension assessment model based on multimodal feature fusion and combining deep reinforcement learning mechanism with user behavior feature modeling technology. The system extracts time-frequency domain features (such as MFCC, fundamental frequency trajectory, high-frequency energy ratio) from the vocal track and integrates contextual information such as singing difficulty coefficient and historical tuning preference to construct a high-dimensional feature vector, which is input into a neural network with an integrated attention gating mechanism to output a continuous tension score. At the same time, a reinforcement learning controller based on a deep Q-network (DQN) is introduced to map the tension assessment results into specific sound control instructions such as equalizer gain adjustment value, dynamic range compression parameter and reverberation feedback intensity, and continuously optimize the strategy network parameters based on the user's real-time feedback and the system's automatic scoring mechanism. This mechanism can not only dynamically identify and alleviate the harshness and vocal fatigue caused by vocal tension during singing, but also personalize the vocal state recognition model and sound optimization module through lightweight online fine-tuning technology, so that the system has the ability to continuously learn the user's singing habits and hearing preferences. The closed-loop intelligent tuning process formed in this way significantly improves the accuracy and naturalness of the tuning response, effectively enhances the user's singing comfort, expressiveness and immersion, and comprehensively improves the interactive experience and user satisfaction in the karaoke scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described below with reference to the accompanying drawings.

[0018] Figure 1This is a flow chart of an intelligent tuning method for a karaoke machine based on AI sound effect optimization according to the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] See also Figure 1 As shown, the present invention is an intelligent tuning method for a karaoke machine based on AI sound effect optimization, comprising the following steps: S1: Obtaining an original mixed audio signal sung by a user, and preprocessing the mixed audio signal to eliminate environmental noise and gain fluctuations; S2: Use a deep neural network model to perform multi-source audio separation on the pre-processed mixed audio signal to obtain multiple independent audio components including vocal tracks, drum tracks, melodic instrument tracks, and low-frequency tracks; S3: extracting time-frequency domain speech feature data from the vocal track, and quantitatively evaluating the tension level of the user's voice based on the trained voice state recognition model; S4: dynamically adjusting equalizer parameters and dynamic range compression parameters in the audio processing module according to the vocal tension evaluation result to achieve adaptive sound effect optimization of the human voice signal; S5: Resynthesizes the optimized vocal signal with the accompaniment track and outputs it to the monitoring device for real-time user feedback; S6: Based on the user's real-time singing performance and historical tuning data, the voice state recognition model and sound effect optimization strategy are fine-tuned online through a reinforcement learning mechanism to improve personalized tuning capabilities.

[0021] In S1, an original mixed audio signal of the user's singing is obtained, and the mixed audio signal is preprocessed to eliminate environmental noise and gain fluctuation, specifically including: Original audio signal acquisition: The original mixed audio signal generated by the user's singing is collected through the built-in high-sensitivity microphone array or external professional vocal pickup equipment; The mixed audio signal includes user voice, background accompaniment, environmental noise, and possible reverberation, echo and other components; The acquisition process supports multiple sampling rates (such as 44.1kHz, 48kHz) and bit depths (such as 16bit, 24bit), ensuring that the audio quality meets professional-level processing requirements.

[0022] Digital noise reduction processing: Use spectral subtraction, Wiener filtering, or deep learning-based speech enhancement models (such as SEGAN and RNNoise) to reduce noise on the collected audio signals. Detect and suppress non-target sound sources (such as air conditioning noise, room reverberation, traffic noise, etc.), preserving clear vocals and accompaniment components; For sudden interference noise (such as knocking and blasting sounds), a short-time energy detection mechanism is used to suppress transients to prevent them from affecting subsequent analysis modules.

[0023] Gain Normalization and Dynamic Range Control: Perform automatic gain control (AGC) on audio signals to adjust input signals of varying strengths to a uniform target loudness level; A compressor is used to moderately compress the dynamic range to account for loudness fluctuations caused by factors such as the user's distance from the microphone and differences in vocalization methods. Set adjustable threshold and ratio parameters to prevent over-compression and artifacts while ensuring consistent and stable output audio.

[0024] Phase correction and channel balance processing (optional): In multi-channel input scenarios, the phase difference between the left and right channels is corrected to improve the consistency of stereo imaging; Adjust the relative levels between channels to eliminate sound field shifts caused by improper microphone placement and enhance the monitoring experience.

[0025] Through the above preprocessing process, the system can obtain high-quality, low-noise, and stable loudness mixed audio signals, providing a reliable data foundation for subsequent multi-source audio separation, voice state recognition, and personalized tuning, thereby improving the robustness of the overall system and user experience.

[0026] In S2, a deep neural network model is used to perform multi-source audio separation on the pre-processed mixed audio signal to obtain multiple independent audio components including vocal tracks, drum tracks, melodic instrument tracks, and low-frequency tracks. Specifically, the following components are obtained: Perform short-time Fourier transform (STFT) on the preprocessed mixed audio signal to convert the one-dimensional time-domain audio signal into a two-dimensional time-spectrum graph to obtain the amplitude spectrum and phase information; The time-frequency spectrum is saved in complex form and can be further decomposed into two independent channels of amplitude and phase for subsequent model input.

[0027] The spectrogram is input into a deep network structure consisting of multi-scale convolutional modules and gated recurrent units (GRU): The multi-scale convolution module uses convolution kernels with different receptive fields to process the spectrogram in parallel to capture local frequency band features; The GRU layer is responsible for modeling continuity in the time dimension and capturing dynamic features such as singing rhythm and pitch changes; By stacking multiple convolution-GRU modules, a separation backbone network with strong spatiotemporal modeling capabilities is constructed.

[0028] A mask estimation mechanism is introduced at the output of the network to generate spectral masks corresponding to human voice, drum beats, melodic instruments and low-frequency components respectively: Each mask represents the energy distribution weight of a specific audio component in the original spectrum; Perform element-wise multiplication of the mask and the original mixed spectrum to reconstruct each independent audio track; The original phase information is retained during the reconstruction process to ensure a natural listening experience after audio synthesis.

[0029] Attention mechanism enhances audio separation performance.

[0030] Introducing a multi-head self-attention mechanism in deep networks to model long-range dependencies between different audio sources: The attention mechanism dynamically assigns attention weights based on the contextual information of the current frame; Special attention is paid to the human voice frequency range (approximately 85Hz to 250Hz for male voices and 165Hz to 260Hz for female voices); The attention-weighted feature vector is used to perform post-processing optimization on each audio track to improve the clarity of the human voice and the accuracy of accompaniment separation.

[0031] During the training phase, each audio component is labeled with corresponding semantic label information, including: Vocal type: a cappella, harmony; Percussion types: kick drum, snare drum; Melodic instrument types: guitar, piano; Build a multi-task learning framework with a semantic embedding layer to simultaneously predict the audio component categories during the separation process; Joint optimization of separation and classification is achieved by sharing feature extraction layers and task-specific decoding layers; The classification results serve as auxiliary information to guide parameter adjustment during the separation process.

[0032] During the inference phase, the target separation channel is dynamically selected based on the predicted audio component type, and the energy output of non-target components is suppressed: If the user only wants to highlight the vocal performance, the energy ratio of the drum set and melodic instruments will be automatically reduced; If the current song style is detected as rock, the frequency bands of electric guitar and bass drum will be retained first; The final output is a semantically interpretable separated audio track, which significantly improves the system's adaptability and stability in complex accompaniment scenarios.

[0033] The above technical solution realizes the complete process from raw audio acquisition to multi-source audio separation. It not only improves the separation accuracy of vocals and accompaniment in karaoke scenarios, but also enhances the intelligence and personalization capabilities of the system by introducing attention mechanism and semantic-assisted learning. It has good engineering application prospects and technological innovation value.

[0034] In S3, time-frequency domain speech feature data is extracted from the vocal track, and the user's vocal tension is quantitatively assessed based on the trained voice state recognition model, specifically including: Specific implementation methods for voice state recognition and tension level assessment: In the AI-based sound optimization-based intelligent tuning method for karaoke machines described in the present invention, multi-level feature analysis and model evaluation are performed on the vocal track to achieve accurate recognition of the user's vocal state and adaptive tuning control. The method specifically includes the following steps: Perform short-time Fourier transform (STFT) and wavelet packet decomposition on the vocal track output from the multi-source separation module: STFT is used to generate a linear spectrum to capture the overall energy distribution; Wavelet packet decomposition extracts nonlinear time-frequency features and enhances sensitivity to transient vocal states (such as crackling and jitter). The two representations together serve as the basic input for subsequent feature fusion.

[0035] The following three types of basic physiological acoustic features are extracted from the above time-frequency representation: Mel-frequency cepstral coefficients (MFCC): used to characterize the formant structure of sound; Fundamental frequency trajectory (F0 contour): reflects the user's current pitch change trend and vocal stability; High-frequency Energy Ratio (HNR): This measures the clarity and turbidity of the sound, helping to determine whether the voice is hoarse or cracked. These features together form a multidimensional feature vector, which is used for subsequent classification and scoring tasks.

[0036] Construct a feature fusion network with parallel convolution branches and attention gating mechanism: The parallel convolution branches process different modal data such as linear spectrum, wavelet packet coefficients and time domain MFCC respectively; The attention gating mechanism dynamically weights and fuses the features of each branch according to the importance of the current speech segment; The fused high-order feature vector is input into the fully connected layer or lightweight Transformer module to output a preliminary voice state prediction.

[0037] During the model training phase, a labeled dataset covering various voice states is constructed. The labeled content includes: “Relaxed”: The user speaks naturally, without noticeable muscle tension; “Moderate”: normal singing state; "Tense": high vocal cord tension, accompanied by mild throat discomfort; "Tendency to break vocalization": an unstable vocal state that is about to be distorted or broken; the dataset covers samples of professional singers, ordinary users, and users of different genders, and is labeled through a combination of manual listening and automatic detection.

[0038] Using a transfer learning strategy, the model pre-trained on large-scale singing speech data is used as the initial weight; Then, fine-tune the system for specific user groups in karaoke scenarios (such as ordinary consumers and non-professional singers); By introducing cross-gender, cross-voice, and cross-style data augmentation strategies, the model's adaptability to diverse user behaviors is improved.

[0039] A weighted cross-entropy loss function is introduced in the model output layer to assign higher weights to samples in the "nervous" and "breaking voice" categories; At the same time, a label smoothing mechanism is introduced to prevent overfitting to a small number of extreme samples; The final output of the model is a four-category result and a corresponding continuous tension score (such as a confidence value between 0 and 1) for use in the subsequent tuning module.

[0040] During real-time operation, the system continuously monitors the target pitch curve of the current singing segment and compares it with the user's historical singing ability: If the target pitch is much higher than the user's average singing level, it is judged as a difficult passage; The singing difficulty coefficient is calculated based on the deviation rate between the actual singing pitch and the target pitch; Combine the current pitch position with a pre-defined vocal cord tension model (e.g., a statistical model based on age, gender, and vocal range) to predict potential vocal burden. Input the singing difficulty coefficient and vocal burden into a time series neural network (such as LSTM or GRU) to dynamically adjust the vocal tension evaluation weight; The final output is a personalized vocal tension assessment result, which supports the implementation of tuning strategies that vary from person to person.

[0041] The above solution realizes the complete process from original human voice signal to voice status recognition. Through key technical means such as multimodal feature fusion, semantic label modeling, transfer learning and dynamic evaluation mechanism, it significantly improves the karaoke all-in-one machine's perception ability and response accuracy of the user's vocal status in complex singing scenarios, and has good practicality and scalability.

[0042] In S4, according to the vocal tension evaluation result, the equalizer parameters and dynamic range compression parameters in the audio processing module are dynamically adjusted to achieve adaptive sound optimization of the human voice signal, specifically including: Construct a nonlinear mapping function based on polynomial fitting or neural network regression to map the vocal tension score (e.g., a continuous value between 0 and 1) into a set of executable audio processing parameters: Mapping output includes equalizer gain adjustment values ​​for each frequency band, dynamic range compressor threshold offset, attack time and recovery time, etc. The mapping function is generated by training a large amount of user test data to ensure that the parameter adjustment strategy has good perceptual consistency under different stress levels; The high tension state corresponds to a more inhibitory parameter configuration, while the low tension state maintains natural vocal performance.

[0043] When the system detects that the user is in a high-stress state (for example, a stress score > 0.7), it automatically performs the following audio parameter adjustments: The equalizer module reduces the gain of the high frequency band from 3kHz to 8kHz (typically -2dB to -6dB) to alleviate the harshness caused by tight vocal cords; At the same time, the mid-frequency gain of 500Hz to 2kHz is moderately increased (+1dB to +3dB) to enhance voice clarity and resonance; The above adjustments are achieved through a smooth transition of sliding windows to avoid auditory discomfort caused by sudden changes in parameters.

[0044] In the dynamic range compressor module, the system adjusts its core parameters in real time based on the tension score: Increase the compression threshold and shorten the attack time, making the compressor respond faster to transient energy in the high range; Prolong recovery time to prevent the "pumping effect" caused by rapid release of sound; Adjust the compression ratio and knee parameters to make the compression process smoother and more natural, effectively controlling the risk of distortion. Especially for the high pitch area (such as F0>300Hz), local compression enhancement is performed to retain the overall tone balance.

[0045] Write the adjusted audio parameters to the built-in digital signal processing (DSP) module or software audio engine in real time through the API interface: Use low-latency communication protocols (such as I²C, SPI, or ASIO) to ensure parameter updates are synchronized with audio playback; After each parameter update, the current audio state and user feedback are recorded for subsequent iterative optimization of the reinforcement learning model; The system also provides a user-adjustable "tuning style preference" option, allowing users to switch between modes such as "natural", "soft", and "professional" to enhance the personalized experience.

[0046] The above solution realizes a complete closed-loop process from voice status recognition to automatic adjustment of audio parameters. By introducing key technical means such as nonlinear mapping functions, multi-band equalization control, and dynamic compression optimization, it significantly improves the perception ability and response accuracy of the user's vocal status in the karaoke scenario, enhances the system's intelligence level and user experience, and has good practicality and scalability.

[0047] In S5, the optimized vocal signal is resynthesized with the accompaniment track and output to the monitoring device for real-time user feedback, including: Before combining the optimized vocal signal with the accompaniment track, the system first performs precise time alignment: By detecting the time domain offset (Latency Offset) between audio tracks, we can identify the system delay introduced by audio separation, feature extraction and parameter adjustment. Calculate the relative delay between tracks using sliding window correlation analysis or peak detection algorithm; Perform frame-level interpolation or cropping on vocal tracks to keep them in strict sync with accompaniment tracks; The delay compensation accuracy is controlled within 5ms, ensuring the consistency of sound and picture synchronization and singing rhythm perception.

[0048] Before the synthesized audio signal is output, the virtual sound field rendering algorithm is applied to spatialize the mixed signal: Use head-related transfer function (HRTF) models or convolution-based spatial filters to simulate the three-dimensional sound field characteristics in different listening environments; Preset multiple spatial rendering modes for typical scenes such as KTV boxes, stage scenes, and recording studios; The vocals are placed in the center channel, and the accompanying instruments are distributed to the left, right, or back space according to the style, enhancing the three-dimensional sense and immersive experience; The rendering process supports dual-mode adaptation for headphones and speakers, avoiding spatial perception distortion caused by differences in playback devices.

[0049] The system automatically identifies the type of monitoring device the user is currently using (such as in-ear headphones, open headphones, or built-in speakers) and dynamically adjusts the following audio rendering parameters: If playing with headphones, moderately reduce the reverberation feedback intensity to prevent excessive envelopment and enhance mid- and high-frequency clarity. If playing through speakers, add room reflection modeling to enhance low-frequency resonance and create a more realistic KTV atmosphere; Automatically correct stereo image distribution based on the device's frequency response curve to ensure accurate vocal positioning and balanced accompaniment distribution; The device identification can be achieved through hardware interface detection or user manual selection.

[0050] After completing the above processing, the system outputs the final synthesized audio signal to the monitoring device in real time through the low-latency audio engine: Use low-latency audio interface protocols such as ASIO, WASAPI, or Android AAudio to ensure end-to-end latency is less than 50ms; The output channel supports mono, stereo and virtual surround sound formats to meet different user needs; Provides an adjustable dry / wet ratio control option, allowing users to independently determine the mixing ratio of vocals and accompaniment; Monitor the audio output status in real time. If an interruption or abnormal delay is detected, it will automatically switch to the backup buffer mechanism to maintain the smoothness of the performance.

[0051] The above solution realizes the complete process from multi-track synthesis to monitoring feedback. By introducing key technical means such as time alignment, virtual sound field rendering, device adaptive reverberation control and low-latency output mechanism, it significantly improves the immersive and interactive experience of the karaoke all-in-one machine in real-time singing scenarios, and has good practicality and scalability.

[0052] In S6, based on the user's real-time singing performance and historical tuning data, the voice state recognition model and sound optimization strategy are fine-tuned online through a reinforcement learning mechanism to improve personalized tuning capabilities, including: The system continuously collects multi-dimensional singing behavior data of users and generates personalized feature vectors based on historical tuning parameters: Pitch deviation: The current pitch of the sung note is extracted using a pitch detection algorithm (such as YIN, SWIPE, or a CNN-based pitch estimation model), and compared with the target pitch of the accompaniment to calculate the error value; Rhythm stability: Analyzes the offset between the user's actual rhythm and the accompaniment beat to identify whether there is any rhythmic overshoot, dragging, or instability; Voice status changes: Combined with the voice tension score, probability of breaking voice, and vocal cord fatigue index, the user's voice health is assessed; Historical tuning parameters: This records the user's past usage of the equalizer settings, compressor threshold, reverberation intensity, and other parameter combinations. This information is normalized and constructed into a dimensionally scalable user behavior feature vector, which is used to drive the subsequent reinforcement learning controller.

[0053] The feature vector is input into the pre-trained deep Q network controller to generate the optimal tuning action strategy for the current singing state: The DQN controller uses a convolutional neural network (CNN) or a fully connected network (FCN) as a feature encoder to output the value function of each tuning parameter adjustment action; The action space includes multiple dimensions such as equalizer band gain adjustment, dynamic range compressor threshold offset, and reverberation dry-wet ratio adjustment; The controller selects the tuning action that maximizes the expected reward based on the current state, forming a personalized sound optimization plan; To avoid sudden changes in auditory perception during the exploration process, the system adopts the ε-greedy strategy and sliding average mechanism to achieve smooth transition parameter updates.

[0054] During reinforcement learning, the system obtains feedback signals in two ways to update the policy network parameters: Subjective feedback mechanism: Users can rate or confirm the current tuning effect through physical knobs, touch interface or voice commands, and the system converts the rating results into positive / negative reward signals; Automatic scoring mechanism: This system objectively evaluates the tuned vocals based on audio quality assessment models (such as PESQ, STOI, or custom perceptual models) and generates a continuous reward value. The model uses Experience Replay and Target Network mechanisms to stabilize the training process; The gradient descent method is used to update the DQN policy network parameters to ensure that the model can continuously adapt to the user's singing style and auditory preferences.

[0055] Based on the updated reinforcement learning strategy, the system performs online fine-tuning operations on multiple key modules: Fine-tune the parameters of some layers of the multi-source audio separation model (such as the attention weight layer and the mask prediction layer) in a frozen manner to improve the accuracy of extracting specific vocal components; Locally adjust the EQ and compressor parameter mapping functions in the sound optimization module to make them more suitable for the user's current vocal characteristics; The fine-tuning process uses a differential privacy protection mechanism and an incremental update strategy to prevent model overfitting and maintain low computational overhead; All updates run in a lightweight form on edge devices, allowing personalized evolution to be completed without relying on cloud services.

[0056] The above solution realizes the complete process from user behavior perception to autonomous evolution of tuning strategy. By introducing key technical means such as deep Q network, multi-dimensional feature modeling and online fine-tuning mechanism, it significantly improves the karaoke machine's ability to understand and respond to users' personalized needs during long-term use, enhances the system's intelligence level and user experience, and has good practicality and scalability.

[0057] The working principle of the invention is as follows: The invention obtains the original mixed audio signal of the user's singing and performs preprocessing such as noise reduction and gain normalization to extract a pure human voice signal; uses a deep neural network to perform multi-source separation on the mixed audio to accurately extract the human voice track and accompaniment components; extracts time-frequency domain features from the human voice and combines them with a trained voice state recognition model to quantitatively evaluate the user's vocal tension; dynamically adjusts the equalizer and compressor parameters based on the evaluation results to achieve adaptive sound effect optimization of the human voice; synthesizes the optimized human voice and accompaniment and outputs it to the monitoring device, providing low-latency, highly immersive real-time feedback; and constructs personalized feature vectors by collecting user singing performance and historical tuning data. The deep Q network reinforcement learning mechanism is introduced to continuously fine-tune the voice state recognition model and sound effect strategy online to enhance the system's personalized tuning capabilities. This technical solution realizes a complete closed-loop process from audio acquisition, intelligent analysis to adaptive tuning feedback, integrating multiple advanced technologies such as multimodal feature extraction, semantic-assisted separation, attention mechanism, virtual sound field rendering and edge-end online learning, significantly improving the intelligent level of sound effect processing and user experience in karaoke scenarios.

[0058] The above is a detailed description of an embodiment of the present invention. However, the content described is only a preferred embodiment of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. An intelligent tuning method for a karaoke machine based on AI sound effect optimization, characterized in that: The following steps are involved: S1: Obtaining an original mixed audio signal sung by a user, and preprocessing the mixed audio signal to eliminate environmental noise and gain fluctuations; S2: Use a deep neural network model to perform multi-source audio separation on the pre-processed mixed audio signal to obtain multiple independent audio components including vocal tracks, drum tracks, melodic instrument tracks, and low-frequency tracks; S3: extracting time-frequency domain speech feature data from the vocal track, and quantitatively evaluating the tension level of the user's voice based on the trained voice state recognition model; S4: dynamically adjusting equalizer parameters and dynamic range compression parameters in the audio processing module according to the vocal tension evaluation result to achieve adaptive sound effect optimization of the human voice signal; S5: Resynthesizes the optimized vocal signal with the accompaniment track and outputs it to the monitoring device for real-time user feedback; S6: Based on the user's real-time singing performance and historical tuning data, the voice state recognition model and sound optimization strategy are fine-tuned online through a reinforcement learning mechanism to improve personalized tuning capabilities.

2. The intelligent tuning method for a karaoke machine based on AI sound effect optimization according to claim 1, characterized in that: The deep neural network model is a multi-source audio separation model built based on D3Net, which includes the following processing steps: Perform short-time Fourier transform on the preprocessed mixed audio signal to obtain time-frequency domain representation; Inputting the time-frequency domain representation into a deep network structure comprising a multi-scale convolutional module and a gated recurrent unit; The mask estimation mechanism is used to generate spectral masks corresponding to human voice, drum beats, melodic instruments and low-frequency components respectively; The original mixed spectrum is reconstructed based on the spectrum mask, and multiple independent audio tracks are output.

3. The intelligent tuning method for a karaoke machine based on AI sound effect optimization according to claim 1, characterized in that: The process of performing multi-source audio separation also includes the following enhancement steps: Introducing a multi-head self-attention mechanism in deep neural networks to model long-range dependencies between different audio sources; Dynamically allocate attention weights based on audio content to strengthen focus on the human voice frequency band; Use attention-weighted feature vectors to perform post-processing optimization on each audio track; The optimized vocal track is output for subsequent voice status identification and tuning parameter adjustment.

4. The intelligent tuning method for a karaoke machine based on AI sound effect optimization according to claim 1, characterized in that: The multi-source audio separation process further includes the following semantic assistance steps: During the training phase, each audio component is labeled with corresponding semantic label information, including vocal type (capella or harmony), percussion type (bass drum or snare drum), and melodic instrument type (guitar or piano). Build a multi-task learning framework with a semantic embedding layer to simultaneously predict the audio component categories during the separation process; In the inference stage, the target separation channel is dynamically selected according to the prediction results, and the energy output of non-target components is suppressed; The final output is a semantically interpretable separated audio track, which improves the system's adaptability to complex accompaniment scenarios.

5. The intelligent tuning method for a karaoke machine based on AI sound effect optimization according to claim 1, characterized in that: The process of extracting the speech features includes the following steps: Performing short-time Fourier transform and wavelet packet decomposition on the vocal track to obtain linear spectrum and nonlinear time-frequency features respectively; Extract Mel-frequency cepstral coefficients, fundamental frequency trajectory and high-frequency energy ratio as basic physiological acoustic features; Inputting multiple types of features into a fusion network, wherein the fusion network includes parallel convolution branches and an attention gating mechanism; The vocal tension score is output based on the fusion features, and the score result is used to drive the subsequent adaptive tuning module.

6. The intelligent tuning method for a karaoke machine based on AI sound effect optimization according to claim 1, characterized in that: The voice state recognition model adopts the following enhancement strategy during the training process: Construct a dataset containing various voice states; Using transfer learning methods, we conduct cross-group generalization training on singing samples of professional singers, ordinary users, and users of different genders; A weighted cross-entropy loss function is introduced in the model output layer to enhance the recognition sensitivity of nervousness and broken voice categories; The trained model is used to classify and score real-time human voices, and a continuous tension value is output for tuning control.

7. The intelligent tuning method for a karaoke machine based on AI sound effect optimization according to claim 1, characterized in that: The quantitative evaluation of the user's vocal tension also includes the following dynamic analysis steps: Real-time detection of the target pitch curve of the current singing segment and comparison with the user's historical singing ability; The singing difficulty coefficient is calculated based on the deviation rate between the target pitch and the actual singing pitch; Combining the current pitch position with the vocal fold tension model to predict potential phonation burden; The singing difficulty coefficient and vocal burden are input into the timing neural network, the vocal tension assessment weight is dynamically adjusted, and personalized assessment results are output.

8. The intelligent tuning method for a karaoke machine based on AI sound effect optimization according to claim 1, characterized in that: The adaptive sound optimization of the human voice signal is implemented, specifically including: A nonlinear mapping function is established between vocal tension and audio parameters, and the tension score is converted into an equalizer band gain adjustment value and a compressor threshold offset. When it detects that the user is in a state of high tension, it automatically reduces the high-frequency gain and moderately improves the clarity of the mid-frequency band to alleviate the harshness; Dynamically adjust the attack and recovery time of the dynamic range compressor to enhance the control of transient energy in the high-pitched range; The adjusted parameters are loaded into the digital signal processing module in real time to complete the adaptive sound optimization of the vocal signal, improving auditory comfort and singing stability.

9. The intelligent tuning method for a karaoke machine based on AI sound effect optimization according to claim 1, characterized in that: The optimized vocal signal is resynthesized with the accompaniment track and outputted to a monitoring device for real-time user feedback, specifically including: Time-align the optimized vocal signal with the accompaniment track to compensate for system delays introduced by audio separation and processing; Apply virtual sound field rendering algorithm to perform spatial processing on the synthesized audio signal to simulate the auditory perception effect in KTV box or stage environment; Dynamically adjust the reverberation feedback intensity and stereo image distribution based on the type of monitoring device the user is wearing, whether it is headphones or speakers; The processed synthesized audio is output to the monitoring device in real time to provide a low-latency, highly immersive singing feedback experience.

10. The intelligent tuning method for a karaoke machine based on AI sound effect optimization according to claim 1, characterized in that: The online fine-tuning of the voice state recognition model and the sound effect optimization strategy through the reinforcement learning mechanism specifically includes: Real-time collection of user pitch deviation, rhythm stability, and vocal state change information, combined with historical tuning parameters to build a personalized user behavior feature vector; Inputting the feature vector into a reinforcement learning controller based on a deep Q network to generate an optimal tuning action strategy for the current singing state; Dynamically update the policy network parameters of the reinforcement learning model based on the user's subjective feedback on the tuning results or the reward signal provided by the system scoring mechanism; Based on the updated strategy, lightweight online fine-tuning of the multi-source audio separation model and sound effect optimization module is performed to achieve continuous personalized evolution of the tuning system.

Images