Intelligent lyric voice recognition method and device, equipment and medium

Abstract

The invention discloses an intelligent lyric voice recognition method, device and equipment and a medium, and the method comprises the steps: receiving an original audio signal, and carrying out the preprocessing of the original audio signal; inputting the preprocessed audio into an audio source separation model based on deep learning, generating a time-frequency mask for separation, and separating a target human voice spectrum and a background accompaniment spectrum by using the mask; processing the separated target human voice track by adopting a detection module based on a sequence memory network, and combining continuous effective human voice frames to obtain effective human voice fragments when the target human voice track is detected and judged to be effective human voice frames; extracting audio features from the effective human voice segments to obtain a feature sequence, and inputting the feature sequence into an encoder-decoder architecture recognition model for lyric text recognition; and generating time alignment information for the recognized lyric text, and outputting a lyric recognition result with a timestamp. The invention aims to solve the problem that the recognition rate is reduced due to overlapping of human voice and accompaniment in the mixed audio.

Description

Technical Field

[0001] This application belongs to the field of speech recognition and audio processing, and in particular relates to an intelligent lyrics voice recognition method, device, equipment and medium. Background Technology

[0002] Current AI-based lyrics recognition technology relies primarily on end-to-end speech recognition models. Through audio preprocessing, acoustic feature extraction, and speech recognition decoding modules, it directly processes mixed audio including accompaniment, achieving an end-to-end mapping from audio signals to lyrics text. However, this technology suffers from several critical drawbacks that severely limit recognition effectiveness and application scenarios: the accompaniment and vocals highly overlap in the frequency domain, creating strong background noise that significantly reduces recognition accuracy; general models are not optimized for the unique acoustic characteristics of singing voices, such as pitch variations and vocal sustain, resulting in insufficient model adaptability; the output is merely the raw text, completely lacking precise timestamp localization and cross-language adaptive performance; and the system has weak fault tolerance, with the error rate spikeing significantly when audio quality is poor or when faced with complex accompaniment. These issues collectively make it difficult for existing technologies to meet the recognition needs of practical applications. Summary of the Invention

[0003] This application proposes an intelligent lyrics voice recognition method, device, equipment, and medium to solve the problem of decreased recognition rate caused by the overlap of human voice and accompaniment in mixed audio, as well as the technical bottleneck of existing technologies that cannot provide accurate timestamps and cross-language adaptability.

[0004] This invention provides an intelligent lyrics voice recognition method, comprising:

[0005] Receive the original audio signal and preprocess the original audio signal, including sampling rate unification, channel conversion and amplitude normalization;

[0006] The preprocessed audio input is used to generate a time-frequency mask for separation based on a deep learning audio source separation model. The mask is then used to separate the target human voice spectrum from the background accompaniment spectrum.

[0007] The separated target human voice track is processed by a detection module based on a sequence memory network. When the detection determines that it is a valid human voice frame, consecutive valid human voice frames are merged to obtain a valid human voice segment.

[0008] Audio features are extracted from the effective human voice segments to obtain a feature sequence, and the feature sequence is input into the encoder-decoder architecture recognition model for lyrics text recognition.

[0009] Generate time alignment information for the identified lyrics text and output the lyrics recognition results with timestamps.

[0010] According to the intelligent lyrics vocal recognition method provided by the present invention, the amplitude normalization specifically includes:

[0011] x norm =(x original -μ) / σ

[0012] Where, x norm Let x represent the original audio signal, μ and σ be the mean and standard deviation of the signal, respectively. norm This is the normalized signal.

[0013] According to the present invention, an intelligent lyrics voice recognition method is provided, wherein the deep learning-based audio source separation model includes:

[0014] The audio source separation model adopts an encoder-decoder architecture, wherein the encoder is composed of a multi-layer convolutional neural network and is used to extract high-level time-frequency features of the mixed audio.

[0015] The decoder is used to upsample and reconstruct the time-frequency features to generate a soft mask, which is used to separate the target vocal spectrum and the background accompaniment spectrum from the mixed audio spectrum.

[0016] According to the intelligent lyrics voice recognition method provided by the present invention, the detection module based on the sequence memory network is a feedforward sequence memory network, and its detection process is expressed as follows:

[0017]

[0018] p t =σ(W·h t +b)

[0019] Where, x t Given the audio features at time t, h t It is the hidden state of the network at time t, FSMN Layer p represents a network layer containing memory modules. t Let σ be the probability that the current frame predicted by the model belongs to a valid human voice, σ be the sigmoid activation function, and W and b be learnable parameters. t If the threshold is exceeded, the frame is determined to be a valid human voice frame.

[0020] According to the intelligent lyrics voice recognition method provided by the present invention, the audio feature extraction specifically includes:

[0021] The effective human voice segments are pre-emphasized, framed, and windowed.

[0022] Perform a short-time Fourier transform on each frame of the signal and calculate the power spectrum;

[0023] The power spectrum is passed through a triangular filter bank of Mel-scale, and the output energy of each filter is calculated.

[0024] The natural logarithm of the output energy of each filter is taken to obtain the feature vector, and all feature vectors are arranged in chronological order to form the feature sequence.

[0025] According to the present invention, an intelligent lyrics voice recognition method is provided, which uses an encoder-decoder architecture recognition model to recognize lyrics text, specifically including:

[0026] The encoder-decoder architecture recognition model includes an attention-based neural network architecture;

[0027] The encoder performs contextual modeling on the feature sequence through multiple self-attention layers and a feedforward neural network to generate a context-related representation that includes temporal correlations;

[0028] The decoder employs an autoregressive approach, using a cross-attention layer to associate the context-related representation with the generated lyric tag sequence, predicting the current lyric tag digit by digit. The model optimizes parameters by minimizing the prediction loss.

[0029] According to the intelligent lyrics voice recognition method provided by the present invention, the recognized lyrics text generation time alignment information specifically includes:

[0030] A time-series classification mechanism is used to generate timestamps for the lyrics text, and the output includes the lyrics text and its corresponding start and end times in the original audio.

[0031] The present invention also provides an intelligent lyrics voice recognition device, the device comprising:

[0032] The receiving and processing module is used to receive the original audio signal and preprocess the original audio signal, including sampling rate unification, channel conversion and amplitude normalization.

[0033] The human voice separation module is used to take the preprocessed audio input based on a deep learning audio source separation model and generate a time-frequency mask for separation, and use the mask to separate the target human voice spectrum from the background accompaniment spectrum;

[0034] The speech detection module is used to process the separated target human voice track using a detection module based on a sequence memory network. When the detection determines that it is a valid human voice frame, the consecutive valid human voice frames are merged to obtain a valid human voice segment.

[0035] The extraction and recognition module is used to extract audio features from the effective human voice segments to obtain a feature sequence, and input the feature sequence into the encoder-decoder architecture recognition model for lyrics text recognition;

[0036] The alignment output module is used to generate time alignment information for the recognized lyrics text and output the lyrics recognition results with timestamps.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the intelligent lyrics voice recognition method as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent lyrics voice recognition method as described above.

[0039] This invention provides an intelligent lyrics vocal recognition method, apparatus, device, and medium. The method involves receiving raw audio signals and preprocessing them; inputting the preprocessed audio into a deep learning-based audio source separation model to generate a time-frequency mask for separation; using the mask to separate the target vocal spectrum from the background accompaniment spectrum; processing the separated target vocal track using a detection module based on a sequence memory network; merging consecutive valid vocal frames to obtain valid vocal segments when a valid vocal frame is detected; extracting audio features from the valid vocal segments to obtain a feature sequence; inputting the feature sequence into an encoder-decoder architecture recognition model for lyrics text recognition; generating time alignment information for the recognized lyrics text; and outputting a timestamped lyrics recognition result. This invention achieves accurate separation of target vocals and background accompaniment by utilizing a deep learning-based audio source separation model and soft masking technology. This addresses the issue of decreased recognition rate caused by overlap between vocals and accompaniment in mixed audio, thereby reducing noise interference from frequency domain overlap and significantly improving lyric recognition accuracy. Through voice activity detection using a feedforward sequence memory network, effective vocal segments are precisely selected, reducing interference from invalid information and further optimizing recognition stability. Employing an audio feature extraction method adapted to the characteristics of singing voices and an encoder-decoder architecture recognition model effectively overcomes the bottleneck of insufficient cross-language adaptability in existing technologies, enabling efficient handling of multilingual and mixed-language lyric recognition scenarios. Generating precise timestamps through a temporal classification mechanism compensates for the technical shortcomings of existing technologies in providing precise time alignment information, achieving accurate correspondence between lyric text and original audio, and meeting the needs of various applications requiring temporal correlation. The system maintains stable recognition performance even in environments with poor audio quality or complex accompaniment, significantly enhancing fault tolerance. Furthermore, the modular design allows for independent optimization and upgrades of each functional module, improving the scalability and maintenance convenience of the technology and comprehensively expanding the application scenarios and practical value of lyric recognition technology. Attached Figure Description

[0040] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating an intelligent lyrics voice recognition method provided in an embodiment of this application;

[0042] Figure 2 This is a schematic diagram of the structure of an intelligent lyrics voice recognition device provided in an embodiment of this application;

[0043] Figure 3 This is a schematic diagram of the electronic device structure provided in the embodiments of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0045] To address the problems in existing technologies, this invention proposes an intelligent lyrics vocal recognition method to solve the problem of decreased recognition rate caused by overlap between vocals and accompaniment in mixed audio, as well as the technical bottlenecks of existing technologies that cannot provide accurate timestamps and cross-language adaptability. Figure 1 As shown, including but not limited to the following steps:

[0046] Step 110: Receive the original audio signal and preprocess the original audio signal, including sampling rate unification, channel conversion and amplitude normalization.

[0047] In step 110, raw music audio signals in formats such as MP3 and WAV are first received to ensure compatibility with mainstream audio file types. Next, preprocessing is performed: the sampling rate is standardized to 16kHz for audio from different sources to eliminate subsequent processing deviations caused by sampling rate differences; channel conversion converts any stereo audio to mono, simplifying the audio data structure and reducing the computational complexity of subsequent modules.

[0048] As a further optional embodiment, the amplitude normalization specifically includes:

[0049] x norm =(x original -μ) / σ

[0050] Where, x norm Let x represent the original audio signal, μ and σ be the mean and standard deviation of the signal, respectively. norm This is the normalized signal.

[0051] Furthermore, through this normalization operation, the amplitude range of the original audio signal can be standardized to a unified range, effectively suppressing amplitude fluctuations caused by differences in recording devices and volume settings of different audio signals. This further improves the consistency of input data for subsequent modules such as voice separation and feature extraction, helping the model to learn audio feature patterns more stably, indirectly optimizing the overall accuracy of lyrics recognition, and contributing to improving the numerical stability and convergence speed of subsequent deep learning models.

[0052] Step 120: Input the preprocessed audio into a deep learning-based audio source separation model and generate a time-frequency mask for separation. Use the mask to separate the target vocal spectrum from the background accompaniment spectrum.

[0053] As a further optional embodiment, the deep learning-based audio source separation model includes:

[0054] The audio source separation model adopts an encoder-decoder architecture, wherein the encoder is composed of a multi-layer convolutional neural network and is used to extract high-level time-frequency features of the mixed audio.

[0055] The decoder is used to upsample and reconstruct the time-frequency features to generate a soft mask, which is used to separate the target vocal spectrum and the background accompaniment spectrum from the mixed audio spectrum.

[0056] In step 120, the preprocessed mixed audio signal is input into a deep learning-based audio source separation model. This model typically employs an encoder-decoder architecture. The encoder extracts deep time-frequency features of the audio spectrum through multi-layer convolutional or recurrent neural networks, capturing the differences in frequency distribution and temporal dynamics between vocals and background sounds. The decoder then uses these features to generate a soft mask through deconvolution or upsampling operations. This mask amplifies the time-frequency units belonging to vocals while suppressing the time-frequency units belonging to accompaniment.

[0057] Furthermore, the core objective of the model is to extract the spectral features Y of the mixed audio. mix Precisely disassembled into an independent human voice spectrum Y vocal Accompaniment spectrum Y accompaniment Because audio signals exhibit linear superposition characteristics in the time-frequency domain, the three satisfy an approximate relationship Y. mix ≈Y vocal +Y accompaniment This provides a theoretical basis for the separation task.

[0058] To achieve this decomposition, the separation model learns to generate a mask M that highly matches the characteristics of human voices through training on a large amount of labeled data. vocal The mask is applied to the mixed spectrum Y through an element-wise multiplication (⊙) operation. mix , i.e. Y vocal =M vocal ⊙Y mix This allows for the selection and retention of time-frequency units dominated by vocals, while suppressing accompaniment components. The accompaniment spectrum can then be determined via Y... accompaniment =Y mix -Y vocal It was derived indirectly.

[0059] This embodiment particularly favors a dual-track output mode, simultaneously generating two independent audio tracks: vocal and accompaniment. This design minimizes signal loss during the separation process, ensures higher purity of the extracted vocal spectrum, and effectively avoids interference from background noise and instrument sounds on subsequent lyric recognition, feature analysis, and other stages, providing crucial support for the accuracy of the entire audio processing workflow.

[0060] Step 130: The separated target human voice track is processed by a detection module based on a sequence memory network. When the detection determines that it is a valid human voice frame, the consecutive valid human voice frames are merged to obtain a valid human voice segment.

[0061] In step 130, after acquiring the separated vocal tracks, valid singing segments need to be located using Voice Activity Detection (VAD) to filter out irrelevant segments such as silence and pure accompaniment. This invention employs a VAD module based on a feedforward sequential memory network (FSMN), whose core advantage lies in its ability to efficiently capture long-range temporal dependencies in audio signals, making it particularly suitable for processing continuous acoustic features in singing, such as sustained notes and transitions.

[0062] As a further optional embodiment, the detection module based on the sequence memory network is a feedforward sequence memory network, and its detection process is expressed as follows:

[0063]

[0064] p t =σ(W·h t +b)

[0065] Where, x t Given the audio features at time t, h t It is the hidden state of the network at time t, FSMN Layer p represents a network layer containing memory modules. tLet σ be the probability that the current frame predicted by the model belongs to a valid human voice, σ be the sigmoid activation function, and W and b be learnable parameters. t If the threshold is exceeded, the frame is determined to be a valid human voice frame.

[0066] Furthermore, the audio features x at time t t Enter FSMN Layer This network layer combines historical hidden states h t-1 (Storing feature memories from past frames) is used to calculate and generate the hidden state h at the current moment. t This enables modeling of the correlation between consecutive frames; subsequently, a linear transformation (W·h) is performed using the Sigmoid activation function σ. t The result of +b) is processed to obtain the probability p that the current frame belongs to a valid human voice. t (Value range: 0-1).

[0067] Step 140: Extract audio features from the effective human voice segment to obtain a feature sequence, and input the feature sequence into the encoder-decoder architecture recognition model for lyrics text recognition.

[0068] In step 140, audio features are first extracted from the separated and filtered valid vocal segments. Combining the prosodic characteristics of the singing speech with the requirements for lyric recognition, Mel frequency cepstral coefficients (MFCC), Mel spectrogram features, or logarithmic Mel spectrogram features are preferentially extracted. These features effectively preserve key information such as pitch and rhythm of the vocals while reducing irrelevant noise interference. By dividing the valid vocal segments into frames with fixed frame lengths and frame shifts, feature calculations are performed on each frame, ultimately forming a feature sequence with uniform dimensions and temporal coherence.

[0069] As a further optional embodiment, the audio feature extraction specifically includes:

[0070] The effective human voice segments are pre-emphasized, framed, and windowed.

[0071] Perform a short-time Fourier transform on each frame of the signal and calculate the power spectrum;

[0072] The power spectrum is passed through a triangular filter bank of Mel-scale, and the output energy of each filter is calculated.

[0073] The natural logarithm of the output energy of each filter is taken to obtain the feature vector, and all feature vectors are arranged in chronological order to form the feature sequence.

[0074] Furthermore, the effective human voice segments are first pre-emphasized, and high-frequency components are boosted by a first-order high-pass filter to compensate for the high-frequency attenuation of the vocal system. Then, the segments are divided into frames with a fixed frame length and frame shift, and a Hamming window or Hanning window is applied to each frame to reduce spectral leakage. After that, a short-time Fourier transform is performed on each frame to convert the time-domain signal into a frequency-domain signal and calculate the power spectrum. The power spectrum is then input into a Mel-scale triangular filter bank. The frequency response of this filter bank, which conforms to the characteristics of human hearing perception, is used to filter key frequency band information. The output energy of each filter is calculated, and finally, the natural logarithm of the output energy of each filter is taken to obtain the single-frame feature vector. All feature vectors are arranged in the original time order to form a temporal feature sequence that meets the subsequent recognition requirements.

[0075] As a further optional embodiment, the encoder-decoder architecture recognition model performs lyrics text recognition, specifically including:

[0076] The encoder-decoder architecture recognition model includes an attention-based neural network architecture;

[0077] The encoder performs contextual modeling on the feature sequence through multiple self-attention layers and a feedforward neural network to generate a context-related representation that includes temporal correlations;

[0078] The decoder employs an autoregressive approach, using a cross-attention layer to associate the context-related representation with the generated lyric tag sequence, predicting the current lyric tag digit by digit. The model optimizes parameters by minimizing the prediction loss.

[0079] Furthermore, the recognition model employs a neural network architecture that integrates attention mechanisms. The encoder consists of multiple layers of self-attention layers and feedforward neural networks stacked alternately. The self-attention layers capture the long-term and short-term temporal dependencies within the feature sequence, while the feedforward neural network performs nonlinear transformations and feature enhancements to complete contextual modeling of the feature sequence and generate a context-related representation containing global temporal correlation information. The decoder adopts an autoregressive generation mode. When predicting the lyrics tag at the current moment, it dynamically associates the context-related representation output by the encoder with the previously generated lyrics tag sequence through a cross-attention layer, accurately focusing on key acoustic features. Then, the Softmax layer calculates the probability distribution of candidate lyrics tags and selects the tag with the highest probability as the current output. During training, the model aims to minimize the prediction loss and iteratively updates all learnable parameters through the backpropagation algorithm to continuously optimize the mapping accuracy and achieve efficient and accurate recognition of lyrics text.

[0080] Optionally, for each valid human voice segment detected by VAD, its filter bank features are extracted. FBank features simulate the frequency perception characteristics of the human cochlea and often exhibit better robustness in noisy environments compared to features such as MFCC. The extraction process includes:

[0081] (1) Pre-emphasis: A first-order high-pass filter is used to boost the high-frequency components. The main purpose is to boost the high-frequency part through the high-pass filter to compensate for the suppression of high-frequency components by the sound system.

[0082] (2) Framing and windowing: The audio signal is divided into short frames (e.g., frame length 25ms, frame shift 10ms), and a Hamming window is applied to each frame to reduce spectral leakage.

[0083] (3) Short-time Fourier transform (STFT) and power spectrum calculation: Perform STFT on each frame of signal and calculate the power spectrum P(f).

[0084] (4) Mel filter bank processing: The power spectrum is passed through a set of Mel-scaled triangular filters. The conversion relationship between Mel frequency Mel(f) and linear frequency f(Hz) is as follows:

[0085] Mel(f) = 2595 * log10(1 + f / 700)

[0086] Suppose there are M filters in total, and the output energy of each filter is E. m , where m is the filter index.

[0087] (5) Calculate the logarithmic energy by taking the natural logarithm of the output energy of each filter to obtain the FBank eigenvector:

[0088] FBank m =ln(E m )

[0089] Ultimately, each audio frame is converted into an M-dimensional FBank feature vector. The feature vectors of all frames are arranged in chronological order, forming the feature sequence X = {x1, x2, ..., x...} for the human voice segment. T}, where T is the total number of frames, x T Let be the M-dimensional feature vector of the T-th frame.

[0090] Optionally, the FBank feature sequence X is input into an end-to-end automatic speech recognition model for lyrics transcription. This ASR model employs a Transformer-based encoder-decoder architecture. The encoder, composed of multiple layers of self-attention layers and a feedforward neural network, is responsible for encoding the input acoustic feature sequence and learning its context-dependent representation H. enc :

[0091] H enc =Encoder(X)

[0092] The decoder is based on the encoder output H enc and the lyric marker sequence y generated at the previous time stept<x Autoregressively predict the most likely lyric token y at the current moment t :

[0093] P(y t |y t<x},X)=Decoder(y t<x H enc )

[0094] During training, the model optimizes its parameters by minimizing the CTC loss, thereby accurately mapping the singing signal into a text sequence.

[0095] Step 150: Generate time alignment information for the identified lyrics text and output the lyrics recognition result with timestamp.

[0096] In step 150, after the lyrics text is transcribed by the recognition model, precise time alignment information needs to be generated for each lyric, word, or syllable, ultimately outputting a timestamped lyrics recognition result. This process relies on the mapping relationship between the connectionist temporal classification (CTC) mechanism or attention mechanism and the audio frame temporal sequence: First, by combining the frame-level correspondence between the lyrics marker sequence output by the ASR model and the input feature sequence, the temporal position of each lyrics marker in the original valid vocal segment is traced; second, by using the timestamp mapping rules between the feature sequence and the original audio, the start and end times of each lyrics marker are calculated, clarifying its specific time interval in the original music audio; finally, the lyrics text content and the corresponding timestamp information are structurally associated to form a standardized output format of [(word 1, start time 1, end time 1), (word 2, start time 2, end time 2),...]. This timestamped recognition result not only achieves precise synchronization between the lyrics text and the audio, but also meets diverse application needs such as automatic lyrics subtitle matching and music content segment retrieval, improving the practicality and adaptability of the lyrics recognition result.

[0097] As a further optional embodiment, the identified lyrics text generation time alignment information specifically includes:

[0098] A time-series classification mechanism is used to generate timestamps for the lyrics text, and the output includes the lyrics text and its corresponding start and end times in the original audio.

[0099] Furthermore, in the lyrics text and timestamp output stage, the ASR model, while transcribing the lyrics text, simultaneously generates precise timestamp information for each identified word or syllable, accurately marking its start and end times in the original audio. This precise temporal alignment function is typically achieved through core mechanisms such as Connectionist Temporal Classification (CTC), effectively solving the length mismatch problem between the acoustic feature sequence and the lyrics text sequence, ensuring the accuracy of temporal association. The final output is a structured lyrics text with timestamps, which can be represented as a series of tuples [(word_1,start_time_1,end_time_1),(word_2,start_time_2,end_time_2),...]. This standardized output format is highly adaptable, facilitating diverse applications, such as automatic synchronized display of lyrics subtitles, precise segment retrieval of music content, or providing basic data support for more in-depth music content analysis such as music sentiment analysis and lyric prosody research.

[0100] This invention provides an intelligent lyrics vocal recognition method, which involves receiving raw audio signals and preprocessing them; inputting the preprocessed audio into a deep learning-based audio source separation model to generate a time-frequency mask for separation, using the mask to separate the target vocal spectrum from the background accompaniment spectrum; processing the separated target vocal track using a detection module based on a sequence memory network, and merging consecutive valid vocal frames to obtain valid vocal segments when the detection determines they are valid vocal frames; extracting audio features from the valid vocal segments to obtain feature sequences, and inputting the feature sequences into an encoder-decoder architecture recognition model for lyrics text recognition; generating time alignment information for the recognized lyrics text, and outputting the lyrics recognition result with a timestamp. This invention achieves accurate separation of target vocals and background accompaniment by utilizing a deep learning-based audio source separation model and soft masking technology. This addresses the issue of decreased recognition rate caused by overlap between vocals and accompaniment in mixed audio, thereby reducing noise interference from frequency domain overlap and significantly improving lyric recognition accuracy. Through voice activity detection using a feedforward sequence memory network, effective vocal segments are precisely selected, reducing interference from invalid information and further optimizing recognition stability. Employing an audio feature extraction method adapted to the characteristics of singing voices and an encoder-decoder architecture recognition model effectively overcomes the bottleneck of insufficient cross-language adaptability in existing technologies, enabling efficient handling of multilingual and mixed-language lyric recognition scenarios. Generating precise timestamps through a temporal classification mechanism compensates for the technical shortcomings of existing technologies in providing precise time alignment information, achieving accurate correspondence between lyric text and original audio, and meeting the needs of various applications requiring temporal correlation. The system maintains stable recognition performance even in environments with poor audio quality or complex accompaniment, significantly enhancing fault tolerance. Furthermore, the modular design allows for independent optimization and upgrades of each functional module, improving the scalability and maintenance convenience of the technology and comprehensively expanding the application scenarios and practical value of lyric recognition technology.

[0101] Example 2

[0102] The intelligent lyrics voice recognition method and apparatus provided by the present invention are described below, such as... Figure 2 As shown, the intelligent lyrics voice recognition method device described below can be referred to in correspondence with the intelligent lyrics voice recognition method described above.

[0103] A smart lyrics voice recognition device, the device comprising:

[0104] The receiving and processing module is used to receive the original audio signal and preprocess the original audio signal, including sampling rate unification, channel conversion and amplitude normalization.

[0105] The human voice separation module is used to take the preprocessed audio input based on a deep learning audio source separation model and generate a time-frequency mask for separation, and use the mask to separate the target human voice spectrum from the background accompaniment spectrum;

[0106] The speech detection module is used to process the separated target human voice track using a detection module based on a sequence memory network. When the detection determines that it is a valid human voice frame, the consecutive valid human voice frames are merged to obtain a valid human voice segment.

[0107] The extraction and recognition module is used to extract audio features from the effective human voice segments to obtain a feature sequence, and input the feature sequence into the encoder-decoder architecture recognition model for lyrics text recognition;

[0108] The alignment output module is used to generate time alignment information for the recognized lyrics text and output the lyrics recognition results with timestamps.

[0109] Example 3

[0110] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute an intelligent lyrics voice recognition method, which includes:

[0111] Receive the original audio signal and preprocess the original audio signal, including sampling rate unification, channel conversion and amplitude normalization;

[0112] The preprocessed audio input is used to generate a time-frequency mask for separation based on a deep learning audio source separation model. The mask is then used to separate the target human voice spectrum from the background accompaniment spectrum.

[0113] The separated target human voice track is processed by a detection module based on a sequence memory network. When the detection determines that it is a valid human voice frame, consecutive valid human voice frames are merged to obtain a valid human voice segment.

[0114] Audio features are extracted from the effective human voice segments to obtain a feature sequence, and the feature sequence is input into the encoder-decoder architecture recognition model for lyrics text recognition.

[0115] Generate time alignment information for the identified lyrics text and output the lyrics recognition results with timestamps.

[0116] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent lyrics voice recognition method provided by the methods described above, the method comprising:

[0118] Receive the original audio signal and preprocess the original audio signal, including sampling rate unification, channel conversion and amplitude normalization;

[0119] The preprocessed audio input is used to generate a time-frequency mask for separation based on a deep learning audio source separation model. The mask is then used to separate the target human voice spectrum from the background accompaniment spectrum.

[0120] The separated target human voice track is processed by a detection module based on a sequence memory network. When the detection determines that it is a valid human voice frame, consecutive valid human voice frames are merged to obtain a valid human voice segment.

[0121] Audio features are extracted from the effective human voice segments to obtain a feature sequence, and the feature sequence is input into the encoder-decoder architecture recognition model for lyrics text recognition.

[0122] Generate time alignment information for the identified lyrics text and output the lyrics recognition results with timestamps.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent lyrics voice recognition, characterized in that, include: Receive the original audio signal and preprocess the original audio signal, including sampling rate unification, channel conversion and amplitude normalization; The preprocessed audio input is used to generate a time-frequency mask for separation based on a deep learning audio source separation model. The mask is then used to separate the target human voice spectrum from the background accompaniment spectrum. The separated target human voice track is processed by a detection module based on a sequence memory network. When the detection determines that it is a valid human voice frame, consecutive valid human voice frames are merged to obtain a valid human voice segment. Audio features are extracted from the effective human voice segments to obtain a feature sequence, and the feature sequence is input into the encoder-decoder architecture recognition model for lyrics text recognition. Generate time alignment information for the identified lyrics text and output the lyrics recognition results with timestamps.

2. The intelligent lyrics voice recognition method according to claim 1, characterized in that, The amplitude normalization specifically includes: x norm =(x original -m) / s Where, x norm Let x represent the original audio signal, μ and σ be the mean and standard deviation of the signal, respectively. norm This is the normalized signal.

3. The intelligent lyrics voice recognition method according to claim 1, characterized in that, The deep learning-based audio source separation model includes: The audio source separation model adopts an encoder-decoder architecture, wherein the encoder is composed of a multi-layer convolutional neural network and is used to extract high-level time-frequency features of the mixed audio. The decoder is used to upsample and reconstruct the time-frequency features to generate a soft mask, which is used to separate the target vocal spectrum and the background accompaniment spectrum from the mixed audio spectrum.

4. The intelligent lyrics voice recognition method according to claim 1, characterized in that, The detection module based on the sequence memory network is a feedforward sequence memory network, and its detection process is as follows: p t =σ(W·h t +b) Where, x t Given the audio features at time t, h t It is the hidden state of the network at time t, FSMN Layer p represents a network layer containing memory modules. t Let σ be the probability that the current frame predicted by the model belongs to a valid human voice, σ be the sigmoid activation function, and W and b be learnable parameters. t If the threshold is exceeded, the frame is determined to be a valid human voice frame.

5. The intelligent lyrics voice recognition method according to claim 1, characterized in that, The audio feature extraction specifically includes: The effective human voice segments are pre-emphasized, framed, and windowed. Perform a short-time Fourier transform on each frame of the signal and calculate the power spectrum; The power spectrum is passed through a triangular filter bank of Mel-scale, and the output energy of each filter is calculated. The natural logarithm of the output energy of each filter is taken to obtain the feature vector, and all feature vectors are arranged in chronological order to form the feature sequence.

6. The intelligent lyrics voice recognition method according to claim 1, characterized in that, The encoder-decoder architecture recognition model is used for lyrics text recognition, specifically including: The encoder-decoder architecture recognition model includes an attention-based neural network architecture; The encoder performs contextual modeling on the feature sequence through multiple self-attention layers and a feedforward neural network to generate a context-related representation that includes temporal correlations; The decoder employs an autoregressive approach, using a cross-attention layer to associate the context-related representation with the generated lyric tag sequence, predicting the current lyric tag digit by digit. The model optimizes parameters by minimizing the prediction loss.

7. The intelligent lyrics voice recognition method according to claim 1, characterized in that, The identified lyrics text generation time alignment information specifically includes: A time-series classification mechanism is used to generate timestamps for the lyrics text, and the output includes the lyrics text and its corresponding start and end times in the original audio.

8. An intelligent lyrics and voice recognition device, characterized in that, include: The receiving and processing module is used to receive the original audio signal and preprocess the original audio signal, including sampling rate unification, channel conversion and amplitude normalization. The human voice separation module is used to take the preprocessed audio input based on a deep learning audio source separation model and generate a time-frequency mask for separation, and use the mask to separate the target human voice spectrum from the background accompaniment spectrum; The speech detection module is used to process the separated target human voice track using a detection module based on a sequence memory network. When the detection determines that it is a valid human voice frame, the consecutive valid human voice frames are merged to obtain a valid human voice segment. The extraction and recognition module is used to extract audio features from the effective human voice segments to obtain a feature sequence, and input the feature sequence into the encoder-decoder architecture recognition model for lyrics text recognition; The alignment output module is used to generate time alignment information for the recognized lyrics text and output the lyrics recognition results with timestamps.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the intelligent lyrics voice recognition method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the intelligent lyrics voice recognition method as described in any one of claims 1 to 7.

Images