A pitch location recognition method based on human voice separation

Abstract

The application discloses a pitch positioning and recognizing method based on human voice separation, which comprises the following steps: firstly, selecting a corresponding real contrast file according to a to-be-recognized singing video and preprocessing; then, adopting an end-to-end human voice separation technology to separate human voice from the processed data to obtain a human voice file; constructing a pitch positioning and recognizing network, which comprises an input layer, a relationship layer and an output layer; finally, inputting the human voice file obtained by the human voice separation into the constructed pitch positioning and recognizing network to realize pitch positioning and recognizing. In the method, the pitch positioning and recognizing network is constructed, the pitch and a standard line are visualized, the curve comparison between the human voice and the sound synthesized after pitch correction can be directly observed, the rights and interests of audience fans are maintained, and the universality and accuracy are improved compared with the prior art.

Description

Technical Field

[0001] This invention relates to the field of multimedia processing, and specifically to a pitch localization and recognition method based on human voice separation. Background Technology

[0002] Lip-syncing is a complex and widely debated topic in music performances. Lip-syncing, also known as matching vocals to a pre-recorded song, refers to a singer performing a song live while simultaneously mimicking the sounds of their lips without actually singing. Legally, lip-syncing is considered deceiving the audience. However, with technological advancements, the use of pre-recorded backing tracks or real-time AI-assisted audio editing in live performances has become commonplace. While these practices aim to enhance performance quality, they also blur the lines of what constitutes lip-syncing.

[0003] Current technology often involves extracting the first audio feature from the audio recording of the singing segment, then extracting the second audio feature from the original recording of the song that also belongs to the singing segment, and calculating the similarity between the first and second audio features. This technique tends to assume that perfect pitch is impossible in a live performance, and that any overly perfect pitch could be considered lip-syncing. However, highly skilled singers can achieve near-perfect pitch without pre-recording, potentially leading to misjudgments. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a pitch localization and recognition method based on voice separation. This method utilizes existing voice separation technology to accurately locate the pitch of the voice by separating the voice from the accompaniment. It uses end-to-end voice separation technology to separate the voice, instruments, and background sounds in the audio file, and then uses the Parselmouth library for pitch analysis.

[0005] A pitch localization and recognition method based on human voice separation includes the following steps:

[0006] Step 1: Prepare data and perform data preprocessing. The specific steps are as follows:

[0007] The corresponding real comparison file needs to be selected based on the singing video to be identified; the singing video to be identified and the corresponding real comparison file need to be preprocessed, mainly focusing on the audio, including frame segmentation, windowing, endpoint detection and noise reduction.

[0008] First, endpoint detection: determine the start and end points of the audio signal.

[0009] Then, a frame segmentation operation is performed to divide the audio signal into short frames, with the interval between each frame segmentation being 10-30 milliseconds.

[0010] Next, windowing is applied: a window function is applied to each frame to reduce discontinuities at frame edges.

[0011] Finally, noise reduction: Use existing noise reduction algorithms to reduce background noise and improve speech clarity.

[0012] Step 2: Use end-to-end voice separation technology to separate the voices from the processed data to obtain the voice file;

[0013] Step 3: Construct a pitch localization and recognition network. The specific steps are as follows:

[0014] The pitch localization and recognition network consists of an input layer, a relational layer, and an output layer.

[0015] The input layer is used to read human voice files, employing traditional audio feature extraction methods, such as Mel-frequency cepstral coefficients, to obtain audio features. These features capture key information in the audio signal, aiding in pitch recognition. Audio feature transformation methods are then used to process the obtained audio features, such as converting them into Mel spectrograms, which provide a frequency domain representation of the audio signal. Finally, the audio features are transformed into readable frequency domain features.

[0016] The aforementioned relationship layer defines a set of tones (i.e., major keys in music theory), specifies the names of the tones, and uses list mapping to correspond to the relationship between tones and frequencies.

[0017] The output layer, based on the frequency domain features read from the input layer by the relation layer, processes and outputs pitch video, which includes a standard line between pitch and frequency, as well as the frequency points of each pitch.

[0018] Step 4: Implement pitch localization and recognition through the constructed pitch localization and recognition network. The specific steps are as follows:

[0019] The vocal file obtained in step 2 is input into the constructed pitch localization and recognition network, which then outputs a pitch video. Compared to the frequency curve of live singing, the frequency distribution of lip-synced vocals is more uniform and dense, and the connections between the points are smoother.

[0020] Furthermore, the comparison files used are relevant music videos from official music platforms, or videos of music programs and competitions. The video requirements are: clear vocals and relatively low background noise (signal-to-noise ratio above 70dB).

[0021] The beneficial effects of this invention are:

[0022] To address the issue of lip-syncing in music performances, computer-based evaluation and verification can be used. In this invention, a pitch localization and recognition network is constructed to visualize pitch and the standard line, allowing for a direct comparison of the curves of the human voice and the synthesized, edited sound. This protects the rights of the audience and fans, while also improving versatility and accuracy compared to existing technologies. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the method flow of an embodiment of the present invention;

[0024] Figure 2 Two pitch comparison videos were generated for this invention. Detailed Implementation

[0025] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] End-to-end voice separation technology is a method that builds a deep learning model. The voices of multiple speakers to be separated are input into the model, and the output of the model is an audio that contains the voices of each individual speaker.

[0027] Parselmouth is a Python library for the Praat software. Praat is an open-source software for audio analysis and processing, primarily used in phonetics, phonology, and musicology. It offers a rich set of features, including spectroscopic analysis, resonance analysis, speech synthesis, and pitch analysis. Parselmouth aims to provide a Python interface for Praat, allowing users to utilize Praat's functionalities within a Python environment.

[0028] like Figure 1 As shown, a pitch localization and recognition method based on human voice separation includes the following steps:

[0029] 1. Download two videos according to the video requirements: one is a performance video by a powerful singer (i.e., the comparison file), and the other is a lip-synced video that has obviously been edited (i.e., the performance video to be identified). Then preprocess the videos, including frame segmentation, windowing, endpoint detection, and noise reduction.

[0030] Endpoint detection: Determines the start and end points of an audio signal.

[0031] Frame segmentation was performed, with each frame lasting 10ms. When applying windowing, the Hanning window was selected.

[0032]

[0033] For each frame of signal x[n], apply a window function w[n], where n = 0, 1, 2, ..., N-1. The windowed signal y[n] can be expressed as:

[0034] y[n] = x[n]·w[n]

[0035] Where y[n] is the windowed signal, x[n] is the k-th frame of the original signal, and w[n] is the window function.

[0036] Regarding the noise reduction method, wavelet transform is chosen here. Wavelet transform is used to perform time-frequency analysis on the signal, and then the wavelet coefficients are thresholded to reduce noise.

[0037] 2. Download third-party software that uses end-to-end vocal separation technology for vocal separation. This example uses Ultimate Vocal Remover, which is free and open-source. After downloading and installing, open the software, click "Select Input," select a singing video, then click "Select Output," select a location path to generate the vocal file, choose MP3 as the output format, and select "Vocals Only." Methods used include VR-Architechture, MDX-Net, Demucs, Ensemble Mode, and Audio Tools; here, we choose the default MDX-Net. The model is also selected as default. Finally, click "Start Processing" to generate an MP3 file, which is the vocal file. At this point, we will have two new vocal files.

[0038] III. Constructing a pitch localization and recognition network;

[0039] The pitch localization and recognition network consists of an input layer, a relational layer, and an output layer.

[0040] The input layer is used to read human voice files. Based on traditional audio feature extraction methods, such as Mel-frequency cepstral coefficients (MFCC), it can extract key information from the audio signal. Through audio feature transformation methods, such as converting to a Mel spectrogram, we can obtain the frequency domain representation of the audio signal. Finally, combining this feature information, we can return the corresponding frequency value according to the inverse Mel-frequency transform formula.

[0041]

[0042] Where f Hz It is the linear frequency value (in Hertz), f mel It is the Mel frequency value. And it calculates the pitch and frequency of all notes within a given minimum and maximum frequency range.

[0043] The relationship layer first defines a set of tones (i.e., major keys in music theory), clarifies the names of the tones, then processes the relationship between tones and frequencies, sets a tone list (dif) to represent the differences between tones, and sets a dictionary (tone_fre_map) to map tones and frequencies.

[0044] The output layer, based on the frequency domain features read from the input layer by the relation layer, processes and outputs a pitch video, which includes a standard line between pitch and frequency, as well as the frequency points of each pitch. The specific operation is as follows:

[0045] First, the parslmouth library was used to load the input vocal file and call its to_pitch_ac() method to extract the pitch information of the audio. This method returns an object containing the time series and the corresponding frequency values.

[0046] Next, a canvas is created using the Matplotlib library, and standard pitch lines are drawn on the canvas, which are pitch lines within a specified pitch range. These standard pitch lines will serve as a reference for displaying pitch information in the video.

[0047] Next, using the animation features provided by Matplotlib, an animation object is created, and the animation function `animate()` is written. This function updates the display of pitch information in each frame, obtains the pitch data for the current time period through the pitch information returned by Parselmouth, and updates the pitch curve and current time indicator line displayed in the animation based on this data. The specific drawing operations are as follows:

[0048] Draw a baseline between pitch and frequency and add a pitch label. Define a function with three parameters: `tone` (pitch), `min_freq` (minimum frequency), and `max_freq` (maximum frequency). Draw a straight line from (0, f) to (1, f) with a width of 2. Add a text label at the leftmost coordinate (1.01, f) of the line, with the content "tone", horizontal alignment left-aligned, vertical alignment bottom-aligned, and font size 18. `f` is the loop parameter in the pitch list.

[0049] Plotting frequency data: Obtain the frequency points of the pitch. Define a time window using the start and end points of the audio signal. Filter the frequency data within this window to obtain the frequency points of the pitch. Plot the time series of frequency points as the x-axis value and the frequency values ​​as the y-axis value to show the frequency changes of a series of tones, demonstrating how pitch changes over time in a musical performance.

[0050] Draw a vertical line at the current time point: Draw a vertical line that starts from the current time point and extends through the entire vertical height of the chart. In music analysis or video analysis applications, such vertical lines are often used to highlight or mark specific time points, such as a beat, the start or end of an event, or a specific frame in a video.

[0051] Next, we use the FuncAnimation class provided by Matplotlib to associate the animation function with the canvas object, generate an animation object, and use the save() method of the animation object to save the animation as a temporary audio-visual file.

[0052] Finally, the FFmpeg library is used to merge the original video file and the generated pitch video file into a new video file. During the merging process, parameters such as the size and position of the pitch video, as well as the output path of the merged video, are set. The final video file will display a graphical representation of the original video content combined with the pitch information of the audio.

[0053] Fourth, place the vocal files of the singing video to be identified and the comparison file into the specified path, determine the predefined pitch set according to the major key of the song sung by the vocals, and finally output the pitch comparison video through the constructed pitch localization and recognition network.

[0054] At this point, two pitch comparison videos have been generated, as follows: Figure 2 As shown, Figure 2 (a) shows the live singing part, and (b) shows the lip-syncing part.

[0055] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0056] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A pitch localization and recognition method based on human voice separation, characterized in that, Includes the following steps: Step 1: Prepare data and perform data preprocessing. The specific steps are as follows: The corresponding real comparison file needs to be selected based on the singing video to be identified; the singing video to be identified and the corresponding real comparison file are preprocessed, and the audio is processed, specifically including frame segmentation, windowing, endpoint detection and noise reduction. First, endpoint detection: determining the start and end points of the audio signal; Then, a frame segmentation operation is performed to divide the audio signal into short frames, with the interval between each frame segmentation being 10-30 milliseconds. Next, windowing is applied: a window function is applied to each frame to reduce discontinuities at frame edges; Finally, noise reduction: noise reduction algorithms are used to reduce background noise and improve speech clarity; Step 2: Use end-to-end voice separation technology to separate the voices from the processed data to obtain the voice file; Step 3: Construct a pitch localization and recognition network. The specific steps are as follows: The pitch localization and recognition network consists of an input layer, a relational layer, and an output layer. The input layer is used to read human voice files, obtain audio features using audio feature extraction methods, process the obtained audio features using audio feature conversion methods, and finally convert the audio features into readable frequency domain features. The aforementioned relationship layer defines a set of tones, specifies the names of the tones, and uses list mapping to correspond to the relationship between tones and frequencies; The output layer, based on the frequency domain features read from the input layer by the relation layer, processes and outputs pitch video, which includes a standard line between pitch and frequency, as well as the frequency points of each pitch. Step 4: Implement pitch localization and recognition through the constructed pitch localization and recognition network. The specific steps are as follows: The human voice file obtained in step 2 is input into the constructed pitch localization and recognition network, and the pitch localization and recognition network outputs pitch video.

2. The pitch localization and recognition method based on human voice separation according to claim 1, characterized in that, The comparison files used are relevant music videos from official music platforms, or videos of music programs and competitions. The video requirements are: clear vocals and a background noise signal-to-noise ratio of over 70dB.

3. The pitch localization and recognition method based on human voice separation according to claim 1, characterized in that, The specific preprocessing steps are as follows: Endpoint detection: Determines the start and end points of an audio signal; Frame segmentation was performed, with each frame lasting 10ms. When applying windowing, the Hanning window was selected. For each frame of signal x[n], apply the window function w[n], where n = 0, 1, 2, ..., N-1; The windowed signal y[n] can be represented as: Where y[n] is the windowed signal, x[n] is the k-th frame of the original signal, and w[n] is the window function; Denoising is achieved by using wavelet transform to perform time-frequency analysis on the signal, and then thresholding the wavelet coefficients to reduce noise.

4. A pitch localization and recognition method based on human voice separation according to any one of claims 1-3, characterized in that, The output layer, based on the frequency domain features read from the input layer by the relation layer, processes and outputs a pitch video, which includes a standard line between pitch and frequency, as well as the frequency points of each pitch. The specific operation is as follows: First, the parslmouth library was used to load the input vocal file and call its to_pitch_ac() method to extract the pitch information of the audio. This method returns an object containing the time series and the corresponding frequency values. Next, a canvas is created using the Matplotlib library, and standard pitch lines are drawn on the canvas, which are pitch lines within a specified pitch range; these standard pitch lines will serve as a reference for displaying pitch information in the video. Then, using the animation features provided by Matplotlib, an animation object is created, and the animation function `animate()` is written. This function updates the display of pitch information in each frame, obtains the pitch data within the current time period through the pitch information returned by Parselmouth, and updates the pitch curve and the current time indicator line displayed in the animation based on this data. The specific drawing operations are as follows: Draw a standard line between pitch and frequency and add a pitch label. Define a function with three parameters: pitch tone, minimum frequency min_freq, and maximum frequency max_freq. Draw a straight line from (0, f) to (1, f). Add a text label with the content "pitch tone" at the leftmost end of the line. f is the loop parameter in the pitch list. Plotting frequency data: Obtain the frequency points of the pitch, define a time window using the start and end points of the audio, and filter the frequency domain features, i.e., the frequency data, within this window to obtain the frequency points of the pitch; plot a series of pitch frequency changes by using the time series of frequency points as the values ​​on the horizontal axis and the frequency values ​​as the values ​​on the vertical axis. Draw a vertical line at the current time: Draw a vertical line that starts from the current time and extends through the entire vertical height of the chart; Next, we use the FuncAnimation class provided by Matplotlib to associate the animation function with the canvas object, generate an animation object, and use the save() method of the animation object to save the animation as a temporary audio-visual file; Finally, the FFmpeg library is called to merge the original video file and the generated pitch video file into a new video file; During the merging process, the size and position of the audio and video are set, as well as the output path of the merged video. The final video file will display a graphical representation of the original video content combined with the audio pitch information.

Images