AI pitch detection method for singing feedback and terminal

Abstract

The invention discloses an AI pitch detection method for singing feedback and a terminal, and the method comprises the steps: carrying out the rearrangement and grouping of the combined frequency and periodic data belonging to the same tone in a received singing audio, and obtaining octave feature data; extracting a first saliency feature map from the combined frequency and periodic data, predicting a tone feature by using the first saliency feature map, extracting a second saliency feature map from the octave feature data, and predicting an octave feature by using the second saliency feature map; and obtaining a combined feature according to the first saliency feature map and the second saliency feature map, combining the combined feature, the tone feature and the octave feature to obtain a fusion feature, inputting the fusion feature into a convolutional neural network layer, and obtaining, feeding back and outputting a pitch detection result. In this way, pitch detection is firstly decomposed into independent tone and octave prediction, and then pitch detection is performed by fusing the multi-dimensional features, so that octave errors in the detection process can be reduced, and the pitch detection precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pitch detection, and in particular to an AI pitch detection method and a terminal for singing feedback. Background Art

[0002] As the core technology in the singing feedback system, pitch detection is an important technology in music and speech signal processing, which aims to extract the fundamental frequency, that is, the main frequency component of the sound, from the audio signal. Pitch detection is widely used in many fields such as music teaching, instrument tuning, and scale games. As a basic algorithm, accurate pitch detection results directly affect the performance of related applications. In music singing teaching, the pitch detection algorithm is used to obtain the pitch value of the singer and the pitch value of the original singer / simplified score, so as to estimate the accuracy of the singer's pitch.

[0003] At present, the three traditional methods of pitch detection are time domain method, frequency domain method and time-frequency domain combined method. However, existing methods may be insufficient in accuracy and efficiency. In order to find an efficient method for extracting the fundamental frequency of human voice, researchers have utilized deep neural networks, especially the powerful learning ability of convolutional neural networks. In addition, musical and structural prior knowledge is combined to further improve the representation effect. In addition, there are frequency domain representations to estimate the frequency of the singing. However, these designs usually cannot reach the level of human perception of pitch and octave melodic information.

[0004] First, current methods can easily mispredict pitches that are within the correct octave range, a type of error often referred to as a pitch error, e.g., mispredicting A4 as G4 or A#4. Second, existing methods can make octave errors even when they correctly predict the pitch being sung, e.g., mispredicting A4 as A3 or A5.

[0005] Although combined frequency and periodicity (CFP) data is widely used as model input representation features in the field of pitch detection due to its powerful audio representation ability, most existing feature representations, including CFP, cannot capture harmonic relationships well due to the limitation of the local receptive field size of CNN. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide an AI pitch detection method and terminal for singing feedback, which can improve the pitch detection accuracy and reduce the pitch missed detection rate.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: An AI pitch detection method for singing feedback comprises the steps of: S1, receiving singing audio, rearranging and grouping the frequency bins of the combined frequency and periodic data belonging to the same tone in the singing audio, and obtaining octave feature data; S2, extracting a first significant feature map from the combined frequency and periodicity data, using the first significant feature map to predict pitch features, extracting a second significant feature map from the octave feature data, and using the second significant feature map to predict octave features; S3. Obtain a combined feature according to the first significant feature map and the second significant feature map, obtain a fused feature by combining the combined feature, the pitch feature and the octave feature, input the fused feature into a convolutional neural network layer, and obtain and feed back an output pitch detection result.

[0008] In order to solve the above technical problems, another technical solution adopted by the present invention is: An AI pitch detection terminal for singing feedback includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the above-mentioned AI pitch detection method for singing feedback is implemented.

[0009] The beneficial effects of the present invention are: the frequency bins of the combined frequency and periodic data belonging to the same tone in the received singing audio are rearranged and grouped to obtain octave feature data; a first significant feature map is extracted from the combined frequency and periodic data, and the tone feature is predicted using the first significant feature map; a second significant feature map is extracted from the octave feature data, and the octave feature is predicted using the second significant feature map; a combined feature is obtained according to the first significant feature map and the second significant feature map, and a fusion feature is obtained by combining the combined feature, the tone feature and the octave feature, and the fusion feature is input into the convolutional neural network layer to obtain and feed back the output pitch detection result. In this way, deep learning (AI) technology is used to first decompose pitch detection into separate tone and octave predictions, and then multi-dimensional features are fused for pitch detection, which can reduce the octave error in the detection process to improve the accuracy of pitch detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of an AI pitch detection method for singing feedback according to an embodiment of the present invention; Figure 2 Schematic diagram of an AI pitch detection terminal for singing feedback according to an embodiment of the present invention; Figure 3 Schematic diagram of the encoder network model structure based on FTAnet according to an embodiment of the present invention; Figure 4Schematic diagram of a decoder module based on a self-attention mechanism according to an embodiment of the present invention; Figure 5 The present invention is a flowchart of the specific steps of an AI pitch detection method for singing feedback according to an embodiment of the present invention.

[0011] Description of labels: 1. An AI pitch detection terminal for singing feedback; 2. Memory; 3. Processor. DETAILED DESCRIPTION

[0012] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0013] Please refer to Figure 1 , an embodiment of the present invention provides an AI pitch detection method for singing feedback, comprising the steps of: S1, receiving singing audio, rearranging and grouping the frequency bins of the combined frequency and periodic data belonging to the same tone in the singing audio, and obtaining octave feature data; S2, extracting a first significant feature map from the combined frequency and periodicity data, using the first significant feature map to predict pitch features, extracting a second significant feature map from the octave feature data, and using the second significant feature map to predict octave features; S3. Obtain a combined feature according to the first significant feature map and the second significant feature map, obtain a fused feature by combining the combined feature, the pitch feature and the octave feature, input the fused feature into a convolutional neural network layer, and obtain and feed back an output pitch detection result.

[0014] From the above description, it can be seen that the beneficial effects of the present invention are: the frequency bins of the combined frequency and periodic data belonging to the same tone in the received singing audio are rearranged and grouped to obtain octave feature data; a first significant feature map is extracted from the combined frequency and periodic data, and the tone feature is predicted using the first significant feature map, and a second significant feature map is extracted from the octave feature data, and the octave feature is predicted using the second significant feature map; a combined feature is obtained according to the first significant feature map and the second significant feature map, and a fusion feature is obtained by combining the combined feature, the tone feature and the octave feature, and the fusion feature is input into the convolutional neural network layer to obtain and feedback the output pitch detection result. In this way, using deep learning (AI) technology to first decompose pitch detection into separate tone and octave predictions, and then fusing multi-dimensional features for pitch detection, the octave error in the detection process can be reduced to improve the accuracy of pitch detection.

[0015] Further, step S1 includes: Obtain the frequency bins of the combined frequency and periodic data belonging to the same tone in the singing audio, and rearrange the tone harmonics into bins: Sk = { spec_i | i ≡ k mod L}, k = 0, 1, 2, ..., L – 1, i = 0, 1, 2, ..., F – 1; In the formula, Sk Indicates k non-overlapping sets, spec_i Represented as a number i The combined frequency and periodicity data, L represents the number of frequency bins per octave, F represents the total number of frequency bins; All non-overlapping sets Sk according to k The values ​​are sorted, and the octave feature data is obtained according to the sorting result.

[0016] From the above description, it can be seen that the octave feature data is constructed by rearranging the order of the combined frequency and periodicity data on the frequency axis. Through this rearrangement, the frequency bins corresponding to the harmonics of the same tone are grouped in the same area, allowing the subsequent neural network to obtain the harmonic relationship in its receptive field.

[0017] Further, step S2 includes: encoding the combined frequency and periodicity data to extract a first significant feature map of size (F+1, T); encoding the octave feature data to extract a second significant feature map of size (F+1, T), where T represents the number of time frames; Step S3 includes: obtaining a combined feature (2F+2, T) according to the first significant feature map and the second significant feature map, and combining the combined feature, the pitch feature and the octave feature to obtain a fusion feature (2F+2+P+O, T), where P represents the pitch feature number and O represents the octave feature number.

[0018] Furthermore, in step S3, the fusion feature is input into the convolutional neural network layer to obtain and feed back the output pitch detection result, including: The fused features are input into a one-dimensional convolutional neural network layer on the time axis, and a pitch detection result with a dimension of (F, T) is output.

[0019] From the above description, it can be seen that by fusing the first significant feature map, the second significant feature map, the pitch feature and the octave feature, a fusion feature is obtained, so that the subsequent neural network model can learn to obtain a more accurate pitch prediction value.

[0020] Furthermore, step S3 includes: Performing endpoint detection on the singing audio to identify each note endpoint, wherein the note endpoint includes a start endpoint and an end endpoint; The note endpoints are aligned with the lyrics timestamps corresponding to the singing audio.

[0021] From the above description, it can be seen that by identifying the endpoint of each note based on endpoint detection and aligning the note endpoint with the lyrics timestamp, the accuracy of pitch detection can be significantly improved and the system missed detection rate can be reduced.

[0022] The above-mentioned AI pitch detection method and terminal for singing feedback of the present invention are suitable for improving the pitch detection accuracy and reducing the pitch missed detection rate, and are described below through specific implementation methods: Please refer to Figure 1 and Figure 5 , Embodiment 1 of the present invention is: An AI pitch detection method for singing feedback comprises the steps of: S1. Receive singing audio, rearrange and group the frequency bins of combined frequencies and periodic data belonging to the same tone in the singing audio, and obtain octave feature data.

[0023] Specifically, the combined frequency and periodic data CFP frequency bins belonging to the same tone in the singing audio are obtained, and the tone harmonics are rearranged into bins: Sk = { spec_i | i ≡ k mod L}, k = 0, 1, 2, ..., L – 1, i = 0, 1, 2, ..., F – 1; In the formula, Sk Indicates k non-overlapping sets, spec_i Represented as a number i The combined frequency and periodicity data, L represents the number of frequency bins per octave, and F represents the total number of frequency bins. spec_i After grouping into different sets, according to all sets Sk The indexes are sorted to obtain the octave feature data TCFP.

[0024] It can be seen that in this embodiment, TCFP is L in CFP. spec_i For example, L=3, F=9, then the non-overlapping sets Sk There should be 3 subsets. Each subset is S 0= {0,3,6}, S 1= {1,4,7}, S 2 = {2,5,8}.

[0025] Therefore, TCFP is constructed by rearranging the order of CFP on the frequency axis, so that the frequency bins corresponding to the harmonics of the same tone can be grouped in the same area to facilitate being captured by the local receptive field of the CNN kernel.

[0026] S2. Extracting a first significant feature map from the combined frequency and periodicity data, and using the first significant feature map to predict pitch features, extracting a second significant feature map from the octave feature data, and using the second significant feature map to predict octave features.

[0027] Specifically, the combined frequency and periodicity data are encoded to extract a first significant feature map of size (F+1, T); the octave feature data are encoded to extract a second significant feature map of size (F+1, T).

[0028] In this embodiment, the encoder consists of a pitch salient feature extraction branch and an octave salient feature extraction branch. One part takes CFP features as input, and the other part takes TCFP features as input. They do not share the same weights and independently output a salient feature map of size (F+1,T). Where F represents the number of frequency bins, and T represents the number of time frames. In order to increase the model's anti-noise ability, the non-human voice part is taken as a separate dimension. Then, the feature maps of CFP and TCFP are superimposed on the frequency dimension to form a combined feature map (2F+2,T). It proceeds to the next prediction step through the pitch decoder and the octave decoder. This embodiment uses FTANet as the encoder network model structure, and the specific network structure is as follows: Figure 3 shown.

[0029] Furthermore, the salient features (F+1, T) of pitch and octave extracted by the encoder are passed to two decoder branches: pitch decoder and octave decoder, respectively.

[0030] For each branch, we adopt the self-attention module of Transformer-encoder to obtain the temporal dependency of salient features. It contains a multi-head self-attention layer, a Feed-Forward, and two normalization layers that connect the residual information. The absolute position encoding is an embedding input layer as an addition vector, representing the position information of the input token. Multiple self-attention modules can be repeatedly connected N times. For the hyperparameters of the model, the context dimension dk = 1024 and the multi-head size h = 8 are set in each self-attention layer, where each head vector is 1024 / 8 = 128 dimensions. In this embodiment, the number of modules is set to N=2. After obtaining the context representation output C, a fully connected layer is used to decode the pitch / octave into a presence probability map. The decoder network structure is as follows Figure 4 shown.

[0031] S3. Obtain a combined feature according to the first significant feature map and the second significant feature map, obtain a fused feature by combining the combined feature, the pitch feature and the octave feature, input the fused feature into a convolutional neural network layer, and obtain and feed back an output pitch detection result.

[0032] Specifically, a combined feature (2F+2, T) is obtained according to the first significant feature map and the second significant feature map, and a fused feature (2F+2+P+O, T) is obtained by combining the combined feature, the pitch feature and the octave feature, where P represents the number of pitch features and O represents the number of octave features. Afterwards, the fused feature is input into a one-dimensional convolutional neural network layer on the time axis, and a pitch detection result with a dimension of (F, T) is obtained as an output.

[0033] In this embodiment, the pitch and octave features are fused into the final output. The model receives more pitch and octave information, thereby improving the prediction accuracy of pitch and octave. The decoder outputs the predicted features of pitch and octave with the same time frame length respectively. First, the pitch feature, octave feature and the previous combined feature map are connected as (2F+2+P+O,T). Wherein, P and O are the number of features of pitch and octave. Further, the fused feature map is calculated using the time axis 1D CNN layer, where the convolution kernel size is 5 and the padding is 2. The pitch prediction value with the dimension of (F,T) is finally output by the calculation formula T=(T-5+2*2) / 2+1. That is, the (2F+2+P+O,T) feature is sent to the model for processing, and then the model returns the pitch feature in (F,T) format.

[0034] Furthermore, the application scenarios of this embodiment may include: (1) Pitch detection in Karaoke system In the Karaoke system, pitch detection technology is widely used for real-time scoring and feedback, helping users improve their singing skills and enhancing their entertainment experience. The Karaoke system uses pitch detection technology to analyze the pitch of the user's singing in real time and compare it with the original singer's pitch. The system detects whether the user's pitch is accurate and whether it matches the original singer's notes, and then scores based on the degree of match. Real-time scoring not only increases the fun of Karaoke, but also stimulates users' sense of competition and motivation to improve.

[0035] Through pitch detection, the Karaoke system can also display the user's current pitch curve on the screen and compare it with the original singer's pitch curve. Users can intuitively see which parts they sing accurately and which parts need improvement. For example, the system can mark pitch deviations with different colors, red indicates a large deviation, and green indicates accuracy. This visual feedback helps users make instant adjustments during singing and improve their pitch control ability.

[0036] Some advanced Karaoke systems also provide a teaching mode. Through pitch detection technology, the system can guide users on how to correctly pronounce and adjust pitch. For example, the system can pause the playback when the user sings the wrong pitch and provide a correct pitch demonstration to help the user gradually correct the mistake. This interactive teaching mode not only improves the user's singing skills, but also increases the educational value of Karaoke.

[0037] Modern Karaoke systems often integrate social functions, allowing users to share their singing videos and scores. Through pitch detection technology, the system can generate detailed singing reports, including scores for pitch accuracy, rhythm control, emotional expression, etc. Users can share these reports with friends or upload them to social media to communicate and compare with other users to further improve their singing skills.

[0038] In conclusion, pitch detection technology plays an important role in Karaoke system, helping users improve their singing skills and enhance entertainment experience and social interaction through real-time scoring, visual feedback, teaching mode and social functions.

[0039] (2) Pitch detection in music teaching In music teaching, pitch detection technology is widely used in instrument performance and vocal training to help students improve their pitch and musical expression.

[0040] In instrument performance training, pitch detection technology can monitor in real time whether the pitch played by students is accurate. For example, when learning instruments such as piano, violin or flute, students can use pitch detection applications or devices to ensure that each note is played correctly. Pitch detection tools can display the pitch deviation of each note and provide instant feedback to help students find and correct mistakes. For example, when a student plays a note, the application will display the difference between the actual pitch of the note and the target pitch, and remind the student to adjust the finger position or breath control with visual or auditory prompts.

[0041] In vocal training, pitch detection technology is equally important. Singers can use pitch detection tools to practice scales, intervals, and songs to ensure that their singing pitch is accurate. Pitch detection applications can display pitch deviations in real time, helping singers adjust their vocal techniques and reach a higher level of singing. For example, the application can display the pitch curve of the singer's singing on the screen and compare it with the target pitch curve to help the singer find pitch deviations and make adjustments. In addition, pitch detection tools can also be used in group training, such as choir rehearsals, to help each member maintain a harmonious pitch, thereby improving the overall singing quality.

[0042] Pitch detection technology also plays an important role in self-study and distance learning. Students can use pitch detection applications for self-practice, and the application will provide detailed feedback and practice suggestions to help students gradually improve their pitch and musical expression. In distance learning, teachers can use pitch detection tools to monitor students' singing or performance in real time and provide instant feedback and guidance. For example, teachers can share the interface of the pitch detection application through video conferencing software, view students' pitch curves in real time, and make comments and corrections.

[0043] In music assessments and exams, pitch detection technology can provide an objective basis for scoring. Teachers can use pitch detection tools to evaluate students' singing or playing intonation and score them based on pitch accuracy, stability, and expressiveness. Pitch detection tools can generate detailed evaluation reports, including the pitch deviation and overall accuracy of each note, to help teachers understand students' musical level more comprehensively and provide targeted guidance and suggestions.

[0044] Embodiment 2 of the present invention is: Based on the first embodiment, it is further defined that after step S3, the following steps are further included: Endpoint detection is performed on the singing audio to identify each note endpoint, wherein the note endpoint includes a start endpoint and an end endpoint; and the note endpoint is aligned with a lyrics timestamp corresponding to the singing audio.

[0045] Specifically, in this embodiment, endpoint detection is used to identify the start and end points in the audio signal. In music processing, endpoint detection can help accurately locate the start and end time of each note, thereby aligning the timestamps of the lyrics and the notes.

[0046] First, the audio signal is preprocessed by endpoint detection technology to identify the start and end time of each note. In this embodiment, the CNNOnsetProcessor method in the open source package madmom provided by python is directly used to obtain the endpoint of each note.

[0047] Secondly, align the detected note endpoints with the lyrics. Usually, each word corresponds to a timestamp. By comparing the note endpoints and the lyrics timestamps, each note can be aligned with the corresponding lyrics. This step can rely on the Viterbi algorithm to find the best matching path between the note and the corresponding lyrics. Specifically, the Viterbi algorithm constructs a dynamic programming table to gradually calculate the optimal path at each time point.

[0048] Therefore, the lyrics and note timestamp alignment technology based on endpoint detection and Viterbi algorithm can significantly improve the accuracy of pitch detection and reduce the system missed detection rate by accurately locating the note endpoints and aligning them with the lyrics.

[0049] Please refer to Figure 2 , Embodiment 3 of the present invention is: An AI pitch detection terminal 1 for singing feedback includes a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, each step of an AI pitch detection method for singing feedback in embodiment one or two is implemented.

[0050] In summary, the present invention provides an AI pitch detection method for singing feedback, which rearranges and groups the frequency bins of the combined frequency and periodic data belonging to the same tone in the received singing audio to obtain octave feature data; extracts a first significant feature map from the combined frequency and periodic data, uses the first significant feature map to predict the tone feature, extracts a second significant feature map from the octave feature data, and uses the second significant feature map to predict the octave feature; obtains a combined feature based on the first significant feature map and the second significant feature map, obtains a fused feature by combining the combined feature, the tone feature and the octave feature, inputs the fused feature into the convolutional neural network layer, and obtains and feeds back the output pitch detection result.

[0051] The method provided by the present invention can improve the effect of intelligent online music education and practice, that is, it can analyze the students' singing or playing pitch in real time, and provide instant feedback to help students find and correct pitch deviations. This real-time feedback can significantly improve students' learning efficiency and pitch control ability; through pitch detection technology, teachers can formulate personalized teaching plans according to the specific problems of each student, provide targeted practice and guidance, thereby improving teaching effects; for self-learners, the pitch detection application can provide detailed practice reports and improvement suggestions to help them improve their music skills independently.

[0052] The method provided by the present invention can also improve the user experience of the Karaoke system, that is, the singing pitch of the user can be accurately analyzed and compared with the original singer, and accurate scoring and ranking can be provided. This function not only increases the fun of Karaoke, but also stimulates the user's sense of competition and motivation for improvement; real-time visual and auditory feedback can be provided through pitch detection technology to help users adjust the pitch in real time during the singing process and improve the singing quality; a teaching mode can be integrated to guide users on how to correctly pronounce and adjust the pitch through pitch detection technology, and help users gradually improve their singing skills.

[0053] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An AI pitch detection method for singing feedback, characterized in that: Includes steps: S1, receiving singing audio, rearranging and grouping the frequency bins of the combined frequency and periodic data belonging to the same tone in the singing audio, and obtaining octave feature data; S2, extracting a first significant feature map from the combined frequency and periodicity data, using the first significant feature map to predict pitch features, extracting a second significant feature map from the octave feature data, and using the second significant feature map to predict octave features; S3. Obtain a combined feature according to the first significant feature map and the second significant feature map, obtain a fused feature by combining the combined feature, the pitch feature and the octave feature, input the fused feature into a convolutional neural network layer, and obtain and feed back an output pitch detection result.

2. The AI ​​pitch detection method for singing feedback according to claim 1, characterized in that: Step S1 includes: Obtain the frequency bins of the combined frequency and periodic data belonging to the same tone in the singing audio, and rearrange the tone harmonics into bins: Sk = { spec_i | i ≡ k against L }, k = 0, 1, 2, ..., L – 1, i = 0, 1, 2, ..., F – 1; In the formula, Sk Indicates k non-overlapping sets, spec_i Represented as a number i The combined frequency and periodicity data, L represents the number of frequency bins per octave, F represents the total number of frequency bins; All non-overlapping sets Sk according to k The values ​​are sorted, and the octave feature data is obtained according to the sorting result.

3. A method for AI pitch detection for singing feedback according to claim 2, characterized in that: Step S2 includes: encoding the combined frequency and periodicity data to extract a first significant feature map of size (F+1, T); encoding the octave feature data to extract a second significant feature map of size (F+1, T), where T represents the number of time frames; Step S3 includes: obtaining a combined feature (2F+2, T) according to the first significant feature map and the second significant feature map, and combining the combined feature, the pitch feature and the octave feature to obtain a fusion feature (2F+2+P+O, T), where P represents the pitch feature number and O represents the octave feature number.

4. The AI ​​pitch detection method for singing feedback according to claim 1, characterized in that: In step S3, the fusion feature is input into the convolutional neural network layer to obtain and feed back the output pitch detection result, including: The fused features are input into a one-dimensional convolutional neural network layer on the time axis, and a pitch detection result with a dimension of (F, T) is output.

5. The AI ​​pitch detection method for singing feedback according to claim 1, characterized in that: Step S3 includes: Performing endpoint detection on the singing audio to identify each note endpoint, wherein the note endpoint includes a start endpoint and an end endpoint; The note endpoints are aligned with the lyrics timestamps corresponding to the singing audio.

6. An AI pitch detection terminal for singing feedback, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: S1, receiving singing audio, rearranging and grouping the frequency bins of the combined frequency and periodic data belonging to the same tone in the singing audio, and obtaining octave feature data; S2, extracting a first significant feature map from the combined frequency and periodicity data, using the first significant feature map to predict pitch features, extracting a second significant feature map from the octave feature data, and using the second significant feature map to predict octave features; S3. Obtain a combined feature according to the first significant feature map and the second significant feature map, obtain a fused feature by combining the combined feature, the pitch feature and the octave feature, input the fused feature into a convolutional neural network layer, and obtain and feed back an output pitch detection result.

7. An AI pitch detection terminal for singing feedback according to claim 6, characterized in that: Step S1 includes: Obtain the frequency bins of the combined frequency and periodic data belonging to the same tone in the singing audio, and rearrange the tone harmonics into bins: Sk = { spec_i | i ≡ k against L }, k = 0, 1, 2, ..., L – 1, i = 0, 1, 2, ..., F – 1; In the formula, Sk Indicates k non-overlapping sets, spec_i Represented as a number i The combined frequency and periodicity data, L represents the number of frequency bins per octave, F represents the total number of frequency bins; All non-overlapping sets Sk according to k The values ​​are sorted, and the octave feature data is obtained according to the sorting result.

8. An AI pitch detection terminal for singing feedback according to claim 7, characterized in that: Step S2 includes: encoding the combined frequency and periodicity data to extract a first significant feature map of size (F+1, T); encoding the octave feature data to extract a second significant feature map of size (F+1, T), where T represents the number of time frames; Step S3 includes: obtaining a combined feature (2F+2, T) according to the first significant feature map and the second significant feature map, and combining the combined feature, the pitch feature and the octave feature to obtain a fusion feature (2F+2+P+O, T), where P represents the pitch feature number and O represents the octave feature number.

9. The AI ​​pitch detection terminal for singing feedback according to claim 6, characterized in that: In step S3, the fusion feature is input into the convolutional neural network layer to obtain and feed back the output pitch detection result, including: The fused features are input into a one-dimensional convolutional neural network layer on the time axis, and a pitch detection result with a dimension of (F, T) is output.

10. The AI ​​pitch detection terminal for singing feedback according to claim 6, characterized in that: Step S3 includes: Performing endpoint detection on the singing audio to identify each note endpoint, wherein the note endpoint includes a start endpoint and an end endpoint; The note endpoints are aligned with the lyrics timestamps corresponding to the singing audio.

Images