Piano note identification method and terminal

Abstract

The invention discloses a piano note identification method and a terminal, and the method comprises the steps: sequentially carrying out the strong sound detection and background sound detection of an audio data stream, obtaining an audio frame belonging to a foreground sound, inputting the audio frame belonging to the foreground sound into a feature extraction module in a convolutional neural network for feature extraction, and obtaining a shallow feature; and the shallow layer features are respectively input into a piano sound detection module and a piano pitch recognition module in the convolutional neural network to obtain a piano sound detection result and a piano pitch recognition result, so that background sound in the audio data stream is effectively filtered out, interference of the background sound on subsequent recognition is avoided, the robustness of a recognition algorithm is improved, and the recognition efficiency is improved. And the convolutional neural network does not use a mode of cascading a plurality of hierarchical networks, and adopts a single neural network architecture sharing a hierarchy (i.e., a feature extraction module) and branches (i.e., a piano sound detection module and a piano pitch recognition module), so that the complexity is relatively low, the recognition precision is improved, and the algorithm complexity is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of audio analysis, and in particular to a piano note recognition method and a terminal. Background Art

[0002] Piano note recognition is an application scenario in the field of audio analysis. Its main purpose is to detect which audio frame contains a piano note from the audio data and identify the note of the piano note.

[0003] In the prior art, when identifying musical sounds, the audio data is directly analyzed and processed, and the relevant features are extracted and input into the corresponding algorithm for judgment. No filtering or screening is performed, which is more suitable for electronic music or music audio with good quality recorded in recording studios and other environments. The audio data recorded in actual usage scenarios contains a large amount of background sound (such as human voice, environmental sound, etc.), and many audio frames do not contain musical sounds, only background sounds. If exclusion and screening are not performed, some single-frequency background sounds in the audio frame are easily identified as notes with similar frequencies, thereby polluting the note relationship table.

[0004] In addition, the existing technology uses a recurrent neural network to segment notes, and then uses 88 neural networks to determine which single note it is. This results in too high time complexity. The notes must be determined by 88 neural networks, which requires too much computing power and makes it difficult to achieve real-time note recognition on embedded devices with limited computing power.

[0005] In summary, the existing note recognition methods have low recognition accuracy and high algorithm complexity. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a piano note recognition method and terminal, which can improve the recognition accuracy and reduce the algorithm complexity.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: A piano note recognition method comprises the steps of: Collect audio data stream; Performing strong sound detection on the audio data stream to obtain a valid audio frame; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the foreground sound; The audio frame belonging to the foreground sound is input into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and the shallow features are respectively input into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain piano sound detection results and piano pitch recognition results.

[0008] In order to solve the above technical problems, another technical solution adopted by the present invention is: A piano note recognition terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: Collect audio data stream; Performing strong sound detection on the audio data stream to obtain a valid audio frame; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the foreground sound; The audio frame belonging to the foreground sound is input into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and the shallow features are respectively input into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain piano sound detection results and piano pitch recognition results.

[0009] The beneficial effects of the present invention are as follows: strong sound detection and background sound detection are performed on the audio data stream in sequence to obtain audio frames belonging to foreground sound, the audio frames belonging to foreground sound are input into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and the shallow features are respectively input into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain piano sound detection results and piano pitch recognition results, effectively filtering out the background sound in the audio data stream, avoiding the interference of the background sound on subsequent recognition, and improving the robustness of the recognition algorithm. The convolutional neural network does not use a plurality of hierarchical network cascades, but adopts a single neural network architecture of a common layer (i.e., feature extraction module) + branch (i.e., piano sound detection module and piano pitch recognition module), which has low complexity, thereby improving the recognition accuracy and reducing the algorithm complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a piano note recognition method according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a piano note recognition terminal according to an embodiment of the present invention; Figure 3 is a recognition flow chart of a piano note recognition method according to an embodiment of the present invention; Figure 4 A schematic diagram of background sound detection in a piano note recognition method according to an embodiment of the present invention; Figure 5 Schematic diagram of the structure of a convolutional neural network in a piano note recognition method according to an embodiment of the present invention; Figure 6 Schematic diagram of the background sound detection effect in the piano note recognition method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0011] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0012] Please refer to Figure 1 , a piano note recognition method, comprising the steps of: Collect audio data stream; Performing strong sound detection on the audio data stream to obtain a valid audio frame; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the foreground sound; The audio frame belonging to the foreground sound is input into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and the shallow features are respectively input into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain piano sound detection results and piano pitch recognition results.

[0013] From the above description, it can be seen that the beneficial effects of the present invention are: performing strong sound detection and background sound detection on the audio data stream in turn to obtain audio frames belonging to foreground sounds, inputting the audio frames belonging to foreground sounds into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and inputting the shallow features into the piano sound detection module and the piano pitch recognition module in the convolutional neural network respectively to obtain piano sound detection results and piano pitch recognition results, thereby effectively filtering out the background sound in the audio data stream, avoiding the interference of background sound on subsequent recognition, and improving the robustness of the recognition algorithm. The convolutional neural network does not use a cascade of multiple hierarchical networks, but adopts a single neural network architecture of a shared layer (i.e., feature extraction module) + branch (i.e., piano sound detection module and piano pitch recognition module), which has low complexity, thereby improving recognition accuracy and reducing algorithm complexity.

[0014] Further, the performing strong sound detection on the audio data stream to obtain a valid audio frame includes: Dividing the audio data stream according to a preset subframe length to obtain an audio subframe sequence; Strong sound detection is performed on each audio subframe in the audio subframe sequence in turn. If strong sound is detected in the audio subframe, the audio subframe with the strong sound is used as the starting frame, and a preset number of audio subframes after the starting frame are spliced ​​to generate a valid audio frame. If no strong sound is detected in the audio subframe, strong sound detection is performed on the next audio subframe.

[0015] From the above description, it can be seen that after the audio data stream is divided, an audio subframe sequence is obtained, and strong sound detection is performed on each audio subframe in the audio subframe sequence in sequence. The audio subframe with strong sound is used as the starting frame, and a preset number of audio subframes after the starting frame are spliced ​​to generate a valid audio frame. The audio frame without strong sound is probably the background sound. Through strong sound detection, the valid audio frames can be screened out from the audio data stream, thereby improving the efficiency and accuracy of subsequent processing.

[0016] Further, the performing background sound detection on the valid audio frame to obtain the audio frame belonging to the foreground sound includes: Calculating the short-time energy and the zero-crossing rate of the valid audio frame, and performing nonlinear transformation on the short-time energy and the zero-crossing rate to obtain transformed short-time energy and zero-crossing rate; The background sound detection is performed on the valid audio frame according to the transformed short-time energy and zero-crossing rate, and the audio frame belonging to the foreground sound is extracted.

[0017] From the above description, it can be seen that the short-time energy and zero-crossing rate of the valid audio frame are transformed nonlinearly, and the background sound detection is performed on the valid audio frame based on the transformed short-time energy and zero-crossing rate, and the audio frame belonging to the foreground sound is extracted. Since the sounds of musical instruments with different pitches have different short-time energy and zero-crossing rate performances, the short-time energy and zero-crossing rate features are fused and judged through nonlinear transformation to determine whether the audio frame is foreground sound or background sound, thereby retaining most of the piano sound while effectively filtering out most of the background sound to ensure subsequent recognition accuracy.

[0018] Furthermore, the feature extraction module includes a convolutional layer and a fully connected layer connected in sequence; The audio frame belonging to the foreground sound is input into the feature extraction module in the convolutional neural network for feature extraction, and the shallow features obtained include: Performing a fast Fourier transform of a preset dimension on the audio frame belonging to the foreground sound to obtain frequency domain features; Performing white noise filtering on the frequency domain features to obtain filtered frequency domain features; Performing feature fusion on the filtered frequency domain features to obtain fused frequency domain features, and normalizing the fused frequency domain features to obtain normalized frequency domain features; The normalized frequency domain features are input into a feature extraction module in a convolutional neural network, and feature extraction is performed through the convolutional layer and the fully connected layer in sequence to obtain shallow features.

[0019] From the above description, it can be seen that the audio frames belonging to the foreground sound are subjected to fast Fourier transform of preset dimensions, white noise filtering, feature fusion and normalization, which further improves the purity of the audio data subsequently input into the neural network, thereby improving the recognition accuracy.

[0020] Further, the piano sound detection module includes a connection layer; The piano pitch recognition module includes two fully connected layers; The step of inputting the shallow features into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain the piano sound detection result and the piano pitch recognition result comprises: The shallow features are input into the piano sound detection module in the convolutional neural network, and after the feature correlation is extracted by the first connection layer, the piano sound detection result is output through the Sigmoid layer. At the same time, the shallow features are input into the piano pitch recognition module in the convolutional neural network, and after the classification is obtained through the two fully connected layers, the piano pitch recognition result is output through the Softmax layer.

[0021] From the above description, it can be seen that the shallow features are input into the piano sound detection module, and after a connection layer extracts the feature correlation, the piano sound detection result is output through the Sigmoid layer. At the same time, the shallow features are input into the piano pitch recognition module, and after being classified through two fully connected layers, the piano pitch recognition result is output through the Softmax layer. There is no need to repeatedly extract the shallow features, which saves computing power overhead and is easier to deploy on embedded devices with limited computing power. The piano sound detection module and the piano pitch recognition module can more accurately identify the piano sound and pitch.

[0022] Please refer to Figure 2 , a piano note recognition terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Collect audio data stream; Performing strong sound detection on the audio data stream to obtain a valid audio frame; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the foreground sound; The audio frame belonging to the foreground sound is input into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and the shallow features are respectively input into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain piano sound detection results and piano pitch recognition results.

[0023] From the above description, it can be seen that the beneficial effects of the present invention are: performing strong sound detection and background sound detection on the audio data stream in turn to obtain audio frames belonging to foreground sounds, inputting the audio frames belonging to foreground sounds into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and inputting the shallow features into the piano sound detection module and the piano pitch recognition module in the convolutional neural network respectively to obtain piano sound detection results and piano pitch recognition results, thereby effectively filtering out the background sound in the audio data stream, avoiding the interference of background sound on subsequent recognition, and improving the robustness of the recognition algorithm. The convolutional neural network does not use a cascade of multiple hierarchical networks, but adopts a single neural network architecture of a shared layer (i.e., feature extraction module) + branch (i.e., piano sound detection module and piano pitch recognition module), which has low complexity, thereby improving recognition accuracy and reducing algorithm complexity.

[0024] Further, the performing strong sound detection on the audio data stream to obtain a valid audio frame includes: Dividing the audio data stream according to a preset subframe length to obtain an audio subframe sequence; Strong sound detection is performed on each audio subframe in the audio subframe sequence in turn. If strong sound is detected in the audio subframe, the audio subframe with the strong sound is used as the starting frame, and a preset number of audio subframes after the starting frame are spliced ​​to generate a valid audio frame. If no strong sound is detected in the audio subframe, strong sound detection is performed on the next audio subframe.

[0025] From the above description, it can be seen that after the audio data stream is divided, an audio subframe sequence is obtained, and strong sound detection is performed on each audio subframe in the audio subframe sequence in sequence. The audio subframe with strong sound is used as the starting frame, and a preset number of audio subframes after the starting frame are spliced ​​to generate a valid audio frame. The audio frame without strong sound is probably the background sound. Through strong sound detection, the valid audio frames can be screened out from the audio data stream, thereby improving the efficiency and accuracy of subsequent processing.

[0026] Further, the performing background sound detection on the valid audio frame to obtain the audio frame belonging to the foreground sound includes: Calculating the short-time energy and the zero-crossing rate of the valid audio frame, and performing nonlinear transformation on the short-time energy and the zero-crossing rate to obtain transformed short-time energy and zero-crossing rate; The background sound detection is performed on the valid audio frame according to the transformed short-time energy and zero-crossing rate, and the audio frame belonging to the foreground sound is extracted.

[0027] From the above description, it can be seen that the short-time energy and zero-crossing rate of the valid audio frame are transformed nonlinearly, and the background sound detection is performed on the valid audio frame based on the transformed short-time energy and zero-crossing rate, and the audio frame belonging to the foreground sound is extracted. Since the sounds of musical instruments with different pitches have different short-time energy and zero-crossing rate performances, the short-time energy and zero-crossing rate features are fused and judged through nonlinear transformation to determine whether the audio frame is foreground sound or background sound, thereby retaining most of the piano sound while effectively filtering out most of the background sound to ensure subsequent recognition accuracy.

[0028] Furthermore, the feature extraction module includes a convolutional layer and a fully connected layer connected in sequence; The audio frame belonging to the foreground sound is input into the feature extraction module in the convolutional neural network for feature extraction, and the shallow features obtained include: Performing a fast Fourier transform of a preset dimension on the audio frame belonging to the foreground sound to obtain frequency domain features; Performing white noise filtering on the frequency domain features to obtain filtered frequency domain features; Performing feature fusion on the filtered frequency domain features to obtain fused frequency domain features, and normalizing the fused frequency domain features to obtain normalized frequency domain features; The normalized frequency domain features are input into a feature extraction module in a convolutional neural network, and feature extraction is performed through the convolutional layer and the fully connected layer in sequence to obtain shallow features.

[0029] From the above description, it can be seen that the audio frames belonging to the foreground sound are subjected to fast Fourier transform of preset dimensions, white noise filtering, feature fusion and normalization, which further improves the purity of the audio data subsequently input into the neural network, thereby improving the recognition accuracy.

[0030] Further, the piano sound detection module includes a connection layer; The piano pitch recognition module includes two fully connected layers; The step of inputting the shallow features into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain the piano sound detection result and the piano pitch recognition result comprises: The shallow features are input into the piano sound detection module in the convolutional neural network, and after the feature correlation is extracted by the first connection layer, the piano sound detection result is output through the Sigmoid layer. At the same time, the shallow features are input into the piano pitch recognition module in the convolutional neural network, and after the classification is obtained through the two fully connected layers, the piano pitch recognition result is output through the Softmax layer.

[0031] From the above description, it can be seen that the shallow features are input into the piano sound detection module, and after a connection layer extracts the feature correlation, the piano sound detection result is output through the Sigmoid layer. At the same time, the shallow features are input into the piano pitch recognition module, and after being classified through two fully connected layers, the piano pitch recognition result is output through the Softmax layer. There is no need to repeatedly extract the shallow features, which saves computing power overhead and is easier to deploy on embedded devices with limited computing power. The piano sound detection module and the piano pitch recognition module can more accurately identify the piano sound and pitch.

[0032] The piano note recognition method and terminal of the present invention are deployed on an embedded device running RTOS (Real Time Operating System), and can be applied to piano note recognition scenarios. The following is an explanation through specific implementation methods: Please refer to Figure 1 , Figure 3-Figure 6 , Embodiment 1 of the present invention is: A piano note recognition method comprises the steps of: S1. Collect audio data stream.

[0033] Since the highest note of the piano is C8#, the fundamental frequency is 4186Hz, and considering that its second harmonic frequency is 8372Hz, according to the Nyquist sampling principle, the sampling rate must be no less than 2 times the data spectrum range to ensure that there is no distortion. In an optional implementation, the microphone device is set to mono sampling, the sampling rate is set to no less than 16KHz, and the sampling bit width is set to 8bit or 16bit, and the microphone device is used to collect the audio data stream according to the sampling rate and the sampling bit width.

[0034] S2, perform strong sound detection on the audio data stream to obtain a valid audio frame, such as Figure 3 As shown, specifically including S21-S22: In an optional implementation manner, the length of an audio frame is determined to be 4096 samples, and the preset subframe length is determined to be 32 samples, then the preset number is 4096 / 32=128.

[0035] S21. Divide the audio data stream according to a preset subframe length to obtain an audio subframe sequence.

[0036] In an optional implementation, the audio data stream is divided into 32 samples to obtain an audio subframe sequence.

[0037] S22. Perform strong sound detection on each audio subframe in the audio subframe sequence in turn. If a strong sound is detected in the audio subframe, the audio subframe with the strong sound is used as the starting frame, and a preset number of audio subframes after the starting frame are spliced ​​to generate a valid audio frame. If it is detected that there is no strong sound in the audio subframe, perform strong sound detection on the next audio subframe.

[0038] Specifically, starting from the i-th audio subframe in the audio subframe sequence, the initial value of i is 0, and the short-time energy of the i-th audio subframe is counted. If the short-time energy reaches a preset value, it is considered that there is a strong sound in the i-th audio subframe, and the i-th audio subframe with the strong sound is used as the starting frame. The 128 audio subframes after the starting frame are spliced ​​to generate a valid audio frame. If the short-time energy does not reach the preset value, it is considered that there is no strong sound in the i-th audio subframe, and the strong sound detection is performed on the i+1-th audio subframe.

[0039] S3, performing background sound detection on the valid audio frame to obtain an audio frame belonging to the foreground sound, such as Figure 4 As shown, specifically including S31-S32: S31, calculating the short-time energy and the zero-crossing rate of the valid audio frame, and performing nonlinear transformation on the short-time energy and the zero-crossing rate to obtain transformed short-time energy and zero-crossing rate.

[0040] In an optional implementation, after taking the logarithm of the short-time energy STE, a binary K-order polynomial function is used as a nonlinear function to perform a nonlinear transformation on the short-time energy STE and the zero-crossing rate ZCR to obtain the transformed short-time energy and zero-crossing rate. The order K=2, specifically: ; In the formula, s=log(STE), z=ZCR, the goal is to fit and find the coefficients of the polynomial The value of .

[0041] When fitting the nonlinear function, increase the weight of the piano sound positive sample so that the fitting function can improve the recall rate as much as possible. The recall rate Recall = TP / (TP+FN), that is, try to avoid missing the piano audio frame. Figure 6 As shown, the coarse detection algorithm can effectively filter out most of the background sounds while retaining most of the piano sounds.

[0042] S32: Perform background sound detection on the valid audio frame according to the transformed short-time energy and zero-crossing rate, and extract the audio frame belonging to the foreground sound.

[0043] The effective audio frames of piano sound and background sound are used as positive and negative samples. On the two-dimensional plane of ZCR-log (STE), positive and negative sample points are scattered. A nonlinear function curve is fitted as the decision function to divide the positive and negative samples. The nonlinear function curve can roughly but quickly divide the suspected piano sound and background sound. Subsequently, the audio frame suspected of piano sound is input into the neural network for fine detection to determine whether the audio frame is background sound. This rough detection-fine detection strategy can effectively filter simple background audio frames and prevent them from being input into the neural network, thereby greatly reducing the recognition time.

[0044] S4, input the audio frame belonging to the foreground sound into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and input the shallow features into the piano sound detection module and the piano pitch recognition module in the convolutional neural network respectively to obtain piano sound detection results and piano pitch recognition results, such as Figure 3 As shown, specifically including S41-S45: Among them, Figure 5 As shown, the feature extraction module includes a one-dimensional convolutional layer and two fully connected layers connected in sequence, and different layers are connected through the activation function Relu; the piano sound detection module includes a connection layer; and the piano pitch recognition module includes two fully connected layers.

[0045] S41, performing a fast Fourier transform of a preset dimension on the audio frame belonging to the foreground sound to obtain frequency domain features.

[0046] In an optional implementation, the preset dimension is 4096 dimensions.

[0047] S42, performing white noise filtering on the frequency domain features to obtain filtered frequency domain features.

[0048] S43, performing feature fusion on the filtered frequency domain features to obtain fused frequency domain features, and normalizing the fused frequency domain features to obtain normalized frequency domain features.

[0049] S44, inputting the normalized frequency domain features into a feature extraction module in a convolutional neural network, and performing feature extraction through the one-dimensional convolutional layer and the two fully connected layers in sequence to obtain shallow features.

[0050] S45. Input the shallow features into the piano sound detection module in the convolutional neural network, extract the feature correlation through the first connection layer, and output the piano sound detection result through the Sigmoid layer. At the same time, input the shallow features into the piano pitch recognition module in the convolutional neural network, analyze and classify through the two fully connected layers, and output the piano pitch recognition result through the Softmax layer.

[0051] The piano sound detection result includes piano sound or background sound. The piano pitch recognition result is a specific one of 88 pitches.

[0052] Please refer to Figure 2 , Embodiment 2 of the present invention is: A piano note recognition terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the piano note recognition method in the first embodiment is implemented.

[0053] In summary, the piano note recognition method and terminal provided by the present invention sequentially perform strong sound detection and background sound detection on the audio data stream to obtain an audio frame belonging to the foreground sound, input the audio frame belonging to the foreground sound into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and respectively input the shallow features into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain piano sound detection results and piano pitch recognition results, effectively filtering out the background sound in the audio data stream, avoiding the interference of the background sound on subsequent recognition, and improving the robustness of the recognition algorithm. In addition, the convolutional neural network does not use a plurality of hierarchical network cascades, but adopts a shared hierarchy. The architecture of a single neural network consisting of a feature extraction module (i.e., a piano sound detection module and a piano pitch recognition module) + a branch (i.e., a piano sound detection module and a piano pitch recognition module) has low complexity, thereby improving recognition accuracy and reducing algorithm complexity; in addition, the short-time energy and zero-crossing rate features are fused and judged through nonlinear transformation to determine whether the audio frame is foreground sound or background sound, thereby effectively filtering out most of the background sound while retaining most of the piano sound, thereby ensuring subsequent recognition accuracy; and, the audio frames belonging to the foreground sound are subjected to fast Fourier transform of preset dimensions, white noise filtering, feature fusion and normalization, thereby further improving the purity of the audio data subsequently input into the neural network, thereby improving recognition accuracy.

[0054] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A piano note recognition method, characterized in that: Includes steps: Collect audio data stream; Performing strong sound detection on the audio data stream to obtain a valid audio frame; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the foreground sound; The audio frame belonging to the foreground sound is input into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and the shallow features are respectively input into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain piano sound detection results and piano pitch recognition results.

2. A piano note recognition method according to claim 1, characterized in that: The performing strong sound detection on the audio data stream to obtain a valid audio frame comprises: Dividing the audio data stream according to a preset subframe length to obtain an audio subframe sequence; Strong sound detection is performed on each audio subframe in the audio subframe sequence in turn. If strong sound is detected in the audio subframe, the audio subframe with the strong sound is used as the starting frame, and a preset number of audio subframes after the starting frame are spliced ​​to generate a valid audio frame. If no strong sound is detected in the audio subframe, strong sound detection is performed on the next audio subframe.

3. A piano note recognition method according to claim 1, characterized in that: The performing background sound detection on the valid audio frame to obtain an audio frame belonging to the foreground sound comprises: Calculating the short-time energy and the zero-crossing rate of the valid audio frame, and performing nonlinear transformation on the short-time energy and the zero-crossing rate to obtain transformed short-time energy and zero-crossing rate; The background sound detection is performed on the valid audio frame according to the transformed short-time energy and zero-crossing rate, and the audio frame belonging to the foreground sound is extracted.

4. A piano note recognition method according to claim 1, characterized in that: The feature extraction module includes a convolutional layer and a fully connected layer connected in sequence; The audio frame belonging to the foreground sound is input into the feature extraction module in the convolutional neural network for feature extraction, and the shallow features obtained include: Performing a fast Fourier transform of a preset dimension on the audio frame belonging to the foreground sound to obtain frequency domain features; Performing white noise filtering on the frequency domain features to obtain filtered frequency domain features; Performing feature fusion on the filtered frequency domain features to obtain fused frequency domain features, and normalizing the fused frequency domain features to obtain normalized frequency domain features; The normalized frequency domain features are input into a feature extraction module in a convolutional neural network, and feature extraction is performed through the convolutional layer and the fully connected layer in sequence to obtain shallow features.

5. A piano note recognition method according to claim 1, characterized in that: The piano sound detection module includes a connection layer; The piano pitch recognition module includes two fully connected layers; The step of inputting the shallow features into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain the piano sound detection result and the piano pitch recognition result comprises: The shallow features are input into the piano sound detection module in the convolutional neural network, and after the feature correlation is extracted by the first connection layer, the piano sound detection result is output through the Sigmoid layer. At the same time, the shallow features are input into the piano pitch recognition module in the convolutional neural network, and after the classification is obtained through the two fully connected layers, the piano pitch recognition result is output through the Softmax layer.

6. A piano note recognition terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: Collect audio data stream; Performing strong sound detection on the audio data stream to obtain a valid audio frame; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the foreground sound; The audio frame belonging to the foreground sound is input into the feature extraction module in the convolutional neural network for feature extraction to obtain shallow features, and the shallow features are respectively input into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain piano sound detection results and piano pitch recognition results.

7. A piano note recognition terminal according to claim 6, characterized in that: The performing strong sound detection on the audio data stream to obtain a valid audio frame comprises: Dividing the audio data stream according to a preset subframe length to obtain an audio subframe sequence; Strong sound detection is performed on each audio subframe in the audio subframe sequence in turn. If strong sound is detected in the audio subframe, the audio subframe with the strong sound is used as the starting frame, and a preset number of audio subframes after the starting frame are spliced ​​to generate a valid audio frame. If no strong sound is detected in the audio subframe, strong sound detection is performed on the next audio subframe.

8. A piano note recognition terminal according to claim 6, characterized in that: The performing background sound detection on the valid audio frame to obtain an audio frame belonging to the foreground sound comprises: Calculating the short-time energy and the zero-crossing rate of the valid audio frame, and performing nonlinear transformation on the short-time energy and the zero-crossing rate to obtain transformed short-time energy and zero-crossing rate; The background sound detection is performed on the valid audio frame according to the transformed short-time energy and zero-crossing rate, and the audio frame belonging to the foreground sound is extracted.

9. A piano note recognition terminal according to claim 6, characterized in that: The feature extraction module includes a convolutional layer and a fully connected layer connected in sequence; The audio frame belonging to the foreground sound is input into the feature extraction module in the convolutional neural network for feature extraction, and the shallow features obtained include: Performing a fast Fourier transform of a preset dimension on the audio frame belonging to the foreground sound to obtain frequency domain features; Performing white noise filtering on the frequency domain features to obtain filtered frequency domain features; Performing feature fusion on the filtered frequency domain features to obtain fused frequency domain features, and normalizing the fused frequency domain features to obtain normalized frequency domain features; The normalized frequency domain features are input into a feature extraction module in a convolutional neural network, and feature extraction is performed through the convolutional layer and the fully connected layer in sequence to obtain shallow features.

10. A piano note recognition terminal according to claim 6, characterized in that: The piano sound detection module includes a connection layer; The piano pitch recognition module includes two fully connected layers; The step of inputting the shallow features into the piano sound detection module and the piano pitch recognition module in the convolutional neural network to obtain the piano sound detection result and the piano pitch recognition result comprises: The shallow features are input into the piano sound detection module in the convolutional neural network, and after the feature correlation is extracted by the first connection layer, the piano sound detection result is output through the Sigmoid layer. At the same time, the shallow features are input into the piano pitch recognition module in the convolutional neural network, and after the classification is obtained through the two fully connected layers, the piano pitch recognition result is output through the Softmax layer.

Images