Musical instrument sound detection method and terminal

Abstract

The invention discloses a musical instrument sound detection method and a terminal, and the method comprises the steps: carrying out the strong sound detection of a collected audio stream, extracting an effective audio frame with the strong sound, carrying out the background sound detection of the effective audio frame, screening out an audio frame belonging to a foreground sound, and carrying out the musical instrument sound detection based on the audio frame belonging to the foreground sound through employing a lightweight neural network. Through strong sound detection and background sound detection, most background sounds can be effectively filtered out, the lightweight neural network has the characteristics of shallow hierarchy and simplified parameters, and then the lightweight neural network is used for detecting the musical instrument sounds, so that the musical instrument sound recognition accuracy is improved, the algorithm complexity is reduced, and the accuracy of musical instrument sound recognition is improved. And the method is easier to deploy on embedded equipment with limited computing power.

Description

Technical Field

[0001] The present invention relates to the technical field of audio analysis, and in particular to a musical instrument sound detection method and a terminal. Background Art

[0002] Musical instrument detection is an application scenario in the field of audio analysis. Its main purpose is to detect which audio frame contains the sound of a specified musical instrument from the audio data.

[0003] In the prior art, when recognizing the sound of musical instruments, the audio data is directly extracted with relevant features and then input into the corresponding neural network for judgment, which has the following two problems: (1) Since the collected audio data usually contains background sound, that is, sound that is useless for detecting musical instrument sounds, there may even be a situation where there is only background sound but no musical instrument sounds. In this case, directly extracting relevant features from the audio data and inputting them into the corresponding neural network for judgment will result in poor recognition accuracy for musical instrument sounds. (2) The neural network used in existing technologies is deep and the algorithm is highly complex, which requires higher computing power for the deployed equipment. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a musical instrument sound detection method and terminal, which can improve the accuracy of musical instrument sound recognition and reduce the complexity of the algorithm.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for detecting musical instrument sound comprises the steps of: Collect the audio stream to be detected; Performing strong sound detection on the audio stream to extract valid audio frames containing strong sounds; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound; Based on the audio frame belonging to the foreground sound, a lightweight neural network is used to detect musical instrument sounds to obtain a musical instrument sound detection result.

[0006] In order to solve the above technical problems, another technical solution adopted by the present invention is: A musical instrument sound detection terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: Collect the audio stream to be detected; Performing strong sound detection on the audio stream to extract valid audio frames containing strong sounds; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound; Based on the audio frame belonging to the foreground sound, a lightweight neural network is used to detect musical instrument sounds to obtain a musical instrument sound detection result.

[0007] The beneficial effects of the present invention are as follows: strong sound detection is first performed on the collected audio stream, valid audio frames with strong sounds are extracted, background sound detection is performed on the valid audio frames, audio frames belonging to foreground sounds are screened out, and musical instrument sound detection is performed using a lightweight neural network based on the audio frames belonging to foreground sounds to obtain musical instrument sound detection results. Most of the background sounds can be effectively filtered out through strong sound detection and background sound detection. The lightweight neural network has the characteristics of shallow hierarchy and streamlined parameters. The lightweight neural network is then used to detect musical instrument sounds, thereby improving the accuracy of musical instrument sound recognition, reducing algorithm complexity, and being easier to deploy on embedded devices with limited computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flowchart of a method for detecting musical instrument sound according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a musical instrument sound detection terminal according to an embodiment of the present invention; Figure 3 is a detection flow chart of a method for detecting musical instrument sound according to an embodiment of the present invention; Figure 4 A schematic diagram of background sound detection in a method for detecting musical instrument sound according to an embodiment of the present invention; Figure 5 Schematic diagram of frequency domain feature preprocessing in a musical instrument sound detection method according to an embodiment of the present invention; Figure 6 A diagram of a lightweight neural network structure in a musical instrument sound detection method according to an embodiment of the present invention; Figure 7 Schematic diagram of the background sound detection effect in the instrument sound detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0009] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0010] Please refer to Figure 1 , a musical instrument sound detection method, comprising the steps of: Collect the audio stream to be detected; Performing strong sound detection on the audio stream to extract valid audio frames containing strong sounds; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound; Based on the audio frame belonging to the foreground sound, a lightweight neural network is used to detect musical instrument sounds to obtain a musical instrument sound detection result.

[0011] From the above description, it can be seen that the beneficial effects of the present invention are: firstly performing strong sound detection on the collected audio stream, extracting the valid audio frames with strong sounds, performing background sound detection on the valid audio frames, screening out the audio frames belonging to the foreground sound, and using a lightweight neural network to perform instrument sound detection based on the audio frames belonging to the foreground sound to obtain the instrument sound detection result. Through strong sound detection and background sound detection, most of the background sounds can be effectively filtered out. The lightweight neural network has the characteristics of shallow hierarchy and streamlined parameters. The lightweight neural network is then used to detect instrument sounds, thereby improving the accuracy of instrument sound recognition, reducing the complexity of the algorithm, and being easier to deploy on embedded devices with limited computing power.

[0012] Furthermore, the lightweight neural network includes a one-dimensional convolutional layer and two fully connected layers; The detecting of musical instrument sound using a lightweight neural network based on the audio frame belonging to the foreground sound, and obtaining the musical instrument sound detection result includes: Performing a fast Fourier transform of a preset dimension on the audio frame belonging to the foreground sound to obtain frequency domain features; Preprocessing the frequency domain features to obtain preprocessed frequency domain features; The preprocessed frequency domain features are input into a lightweight neural network, and sequentially pass through the one-dimensional convolutional layer and the two fully connected layers to output instrument sound detection results.

[0013] From the above description, it can be seen that the preprocessed frequency domain features are input into the lightweight neural network, and then pass through a one-dimensional convolution layer and two fully connected layers in sequence to output the instrument sound detection results. The design of the one-dimensional convolution layer can effectively extract the spectral characteristics of the fundamental and harmonic waves of each note of the instrument. Combined with the data after filtering out the background sound and the noise, it can accurately realize the judgment of the type of instrument sound.

[0014] Further, the preprocessing the frequency domain features to obtain the preprocessed frequency domain features includes: Filtering the white noise in the frequency domain feature to obtain the filtered frequency domain feature; Performing feature fusion on the filtered frequency domain features to obtain fused frequency domain features; The fused frequency domain features are normalized to obtain preprocessed frequency domain features.

[0015] From the above description, it can be seen that the white noise in the frequency domain features is filtered out, the filtered frequency domain features are fused, and finally the fused frequency domain features are normalized, which further improves the purity of the audio data subsequently input into the neural network, thereby improving the accuracy of instrument sound detection.

[0016] Furthermore, the performing strong sound detection on the audio stream and extracting valid audio frames with strong sounds includes: Dividing the audio stream according to a preset subframe length to obtain an audio subframe sequence; The short-time energy of each audio subframe in the audio subframe sequence is calculated in turn to determine whether the short-time energy exceeds a preset value. If so, the audio subframe whose short-time energy exceeds the preset value is used as the starting frame, and a preset number of audio subframes starting from the starting frame are spliced ​​to generate a valid audio frame with a strong sound.

[0017] From the above description, it can be seen that the audio subframe whose short-time energy exceeds the preset value is used as the starting frame, and a preset number of audio subframes starting from the starting frame are spliced ​​to generate a valid audio frame with a strong sound. This can accurately locate the position of the valid audio frame, which is beneficial to improving the efficiency and accuracy of instrument sound recognition.

[0018] Further, the performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound includes: Calculating the short-time energy and zero-crossing rate of the valid audio frame; Performing nonlinear transformation on the short-time energy and zero-crossing rate to obtain transformed short-time energy and zero-crossing rate; Audio frames belonging to background sounds are screened out from the valid audio frames according to the transformed short-time energy and zero-crossing rate, and the audio frames belonging to background sounds are removed from the valid audio frames to obtain audio frames belonging to foreground sounds.

[0019] From the above description, it can be seen that a nonlinear transformation is performed on the short-time energy and the zero-crossing rate, and audio frames belonging to the foreground sound are screened out from the valid audio frames based on the transformed short-time energy and the zero-crossing rate. The nonlinear transformation can more accurately screen the audio frames belonging to the foreground sound. Such audio frames are most likely to be musical instrument sounds, which facilitates the subsequent efficient identification of musical instrument sounds.

[0020] Please refer to Figure 2 , a musical instrument sound detection terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Collect the audio stream to be detected; Performing strong sound detection on the audio stream to extract valid audio frames containing strong sounds; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound; Based on the audio frame belonging to the foreground sound, a lightweight neural network is used to detect musical instrument sounds to obtain a musical instrument sound detection result.

[0021] From the above description, it can be seen that the beneficial effects of the present invention are: firstly performing strong sound detection on the collected audio stream, extracting the valid audio frames with strong sounds, performing background sound detection on the valid audio frames, screening out the audio frames belonging to the foreground sound, and using a lightweight neural network to perform instrument sound detection based on the audio frames belonging to the foreground sound to obtain the instrument sound detection result. Through strong sound detection and background sound detection, most of the background sounds can be effectively filtered out. The lightweight neural network has the characteristics of shallow hierarchy and streamlined parameters. The lightweight neural network is then used to detect instrument sounds, thereby improving the accuracy of instrument sound recognition, reducing the complexity of the algorithm, and being easier to deploy on embedded devices with limited computing power.

[0022] Furthermore, the lightweight neural network includes a one-dimensional convolutional layer and two fully connected layers; The detecting of musical instrument sound using a lightweight neural network based on the audio frame belonging to the foreground sound, and obtaining the musical instrument sound detection result includes: Performing a fast Fourier transform of a preset dimension on the audio frame belonging to the foreground sound to obtain frequency domain features; Preprocessing the frequency domain features to obtain preprocessed frequency domain features; The preprocessed frequency domain features are input into a lightweight neural network, and sequentially pass through the one-dimensional convolutional layer and the two fully connected layers to output instrument sound detection results.

[0023] From the above description, it can be seen that the preprocessed frequency domain features are input into the lightweight neural network, and then pass through a one-dimensional convolution layer and two fully connected layers in sequence to output the instrument sound detection results. The design of the one-dimensional convolution layer can effectively extract the spectral characteristics of the fundamental and harmonic waves of each note of the instrument. Combined with the data after filtering out the background sound and the noise, it can accurately realize the judgment of the type of instrument sound.

[0024] Further, the preprocessing the frequency domain features to obtain the preprocessed frequency domain features includes: Filtering the white noise in the frequency domain feature to obtain the filtered frequency domain feature; Performing feature fusion on the filtered frequency domain features to obtain fused frequency domain features; The fused frequency domain features are normalized to obtain preprocessed frequency domain features.

[0025] From the above description, it can be seen that the white noise in the frequency domain features is filtered out, the filtered frequency domain features are fused, and finally the fused frequency domain features are normalized, which further improves the purity of the audio data subsequently input into the neural network, thereby improving the accuracy of instrument sound detection.

[0026] Furthermore, the performing strong sound detection on the audio stream and extracting valid audio frames with strong sounds includes: Dividing the audio stream according to a preset subframe length to obtain an audio subframe sequence; The short-time energy of each audio subframe in the audio subframe sequence is calculated in turn to determine whether the short-time energy exceeds a preset value. If so, the audio subframe whose short-time energy exceeds the preset value is used as the starting frame, and a preset number of audio subframes starting from the starting frame are spliced ​​to generate a valid audio frame with a strong sound.

[0027] From the above description, it can be seen that the audio subframe whose short-time energy exceeds the preset value is used as the starting frame, and a preset number of audio subframes starting from the starting frame are spliced ​​to generate a valid audio frame with a strong sound. This can accurately locate the position of the valid audio frame, which is beneficial to improving the efficiency and accuracy of instrument sound recognition.

[0028] Further, the performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound includes: Calculating the short-time energy and zero-crossing rate of the valid audio frame; Performing nonlinear transformation on the short-time energy and zero-crossing rate to obtain transformed short-time energy and zero-crossing rate; Audio frames belonging to background sounds are screened out from the valid audio frames according to the transformed short-time energy and zero-crossing rate, and the audio frames belonging to background sounds are removed from the valid audio frames to obtain audio frames belonging to foreground sounds.

[0029] From the above description, it can be seen that a nonlinear transformation is performed on the short-time energy and the zero-crossing rate, and audio frames belonging to the foreground sound are screened out from the valid audio frames based on the transformed short-time energy and the zero-crossing rate. The nonlinear transformation can more accurately screen the audio frames belonging to the foreground sound. Such audio frames are most likely to be musical instrument sounds, which facilitates the subsequent efficient identification of musical instrument sounds.

[0030] The above-mentioned instrument sound detection method and terminal of the present invention are deployed on an embedded device running RTOS (Real Time Operating System, real-time operating system), which can be applied to audio analysis scenarios requiring instrument sound detection, and are described below through specific implementation methods: Please refer to Figure 1 , Figure 3-Figure 7 , Embodiment 1 of the present invention is: A method for detecting musical instrument sound comprises the steps of: S1. Collect the audio stream to be detected.

[0031] In an optional implementation, the microphone is set to mono sampling, and the sampling rate is set to no less than 16KHz and the sampling bit width is set to 8bit or 16bit. The set microphone is used to collect the audio stream to be detected to ensure that the data is not distorted.

[0032] S2, performing strong sound detection on the audio stream, extracting valid audio frames with strong sound, such as Figure 3 As shown, specifically including S21-S22: S21. Divide the audio stream according to a preset subframe length to obtain an audio subframe sequence.

[0033] In an optional implementation manner, the preset audio frame length is 2048 samples, the preset subframe length is 32 samples, and thus the preset number is 2048 / 32=64.

[0034] For example, divide the audio stream into a sequence of audio subframes according to 32 samples.

[0035] S22. Calculate the short-time energy of each audio subframe in the audio subframe sequence in turn, and determine whether the short-time energy exceeds a preset value. If so, use the audio subframe whose short-time energy exceeds the preset value as the starting frame, and splice a preset number of audio subframes starting from the starting frame to generate a valid audio frame with a strong sound.

[0036] For example, an audio subframe whose short-time energy exceeds a preset value is used as a start frame, and 64 audio subframes starting from the start frame are spliced ​​to generate a valid audio frame with a strong sound.

[0037] S3, performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound, such as Figure 4 As shown, specifically including S31-S33: S31. Calculate the short-time energy and zero-crossing rate of the valid audio frame.

[0038] S32, performing nonlinear transformation on the short-time energy and zero-crossing rate to obtain transformed short-time energy and zero-crossing rate.

[0039] In an optional implementation, after taking the logarithm of the short-time energy STE, a binary K-order polynomial function is used as a nonlinear function to perform a nonlinear transformation on the short-time energy STE and the zero-crossing rate ZCR to obtain the transformed short-time energy and zero-crossing rate. The order K=2, specifically: ; In the formula, s=log(STE), z=ZCR, the goal is to fit and find the coefficients of the polynomial The value of .

[0040] When fitting the nonlinear function, the weight of the instrument sound positive samples is increased so that the fitting function can improve the recall rate as much as possible. The recall rate Recall = TP / (TP+FN), that is, try to avoid missing the instrument audio frame. Figure 7 As shown, the coarse detection algorithm can effectively filter out most of the background sounds while retaining most of the instrument sounds.

[0041] S33. Filter out audio frames belonging to background sounds from the valid audio frames according to the transformed short-time energy and zero-crossing rate, and remove the audio frames belonging to background sounds from the valid audio frames to obtain audio frames belonging to foreground sounds.

[0042] The valid audio frames of musical instrument sounds and background sounds are used as positive and negative samples. On the two-dimensional plane of ZCR-log (STE), positive and negative sample points are scattered. A nonlinear function curve is fitted as a decision function to divide the positive and negative samples. The nonlinear function curve can roughly but quickly divide the suspected musical instrument sounds and background sounds. Subsequently, the audio frames of suspected musical instrument sounds are input into the neural network for fine detection to determine whether the audio frames are background sounds. This coarse detection-fine detection strategy can effectively filter simple background audio frames and prevent them from being input into the neural network, thereby greatly reducing the recognition time.

[0043] S4, using a lightweight neural network to perform instrument sound detection based on the audio frame belonging to the foreground sound, to obtain an instrument sound detection result, specifically including S41-S43: Among them, Figure 6 As shown, the lightweight neural network includes a one-dimensional convolution layer and two fully connected layers. Different layers are connected by Relu activation function. The output of the fully connected layer 2 is Softmaxed to obtain the scores of each classification. The convolution kernel size of the one-dimensional convolution layer is set to K / 16, K represents the preset dimension, the convolution step size is set to K / 64, the number of output channels is set to 32, the number of output channels of the fully connected layer 1 is set to 32, and the number of output channels of the fully connected layer 2 is the number of classifications.

[0044] S41, performing a fast Fourier transform of a preset dimension on the audio frame belonging to the foreground sound to obtain frequency domain features.

[0045] In an optional implementation, the preset dimension is 2048 dimensions.

[0046] S42, preprocessing the frequency domain features to obtain preprocessed frequency domain features, such as Figure 5 As shown, specifically including S421 -S423: S421. Filter the white noise in the frequency domain feature to obtain the filtered frequency domain feature.

[0047] S422: Perform feature fusion on the filtered frequency domain features to obtain fused frequency domain features.

[0048] S423: normalize the fused frequency domain features to obtain preprocessed frequency domain features.

[0049] S43, input the preprocessed frequency domain features into a lightweight neural network, pass through the one-dimensional convolution layer and the two fully connected layers in sequence, and output the instrument sound detection result.

[0050] Specifically, Figure 6 As shown in the figure, the preprocessed frequency domain features are input into a lightweight neural network, and then pass through a one-dimensional convolutional layer and two fully connected layers in sequence. The output of the fully connected layer 2 is subjected to Softmax to obtain the classification scores, and the instrument sound detection results are output, including which instrument the audio frame belongs to or whether it is background sound.

[0051] Please refer to Figure 2 , Embodiment 2 of the present invention is: A musical instrument sound detection terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the musical instrument sound detection method in the first embodiment is implemented.

[0052] In summary, the present invention provides a musical instrument sound detection method and terminal, which first performs strong sound detection on the collected audio stream, extracts valid audio frames with strong sounds, performs background sound detection on the valid audio frames, screens out audio frames belonging to foreground sounds, and uses a lightweight neural network to perform musical instrument sound detection based on the audio frames belonging to foreground sounds to obtain musical instrument sound detection results. Through strong sound detection and background sound detection, most of the background sounds can be effectively filtered out. The lightweight neural network has the characteristics of shallow hierarchy and streamlined parameters. The lightweight neural network is then used to detect musical instrument sounds, thereby improving the accuracy of musical instrument sound recognition, reducing the complexity of the algorithm, and being easier to use. It is deployed on embedded devices with limited computing power. In addition, the white noise in the frequency domain features is filtered out, the filtered frequency domain features are fused, and finally the fused frequency domain features are normalized, which further improves the purity of the audio data subsequently input into the neural network, thereby improving the accuracy of instrument sound detection. In addition, the short-time energy and zero-crossing rate are nonlinearly transformed, and the audio frames belonging to the foreground sound are screened out from the valid audio frames according to the transformed short-time energy and zero-crossing rate. The nonlinear transformation can more accurately screen the audio frames belonging to the foreground sound. Such audio frames are most likely instrument sounds, which facilitates the subsequent efficient recognition of instrument sounds.

[0053] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for detecting musical instrument sound, characterized in that: Includes steps: Collect the audio stream to be detected; Performing strong sound detection on the audio stream to extract valid audio frames containing strong sounds; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound; Based on the audio frame belonging to the foreground sound, a lightweight neural network is used to detect musical instrument sounds to obtain a musical instrument sound detection result.

2. A musical instrument sound detection method according to claim 1, characterized in that: The lightweight neural network includes a one-dimensional convolutional layer and two fully connected layers; The detecting of musical instrument sound using a lightweight neural network based on the audio frame belonging to the foreground sound, and obtaining the musical instrument sound detection result includes: Performing a fast Fourier transform of a preset dimension on the audio frame belonging to the foreground sound to obtain frequency domain features; Preprocessing the frequency domain features to obtain preprocessed frequency domain features; The preprocessed frequency domain features are input into a lightweight neural network, and sequentially pass through the one-dimensional convolutional layer and the two fully connected layers to output instrument sound detection results.

3. A musical instrument sound detection method according to claim 2, characterized in that: The preprocessing of the frequency domain features to obtain the preprocessed frequency domain features comprises: Filtering the white noise in the frequency domain feature to obtain the filtered frequency domain feature; Performing feature fusion on the filtered frequency domain features to obtain fused frequency domain features; The fused frequency domain features are normalized to obtain preprocessed frequency domain features.

4. A musical instrument sound detection method according to claim 1, characterized in that: The performing strong sound detection on the audio stream and extracting valid audio frames with strong sound comprises: Dividing the audio stream according to a preset subframe length to obtain an audio subframe sequence; The short-time energy of each audio subframe in the audio subframe sequence is calculated in turn to determine whether the short-time energy exceeds a preset value. If so, the audio subframe whose short-time energy exceeds the preset value is used as the starting frame, and a preset number of audio subframes starting from the starting frame are spliced ​​to generate a valid audio frame with a strong sound.

5. A musical instrument sound detection method according to claim 1, characterized in that: The performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound comprises: Calculating the short-time energy and zero-crossing rate of the valid audio frame; Performing nonlinear transformation on the short-time energy and zero-crossing rate to obtain transformed short-time energy and zero-crossing rate; Audio frames belonging to background sounds are screened out from the valid audio frames according to the transformed short-time energy and zero-crossing rate, and the audio frames belonging to background sounds are removed from the valid audio frames to obtain audio frames belonging to foreground sounds.

6. A musical instrument sound detection terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: Collect the audio stream to be detected; Performing strong sound detection on the audio stream to extract valid audio frames containing strong sounds; Performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound; Based on the audio frame belonging to the foreground sound, a lightweight neural network is used to detect musical instrument sounds to obtain a musical instrument sound detection result.

7. A musical instrument sound detection terminal according to claim 6, characterized in that: The lightweight neural network includes a one-dimensional convolutional layer and two fully connected layers; The detecting of musical instrument sound using a lightweight neural network based on the audio frame belonging to the foreground sound, and obtaining the musical instrument sound detection result includes: Performing a fast Fourier transform of a preset dimension on the audio frame belonging to the foreground sound to obtain frequency domain features; Preprocessing the frequency domain features to obtain preprocessed frequency domain features; The preprocessed frequency domain features are input into a lightweight neural network, and sequentially pass through the one-dimensional convolutional layer and the two fully connected layers to output instrument sound detection results.

8. A musical instrument sound detection terminal according to claim 7, characterized in that: The preprocessing of the frequency domain features to obtain the preprocessed frequency domain features comprises: Filtering the white noise in the frequency domain feature to obtain the filtered frequency domain feature; Performing feature fusion on the filtered frequency domain features to obtain fused frequency domain features; The fused frequency domain features are normalized to obtain preprocessed frequency domain features.

9. A musical instrument sound detection terminal according to claim 6, characterized in that: The performing strong sound detection on the audio stream and extracting valid audio frames with strong sound comprises: Dividing the audio stream according to a preset subframe length to obtain an audio subframe sequence; The short-time energy of each audio subframe in the audio subframe sequence is calculated in turn to determine whether the short-time energy exceeds a preset value. If so, the audio subframe whose short-time energy exceeds the preset value is used as the starting frame, and a preset number of audio subframes starting from the starting frame are spliced ​​to generate a valid audio frame with a strong sound.

10. A musical instrument sound detection terminal according to claim 6, characterized in that: The performing background sound detection on the valid audio frame to obtain an audio frame belonging to the background sound, and removing the audio frame belonging to the background sound from the valid audio frame to obtain an audio frame belonging to the foreground sound comprises: Calculating the short-time energy and zero-crossing rate of the valid audio frame; Performing nonlinear transformation on the short-time energy and zero-crossing rate to obtain transformed short-time energy and zero-crossing rate; Audio frames belonging to background sounds are screened out from the valid audio frames according to the transformed short-time energy and zero-crossing rate, and the audio frames belonging to background sounds are removed from the valid audio frames to obtain audio frames belonging to foreground sounds.

Images