A guzheng playing skill detection method, device, medium and product

Abstract

The application discloses a guzheng playing skill detection method, device, medium and product, relates to the playing skill detection field, and the method comprises the following steps: acquiring playing audio data; adopting a constant q transformation method to perform feature extraction on the playing audio data; inputting the extracted features into a frame level multi-label classification model to obtain a playing skill prediction result; and determining a playing skill classification result according to the playing skill prediction result. The application can automatically and accurately realize the identification and classification of guzheng playing skills.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of playing technique detection, in particular to a guzheng playing technique detection method, device, medium and product. BACKGROUND

[0002] With the progress of deep learning technology, deep neural networks have been increasingly applied to the research work of playing technique detection. The prior art proposes an end-to-end method based on fully convolutional networks (FCN) to detect the playing technique of a segment connected by independent erhu notes, which can classify a single technique. Then, on the basis of the fully convolutional network, a note onset detector based on convolutional neural networks (CNN) is used, and its output is fused with the output prediction of the playing technique detector based on the fully convolutional network to improve the accuracy of frame-level playing technique detection in an audio sequence and output event-level prediction. However, the guzheng can play intervals and chords (multiple notes played at the same time), and there are cases where multiple different playing technique notes are played at the same time in guzheng playing, and there are cases where a single note has multiple playing techniques. However, existing playing technique detection research has focused on playing technique detection of single-tone melody instruments, so the existing playing technique detection method is difficult to directly apply to instruments that use overlapping techniques or mixed techniques, such as guzheng or harp.

[0003] In order to solve the above problems, it is urgent to provide a guzheng playing technique detection method to automatically and accurately realize the identification and classification of guzheng playing techniques. SUMMARY

[0004] The purpose of the present application is to provide a guzheng playing technique detection method, device, medium and product, which can automatically and accurately realize the identification and classification of guzheng playing techniques.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides a guzheng playing technique detection method, which comprises:

[0007] Obtaining playing audio data;

[0008] Performing feature extraction on the playing audio data using a constant q transform method;

[0009] input the extracted features into a frame-level multi-label classification model to obtain a performance technique prediction result; the frame-level multi-label classification model is constructed based on a multi-scale network; the frame-level multi-label classification model comprises a batch normalization module, a first horizontal branch, a second horizontal branch, a third horizontal branch, and a classifier; the first horizontal branch and the second horizontal branch comprise different numbers of residual blocks; the third horizontal branch comprises a residual block and a self-attention module; the first horizontal branch, the second horizontal branch, and the third horizontal branch have scales that increase in turn and are capable of extracting features of different resolutions; the horizontal intersections of the first horizontal branch, the second horizontal branch, and the third horizontal branch are scaled by a max-pooling layer and a transpose convolution layer, and then subjected to splicing processing to finally obtain a feature map; the classifier is configured to obtain the performance technique prediction result according to the feature map.

[0010] A performance technique classification result is determined according to the performance technique prediction result.

[0011] Optionally, the performance audio data comprises a wav file of the performance audio.

[0012] Optionally, the batch normalization module is configured to perform shape changing and batch normalization processing on the extracted features.

[0013] Optionally, the size of the convolution kernel of the transpose convolution layer is 3×1.

[0014] Optionally, the classifier comprises a convolution layer and a sigmoid layer; the size of the convolution kernel of the convolution layer is 3×1.

[0015] Optionally, the frame-level multi-label classification model adopts a weighted binary cross-entropy as a loss function.

[0016] Optionally, the determination of the performance technique classification result according to the performance technique prediction result specifically comprises:

[0017] comparing the performance technique prediction result with a technique threshold value respectively;

[0018] if the current performance technique prediction result is greater than or equal to the technique threshold value, the performance audio data comprises the current performance technique;

[0019] if the current performance technique prediction result is less than the technique threshold value, the performance audio data does not comprise the current performance technique.

[0020] In a second aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the guzheng performance technique detection method.

[0021] In a third aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the guzheng playing technique detection method.

[0022] In a fourth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the guzheng playing technique detection method.

[0023] According to the specific embodiments provided in the present application, the present application has the following technical effects:

[0024] The present application provides a guzheng playing technique detection method, device, medium and product, which can extract features from different scales by using a frame-level multi-label classification model, and can well capture long-distance features and improve the context learning ability by using a self-attention module at the longest distance scale of the multi-scale network; the playing technique classification result is determined according to the playing technique prediction result, and then the present application can effectively identify and classify the guzheng technique of each audio segment, and can be applied to different complexity playing technique detection scenes, and has better generalization. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Figure 1 FIG. 1 is a flowchart of a guzheng playing technique detection method according to an embodiment of the present application;

[0027] Figure 2 FIG. 2 is a structure diagram of a frame-level multi-label classification model according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0029] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0030] In an exemplary embodiment, as shown in FIG. 1, the guzheng playing technique detection method provided by the present application includes the following steps. Figure 1As shown, a guzheng playing technique detection method is provided, which comprises the following S101 to S104. Among them:

[0031] S101, acquiring playing audio data; the playing audio data comprises a wav file of playing audio.

[0032] S102, adopting a constant q transform (CQT) method to perform feature extraction on the playing audio data;

[0033] The constant q transform (CQT) transforms the data sequence into the frequency domain, the original audio is cut into small pieces according to the standard of 3 seconds, and then resampled to 44.1kHz, and the last part of less than 3 seconds of each audio is zero-filled; the constant q transform (CQT) is calculated in the application using the librosa toolkit, and the parameter settings are as follows: the sampling rate is 44100, the step is 512, there are 12 frequency blocks per octave, fmin (minimum frequency) is 27.5Hz, and the number of frequency blocks is 88.

[0034] S103, inputting the extracted features into a frame-level multi-label classification model to obtain playing technique prediction results; the frame-level multi-label classification model is constructed based on a multi-scale network; the frame-level multi-label classification model comprises a batch normalization module, a first horizontal branch, a second horizontal branch, a third horizontal branch and a classifier; the first horizontal branch and the second horizontal branch comprise different numbers of residual blocks; the third horizontal branch comprises a residual block and a self-attention module; the scales of the first horizontal branch, the second horizontal branch and the third horizontal branch increase in turn, and different resolution features can be extracted; as Figure 2 As shown, the resolution of the features in the horizontal branch from top to bottom is from high to low, the second horizontal branch with medium resolution is used as a transition for the fusion between high-resolution features and long-distance features; by down-sampling / up-sampling the features to different scales, long-distance features and high-resolution features can be repeatedly fused;

[0035] The batch normalization module specifically comprises shape changing and batch normalization processing of the extracted features. Among them, the reshape function in the numpy package is used for shape changing processing.

[0036] Specifically, the constant Q transform with a shape of (1, 88, T) is inputted into processing as a sequence with a shape of (88, T, 1);

[0037] The junctions of the first, second and third horizontal branches are scaled by a max-pooling layer and a 3x1 transpose convolution layer, which re-maps the features from different resolutions to a uniform scale and concatenates along the last dimension to get a feature map; the shape of the feature map is (88, T, 1); the classifier is configured to obtain a performance technique prediction result according to the feature map; the classifier comprises a 3x1 convolution layer and a sigmoid layer; the shape of the performance technique prediction result output by the classifier is (7, T), which represents the possibility of the existence of each performance technique in each frame; wherein, 88 and T represent the lengths of the input frequency axis and time axis;

[0038] Figure 2 The numbers in the dashed box in the top row of (a) are the output channel numbers of the residual block in the current column. Figure 2 The numbers in the dashed box in the top row of (a), Figure 2 The numbers in the brackets in (b) represent the shape of the feature map at the corresponding position, Figure 2 The numbers in the brackets in (c) represent the kernel size of the convolution layer, where n is equal to the number of the last dimension of the residual block input. N is the number of performance technique types. The multi-scale network constitutes the structure of the overall frame-level multi-label classification model.

[0039] Figure 2 (c) is the structure of the residual block, and the skip connection in the residual block is used to design a deeper model structure. After feature processing, a feature map with a shape of (88, T, 1) is obtained. Then, the feature map is input into a 3x1 convolution layer and a sigmoid layer, and a prediction with a shape of (7, T) is output, which represents the possibility of the existence of each performance technique in each frame.

[0040] As shown in the part (a) of the application, Figure 2 The self-attention module is used to capture the interaction between different frames, which further enhances the extraction ability of global features. For an input audio sequence X = (x1, x2, …, xT) with a length of T, where each T The input audio sequence X is converted into queries keys and values Each element of them is calculated according to formula (1).

[0041]

[0042] There are 3 trainable parameters to calculate the "query", "key" and "value" respectively. As shown in formula (2), each element O = (o1, o2, …, oT) of the output sequence is calculated according to formula (3).​T ), wherein, is computed as a weighted sum of the "value" elements, and the result is input into a softmax computation. The weight of each "value" element is computed by a scaled dot product between the corresponding "query" element and "key" element.

[0043]

[0044] The output of the self-attention module is shape-adjusted to the same shape as the input of the self-attention block, and then directly added element-wise with the input, and then passed through a batch normalization layer to obtain the final output of the self-attention block.

[0045] The frame-level multi-label classification model adopts weighted binary cross entropy (BCE) as a loss function.

[0046] S104, determining a performance technique classification result according to the performance technique prediction result.

[0047] S104 specifically includes:

[0048] S41, comparing the performance technique prediction result with a technique threshold value respectively; wherein the technique threshold value is 0.5.

[0049] S42, if the current performance technique prediction result is greater than or equal to the technique threshold value, the performance audio data includes the current performance technique;

[0050] S43, if the current performance technique prediction result is less than the technique threshold value, the performance audio data does not include the current performance technique.

[0051] When the musical instrument is a zither, the zither performance technique detection process starts from input audio, and the audio to be detected is cut into 3-second long audio segments. Constant Q transform (CQT) is performed on the cut audio segments to extract spectral features. Then, the features are converted into fixed shape features, and the data distribution is standardized by batch normalization to adapt to the requirements of the subsequent model. After that, the preprocessed features are input into the frame-level multi-label classification model, and the short-time and long-time dependence of the technique is captured through multi-scale convolution and self-attention mechanism. The feature map of the frame-level multi-label classification model is processed in the classifier to generate a prediction score for each technique. In the detection stage, it is determined whether the corresponding performance technique is contained in the segment by judging whether the prediction score reaches the technique threshold value (such as ≥0.5): if the prediction score exceeds the technique threshold value, it is marked that the segment contains this performance technique; otherwise, it is marked as not containing this performance technique. Through the present application, the zither performance technique of each audio segment can be effectively identified and classified.

[0052] In an example embodiment, a computer device is provided, which can be a server or a terminal. The computer device comprises a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a guzheng playing skill detection method.

[0053] In an example embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0054] In an example embodiment, a computer program product is provided, which comprises a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the related data need to comply with relevant regulations.

[0056] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments of each method. Any reference to memory, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0057] The database involved in each embodiment provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in each embodiment provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0058] In the present application, all actions of obtaining signals, information or data are performed in accordance with the corresponding data protection regulations and policies of the country where the device is located, and with the authorization of the owner of the corresponding device.

[0059] Each technical feature of the above embodiments can be combined arbitrarily. In order to make the description simple, not all possible combinations of each technical feature in the above embodiments are described, but as long as the combination of these technical features does not exist, it should be considered as the scope of the present application.

[0060] The principles and implementation manners of the present application are described herein by using specific examples, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will have changes. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A guzheng playing technique detection method, characterized in that, The guzheng playing technique detection method comprises: obtaining playing audio data; extracting features from the playing audio data using a constant q transform method; inputting the extracted features into a frame-level multi-label classification model to obtain a playing technique prediction result; the frame-level multi-label classification model is constructed based on a multi-scale network; the frame-level multi-label classification model comprises a batch normalization module, a first horizontal branch, a second horizontal branch, a third horizontal branch, and a classifier; the first horizontal branch and the second horizontal branch comprise different numbers of residual blocks; the third horizontal branch comprises a residual block and a self-attention module; the scales of the first horizontal branch, the second horizontal branch, and the third horizontal branch increase in turn and can extract features of different resolutions; the intersection of the first horizontal branch, the second horizontal branch, and the third horizontal branch is scaled through a max-pooling layer and a transpose convolution layer, and then subjected to splicing processing to finally obtain a feature map; the classifier is configured to obtain the playing technique prediction result according to the feature map; determining a playing technique classification result according to the playing technique prediction result.

2. The guzheng playing technique detection method according to claim 1, characterized in that, The playing audio data comprises a wav file of the playing audio.

3. The guzheng playing technique detection method according to claim 1, characterized in that, The batch normalization module is configured to perform shape changing and batch normalization processing on the extracted features.

4. The guzheng playing technique detection method according to claim 1, characterized in that, The size of the convolution kernel of the transpose convolution layer is 3x1.

5. The guzheng playing technique detection method according to claim 1, characterized in that, The classifier comprises a convolution layer and a sigmoid layer; the size of the convolution kernel of the convolution layer is 3x1.

6. The guzheng playing technique detection method according to claim 1, characterized in that, The frame-level multi-label classification model uses weighted binary cross-entropy as a loss function.

7. The guzheng playing technique detection method according to claim 1, characterized in that, The determination of the playing technique classification result according to the playing technique prediction result specifically comprises: comparing the playing technique prediction result with a technique threshold value respectively; if the current playing technique prediction result is greater than or equal to the technique threshold value, the playing audio data comprises the current playing technique; if the current playing technique prediction result is less than the technique threshold value, the playing audio data does not comprise the current playing technique.

8. A computer device comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the guzheng playing technique detection method of any one of claims 1-7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the guzheng playing technique detection method of any one of claims 1-7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the guzheng playing technique detection method of any one of claims 1-7.

Images