Zheng playing technique detection method, equipment, medium and product

Abstract

The invention discloses a Chinese zither playing technique detection method and device, a medium and a product, and relates to the field of playing technique detection, and the method comprises the steps: obtaining playing audio data; performing feature extraction on the playing audio data by adopting a constant q transformation method; inputting the extracted features into a frame-level multi-label classification model to obtain a playing technique prediction result; and determining a playing technique classification result according to the playing technique prediction result. According to the invention, the Chinese zither playing technique can be automatically and accurately identified and classified.

Description

Technical Field

[0001] The present application relates to the field of performance technique detection, and in particular to a guzheng performance technique detection method, equipment, medium and product. Background Art

[0002] With the advancement of deep learning technology, deep neural networks have been increasingly used in the research of playing technique detection. The prior art proposes an end-to-end method based on fully convolutional networks (FCN) to detect the playing techniques of fragments connected by independent erhu notes, which can classify single techniques. Afterwards, a note start detector based on convolutional neural networks (CNN) is used on the basis of the fully convolutional network, and its output is fused with the output prediction of the playing technique detector based on the fully convolutional network to improve the accuracy of frame-level playing technique detection in the audio sequence and output event-level predictions. However, the guzheng can play intervals and chords (multiple notes played at the same time). In the guzheng performance, there will be multiple notes of different playing techniques played at the same time, and there are also cases where a single note has multiple playing techniques. However, the existing playing technique detection research focuses on the playing technique detection of monophonic melody instruments. Therefore, the existing playing technique detection methods are difficult to directly apply to instruments that use overlapping techniques or mixed techniques, such as guzheng or harp.

[0003] In order to solve the above problems, it is urgent to provide a guzheng playing technique detection method so as to automatically and accurately realize the recognition and classification of guzheng playing techniques. Summary of the invention

[0004] The purpose of this application is to provide a guzheng playing technique detection method, equipment, medium and product, which can automatically and accurately realize the recognition and classification of guzheng playing techniques.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for detecting guzheng playing techniques, the method comprising:

[0007] Get the performance audio data;

[0008] The constant q transformation method is used to extract features from the performance audio data;

[0009] The extracted features are input into a frame-level multi-label classification model to obtain a performance technique prediction result; the frame-level multi-label classification model is constructed based on a multi-scale network; the frame-level multi-label classification model includes: a batch normalization module, a first horizontal branch, a second horizontal branch, a third horizontal branch and a classifier; the first horizontal branch and the second horizontal branch include different numbers of residual blocks; the third horizontal branch includes a residual block and a self-attention module; the scales of the first horizontal branch, the second horizontal branch and the third horizontal branch are increased in sequence, and features of different resolutions can be extracted; the horizontal intersections of the first horizontal branch, the second horizontal branch and the third horizontal branch are scaled by a maximum pooling layer and a transposed convolution layer, and then spliced ​​to finally obtain a feature map; the classifier is used to obtain a performance technique prediction result according to the feature map;

[0010] The playing technique classification result is determined according to the playing technique prediction result.

[0011] Optionally, the performance audio data includes: a wav file of the performance audio.

[0012] Optionally, the batch normalization module is used to perform shape change and batch normalization processing on the extracted features.

[0013] Optionally, the convolution kernel size of the transposed convolution layer is 3×1.

[0014] Optionally, the classifier includes: a convolution layer and a sigmoid layer; the convolution kernel size of the convolution layer is 3×1.

[0015] Optionally, the frame-level multi-label classification model adopts weighted binary cross entropy as the loss function.

[0016] Optionally, determining the performance technique classification result according to the performance technique prediction result specifically includes:

[0017] The performance technique prediction results are compared with the technique thresholds respectively;

[0018] If the current performance technique prediction result is greater than or equal to the technique threshold, the performance audio data includes the current performance technique;

[0019] If the current performance technique prediction result is less than the technique threshold, the performance audio data does not include the current performance technique.

[0020] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the Guzheng playing technique detection method.

[0021] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the guzheng playing technique detection method when executed by a processor.

[0022] In a fourth aspect, the present application provides a computer program product, including a computer program, which implements the Guzheng playing technique detection method when executed by a processor.

[0023] According to the specific embodiments provided in this application, this application has the following technical effects:

[0024] The present application provides a method, device, medium and product for detecting guzheng playing techniques. By adopting a frame-level multi-label classification model, it is capable of extracting features from different scales, and adopting a self-attention module on the longest distance scale of the multi-scale network, it is capable of capturing long-distance features well and improving contextual learning capabilities. The playing technique classification results are determined according to the playing technique prediction results, and thus the present application can effectively identify and classify the guzheng techniques of each audio clip, and can be applied to playing technique detection scenarios of different complexities, and is more generalizable. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 This is a flow chart of a method for detecting guzheng playing skills in one embodiment of the present application;

[0027] Figure 2 Schematic diagram of the frame-level multi-label classification model structure. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0030] In an exemplary embodiment, Figure 1As shown, a method for detecting guzheng playing techniques is provided, the method comprising the following S101 to S104. Wherein:

[0031] S101, obtaining performance audio data; the performance audio data includes: a wav file of the performance audio.

[0032] S102, using a constant-Q transform (CQT) method to extract features from the performance audio data;

[0033] The constant q transform (CQT) transforms the data sequence into the frequency domain. The original audio is cut into small segments according to the standard of 3 seconds, and then resampled to 44.1kHz. The last part of each audio less than 3 seconds is padded with zeros; the librosa toolkit is used in this application to calculate the constant q transform (CQT), and the parameters are set as follows: the sampling rate is 44100, the step size is 512, there are 12 frequency blocks in each octave, fmin (minimum frequency) is 27.5Hz, and the number of frequency blocks is 88.

[0034] S103, inputting the extracted features into a frame-level multi-label classification model to obtain a performance technique prediction result; the frame-level multi-label classification model is constructed based on a multi-scale network; the frame-level multi-label classification model includes: a batch normalization module, a first horizontal branch, a second horizontal branch, a third horizontal branch and a classifier; the first horizontal branch and the second horizontal branch include different numbers of residual blocks; the third horizontal branch includes a residual block and a self-attention module; the scales of the first horizontal branch, the second horizontal branch and the third horizontal branch are increased in sequence, and features of different resolutions can be extracted; Figure 2 As shown, the features in the horizontal branch have a high to low resolution from top to bottom, and the second horizontal branch with medium resolution is used as a transition between the fusion of high-resolution features and long-distance features; by downsampling / upsampling the features to different scales, the long-distance features can be repeatedly fused with the high-resolution features;

[0035] The batch normalization module specifically includes: performing shape change and batch normalization processing on the extracted features, wherein the reshape function in the numpy package is used for shape change processing.

[0036] Specifically, a constant Q transform input of shape (1, 88, T) is processed into a sequence of shape (88, T, 1);

[0037] The junctions of the first horizontal branch, the second horizontal branch and the third horizontal branch are scaled by a maximum pooling layer and a 3×1 transposed convolution layer, and the features are remapped from different resolutions to a uniform scale, and connected along the last dimension to obtain a feature map; the shape of the feature map is (88, T, 1); the classifier is used to obtain a performance technique prediction result according to the feature map; the classifier includes: a 3×1 convolution layer and a sigmoid layer; the performance technique prediction result output by the classifier has a shape of (7, T) to indicate the possibility of the existence of each performance technique in each frame; wherein 88 and T represent the length of the frequency axis and time axis of the input;

[0038] Figure 2 The number in the dashed box in the top row of part (a) is the number of output channels of the residual block in the current column. Figure 2 Part (a) of Figure 2 The numbers in brackets in part (b) indicate the shape of the feature map at the corresponding position. Figure 2 The numbers in brackets in part (c) of represent the kernel size of the convolutional layer, where n is equal to the number of the last dimension of the residual block input. N is the number of performance technique types. The multi-scale network constitutes the structure of the overall frame-level multi-label classification model.

[0039] Figure 2 Part (c) is the structure of the residual block. The skip connection in the residual block is to design a deeper model structure. After feature processing, a feature map with a shape of (88, T, 1) is obtained. Then, this feature map passes through a 3×1 convolution layer and a sigmoid layer to output a prediction with a shape of (7, T) to indicate the possibility of each performance technique in each frame.

[0040] like Figure 2 As shown in part (a), the self-attention module is used to capture the interaction between different frames, further enhancing the ability to extract global features. For an input audio sequence X = (x1, x2, ..., x T ), where each The input audio sequence X is converted into queries keys and values Each of their elements is calculated according to formula (1).

[0041]

[0042] are three trainable parameters to calculate the query, key, and value respectively. As shown in formula (2), each element in the output sequence O = (o1, o2, ..., oT ),in, is computed as a weighted sum of the "value" elements, and the result is fed into a softmax computation. The weight of each "value" element is computed by the scaled dot product between the corresponding "query" element and the "key" element.

[0043]

[0044] The output of the self-attention module is reshaped to the same shape as the input of the self-attention block, and then directly added to the input element by element, and then passed through the batch normalization layer to obtain the final output of the self-attention block.

[0045] The frame-level multi-label classification model uses weighted binary cross entropy (BCE) as the loss function.

[0046] S104, determining a performance technique classification result according to the performance technique prediction result.

[0047] S104 specifically includes:

[0048] S41, comparing the performance technique prediction results with the technique thresholds respectively; wherein the technique threshold is 0.5.

[0049] S42, if the current performance technique prediction result is greater than or equal to the technique threshold, the performance audio data includes the current performance technique;

[0050] S43, if the current performance technique prediction result is less than the technique threshold, the performance audio data does not include the current performance technique.

[0051] When the instrument is a guzheng, the guzheng playing technique detection process starts with the input audio, the audio to be detected is cut into 3-second audio clips, and the cut audio clips are subjected to constant Q transformation (CQT) to extract spectral features. Subsequently, the features are converted into fixed-shape features, and the data distribution is standardized by batch normalization to meet the requirements of subsequent models. After that, the preprocessed features are input into the frame-level multi-label classification model, and the short-term and long-term dependencies of the techniques are captured by multi-scale convolution and self-attention mechanisms. The feature map of the frame-level multi-label classification model is processed in the classifier to generate a predicted score for each technique. In the detection stage, it is determined whether the corresponding playing technique is contained in the segment by judging whether the predicted score reaches the technique threshold (such as ≥0.5): if the predicted score exceeds the technique threshold, the segment is marked as containing this playing technique; otherwise, it is marked as not containing this playing technique. Through this application, the guzheng playing technique of each audio segment can be effectively identified and classified.

[0052] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for detecting a guzheng playing technique is implemented.

[0053] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0054] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0056] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0057] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0058] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0059] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0060] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for detecting guzheng playing techniques, characterized in that: The guzheng playing technique detection method comprises: Get the performance audio data; The constant q transformation method is used to extract features from the performance audio data; The extracted features are input into a frame-level multi-label classification model to obtain a performance technique prediction result; the frame-level multi-label classification model is constructed based on a multi-scale network; the frame-level multi-label classification model includes: a batch normalization module, a first horizontal branch, a second horizontal branch, a third horizontal branch and a classifier; the first horizontal branch and the second horizontal branch include different numbers of residual blocks; the third horizontal branch includes a residual block and a self-attention module; the scales of the first horizontal branch, the second horizontal branch and the third horizontal branch are increased in sequence, and features of different resolutions can be extracted; the intersections of the first horizontal branch, the second horizontal branch and the third horizontal branch are scaled by a maximum pooling layer and a transposed convolution layer, and then spliced ​​to finally obtain a feature map; the classifier is used to obtain a performance technique prediction result according to the feature map; The playing technique classification result is determined according to the playing technique prediction result.

2. The guzheng playing technique detection method according to claim 1, characterized in that: The performance audio data includes: a wav file of the performance audio.

3. The guzheng playing technique detection method according to claim 1, characterized in that: The batch normalization module is used to perform shape change and batch normalization processing on the extracted features.

4. The guzheng playing technique detection method according to claim 1, characterized in that: The size of the convolution kernel of the transposed convolution layer is 3×1.

5. The guzheng playing technique detection method according to claim 1, characterized in that: The classifier includes: a convolution layer and a sigmoid layer; the convolution kernel size of the convolution layer is 3×1.

6. The guzheng playing technique detection method according to claim 1, characterized in that: The frame-level multi-label classification model uses weighted binary cross entropy as the loss function.

7. The guzheng playing technique detection method according to claim 1, characterized in that: Determining the performance technique classification result according to the performance technique prediction result specifically includes: The performance technique prediction results are compared with the technique thresholds respectively; If the current performance technique prediction result is greater than or equal to the technique threshold, the performance audio data includes the current performance technique; If the current performance technique prediction result is less than the technique threshold, the performance audio data does not include the current performance technique.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the guzheng playing technique detection method described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the guzheng playing technique detection method described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the guzheng playing technique detection method described in any one of claims 1 to 7 is implemented.

Images