Music transcription model training method and device, music transcription method and device and electronic equipment

Abstract

The invention discloses a music transcription model training method and device, a music transcription method and device, and electronic equipment, and the method specifically comprises the steps: obtaining sample audio data corresponding to sample music, a first annotation pitch sequence corresponding to first audio data in the sample audio data, and annotation music score data corresponding to second audio data in the sample audio data, the annotated music score data comprises a second annotated pitch sequence and annotated frame data; based on the sample audio data, performing self-supervised audio representation training on the to-be-trained music transcription model to obtain a first music transcription model; performing music transcription training on the first music transcription model based on the first audio data and the first annotation pitch sequence to obtain a second music transcription model; and based on the second audio data, the second annotation pitch sequence and the annotation frame data, performing music transcription training on the second music transcription model to obtain a target music transcription model. According to the technical scheme, the transcription performance of the music transcription model can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a music transcription model training, a music transcription method, a device and an electronic device. Background Art

[0002] Existing music transcription models usually use strongly labeled music audio data directly for music transcription training. This solution works well in some simple cases, such as sight singing transcription, children's song transcription (without very complicated singing skills), and slow song transcription (music BPM is below 60).

[0003] However, in complex situations (the singer's singing skills are too rich or the song's tempo is too fast), the accuracy of the transcribed music score obtained by this solution is very low. For example, when transcribing a fast song, it is easy to transcribe multiple consecutive words into one note. When transcribing a tremolo, it is easy to transcribe one word into multiple notes of different pitches. Therefore, a more accurate technical solution needs to be provided. Summary of the invention

[0004] The present application provides a music transcription model training, a music transcription method, a device and an electronic device, which can improve the transcription accuracy of the music transcription model by using a multi-stage model training technology and different levels of labeled data. The technical solution of the present application is as follows:

[0005] In one aspect, a music transcription model training method is provided, the method comprising:

[0006] Acquire sample audio data corresponding to the sample music, a first annotated pitch sequence corresponding to the first audio data in the sample audio data, and annotated music score data corresponding to the second audio data in the sample audio data, wherein the annotated music score data includes: a second annotated pitch sequence and annotated frame data;

[0007] Based on the sample audio data, performing self-supervised audio representation training on the music transcription model to be trained to obtain a first music transcription model;

[0008] Based on the first audio data and the first annotated pitch sequence, performing music transcription training on the first music transcription model to obtain a second music transcription model;

[0009] Based on the second audio data, the second annotated pitch sequence and the annotated frame data, music transcription training is performed on the second music transcription model to obtain a target music transcription model.

[0010] In another aspect, a music transcription method is provided, the method comprising:

[0011] Obtain audio data corresponding to the music to be transcribed;

[0012] Inputting the audio data into a music transcription model for music transcription processing to obtain music score data corresponding to the audio data;

[0013] Wherein, the music transcription model is obtained after training based on the music transcription model training method as described above.

[0014] In another aspect, a music transcription model training device is provided, the device comprising:

[0015] A sample acquisition module is used to acquire sample audio data corresponding to sample music, a first annotated pitch sequence corresponding to first audio data in the sample audio data, and annotated music score data corresponding to second audio data in the sample audio data, wherein the annotated music score data includes: a second annotated pitch sequence and annotated frame data;

[0016] A first model training module is used to perform self-supervised audio representation training on the music transcription model to be trained based on the sample audio data to obtain a first music transcription model;

[0017] A second model training module is used to perform music transcription training on the first music transcription model based on the first audio data and the first annotated pitch sequence to obtain a second music transcription model;

[0018] The third model training module is used to perform music transcription training on the second music transcription model based on the second audio data, the second annotated pitch sequence and the annotated frame data to obtain a target music transcription model.

[0019] In another aspect, a music transcription device is provided, the device comprising:

[0020] An audio data acquisition module, used to acquire audio data corresponding to the music to be transcribed;

[0021] A music transcription module, used for inputting the audio data into a music transcription model for music transcription processing to obtain music score data corresponding to the audio data;

[0022] Wherein, the music transcription model is obtained after training based on the music transcription model training device as mentioned above.

[0023] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the music transcription model training method or the music transcription method as described above.

[0024] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the music transcription model training method or the music transcription method as described above.

[0025] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the music transcription model training method or the music transcription method as described above.

[0026] The present application provides a music transcription model training, music transcription method, device and electronic device, which have the following technical effects:

[0027] The present application uses a multi-stage model training technology and different levels of labeled data to perform music transcription training. First, based on the sample audio data corresponding to the sample music, self-supervised audio representation training is performed on the music transcription model to be trained to obtain a first music transcription model, so that the first music transcription model learns the high-quality representation of the sample audio data, thereby having good audio feature representation capabilities. Then, based on the first audio data in the sample audio data and the first labeled pitch sequence corresponding to the first audio data, weakly labeled music transcription training is performed on the first music transcription model to obtain a second music transcription model, so that the second music transcription model can be familiar with the music transcription task and initially have music transcription capabilities. Finally, based on the second audio data in the sample audio data, the second labeled pitch sequence corresponding to the second audio data, and the labeled frame data corresponding to the second audio data, strongly labeled music transcription training is performed on the second music transcription model to obtain a target music transcription model, which can further improve the transcription accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

[0030] Figure 2 It is a flowchart of a music transcription model training method provided in an embodiment of the present application;

[0031] Figure 3 It is a flowchart of performing self-supervised audio representation training on a music transcription model to be trained based on sample audio data to obtain a first music transcription model provided by an embodiment of the present application;

[0032] Figure 4 is a schematic diagram of a self-supervised model audio representation training provided in an embodiment of the present application;

[0033] Figure 5 It is a flowchart of performing music transcription training on a first music transcription model based on first audio data and a first annotated pitch sequence to obtain a second music transcription model, provided by an embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of a weakly labeled model music transcription training provided by this application;

[0035] Figure 7 It is a flowchart of performing music transcription training on a second music transcription model based on second audio data, a second annotated pitch sequence and annotated frame data to obtain a target music transcription model, provided by an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of a process for determining frame loss information based on labeled data and predicted data provided by an embodiment of the present application;

[0037] Fig. 9 It is a schematic diagram of a strongly labeled model music transcription training provided in an embodiment of the present application;

[0038] Fig.10 It is a flowchart of a music transcription method provided in an embodiment of the present application;

[0039] Fig.11 It is a block diagram of a music transcription model training device provided in an embodiment of the present application;

[0040] Fig.12 It is a block diagram of a music transcription device provided in an embodiment of the present application;

[0041] Fig.13 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0043] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0045] To facilitate understanding of the embodiments of the present application, several concepts are briefly introduced below:

[0046] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0047] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0048] Deep learning is a method of characterizing and learning data through artificial neural networks. Different from the traditional method based on prior knowledge and rule building, it is a data-driven algorithm that automatically mines deep Robust features through learning massive data, thus forming an automatic calculation process from input to output without human intervention.

[0049] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0050] See also Figure 1 , Figure 1 It is a schematic diagram of an application environment provided by an embodiment of the present application, and the application environment may include a client 10 and a server 20, and the client 10 and the server 20 may be indirectly connected via wireless communication. The client 10 sends a music transcription model training instruction to the server 20, and the server 20 responds to the music transcription model training instruction to obtain sample audio data corresponding to the sample music, a first annotated pitch sequence corresponding to the first audio data in the sample audio data, and annotated music score data corresponding to the second audio data in the sample audio data, and the annotated music score data may include: a second annotated pitch sequence and annotated frame data, and then based on the sample audio data, a self-supervised audio representation training is performed on the music transcription model to be trained to obtain a first music transcription model, and then based on the first audio data and the first annotated pitch sequence, music transcription training is performed on the first music transcription model to obtain a second music transcription model, and then based on the second audio data, the second annotated pitch sequence and the annotated frame data, music transcription training is performed on the second music transcription model to obtain a target music transcription model, and the target music transcription model is sent to the client 10. It should be noted that Figure 1 Just an example.

[0051] The client can be a physical device such as a smart phone, a computer (such as a desktop computer, a tablet computer, a laptop computer), a digital assistant, an intelligent voice interaction device (such as an intelligent speaker), an intelligent wearable device, a vehicle-mounted terminal, etc., or it can be software running in a physical device, such as a computer program. The operating system corresponding to the first client can be an Android system (Android system), an iOS system (a mobile operating system developed by Apple), a Linux system (an operating system), a Microsoft Windows system (Microsoft Windows operating system), etc.

[0052] The server side can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can include a network communication unit, a processor, and a memory, etc. The server side can provide background services for the corresponding client.

[0053] The client 10 and the server 20 can be used to build a system for music transcription model training, which can be a distributed system. Taking the distributed system as a blockchain system as an example, it is formed by multiple nodes (any form of computing devices in the access network, such as servers, user terminals) and clients, and a peer-to-peer (P2P) network is formed between the nodes. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. In a distributed system, any machine such as a server or a terminal can join and become a node, and the node includes a hardware layer, an intermediate layer, an operating system layer, and an application layer.

[0054] The functions of each node in the above blockchain system include:

[0055] 1) Routing: a basic function of a node, used to support communication between nodes.

[0056] In addition to the routing function, the node can also have the following functions:

[0057] 2) Applications are deployed in the blockchain to implement specific businesses based on actual business needs, record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system for other nodes to add the record data to a temporary block when they successfully verify the source and integrity of the record data.

[0058] 3) Blockchain, including a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.

[0059] It should be noted that the music transcription model training method provided in the present application can be applied on both the client side and the server side, and is not limited to the embodiments of the above-mentioned application environment.

[0060] The following describes a specific embodiment of a music transcription model training method provided by the present application. Figure 2 It is a flowchart of a music transcription model training method provided in an embodiment of the present application. The present application provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment). Specifically, as Figure 2 As shown, the method may include:

[0061] S201, obtaining sample audio data corresponding to the sample music, a first annotated pitch sequence corresponding to the first audio data in the sample audio data, and annotated music score data corresponding to the second audio data in the sample audio data, wherein the annotated music score data includes: a second annotated pitch sequence and annotated frame data.

[0062] In an embodiment of the present specification, the sample audio data corresponding to the sample music may be training data applied to the training of the music transcription model. In a specific embodiment, the data format of the sample audio data may be a waveform file.

[0063] Specifically, the sample music may include: sample songs, sample music and sample singing. Accordingly, the sample audio data corresponding to the sample songs can be applied to the training of the song transcription model, the sample audio data corresponding to the sample music can be applied to the training of the music transcription model, and the sample audio data corresponding to the sample singing can be applied to the training of the singing transcription model.

[0064] In the embodiment of the present specification, the annotated data of the sample audio data may be annotated music score data. Specifically, the annotated music score data may include: annotated pitch sequence and annotated frame data.

[0065] Specifically, the annotated pitch sequence may be a pitch sequence obtained by pitch annotating a sample frame sequence of sample audio data, wherein the sample frame sequence may be a frame sequence obtained by audio frame sampling of the sample audio data, and optionally, the sampling rate of the audio frame may be 44.1kHz.

[0066] Specifically, the annotated frame data may be annotated data obtained by annotating multiple sample frames in the sample frame sequence, and the annotated data of each sample frame includes: pitch annotated data, note start position annotated data, and note end position annotated data.

[0067] In an embodiment of the present specification, a weakly labeled sample set and a strongly labeled sample set can be divided from the sample audio data. Specifically, the weakly labeled sample set can be sample audio data whose corresponding labeled data is weakly labeled data, and the strongly labeled sample set can be sample audio data whose corresponding labeled data is strongly labeled data. In an optional embodiment, the number of samples in the weakly labeled sample set is greater than the number of samples in the strongly labeled sample set.

[0068] In a specific embodiment, the first audio data may be sample audio data in a weakly labeled sample set, and the labeled data of the first audio data may include: a first labeled pitch sequence; the second audio data may be sample audio data in a strongly labeled sample set, and the labeled data of the second audio data may include: a second labeled pitch sequence and labeled frame data.

[0069] S202, based on the sample audio data, performing self-supervised audio representation training on the music transcription model to be trained to obtain a first music transcription model.

[0070] In practical applications, the self-supervised audio representation training of the music transcription model to be trained requires large-scale sample audio data. Optionally, the sample audio data can come from existing audio datasets in the industry, such as the MIR-ST500 dataset for singing transcription and the DALI dataset for lyrics transcription. It can also come from a large number of song waveform files on major music platforms (for example, QQ Music, Youtube, etc.). A large number of audio files can also be automatically generated using a music synthesis model.

[0071] In an embodiment of the present specification, the model structure of the music transcription model to be trained may include: a first audio feature extraction layer, a first audio feature quantization layer, a first context feature fusion layer and a first prediction layer.

[0072] In a specific embodiment, the first audio feature extraction layer can be used to extract potential features of sample audio data. Specifically, the first audio feature extraction layer can use a feature encoder, and the input of the feature encoder can be an audio waveform obtained after normalization. Specifically, the audio waveform here can be a waveform with a mean of 0 and a variance of 1.

[0073] In a specific embodiment, the first audio feature extraction layer may include: a temporal convolution layer, a layer normalization layer and an activation layer. Optionally, the activation layer may use a GELU (Gaussian error linear units) activation function.

[0074] In a specific embodiment, the first audio feature quantization layer may be used to convert continuous audio features into a set of discrete audio feature sequences. Specifically, the first audio feature quantization layer may perform feature discretization processing through a product quantization algorithm.

[0075] In a specific embodiment, the first context feature fusion layer can be used to perform context fusion processing on the potential features of the sample audio data. Specifically, the first context feature fusion layer can adopt a Transformer model, in which relative position encoding can be performed using a convolutional approach.

[0076] In a specific embodiment, the first prediction layer can be used to predict the pitch, the note start position and the note end position. Specifically, the model structure of the first prediction layer may include: a first pitch prediction layer, a first note start position prediction layer and a first note end position prediction layer. Optionally, the first prediction layer may be a linear prediction layer.

[0077] In an embodiment of the present specification, in the self-supervised audio representation training stage, the first audio feature extraction layer, the first audio feature quantization layer and the first context feature fusion layer in the music transcription model to be trained can be trained, and the first music transcription model can be obtained based on the audio feature extraction layer after self-supervised training, the context feature fusion layer after self-supervised training and the original first prediction layer. Specifically, the self-supervised audio representation training can extract common features of a large amount of sample audio data, so that the audio feature extraction layer after self-supervised training and the context feature fusion layer after self-supervised training can learn better audio feature representations, which is beneficial to improving the generalization ability of the model and the accuracy of downstream music transcription tasks.

[0078] In a specific embodiment, the music transcription model to be trained may include: a first audio feature extraction layer, a first audio feature quantization layer and a first context feature fusion layer, such as Figure 3 As shown, the self-supervised audio representation training of the music transcription model to be trained based on the sample audio data to obtain the first music transcription model may include:

[0079] S301, inputting sample audio data into a first audio feature extraction layer to extract audio features and obtain sample audio features.

[0080] Specifically, the sample audio feature may be feature information obtained after extracting potential audio features from the sample audio data. In a specific embodiment, the sample audio feature may be expressed in the form of a feature vector.

[0081] S302: Input the sample audio features into the first audio feature quantization layer for feature discretization processing to obtain the sample audio quantization features.

[0082] Specifically, the sample audio quantization feature may be feature information obtained by performing feature discretization processing on the sample audio feature. In a specific embodiment, the sample audio quantization feature may be expressed in the form of a feature vector.

[0083] Specifically, the first audio feature quantization layer is used to perform vector space decomposition of sample audio features, and use the product quantization codebook and Gumbel Softmax function (Gumbel activation function) to quantize several low-dimensional vector spaces obtained by decomposition, aiming to cluster the same audio features, thereby converting continuous sample audio features into discrete feature sequences. Correspondingly, the sample audio quantization features can be a set of discrete audio feature sequences.

[0084] S303: Input the sample audio features into the first context feature fusion layer to perform context feature fusion to obtain sample audio context features.

[0085] Specifically, the sample audio context feature may be feature information obtained by fusing the sample audio feature with the context feature. In a specific embodiment, the sample audio context feature may be expressed in the form of a feature vector.

[0086] In a specific embodiment, the sample audio features may be partially masked randomly and then input into the first context feature fusion layer for context feature fusion to obtain the sample audio context features.

[0087] S304: Determine audio feature loss information based on the sample audio quantization feature and the sample audio context feature.

[0088] Specifically, the audio feature loss information may characterize the difference between the sample audio quantization feature and the sample audio context feature. In a specific embodiment, the audio feature loss information may include the feature representation difference between the sample audio quantization feature vector and the sample audio context feature vector. Accordingly, the above-mentioned determination of the audio feature loss information based on the sample audio quantization feature and the sample audio context feature may include determining the feature representation difference between the sample audio quantization feature vector and the sample audio context feature vector based on a preset audio feature loss function.

[0089] In a specific embodiment, the preset audio feature loss function may include: a preset contrastive learning loss function. Specifically, the preset contrastive learning loss function may be shown in the following formula 1:

[0090]

[0091] Where t represents the mask time step, c t Represents the sample audio feature z tThe sample audio context feature, q, is partially masked and input to the output of the first context feature fusion layer. t Represents the sample audio feature z t The sample audio quantization features at the same mask time step (i.e., the sample audio features z t The sample audio quantization feature outputted by the first audio feature quantization layer), k represents a non-negative temperature coefficient, z t Represents a set of audio quantization features including positive sample features and negative sample features. The positive sample feature is the sample audio quantization feature q t , the negative sample features are the sample audio quantization features corresponding to the k other mask time steps except the mask time step t, Represents the audio quantization feature set Q t A sample audio quantization feature in , sim(.) represents cosine similarity.

[0092] Specifically, by presetting the contrastive learning loss function, c t and q t The similarity is as large as possible t and Q t The similarity of other sample audio quantization features in , so that the music transcription model to be trained can identify the real sample audio quantization feature q at mask time step t in a set of audio quantization features. t .

[0093] In an optional embodiment, the codebook diversity loss information may also be determined based on quantization codebooks corresponding to each of the multiple vector spaces used by the first audio feature quantization layer.

[0094] Specifically, the codebook diversity loss information may characterize the entropy of the quantization codebooks corresponding to each of the multiple vector spaces. In a specific embodiment, the quantization codebooks corresponding to each of the multiple vector spaces used in the first audio feature quantization layer may include determining the entropy of the quantization codebooks corresponding to each of the multiple vector spaces based on a preset codebook diversity loss function.

[0095] In a specific embodiment, the preset codebook diversity loss function may be shown in the following formula 2:

[0096]

[0097] Among them, g represents the number of vector spaces, v is the number of entries in the quantization codebook corresponding to each vector space, and H(.) represents the information entropy. represents the probability of selecting the vth codebook entry in the gth vector space.

[0098] Specifically, the preset codebook diversity loss function is used to make the quantization codebook entries as diverse as possible, that is, The entropy is maximized, so that the music transcription model to be trained can use the quantized codebook as evenly as possible.

[0099] S305: Based on the audio feature loss information, train the music transcription model to be trained to obtain a first music transcription model.

[0100] In a specific embodiment, training the music transcription model to be trained based on the audio feature loss information to obtain the first music transcription model may include:

[0101] S3051, based on the audio feature loss information, updating the model parameters of the first audio feature extraction layer, the model parameters of the first audio feature quantization layer, and the model parameters of the first context feature fusion layer;

[0102] S3052, based on the updated first audio feature extraction layer, the updated first audio feature quantization layer and the updated first context feature fusion layer, repeating the self-supervised training iteration operation including steps S301-S304 and S3051 until the audio representation convergence condition is reached;

[0103] S3053, using the first audio feature extraction layer obtained when the audio representation convergence condition is met as the second audio feature extraction layer, and using the first context feature fusion layer obtained when the audio representation convergence condition is met as the second context feature fusion layer;

[0104] S3054, obtaining a first music transcription model according to the second audio feature extraction layer, the second context feature fusion layer and the first prediction layer.

[0105] In an optional embodiment, the above-mentioned audio representation convergence condition can be that the number of training iteration operations reaches a preset number of training times. Optionally, the audio representation convergence condition can also be that the audio feature loss information on the validation set no longer decreases. In the embodiments of this specification, the preset number of training times can be pre-set in combination with the training speed and accuracy of the network in actual applications.

[0106] In an optional embodiment, the audio feature loss information and the codebook diversity loss information may be fused to obtain fused loss information, and based on the fused loss information, the music transcription model to be trained may be trained to obtain a first music transcription model.

[0107] Specifically, the fusion loss information L = L m +L d , where α represents a hyperparameter. In practical applications, it can be calculated based on L m and L dThe order of magnitude relationship is used to set the hyperparameter.

[0108] Specifically, here, based on the fusion loss information, the music transcription model to be trained is trained to obtain the self-supervised training process of the first music transcription model, which is similar to the self-supervised training process of the above-mentioned step S305 "based on the audio feature loss information, the music transcription model to be trained is trained to obtain the first music transcription model". Specifically, here, based on the fusion loss information, the music transcription model to be trained is trained to obtain the detailed steps of the first music transcription model. Please refer to steps S3051 to S3054, which will not be repeated here.

[0109] See also Figure 4 , Figure 4 This is a schematic diagram of a self-supervised audio representation training provided by the present application. Specifically, the self-supervised audio representation training can be a loop iterative training. The steps performed during a single loop iterative training process are as follows: 1) Input the audio waveform into the first audio feature extraction layer to extract sample audio features; 2) Input the sample audio features into the first audio feature quantization layer, perform a product quantization operation, and obtain the sample audio quantization features; 3) Randomly perform partial masking on the sample audio features obtained in step 1), then input them into the first context feature fusion layer and output the sample audio context features; 4) Construct fusion loss information based on the sample audio quantization features and the sample audio context features; 5) Update parameters through back propagation.

[0110] It can be seen from the above embodiments that audio representation training is performed through self-supervised learning to extract common features of sample audio data; thereby enabling the music transcription model to be trained to learn better audio feature representations, which is beneficial to improving the generalization ability of the model and the accuracy of downstream music transcription tasks.

[0111] S203: Perform music transcription training on the first music transcription model based on the first audio data and the first annotated pitch sequence to obtain a second music transcription model.

[0112] In practical applications, the music transcription training of the first music transcription model is model fine-tuning training. The training data used in this stage is weakly labeled first audio data. Optionally, some audio data labeled with pitch sequences can be randomly selected from the sample audio data as the first audio data.

[0113] In an embodiment of the present specification, the model structure of the first music transcription model may include: a second audio feature extraction layer (i.e., the audio feature extraction layer after self-supervised training), a second context feature fusion layer (i.e., the context feature fusion layer after self-supervised training) and a first prediction layer, wherein the first prediction layer may include: a first pitch prediction layer, a first note starting position prediction layer and a first note ending position prediction layer.

[0114] In a specific embodiment, the first music transcription model may include: a first pitch prediction layer, such as Figure 5 As shown, the above-mentioned performing music transcription training on the first music transcription model based on the first audio data and the first annotated pitch sequence to obtain the second music transcription model may include:

[0115] S501, inputting first audio data into a first music transcription model, performing pitch prediction based on a first pitch prediction layer, and obtaining a first predicted pitch sequence.

[0116] Specifically, the first predicted pitch sequence may be a predicted pitch sequence generated based on the predicted pitch distribution corresponding to the sample frame sequence of the first audio data, wherein the sample frame sequence may be a frame sequence obtained by sampling audio frames of the first audio data, and the predicted pitch distribution corresponding to the sample frame sequence may include the corresponding predicted pitch distributions of each of the multiple sample frames in the sample frame sequence. Optionally, the sampling rate of the audio frame may be 44.1kHz.

[0117] In a specific embodiment, the first predicted pitch sequence may include: at least one predicted pitch sequence generated based on the predicted pitch distribution corresponding to the sample frame sequence of the first audio data.

[0118] In a specific embodiment, the above-mentioned inputting the first audio data into the first music transcription model, performing pitch prediction based on the first pitch prediction layer, and obtaining a first predicted pitch sequence may include: inputting the first audio data into the second audio feature extraction layer in the first music transcription model to extract audio features and obtain first audio features; inputting the first audio features into the second context feature fusion layer in the first music transcription model to perform context feature fusion and obtain first audio context features; inputting the first audio context features into the first pitch prediction layer in the first music transcription model to perform pitch prediction and obtain a first predicted pitch sequence.

[0119] S502: Determine first sequence loss information based on the first predicted pitch sequence and the first marked pitch sequence.

[0120] Specifically, the first sequence loss information may characterize the sequence difference between the first predicted pitch sequence and the first annotated pitch sequence. In a specific embodiment, the determining of the first sequence loss information based on the first predicted pitch sequence and the first annotated pitch sequence may include determining the sequence difference between the first predicted pitch sequence and the first annotated pitch sequence based on a first preset sequence loss function.

[0121] In a specific embodiment, the first preset sequence loss function may be a CTC (Connectionist Temporal Classification) loss function. Specifically, the CTC loss function may be as shown in the following formula 3:

[0122] L ctc =∑ (X,Y)∈D -logp(Y|X) (Formula 3),

[0123] Wherein, D represents the weakly labeled sample set of the first audio data, X represents the input sequence corresponding to the first audio data, Y represents the first labeled pitch sequence corresponding to the input sequence X, and p(Y|X) is the conditional probability of generating the corresponding first labeled pitch sequence Y based on the input sequence X.

[0124] In a specific embodiment, p(Y|X)=∑π∈φ(Y′)p(π|X), where φ(Y′) represents the path sequence of all possible components of Y in at least one predicted pitch sequence output by the first music transcription model.

[0125] Specifically, the goal of the CTC loss function here is to maximize the path probability and the first annotated pitch sequence output by the first music transcription model. The first audio context feature is input to the first pitch prediction layer to obtain the prediction of the peak value. The prediction result Y′ of the peak value does not require alignment of each frame, but is consistent in the path (i.e., the result of the pitch sequence). In other words, as long as the predicted pitch result is correct, it does not need to be aligned on each frame.

[0126] S503: Based on the first sequence loss information, train the first music transcription model to obtain the second music transcription model.

[0127] In a specific embodiment, based on the first sequence loss information, training the first music transcription model to obtain the second music transcription model may include:

[0128] S5031, based on the first sequence loss information, updating the model parameters of the second audio feature extraction layer, the model parameters of the second context feature fusion layer, and the model parameters of the first pitch prediction layer;

[0129] S5032, based on the updated second audio feature extraction layer, the updated second context feature fusion layer and the updated first pitch prediction layer, repeat the weak labeling training iteration operation including steps S501, S502 and S5031 until the first music transcription convergence condition is reached;

[0130] S5033, using the second audio feature extraction layer obtained when the first music transcription convergence condition is met as the third audio feature extraction layer, using the second context feature fusion layer obtained when the first music transcription convergence condition is met as the third context feature fusion layer, and using the first pitch prediction layer obtained when the first music transcription convergence condition is met as the second pitch prediction layer;

[0131] S5034, obtaining a second music transcription model according to the third audio feature extraction layer, the third context feature fusion layer, the second pitch prediction layer, the first note start position prediction layer and the first note end position prediction layer.

[0132] In an optional embodiment, the first music transcription convergence condition may be achieved when the number of training iterations reaches a preset number of training times. Optionally, the first music transcription convergence condition may also be achieved when the first sequence loss information on the validation set no longer decreases. In the embodiments of this specification, the preset number of training times may be preset in combination with the training speed and accuracy of the network in actual applications.

[0133] See also Figure 6 , Figure 6 It is a schematic diagram of a weakly labeled music transcription training provided by the present application. Specifically, based on the first music transcription model obtained after self-supervised training and the weakly labeled first audio data, a cyclic iterative training method is adopted to train the first music transcription model until the network converges; the following operations are performed during a single cyclic iterative training process: 1) the first audio data is input into the first music transcription model, and the first predicted audio sequence is output; 2) based on the CTC loss function, the first sequence loss information between the first predicted audio sequence and the first labeled audio sequence of the first audio data is determined; 3) the first sequence loss information is used to adjust the relevant parameters of the first music transcription model.

[0134] As can be seen from the above embodiments, by using weakly labeled data that only has the sequence information of the notes, but the start and end time information of each note in the score is missing or inaccurate, the first sequence loss information corresponding to the first music transcription model is obtained, and the first music transcription model is trained for music transcription based on the first sequence loss information to achieve fine-tuning of the model. The training task at this stage is to predict the pitch sequence of the notes, which has a high similarity to the complete music transcription task (predicting the note sequence, including pitch, start position, and end position information), so that the model can be familiar with the transcription task in advance, so as to initially have the ability to transcribe music.

[0135] S204: Perform music transcription training on the second music transcription model based on the second audio data, the second annotated pitch sequence and the annotated frame data to obtain a target music transcription model.

[0136] In practical applications, the music transcription training of the second music transcription model is also model fine-tuning training. The training data used in this stage is strongly labeled second audio data. Optionally, part of the audio data labeled with pitch sequence and frame data can be randomly selected from the sample audio data as the second audio data.

[0137] In an embodiment of the present specification, the model structure of the second music transcription model may include: a third audio feature extraction layer (i.e., the audio feature extraction layer after weak annotation training), a third context feature fusion layer (i.e., the context feature fusion layer after weak annotation training) and a second prediction layer, wherein the second prediction layer may include: a second pitch prediction layer (i.e., the pitch prediction layer after weak annotation training), a first note starting position prediction layer and a first note ending position prediction layer.

[0138] In a specific embodiment, the above-mentioned annotated frame data may include: the annotated data of each of the plurality of sample frames in the second audio data, such as Figure 7 As shown, the above-mentioned music transcription training of the second music transcription model based on the second audio data, the second annotated pitch sequence and the annotated frame data to obtain the target music transcription model may include:

[0139] S701, inputting the second audio data into a second music transcription model for music transcription processing, and obtaining prediction data and a second predicted pitch sequence for each of a plurality of sample frames.

[0140] Specifically, the second predicted pitch sequence may be a predicted pitch sequence generated based on the predicted pitch distribution corresponding to the sample frame sequence of the second audio data, wherein the sample frame sequence may be a frame sequence obtained by sampling audio frames of the second audio data, and the predicted pitch distribution corresponding to the sample frame sequence may include the corresponding predicted pitch distributions of each of the multiple sample frames in the sample frame sequence. Optionally, the sampling rate of the audio frame may be 44.1kHz.

[0141] In a specific embodiment, the second predicted pitch sequence may include: at least one predicted pitch sequence generated based on the predicted pitch distribution corresponding to the sample frame sequence of the second audio data.

[0142] Specifically, the prediction data of each of the multiple sample frames may be prediction data obtained after respectively predicting multiple sample frames in the sample frame sequence of the second audio data, and the prediction data of each sample frame may include: pitch prediction data, note start position prediction data and note end position prediction data.

[0143] Specifically, the pitch prediction data of each sample frame may represent the probability that the notes on the corresponding sample frame belong to each pitch, and the pitch prediction data of each sample frame may include the pitch distribution probability data of each sample frame.

[0144] Specifically, the note start position prediction data of each sample frame can represent the probability that the corresponding sample frame belongs to the note start position, and the note end position prediction data of each sample frame can represent the probability that the corresponding sample frame belongs to the note end position.

[0145] In a specific embodiment, the above-mentioned inputting the second audio data into the second music transcription model for music transcription processing to obtain the prediction data of each of the multiple sample frames and the second predicted pitch sequence may include: inputting the second audio data into the third audio feature extraction layer in the second music transcription model for audio feature extraction to obtain the second audio feature; inputting the second audio feature into the third context feature fusion layer in the second music transcription model for context feature fusion to obtain the second audio context feature; inputting the second audio context feature into the second pitch prediction layer in the second music transcription model for pitch prediction to obtain the second predicted pitch sequence and the pitch prediction data of each of the multiple sample frames; inputting the second audio context feature into the first note starting position prediction layer in the second music transcription model for note starting position prediction to obtain the note starting position prediction data of each of the multiple sample frames; inputting the second audio context feature into the first note ending position prediction layer in the second music transcription model for note ending position prediction to obtain the note ending position prediction data of each of the multiple sample frames.

[0146] S702: Determine second sequence loss information based on the second predicted pitch sequence and the second marked pitch sequence.

[0147] Specifically, the second sequence loss information may characterize the sequence difference between the second predicted pitch sequence and the second annotated pitch sequence. In a specific embodiment, the determining of the second sequence loss information based on the second predicted pitch sequence and the second annotated pitch sequence may include determining the sequence difference between the second predicted pitch sequence and the second annotated pitch sequence based on a second preset sequence loss function.

[0148] In an optional embodiment, the second preset sequence loss function may be the same loss function as the first preset sequence loss function, or may be a loss function different from the first preset sequence loss function.

[0149] In a specific embodiment, the second preset sequence loss function may be a CTC loss function. Specifically, the detailed content when the second preset sequence loss function is the CTC loss function may refer to the detailed content of the CTC loss function in step S502, which will not be repeated here.

[0150] S703: Determine frame loss information based on the labeled data and the predicted data.

[0151] Specifically, the frame loss information may represent the difference between the labeled data of each of the multiple sample frames and the predicted data of each of the multiple sample frames.

[0152] In a specific embodiment, the above-mentioned annotation data may include: pitch annotation data, note start position annotation data and note end position annotation data, and the above-mentioned prediction data may include: pitch prediction data, note start position prediction data and note end position prediction data, such as Figure 8 As shown, the above-mentioned determination of frame loss information based on the labeled data and the predicted data may include:

[0153] S801 : Determine frame pitch loss information based on the pitch marking data of each of the plurality of sample frames and the pitch prediction data of each of the plurality of sample frames.

[0154] Specifically, the frame pitch loss information may characterize the difference between the pitch annotation data of each of the multiple sample frames and the pitch prediction data of each of the multiple sample frames. In a specific embodiment, the above-mentioned determination of the frame pitch loss information based on the pitch annotation data of each of the multiple sample frames and the pitch prediction data of each of the multiple sample frames may include determining the difference between the pitch annotation data of each of the multiple sample frames and the pitch prediction data of each of the multiple sample frames based on a preset pitch loss function.

[0155] In a specific embodiment, the preset pitch loss function may include but is not limited to: a cross entropy loss function, a logical loss function, an exponential loss function, etc.

[0156] S802: Determine frame start position loss information based on the note start position labeling data of each of the plurality of sample frames and the note start position prediction data of each of the plurality of sample frames.

[0157] Specifically, the frame start position loss information can characterize the difference between the note start position labeling data of each of the multiple sample frames and the note start position prediction data of each of the multiple sample frames. In a specific embodiment, the above-mentioned determination of the frame start position loss information based on the note start position labeling data of each of the multiple sample frames and the note start position prediction data of each of the multiple sample frames can include determining the difference between the note start position labeling data of each of the multiple sample frames and the note start position prediction data of each of the multiple sample frames based on a preset note start position loss function.

[0158] In a specific embodiment, the preset note starting position loss function may include but is not limited to: a cross entropy loss function, a logical loss function, an exponential loss function, etc.

[0159] S803, determining frame end position loss information based on the note end position marking data of each of the plurality of sample frames and the note end position prediction data of each of the plurality of sample frames.

[0160] Specifically, the frame end position loss information may characterize the difference between the note end position labeling data of each of the multiple sample frames and the note end position prediction data of each of the multiple sample frames. In a specific embodiment, the above-mentioned determination of the frame end position loss information based on the note end position labeling data of each of the multiple sample frames and the note end position prediction data of each of the multiple sample frames may include determining the difference between the note end position labeling data of each of the multiple sample frames and the note end position prediction data of each of the multiple sample frames based on a preset note end position loss function.

[0161] In a specific embodiment, the preset note end position loss function may include but is not limited to: a cross entropy loss function, a logical loss function, an exponential loss function, etc.

[0162] S804, fusing the frame pitch loss information, the frame start position loss information and the frame end position loss information to obtain frame loss information.

[0163] In a specific embodiment, the frame pitch loss information, the frame starting position loss information and the frame ending position loss information are fused to obtain the frame loss information, which may include: based on a preset pitch weight, a preset starting position weight and a preset ending position weight, the frame pitch loss information, the frame starting position loss information and the frame ending position loss information are weightedly added to obtain the frame loss information.

[0164] Specifically, the preset pitch weight, the preset start position weight, and the preset end position weight can be preset in combination with the importance of the frame pitch loss information, the frame start position loss information, and the frame end position loss information for the model strong labeling training in actual applications. Optionally, the preset pitch weight, the preset start position weight, and the preset end position weight have the same value.

[0165] It can be seen from the above embodiments that the frame pitch loss information between the pitch prediction data and the pitch marking data, the frame starting position loss information between the note starting position prediction data and the note starting position marking data, and the frame ending position loss information between the note ending position prediction data and the note ending position marking data are determined respectively, and then the frame pitch loss information, the frame starting position loss information and the frame ending position loss information are fused to obtain the frame loss information, which can improve the accuracy of the difference representation between the sample frame prediction data and the sample frame marking data.

[0166] S704: Based on the second sequence loss information and the frame loss information, train a second music transcription model to obtain a target music transcription model.

[0167] In a specific embodiment, the training of the second music transcription model based on the second sequence loss information and the frame loss information to obtain the target music transcription model may include:

[0168] 1) Fusing the second sequence loss information and the frame loss information to obtain the target loss information;

[0169] 2) Training a second music transcription model based on the target loss information to obtain a target music transcription model.

[0170] In a specific embodiment, fusing the second sequence loss information and the frame loss information to obtain the target loss information may include: based on a preset sequence loss weight and a preset frame loss weight, respectively performing weighted addition on the second sequence loss information and the frame loss information to obtain the target loss information.

[0171] Specifically, the preset sequence loss weight and the preset frame loss weight can be pre-set based on the importance of the second sequence loss information and frame loss information to the strong annotation training of the model in actual applications and the data levels of the second sequence loss information and the frame loss information.

[0172] In a specific embodiment, based on the second sequence loss information and the frame loss information, training the second music transcription model to obtain the target music transcription model may include:

[0173] S7041, based on the second sequence loss information and the frame loss information, update the model parameters of the third audio feature extraction layer, the model parameters of the third context feature fusion layer, and the model parameters of the second prediction layer;

[0174] S7042, based on the updated third audio feature extraction layer, the updated third context feature fusion layer and the updated second prediction layer, repeating the strong labeling training iterative operation including steps S701 to S703 and S7041 until the second music transcription convergence condition is reached;

[0175] S5033, using the third audio feature extraction layer obtained when the second music transcription convergence condition is met as the fourth audio feature extraction layer, using the third context feature fusion layer obtained when the second music transcription convergence condition is met as the fourth context feature fusion layer, and using the second prediction layer obtained when the second music transcription convergence condition is met as the third prediction layer;

[0176] S5034, obtaining a target music transcription model according to the fourth audio feature extraction layer, the fourth context feature fusion layer and the third prediction layer.

[0177] In a specific embodiment, the second prediction layer includes: a second pitch prediction layer (i.e., a pitch prediction layer after weak labeling training), a first note starting position prediction layer, and a first note ending position prediction layer. Accordingly, updating the model parameters of the second prediction layer may include: updating the model parameters of the second pitch prediction layer, the model parameters of the first note starting position prediction layer, and the model parameters of the first note ending position prediction layer.

[0178] In an optional embodiment, the second music transcription convergence condition may be achieved when the number of training iterations reaches a preset number of training times. Optionally, the second music transcription convergence condition may also be achieved when the second sequence loss information and / or frame loss information on the validation set no longer decreases. In the embodiments of this specification, the preset number of training times may be preset in combination with the training speed and accuracy of the network in actual applications.

[0179] See also Fig. 9 , Fig. 9 It is a schematic diagram of a strongly labeled model music transcription training provided by the present application. Based on the second music transcription model obtained after weakly labeled training and the second audio data with strong labeling, a cyclic iterative training method is adopted to train the second music transcription model until the network converges; the following operations are performed during a single cyclic iterative training process: 1) the second audio data is input into the second music transcription model, and a second predicted audio sequence, a plurality of sample frames' respective pitch prediction data, a plurality of sample frames' respective note start position prediction data and a plurality of sample frames' respective note end position prediction data are output; 2) based on the CTC loss function, the second sequence loss information between the second predicted audio sequence and the second labeled audio sequence of the second audio data is determined; based on the cross entropy loss function, the frame pitch loss information between the pitch prediction data and the pitch labeling data is determined; based on the cross entropy loss function, the frame start position loss information between the note start position prediction data and the note start position labeling data is determined; based on the cross entropy loss function, the frame end position loss information between the note end position prediction data and the note end position labeling data is determined; 3) the relevant parameters of the first music transcription model are adjusted using the second sequence loss information, the frame pitch loss information, the frame start position loss information and the frame end position loss information.

[0180] It can be seen from the above embodiments that the second audio data in the sample audio data is input into the second music transcription model for music transcription, and the predicted data corresponding to the second audio data (including the second predicted pitch sequence and the respective pitch prediction data of multiple sample frames, the note starting position prediction data, and the note ending position prediction data) is obtained, and based on the predicted data and the strong annotation data of the second audio data (including the second annotated pitch sequence and the respective pitch annotation data of multiple sample frames, the note starting position annotation data, and the note annotation position prediction data), the second sequence loss information and frame loss information corresponding to the second music transcription model are determined, wherein the frame loss information is obtained by fusing the frame pitch loss information, the frame starting position loss information, and the frame ending position loss information, which can improve the accuracy of the difference representation between the sample frame prediction data and the sample frame annotation data, and then the second music transcription model is trained for music transcription based on the second sequence loss information and the frame loss information to obtain the target music transcription model, and the transcription accuracy and generalization of the model can be further improved by simultaneously using different loss information for strongly annotated music transcription training.

[0181] Indicatively, taking the sample music as the sample singing as an example, the training process of the singing transcription model may include three stages: In the first stage, a singing transcription model including a feature encoder, a context encoder, a quantization module and a linear prediction layer is trained on large-scale unlabeled singing data using a self-checking and supervised training method. This stage mainly trains the first three modules in the model (feature encoder, context encoder and quantization module), so that the model can learn high-quality representations of singing data from massive unlabeled data and "be familiar" with the singing techniques and singing speed commonly used by humans. In the second stage, the CTC loss is used to fine-tune the model trained in the first stage on weakly labeled singing transcription data. The data used in this stage is weakly labeled data. Here, weakly labeled data refers to data with accurate pitch annotations (the order of notes), but the start and end time information of each note in the score is missing or inaccurate. The amount of these data is less than that used in the first stage (pre-training), but more than that used in the third stage (strongly labeled training). The training task / goal of this stage is to predict the pitch sequence of notes (using CTC loss to train the model), which has a high similarity to the real transcription task (predicting the note sequence, including pitch, note start position, and note end position information). Therefore, the model can be fine-tuned in this stage to make full use of data that is similar to the target task (transcription) but has incomplete labels, so that the model can be familiar with the transcription task in advance. The third stage uses CTC loss and cross entropy loss to train the model trained in the second stage on a strongly labeled dataset to obtain the final model for singing transcription. The dataset used in this stage is a singing transcription dataset with complete labels (strong annotations). The annotation content is all the necessary information contained in a musical score, including the start and end time of each note, as well as the pitch of each note. In this stage, the model is trained through cross entropy loss (. Specifically, the feature representation of each frame of the audio will be input into the linear prediction layer to obtain the probability value of the corresponding category. For example, the pitch linear prediction layer will calculate the probability of each pitch on each frame. The entire singing transcription task is essentially a frame classification task. Through the above three stages of training, the singing transcription model can learn high-quality representations of singing data from massive unlabeled data, and can learn transcription capabilities through cross entropy loss and CTC loss on strongly labeled data and weakly labeled data respectively. Experiments have proved that the singing transcription model trained by the music transcription model training method proposed in the present invention can greatly surpass the traditional singing transcription model, and thanks to the role of big data, it can achieve better transcription effects on singing in various languages.

[0182] It can be seen from the technical solution provided by the above embodiments of the present application that music transcription training is performed by means of multi-stage model training technology and with the help of labeled data of different levels. First, based on the sample audio data corresponding to the sample music, the music transcription model to be trained is subjected to self-supervised audio representation training to obtain a first music transcription model. The common features of the sample audio data can be extracted, so that the first music transcription model can learn better audio feature representations, which is beneficial to improving the generalization ability of the model and the accuracy of downstream music transcription tasks. Then, based on the first audio data in the sample audio data and the first labeled pitch sequence corresponding to the first audio data, the first music transcription model is subjected to weakly labeled music transcription training to obtain a second music transcription model. That is, weakly labeled data that is only labeled with the sequence information of the notes, but the start and end time information of each note in the score is missing or inaccurate is used to obtain the first sequence loss information corresponding to the first music transcription model. Based on the first sequence loss information, the first music transcription model is subjected to music transcription training to achieve fine-tuning of the model, so that the second music transcription model can be familiar with the music transcription task and has preliminary music transcription capabilities. Finally, based on the sample audio data The second audio data, the second annotated pitch sequence corresponding to the second audio data, and the annotated frame data corresponding to the second audio data, perform strongly annotated music transcription training on the second music transcription model, that is, based on the predicted data obtained by inputting the second audio data into the second music transcription model (including the second predicted pitch sequence and the respective pitch prediction data of multiple sample frames, the note starting position prediction data, and the note ending position prediction data) and the strongly annotated data of the second audio data (including the second annotated pitch sequence and the respective pitch annotation data of multiple sample frames, the note starting position annotation data, and the note annotation position prediction data), determine the second sequence loss information and frame loss information corresponding to the second music transcription model, wherein the frame loss information is obtained by fusing the frame pitch loss information, the frame starting position loss information, and the frame ending position loss information, which can improve the accuracy of the difference representation between the sample frame prediction data and the sample frame annotation data, and then perform music transcription training on the second music transcription model based on the second sequence loss information and the frame loss information to obtain the target music transcription model, and by simultaneously using different loss information for strongly annotated music transcription training, the transcription accuracy and generalization of the model can be further improved.

[0183] The present application also provides a music transcription method, such as Fig.10 As shown, the music transcription method may include:

[0184] S1001, obtaining audio data corresponding to the music to be transcribed.

[0185] In the embodiments of the present specification, the music to be transcribed may include: songs to be transcribed, music to be transcribed and singing to be transcribed. Accordingly, the audio data corresponding to the songs to be transcribed needs to be transcribed through a song transcription model, the audio data corresponding to the music to be transcribed needs to be transcribed through a music transcription model, and the audio data corresponding to the singing to be transcribed needs to be transcribed through a singing transcription model.

[0186] In a specific embodiment, the data format of the audio data corresponding to the music to be transcribed can be a waveform file. In an optional embodiment, the audio data corresponding to the music to be transcribed can be a normalized audio waveform, which reduces data complexity and improves the accuracy of music transcription.

[0187] S1002, inputting the audio data into a music transcription model for music transcription processing to obtain music score data corresponding to the audio data;

[0188] Among them, the above-mentioned music transcription model is obtained after training based on the above-mentioned music transcription model training method.

[0189] In a specific embodiment, the formats of the music score data corresponding to the audio data may include: MIDI (Musical Instrument Digital Interface) format and XML (Extensible Markup Language) format, etc.

[0190] It can be seen from the above embodiments that performing music transcription processing based on the music transcription model obtained after training with the above-mentioned music transcription model training method can improve the accuracy of music transcription and further improve the accuracy of music score data.

[0191] In a specific embodiment, the above-mentioned inputting the audio data into the music transcription model for music transcription processing to obtain the music score data corresponding to the audio data may include:

[0192] 1) Input the audio data into the music transcription model for music transcription processing to obtain the pitch prediction data of each audio frame in multiple audio frames, the note starting position prediction data of each audio frame and the note ending position prediction data of each audio frame.

[0193] Specifically, the multiple audio frames may be obtained by sampling the audio data. Optionally, the sampling rate of the audio frames may be 44.1 kHz.

[0194] Specifically, the pitch prediction data of each audio frame can represent the predicted pitch distribution of the corresponding audio frame, that is, the probability of the notes on the corresponding audio frame belonging to each pitch, and the pitch prediction data of each audio frame can include the pitch distribution probability data of each audio frame.

[0195] Specifically, the note start position prediction data of each audio frame can represent the probability that the corresponding audio frame belongs to the note start position, and the note end position prediction data of each audio frame can represent the probability that the corresponding audio frame belongs to the note end position.

[0196] In a specific embodiment, the above-mentioned music transcription model may include: an audio feature extraction layer, a context feature fusion layer and a prediction layer, wherein the prediction layer may include: a pitch prediction layer, a note starting position prediction layer and a note ending position prediction layer; accordingly, the above-mentioned inputting audio data into the music transcription model for music transcription processing to obtain the pitch prediction data of each audio frame in multiple audio frames, the note starting position prediction data of each audio frame and the note ending position prediction data of each audio frame may include: inputting the audio data into the audio feature extraction layer for audio feature extraction to obtain target audio features; inputting the target audio features into the context feature fusion layer for context feature fusion to obtain target audio context features; inputting the target audio context features into the pitch prediction layer for pitch prediction to obtain the pitch prediction data of each audio frame; inputting the target audio context features into the note starting position prediction layer for note starting position prediction to obtain the note starting position prediction data of each audio frame; inputting the target audio context features into the note ending position prediction layer for note ending position prediction to obtain the note ending position prediction data of each audio frame.

[0197] 2) Based on the pitch prediction data of each audio frame, the note start position prediction data of each audio frame and the note end position prediction data of each audio frame, music score generation processing is performed to obtain music score data.

[0198] In a specific embodiment, the music score data obtained by performing the music score generation process based on the pitch prediction data of each audio frame, the note start position prediction data of each audio frame, and the note end position prediction data of each audio frame may include:

[0199] Determine the starting audio frames and the ending audio frames of the plurality of notes in the music score data according to the note starting position prediction data of the plurality of audio frames and the note ending position prediction data of the plurality of audio frames;

[0200] Determine the pitch data corresponding to each note according to the pitch prediction data corresponding to at least one audio frame between the start audio frame corresponding to each note and the end audio frame corresponding to each note;

[0201] The starting audio frame corresponding to each note, the ending audio frame corresponding to each note and the pitch data corresponding to each note are synthesized to obtain music score data.

[0202] Exemplarily, audio frame sampling processing is performed on audio data A to obtain 100 audio frames, and the 100 audio frames are traversed in sequence. According to the note starting position prediction data corresponding to the 5th audio frame, the 5th audio frame is determined to be the starting audio frame of the 1st note. According to the note ending position prediction data corresponding to the 10th audio frame, the 10th audio frame is determined to be the ending audio frame of the 1st note. Then, the pitch prediction data of the 6th to 9th audio frames are obtained, and the mode pitch of the predicted pitches corresponding to the 6th to 9th audio frames is used as the target pitch corresponding to the 1st note. In this way, the start and end audio frames and target pitches of other notes are determined, thereby generating the music score data of audio data A.

[0203] It can be seen from the above embodiments that by performing music score generation processing based on the pitch prediction data of each audio frame, the note starting position prediction data of each audio frame and the note ending position prediction data of each audio frame, the music score data can be obtained, which can improve the integrity and accuracy of the music score data.

[0204] In a specific embodiment, the music to be transcribed includes: songs to be transcribed, and the audio data is input into the music transcription model for music transcription processing, and the music score data corresponding to the audio data is obtained, which may include:

[0205] 1) performing vocal separation processing on the audio data corresponding to the song to be transcribed to obtain vocal audio data;

[0206] 2) Inputting the human voice audio data into the music transcription model for music transcription processing to obtain the human voice music score data corresponding to the human voice audio data;

[0207] 3) Use vocal score data as score data.

[0208] Specifically, the audio data corresponding to the song to be transcribed may be input into a vocal separation module for vocal separation processing. Optionally, the vocal separation module may be a track separation tool, such as Spleeter.

[0209] In a specific embodiment, the music transcription model here may include: a singing transcription model. Accordingly, inputting the human voice audio data into the music transcription model for music transcription processing, and obtaining the human voice music score data corresponding to the human voice audio data may include: inputting the human voice audio data into the singing transcription model for music transcription processing, and obtaining the human voice music score data corresponding to the human voice audio data.

[0210] Illustratively, in music playing application software, the music transcription method provided by the present invention can display the music score of the currently playing song in real time; in KTV or Karaoke application software, the music transcription method provided by the present invention can generate music score data of the user's singing audio in real time, compare the music score data corresponding to the user's singing audio with the music score data corresponding to the standard singing audio, and perform auxiliary evaluation on the user's singing skills or singing level.

[0211] It can be seen from the above embodiments that first separating the human voice audio, then normalizing it, and transcribing the singing voice based on the normalized human voice waveform can reduce data complexity and improve the accuracy of singing voice transcription.

[0212] The present application also provides a music transcription model training device. Fig.11 As shown, the music transcription model training device may include:

[0213] The sample acquisition module 1110 is used to acquire sample audio data corresponding to the sample music, a first annotated pitch sequence corresponding to the first audio data in the sample audio data, and annotated music score data corresponding to the second audio data in the sample audio data, wherein the annotated music score data includes: a second annotated pitch sequence and annotated frame data;

[0214] A first model training module 1120 is used to perform self-supervised audio representation training on the music transcription model to be trained based on the sample audio data to obtain a first music transcription model;

[0215] A second model training module 1130 is used to perform music transcription training on the first music transcription model based on the first audio data and the first annotated pitch sequence to obtain a second music transcription model;

[0216] The third model training module 1140 is used to perform music transcription training on the second music transcription model based on the second audio data, the second annotated pitch sequence and the annotated frame data to obtain a target music transcription model.

[0217] In a specific embodiment, the music transcription model to be trained may include: a first audio feature extraction layer, a first audio feature quantization layer, and a first context feature fusion layer, and the first model training module 1120 may include:

[0218] A sample audio feature unit, used to input the sample audio data into the first audio feature extraction layer to extract audio features and obtain sample audio features;

[0219] A sample audio quantization feature unit, used to input the sample audio feature into the first audio feature quantization layer for feature discretization processing to obtain the sample audio quantization feature;

[0220] A sample audio context feature unit, used to input the sample audio feature into the first context feature fusion layer to perform context feature fusion to obtain the sample audio context feature;

[0221] An audio feature loss information unit, used to determine audio feature loss information based on sample audio quantization features and sample audio context features;

[0222] The first music transcription model unit is used to train the music transcription model to be trained based on the audio feature loss information to obtain the first music transcription model.

[0223] In a specific embodiment, the first music transcription model may include: a first pitch prediction layer, and the second model training module 1130 may include:

[0224] A first predicted pitch sequence unit, used for inputting the first audio data into a first music transcription model, performing pitch prediction based on a first pitch prediction layer, and obtaining a first predicted pitch sequence;

[0225] A first sequence loss information unit, used to determine first sequence loss information based on the first predicted pitch sequence and the first marked pitch sequence;

[0226] The second music transcription model unit is used to train the first music transcription model based on the first sequence loss information to obtain the second music transcription model.

[0227] In a specific embodiment, the annotated frame data may include: annotated data of each of a plurality of sample frames in the second audio data, and the third model training module 1140 may include:

[0228] A music transcription unit, used for inputting the second audio data into a second music transcription model for music transcription processing, and obtaining respective prediction data and a second predicted pitch sequence of a plurality of sample frames;

[0229] A second sequence loss information unit, used to determine second sequence loss information based on the second predicted pitch sequence and the second marked pitch sequence;

[0230] A frame loss information unit, configured to determine frame loss information based on the labeled data and the predicted data;

[0231] The target music transcription model unit is used to train the second music transcription model based on the second sequence loss information and the frame loss information to obtain the target music transcription model.

[0232] In a specific embodiment, the above-mentioned annotation data may include: pitch annotation data, note start position annotation data and note end position annotation data, the above-mentioned prediction data may include: pitch prediction data, note start position prediction data and note end position prediction data, and the above-mentioned frame loss information unit may include:

[0233] A frame pitch loss information unit, used to determine frame pitch loss information based on the pitch annotation data of each of the plurality of sample frames and the pitch prediction data of each of the plurality of sample frames;

[0234] A frame start position loss information unit, used to determine frame start position loss information based on the note start position labeling data of each of the plurality of sample frames and the note start position prediction data of each of the plurality of sample frames;

[0235] A frame end position loss information unit, used to determine frame end position loss information based on the note end position marking data of each of the plurality of sample frames and the note end position prediction data of each of the plurality of sample frames;

[0236] The loss fusion unit is used to fuse the frame pitch loss information, the frame start position loss information and the frame end position loss information to obtain the frame loss information.

[0237] It should be noted that the device and method embodiments in the device embodiment are based on the same inventive concept.

[0238] The present application also provides a music transcription device, such as Fig.12 As shown, the music transcription device may include:

[0239] The audio data acquisition module 1210 is used to acquire the audio data corresponding to the music to be transcribed;

[0240] The music transcription module 1220 is used to input the audio data into the music transcription model for music transcription processing to obtain the music score data corresponding to the audio data;

[0241] Among them, the above-mentioned music transcription model is obtained after training based on the above-mentioned music transcription model training device.

[0242] In a specific embodiment, the music transcription module 1220 may include:

[0243] A music transcription unit, used for inputting audio data into a music transcription model for music transcription processing, and obtaining respective pitch prediction data of each audio frame in a plurality of audio frames, respective note start position prediction data of each audio frame, and respective note end position prediction data of each audio frame;

[0244] The music score generating unit is used to perform music score generating processing based on the pitch prediction data of each audio frame, the note start position prediction data of each audio frame and the note end position prediction data of each audio frame to obtain music score data.

[0245] In a specific embodiment, the music to be transcribed may include: songs to be transcribed, and the music transcription module 1220 may include:

[0246] A vocal separation unit, used for performing vocal separation processing on the audio data corresponding to the song to be transcribed to obtain vocal audio data;

[0247] A vocal transcription unit, used for inputting the vocal audio data into a music transcription model for music transcription processing to obtain vocal music score data corresponding to the vocal audio data;

[0248] The music score data unit is used to use the vocal music score data as the music score data.

[0249] It should be noted that the device and method embodiments in the device embodiment are based on the same inventive concept.

[0250] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement a music transcription model training method or a music transcription method provided in the above method embodiment.

[0251] Further, Fig.13 A hardware structure diagram of an electronic device for implementing the music transcription model training method or music transcription method provided in the embodiment of the present application is shown, and the electronic device may participate in or include the music transcription model training device or music transcription device provided in the embodiment of the present application. Fig.13As shown, the music transcription model training device 130 may include one or more (1302a, 1302b, ..., 1302n are used to illustrate) processors 1302 (the processor 1302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1304 for storing data, and a transmission device 1306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig.13 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.13 More or fewer components as shown, or with Fig.13 Different configurations are shown.

[0252] It should be noted that the one or more processors 1302 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the electronic device 130 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0253] The memory 1304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the music transcription model training method described in the embodiment of the present application or the program instructions / data storage device corresponding to the music transcription method. The processor 1302 executes various functional applications and data processing by running the software programs and modules stored in the memory 1304, that is, realizing the above-mentioned music transcription model training method or music transcription method. The memory 1304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory or other non-volatile solid-state memory. In some instances, the memory 1304 may further include a memory remotely arranged relative to the processor 1302, and these remote memories can be connected to the electronic device 130 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0254] The transmission device 1306 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 130. In one example, the transmission device 1306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one embodiment, the transmission device 1306 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0255] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of the electronic device 130 (or mobile device).

[0256] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program for implementing the at least one instruction or at least one program, which is loaded and executed by the processor to implement the music transcription model training method or music transcription method provided in the above method embodiment.

[0257] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0258] The embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the music transcription model training method or the music transcription method provided in the method embodiment.

[0259] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0260] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0261] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0262] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A music transcription model training method, It is characterized in that The method comprises: Acquire sample audio data corresponding to the sample music, a first annotated pitch sequence corresponding to the first audio data in the sample audio data, and annotated music score data corresponding to the second audio data in the sample audio data, wherein the annotated music score data includes: a second annotated pitch sequence and annotated frame data; Based on the sample audio data, performing self-supervised audio representation training on the music transcription model to be trained to obtain a first music transcription model; Based on the first audio data and the first annotated pitch sequence, performing music transcription training on the first music transcription model to obtain a second music transcription model; Based on the second audio data, the second annotated pitch sequence and the annotated frame data, music transcription training is performed on the second music transcription model to obtain a target music transcription model.

2. The method according to claim 1, It is characterized in that The first music transcription model includes: a first pitch prediction layer, and the first music transcription model is subjected to music transcription training based on the first audio data and the first annotated pitch sequence to obtain a second music transcription model including: Inputting the first audio data into the first music transcription model, performing pitch prediction based on the first pitch prediction layer, and obtaining a first predicted pitch sequence; Determining first sequence loss information based on the first predicted pitch sequence and the first marked pitch sequence; Based on the first sequence loss information, the first music transcription model is trained to obtain the second music transcription model.

3. The method according to claim 1, It is characterized in that The annotated frame data includes: the annotated data of each of the plurality of sample frames in the second audio data; the performing music transcription training on the second music transcription model based on the second audio data, the second annotated pitch sequence and the annotated frame data to obtain the target music transcription model includes: Inputting the second audio data into a second music transcription model for music transcription processing to obtain prediction data and a second predicted pitch sequence for each of the plurality of sample frames; Determining second sequence loss information based on the second predicted pitch sequence and the second marked pitch sequence; Determining frame loss information based on the labeled data and the predicted data; Based on the second sequence loss information and the frame loss information, the second music transcription model is trained to obtain the target music transcription model.

4. The method according to claim 3, It is characterized in that The annotated data includes: pitch annotated data, note start position annotated data and note end position annotated data, the predicted data includes: pitch prediction data, note start position prediction data and note end position prediction data, and the determining of frame loss information based on the annotated data and the predicted data includes: Determining frame pitch loss information based on the pitch marking data of each of the plurality of sample frames and the pitch prediction data of each of the plurality of sample frames; Determining frame start position loss information based on the note start position labeling data of each of the plurality of sample frames and the note start position prediction data of each of the plurality of sample frames; Determining frame end position loss information based on the note end position labeling data of each of the plurality of sample frames and the note end position prediction data of each of the plurality of sample frames; The frame pitch loss information, the frame start position loss information and the frame end position loss information are fused to obtain the frame loss information.

5. The method according to any one of claims 1 to 4, It is characterized in that The music transcription model to be trained includes: a first audio feature extraction layer, a first audio feature quantization layer and a first context feature fusion layer. Based on the sample audio data, the music transcription model to be trained is subjected to self-supervised audio representation training to obtain the first music transcription model, including: Inputting the sample audio data into the first audio feature extraction layer to extract audio features to obtain sample audio features; Inputting the sample audio feature into the first audio feature quantization layer for feature discretization processing to obtain the sample audio quantization feature; Inputting the sample audio feature into the first context feature fusion layer to perform context feature fusion to obtain the sample audio context feature; Determining audio feature loss information based on the sample audio quantization feature and the sample audio context feature; Based on the audio feature loss information, the music transcription model to be trained is trained to obtain the first music transcription model.

6. A music transcription method, It is characterized in that The method comprises: Obtain audio data corresponding to the music to be transcribed; Inputting the audio data into a music transcription model for music transcription processing to obtain music score data corresponding to the audio data; Wherein, the music transcription model is obtained after training based on the music transcription model training method described in any one of claims 1 to 5.

7. The method according to claim 6, It is characterized in that The step of inputting the audio data into a music transcription model for music transcription processing to obtain music score data corresponding to the audio data comprises: Inputting the audio data into the music transcription model for music transcription processing, obtaining respective pitch prediction data of each audio frame in a plurality of audio frames, respective note start position prediction data of each audio frame, and respective note end position prediction data of each audio frame; Music score generation processing is performed based on the pitch prediction data of each audio frame, the note start position prediction data of each audio frame and the note end position prediction data of each audio frame to obtain the music score data.

8. The method according to claim 6 or 7, It is characterized in that The music to be transcribed includes: songs to be transcribed, and the audio data is input into a music transcription model for music transcription processing, and the music score data corresponding to the audio data is obtained, including: Performing vocal separation processing on the audio data corresponding to the song to be transcribed to obtain vocal audio data; Inputting the human voice audio data into the music transcription model for music transcription processing to obtain human voice music score data corresponding to the human voice audio data; The vocal music score data is used as the music score data.

9. A music transcription model training device, It is characterized in that The device comprises: A sample acquisition module is used to acquire sample audio data corresponding to sample music, a first annotated pitch sequence corresponding to first audio data in the sample audio data, and annotated music score data corresponding to second audio data in the sample audio data, wherein the annotated music score data includes: a second annotated pitch sequence and annotated frame data; A first model training module is used to perform self-supervised audio representation training on the music transcription model to be trained based on the sample audio data to obtain a first music transcription model; A second model training module is used to perform music transcription training on the first music transcription model based on the first audio data and the first annotated pitch sequence to obtain a second music transcription model; The third model training module is used to perform music transcription training on the second music transcription model based on the second audio data, the second annotated pitch sequence and the annotated frame data to obtain a target music transcription model.

10. A music transcription device, It is characterized in that The device comprises: An audio data acquisition module, used to acquire audio data corresponding to the music to be transcribed; A music transcription module, used for inputting the audio data into a music transcription model for music transcription processing to obtain music score data corresponding to the audio data; Wherein, the music transcription model is obtained after training based on the music transcription model training device described in claim 9.

11. An electronic device, It is characterized in that The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the music transcription model training method as described in any one of claims 1 to 5 or the music transcription method as described in any one of claims 6 to 8.

12. A computer-readable storage medium, It is characterized in that The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the music transcription model training method as described in any one of claims 1 to 5 or the music transcription method as described in any one of claims 6 to 8.

13. A computer program product, It is characterized in that The computer program product includes at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the music transcription model training method as described in any one of claims 1 to 5 or the music transcription method as described in any one of claims 6 to 8.

Images