Pitch extraction method, apparatus and electronic device

Abstract

Embodiments of the present application disclose a pitch extraction method and device and electronic equipment. A specific embodiment of the method includes: obtaining an audio segment and a timbre feature of a sound emitting object in the audio segment; converting the audio segment into a mel-frequency spectrum, and connecting the timbre feature with the mel-frequency spectrum to obtain a fusion feature; and determining an output result based on the fusion feature and a pre-trained target model, wherein the output result includes pitch information of the sound emitting object. The embodiment can improve the accuracy of pitch extraction in a noisy environment.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, in particular to a pitch extraction method, device and electronic equipment. BACKGROUND

[0002] Pitch is defined as the auditory perception of the highness or lowness of a sound, and is one of the most basic auditory attributes. Pitch is not only crucial to the melody and chord of music, but also can convey rich information when speaking, so the automatic extraction technology of pitch has been widely concerned and occupies an important position in the fields of music information retrieval, speech recognition, speech synthesis, etc. SUMMARY

[0003] This part of the disclosure is provided to briefly introduce the concepts, which will be described in detail in the specific embodiments part. This part of the disclosure is not intended to identify key features or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] In a first aspect, the embodiments of the present disclosure provide a pitch extraction method, comprising: obtaining an audio segment and a timbre feature of a sound object in the audio segment; converting the audio segment into a mel-frequency spectrum, and connecting the timbre feature with the mel-frequency spectrum to obtain a fusion feature; and determining an output result based on the fusion feature and a pre-trained target model, wherein the output result comprises pitch information of the sound object.

[0005] In a second aspect, the embodiments of the present disclosure provide a pitch extraction device, comprising: an obtaining unit configured to obtain an audio segment and a timbre feature of a sound object in the audio segment; a converting unit configured to convert the audio segment into a mel-frequency spectrum, and connect the timbre feature with the mel-frequency spectrum to obtain a fusion feature; and a determining unit configured to determine an output result based on the fusion feature and a pre-trained target model, wherein the output result comprises pitch information of the sound object.

[0006] In a third aspect, the embodiments of the present disclosure provide an electronic device, comprising: at least one processor; and a storage device configured to store at least one program, when the at least one program is executed by the at least one processor, the at least one processor implements the pitch extraction method of the first aspect.

[0007] In a fourth aspect, the embodiments of the present disclosure provide a computer readable medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the pitch extraction method of the first aspect.

[0008] The pitch extraction method, apparatus, and electronic device provided in this disclosure acquire an audio segment and the timbre features of the sound-producing object within the audio segment. Then, the audio segment is converted into a Mel spectrum, and the timbre features are concatenated with the Mel spectrum to obtain a fused feature. Finally, based on the fused feature and a pre-trained target model, an output result including the pitch information of the sound-producing object is determined. This approach improves the accuracy of pitch extraction in noisy environments by using the timbre features of the sound-producing object in the audio segment as auxiliary information for pitch extraction. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0010] Figure 1 This is a flowchart of an embodiment of the pitch extraction method according to the present disclosure;

[0011] Figure 2 This is a flowchart of yet another embodiment of the pitch extraction method according to the present disclosure;

[0012] Figure 3 This is a schematic diagram of an application scenario of the pitch extraction method according to this disclosure;

[0013] Figure 4 This is a flowchart of an embodiment of the pitch extraction method according to the present disclosure, which uses a pitch extraction sub-model to determine pitch information;

[0014] Figure 5 This is a flowchart of an embodiment of the pitch extraction method of the present disclosure, which uses a sound detection sub-model to determine the sound detection result;

[0015] Figure 6 This is a schematic diagram illustrating an application scenario of feature processing in the pitch extraction method of this disclosure;

[0016] Figure 7 This is a flowchart of an embodiment of the pitch extraction method of this disclosure, in which a target model is trained.

[0017] Figure 8 This is a schematic diagram of a structure of an embodiment of the pitch extraction device according to the present disclosure;

[0018] Figure 9 These are exemplary system architecture diagrams to which the various embodiments of this disclosure can be applied;

[0019] Figure 10This is a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present disclosure. Detailed Implementation

[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0021] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0022] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0024] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0027] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] Please refer to Figure 1 The diagram illustrates a flow 100 of an embodiment of a pitch extraction method according to the present disclosure. This pitch extraction method includes the following steps:

[0032] Step 101: Obtain the timbre characteristics of the audio segment and the sound-producing object in the audio segment.

[0033] In this embodiment, the execution entity of the pitch extraction method can acquire an audio segment and the timbre characteristics of the sound-producing object within that audio segment. Here, the audio segment is typically the segment from which pitch information is to be extracted. In scenarios with real-time requirements, the duration of the audio segment is usually short, for example, 64 milliseconds.

[0034] Here, the sounds in the aforementioned audio clips correspond to the sound-producing objects, and the timbre features of these sound-producing objects can also be referred to as timbre embedding. The timbre features of these sound-producing objects can be any features that contain sufficient identifiable timbre information.

[0035] Step 102: Convert the audio segment into a Mel spectrum and connect the timbre features with the Mel spectrum to obtain the fusion features.

[0036] In this embodiment, the execution entity can convert the audio segment into a Mel spectrum. The Mel spectrum can also be called a Mel acoustic spectrum. Here, the execution entity can resample the audio segment and convert it into a Mel spectrum. The Mel spectrum is used because it is similar to human auditory perception and retains enough detail to extract timbre information.

[0037] As an example, the length of the above audio clip can be 64 milliseconds. The 64-millisecond audio clip can be resampled to 16kHz and converted into a Mel spectrum of size (128,3,1), where 128 in (128,3,1) is a vector of frequency dimension and 3 is a vector of time dimension.

[0038] The aforementioned executing entity can then connect the aforementioned timbre features with the aforementioned Mel spectrum to obtain the fusion feature. As an example, if the size of the Mel spectrum is (128,3,1), then the size of the fusion feature is (128,3,2).

[0039] Step 103: Determine the output result based on the fused features and the pre-trained target model.

[0040] In this embodiment, the execution entity can determine the output result based on the aforementioned fusion features and the pre-trained target model. The output result typically includes the pitch information of the sound-producing object. In a sound containing only a single pitch, the pitch can be quantized using the fundamental frequency of that sound, which is the lowest harmonic frequency in the sound signal.

[0041] Here, the target model can be a pitch extraction model, which can be used to represent the correspondence between features and the pitch information of the corresponding audio objects. The execution entity can input the fused features into the pitch extraction model to obtain the pitch information of the audio objects in the audio segment.

[0042] In this embodiment, the target model may include at least two convolutional layers. During the process of inputting the fused features into the target model, for each of the at least two convolutional layers, instance normalization and batch normalization are performed on the output features of that target convolutional layer. The target convolutional layer may be the first preset number of convolutional layers among the at least two convolutional layers. For each convolutional layer other than the target convolutional layer among the at least two convolutional layers, batch normalization is performed on the output features of that convolutional layer.

[0043] As an example, if the target model mentioned above includes six convolutional layers, the output features of the first three convolutional layers can be processed using a combination of instance normalization and batch normalization, while the output features of the last three convolutional layers can be processed using only batch normalization. This enhances the model's generalization ability, enabling it to exhibit a certain degree of noise resistance even when faced with previously unseen noisy environments.

[0044] The method provided in the above embodiments of this disclosure acquires an audio segment and the timbre features of the sound-producing object in the audio segment; then, the audio segment is converted into a Mel spectrum, and the timbre features are concatenated with the Mel spectrum to obtain a fused feature; finally, based on the fused feature and a pre-trained target model, an output result including the pitch information of the sound-producing object is determined. This approach improves the accuracy of pitch extraction in noisy environments by using the timbre features of the sound-producing object in the audio segment as auxiliary information for pitch extraction.

[0045] In some alternative implementations, the aforementioned timbre features can be determined in the following way: the aforementioned audio segment can be input into a pre-trained voice object recognition model, which can be used to characterize the correspondence between the input audio and the corresponding voice object recognition result. That is, after a certain input audio is input into the aforementioned voice object recognition model, the object identifier of the voice object corresponding to the input audio segment can be obtained.

[0046] The aforementioned voice object recognition model can consist of convolutional layers, gated recurrent units (GRUs), and fully connected layers. The input to this model can be an audio segment of arbitrary length (e.g., 3 seconds), and the output can be a probability distribution for voice object recognition. The feature vector output by the target convolutional layer in the model can be determined as the aforementioned timbre feature. This target convolutional layer can be the last convolutional layer in the model; flattening the features output by this last convolutional layer yields the timbre feature of the voice object in the audio segment.

[0047] It should be noted that the aforementioned timbre features can be determined by the execution body of the pitch extraction method as described above, or they can be determined using other electronic devices besides the execution body of the pitch extraction method. After other electronic devices determine the timbre features of the audio segment using the aforementioned method, they can send the timbre features to the execution body of the pitch extraction method.

[0048] In some optional implementations, the aforementioned vocal object recognition model may include a singer recognition model. That is, the aforementioned audio segment can be input into a pre-trained singer recognition model. The aforementioned singer recognition model can be used to represent the correspondence between the input audio and the corresponding singer recognition result. In other words, after a certain input audio is input into the aforementioned singer recognition model, the singer identifier of the speaker or singer corresponding to that input audio segment can be obtained.

[0049] Generally, while the style of a song is crucial for identifying a specific singer, the singer's timbre information is even more indispensable. Therefore, the features output by the intermediate layers of a singer recognition model should contain rich timbre features. The aforementioned singer recognition model can consist of convolutional layers, gated recurrent units, and fully connected layers, and is trained using common datasets for singer recognition tasks. The input to this model can be an audio segment of arbitrary length, and the output can be the probability distribution for singer recognition. The feature vector output by the target convolutional layer in the model can be defined as the aforementioned timbre features. This target convolutional layer can be the last convolutional layer in the model; flattening the features output by this last convolutional layer yields the timbre features of the vocal subject in the audio segment.

[0050] Further reference Figure 2 This illustrates a flow 200 of another embodiment of the pitch extraction method. Flow 200 of this pitch extraction method includes the following steps:

[0051] Step 201: Obtain the timbre characteristics of the audio segment and the sound-producing object in the audio segment.

[0052] Step 202: Convert the audio segment into a Mel spectrum and connect the timbre features with the Mel spectrum to obtain the fusion features.

[0053] In this embodiment, steps 201-202 can be performed in a similar manner to steps 101-102, and will not be described again here.

[0054] Step 203: Input the fused features into the pitch extraction sub-model to obtain the pitch information of the vocal object.

[0055] In this embodiment, the execution entity of the pitch extraction method can input the aforementioned fused features into the pitch extraction sub-model to obtain the pitch information of the vocal object in the aforementioned audio segment. The aforementioned pitch extraction sub-model can be used to characterize the correspondence between the features and the pitch information of the vocal object in the corresponding audio.

[0056] As an example, the output of the above pitch extraction sub-model can be a 359-dimensional vector representing the predicted probabilities of 359 pitch values ​​spaced at intervals of 20 octaves, covering a pitch range of 32.7 Hz to 1975.5 Hz.

[0057] Step 204: Input the fused features into the sound detection sub-model to obtain the sound detection result.

[0058] In this embodiment, the execution entity can input the fused features into the sound detection sub-model to obtain the sound detection result. The sound detection sub-model can be used to characterize the correspondence between the features and the sound detection results of the corresponding audio.

[0059] The sound detection result can include probability values ​​for three states: a first state probability value, a second state probability value, and a third state probability value. The first state probability value can be used to characterize the probability that there is a sound in the audio segment that matches the timbre characteristics of the sound-producing object. The second state probability value can be used to characterize the probability that there is no sound with an extractable pitch in the audio segment. The third state probability value can be used to characterize the probability that the sound with an extractable pitch in the audio segment does not match the timbre characteristics of the sound-producing object.

[0060] Using the aforementioned sound detection sub-model, it is possible to determine whether sound exists in the audio segment and whether the existing sound matches the extracted timbre features.

[0061] from Figure 2 It can be seen from this that, with Figure 1 Compared to the corresponding embodiments, the pitch extraction method in this embodiment, in its flow 200, embodies the steps of determining pitch information using a pitch extraction sub-model and detecting sound using a sound detection sub-model. Therefore, the scheme described in this embodiment integrates the pitch extraction sub-model and the sound detection sub-model into a single large model. To ensure optimal performance for each task, the model adopts a dual-branch structure, improving the practicality of the extraction system.

[0062] In addition, the model integrates features from the pitch branch into the sound branch to assist in sound detection. This is because if there is no sound from which pitch can be extracted, it is difficult for the pitch branch to extract a specific pitch. This information is implicit in the features of the pitch branch and has an auxiliary effect on the sound branch.

[0063] See also Figure 3 , Figure 3 This is a schematic diagram illustrating an application scenario of the pitch extraction method according to this embodiment. Figure 3In the application scenario, the execution body of the pitch extraction method can first obtain a 64-millisecond audio segment 301 and the timbre features 302 of the sound-producing object in the audio segment 301; then, the 64-millisecond audio segment 301 can be resampled to 16kHz and converted into a Mel spectrum 303 of size (128,3,1), and the timbre features 302 and Mel spectrum 303 can be concatenated to obtain a fusion feature 304 of size (128,3,2); then, the fusion feature 304 can be input into the pitch extraction sub-model 305 to obtain the pitch information 307 of the sound-producing object; finally, the fusion feature 304 can be input into the sound detection sub-model 306 to obtain the sound detection result 308.

[0064] Further reference Figure 4 This illustrates a flowchart 400 of an embodiment of a pitch extraction method that utilizes a pitch extraction sub-model to determine pitch information. The flowchart 400, which utilizes a pitch extraction sub-model to determine pitch information, includes the following steps:

[0065] Step 401: In the process of inputting the fused features into the pitch extraction sub-model, for each target convolutional layer in at least two convolutional layers, the output features of the target convolutional layer are subjected to instance normalization and batch normalization.

[0066] In this embodiment, the pitch extraction sub-model typically includes at least two convolutional layers. The execution entity of the pitch extraction method can, during the process of inputting the fused features into the pitch extraction sub-model, perform instance normalization (IN) and batch normalization (BN) processing on the output features of each of the at least two convolutional layers for each target convolutional layer.

[0067] Here, the target convolutional layer can be the first few convolutional layers among the at least two convolutional layers, for example, the target convolutional layer can be the first three convolutional layers of the pitch extraction sub-model. It should be noted that the specific target convolutional layer, which is the first few convolutional layers of the pitch extraction sub-model, can be set according to the actual situation.

[0068] In this embodiment, although the input data is normalized, its distribution is likely to change after matrix multiplication and nonlinear operations. This change in distribution becomes increasingly pronounced with each layer of the deep network. Normalization at the intermediate layers of the network would unify the data in these layers to the same dimension. This method of normalizing data at intermediate layers to improve training performance is called batch normalization. Batch normalization can increase the sensitivity of features to image content but reduce robustness to style class changes.

[0069] The batch normalization algorithm is as follows: First, calculate the mean of the output data of the previous layer; then, calculate the standard deviation of the output data of the previous layer; then, perform normalization processing; finally, reconstruct the data obtained after the above normalization processing.

[0070] Instance normalization, also known as contrast normalization, differs from batch normalization in that it applies to an entire batch of images rather than a single image. Instance normalization normalizes within a single pixel of an image and is primarily used for style transfer. Instance normalization can improve robustness to style class changes.

[0071] Step 402: For each of the at least two convolutional layers other than the target convolutional layer, perform batch normalization on the output features of that convolutional layer.

[0072] In this embodiment, batch normalization is performed on the output features of each of the at least two convolutional layers, excluding the target convolutional layer. For example, if the target convolutional layer is the first three convolutional layers of the pitch extraction sub-model, and the pitch extraction sub-model includes five convolutional layers, then the execution entity can perform batch normalization on the last two convolutional layers of the pitch extraction sub-model.

[0073] The method provided by the above embodiments of this disclosure combines instance normalization and batch normalization in the first few convolutional layers of the pitch extraction sub-model, while using only batch normalization in deeper convolutional layers. This introduces IBN-Net (Instance-Batch Normalization Net) into the model, which can enhance the generalization of the model and enable it to have a certain degree of noise resistance even when facing previously unseen noisy environments.

[0074] Further reference Figure 5 This illustrates a flowchart 500 of an embodiment of a pitch extraction method that utilizes a sound detection sub-model to determine sound detection results. The flowchart 500, which utilizes a sound detection sub-model to determine sound detection results, includes the following steps:

[0075] Step 501: In the process of inputting the fused features into the sound detection sub-model, for each target convolutional layer in at least two convolutional layers, the output features of the target convolutional layer are subjected to instance normalization and batch normalization.

[0076] In this embodiment, the aforementioned sound detection sub-model typically includes at least two convolutional layers. The execution entity of the pitch extraction method can, during the process of inputting the fused features into the sound detection sub-model, perform instance normalization and batch normalization on the output features of each of the at least two convolutional layers.

[0077] Here, the target convolutional layer can be one of the first few convolutional layers from the above at least two convolutional layers, for example, the target convolutional layer can be the first three convolutional layers of the above sound detection sub-model. It should be noted that the specific target convolutional layer, which is the first few convolutional layers of the above pitch extraction sub-model, can be set according to the actual situation.

[0078] In this embodiment, although the input data is normalized, its distribution is likely to change after matrix multiplication and nonlinear operations. This change in distribution becomes increasingly pronounced with each layer of the deep network. Normalization at the intermediate layers of the network would unify the data in these layers to the same dimension. This method of normalizing data at intermediate layers to improve training performance is called batch normalization. Batch normalization can increase the sensitivity of features to image content but reduce robustness to style class changes.

[0079] The batch normalization algorithm is as follows: First, calculate the mean of the output data of the previous layer; then, calculate the standard deviation of the output data of the previous layer; then, perform normalization processing; finally, reconstruct the data obtained after the above normalization processing.

[0080] Instance normalization, also known as contrastive normalization, differs from batch normalization in that it applies to an entire batch of images rather than a single image. Instance normalization normalizes images within a single pixel and is primarily used for style transfer. Instance normalization can improve robustness to style class changes.

[0081] Step 502: For each of the at least two convolutional layers other than the target convolutional layer, perform batch normalization on the output features of that convolutional layer.

[0082] In this embodiment, batch normalization is performed on the output features of each of the at least two convolutional layers, excluding the target convolutional layer. For example, if the target convolutional layer is the first three convolutional layers of the sound detection sub-model, and the sound detection sub-model includes five convolutional layers, then the execution entity can perform batch normalization on the last two convolutional layers of the sound detection sub-model.

[0083] The method provided by the above embodiments of this disclosure combines instance normalization and batch normalization in the first few convolutional layers of the sound detection sub-model, while using only batch normalization in deeper convolutional layers. This introduces IBN-Net (Instance-Batch Normalization Net) into the model, which can enhance the generalization of the model and enable it to have a certain degree of noise resistance even when facing noise environments it has never seen before.

[0084] See also Figure 6 , Figure 6 This is a schematic diagram illustrating an application scenario of feature processing in the pitch extraction method according to this embodiment. Figure 6 In the application scenario, the pitch extraction sub-model 602 includes three convolutional layers. When the fused features 601 are input into the pitch extraction sub-model 602, the output features of the first two convolutional layers can be processed using instance normalization and batch normalization. Similarly, the output features of the third convolutional layer can be processed using batch normalization. The sound detection sub-model 603 also includes three convolutional layers. When the fused features 601 are input into the sound detection sub-model 603, the output features of the first two convolutional layers can be processed using instance normalization and batch normalization. The output features of the third convolutional layer can be processed using batch normalization.

[0085] Further reference Figure 7 This illustrates a process 700 for training a target model in a pitch extraction method. The training process 700 for this target model includes the following steps:

[0086] Step 701: Obtain the first training sample set and the second training sample set.

[0087] In this embodiment, the execution entity that trains the target model can obtain a first training sample set and a second training sample set. The first training sample in the first training sample set is typically used to train the pitch extraction sub-model. The first training sample may include fused feature samples and the pitch information of the emitting object in the audio corresponding to the fused feature samples. The second training sample in the second training sample set is typically used to train the sound detection sub-model. The second training sample may include fused feature samples and the sound detection results of the audio corresponding to the fused feature samples.

[0088] Step 702, perform the following first training step: train the first initial model using the first training sample set; determine whether the first initial model obtained by training has converged; if so, determine the first initial model that has converged as the pitch extraction sub-model, and perform the second training step.

[0089] In this embodiment, step 702 may include sub-steps 7021, 7022, and 7023. Wherein:

[0090] Step 7021: Train the first initial model using the first training sample set.

[0091] In this embodiment, the execution entity for training the target model can use the first training sample set to train the first initial model. Specifically, the execution entity can use the fused feature samples of the first training samples in the first training sample set as the input of the first initial model, and use the pitch information of the sound-producing object in the audio corresponding to the input fused feature samples as the output of the first initial model, thereby training the first initial model.

[0092] It should be noted that the execution entity of the training target model can be the execution entity of the pitch extraction method described above, or it can be any other electronic device besides the execution entity of the pitch extraction method described above. If the execution entity of the training target model is not the execution entity of the pitch extraction method described above, then the execution entity of the training target model can send the trained target model to the execution entity of the pitch extraction method described above.

[0093] Step 7022: Determine whether the first initial model obtained through training has reached convergence.

[0094] In this embodiment, the execution entity can determine whether the trained first initial model has converged. If the trained first initial model has converged, then step 7023 can be executed.

[0095] Step 7023: If the first initial model obtained through training converges, then the first initial model that has converged is determined as the pitch extraction sub-model, and the second training step is executed.

[0096] In this embodiment, if it is determined in step 7022 that the first initial model obtained through training has reached convergence, the above-mentioned execution entity can determine the first initial model that has reached convergence as the pitch extraction sub-model and execute the second training step (i.e., step 703).

[0097] Step 703: Perform the following second training step: fix the parameters of the pitch extraction sub-model, and train the second initial model using the second training sample set; determine whether the trained second initial model has converged; if so, determine the converged second initial model as the sound detection sub-model.

[0098] In this embodiment, step 703 may include sub-steps 7031, 7032, and 7033. Wherein:

[0099] Step 7031: Fix the parameters of the pitch extraction sub-model, and train the second initial model using the second training sample set.

[0100] In this embodiment, the execution entity can fix the parameters of the pitch extraction sub-model and train the second initial model using the second training sample set. Specifically, the execution entity can use the fused feature samples of the second training samples in the second training sample set as the input of the second initial model, and use the sound detection result of the audio corresponding to the input fused feature samples as the output of the second initial model, thereby training the second initial model.

[0101] Step 7032: Determine whether the trained second initial model has converged.

[0102] In this embodiment, the execution entity can determine whether the trained second initial model has converged. If the trained second initial model has converged, step 7033 can be executed.

[0103] Step 7033: If the trained second initial model converges, then the converged second initial model is determined as the sound detection sub-model.

[0104] In this embodiment, if it is determined in step 7032 that the trained second initial model has reached convergence, the above-mentioned execution entity can determine the converged second initial model as the sound detection sub-model.

[0105] It should be noted that the execution entity for training the target model can also first use the second training sample set to train the second initial model until convergence, thus obtaining the sound detection sub-model. Then, the model parameters of the sound detection sub-model can be used with the first training sample set to train the first initial model until convergence, thus obtaining the pitch extraction sub-model.

[0106] The method provided in the above embodiments of this disclosure provides a training method for training a model with two branches by first training one of the sound detection sub-model or the pitch extraction sub-model until convergence, then fixing the model parameters of the trained model, and then training the other model until convergence.

[0107] In some optional implementations, if it is determined in step 7022 that the trained first initial model has not converged, the aforementioned execution entity can adjust the parameters of the first initial model, and use unused first training samples to form a first training sample set, using the adjusted first initial model as the first initial model, and continue executing the aforementioned first training step (i.e., step 702). As an example, the back propagation algorithm (BP algorithm) and gradient descent method (e.g., mini-batch gradient descent algorithm) can be used to adjust the parameters of the first initial model. It should be noted that the back propagation algorithm and gradient descent method are well-known techniques that are currently widely researched and applied, and will not be elaborated upon here.

[0108] In some optional implementations, if it is determined in step 7032 that the trained second initial model has not converged, the aforementioned execution entity can adjust the parameters of the second initial model, and use unused second training samples to form a second training sample set, using the adjusted second initial model as the second initial model, and continue to execute the aforementioned second training step (i.e., step 703). As an example, the backpropagation algorithm and gradient descent method can be used to adjust the parameters of the second initial model.

[0109] Further reference Figure 8 As an implementation of the methods shown in the above figures, this application provides an embodiment of a pitch extraction device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0110] like Figure 8 As shown, the pitch extraction device 800 of this embodiment includes: an acquisition unit 801, a conversion unit 802, and a determination unit 803. The acquisition unit 801 is used to acquire an audio segment and the timbre features of the sound-producing object within the audio segment; the conversion unit 802 is used to convert the audio segment into a Mel spectrum and to concatenate the timbre features with the Mel spectrum to obtain fused features; the determination unit 803 is used to determine the output result based on the fused features and a pre-trained target model, wherein the output result includes the pitch information of the sound-producing object.

[0111] In this embodiment, the specific processing of the acquisition unit 801, conversion unit 802, and determination unit 803 of the pitch extraction device 800 can be referred to Figure 1 The corresponding steps are 101, 102 and 103 in the embodiment.

[0112] In some alternative implementations, the aforementioned timbre features can be determined as follows: the aforementioned audio segment is input into a pre-trained speech object recognition model, and the feature vector output by the target convolutional layer in the speech object recognition model is determined as the aforementioned timbre features, wherein the speech object recognition model is used to characterize the correspondence between the input audio and the speech object recognition result corresponding to the input audio.

[0113] In some optional implementations, the aforementioned vocal object recognition model may include a singer recognition model, which is used to characterize the correspondence between the input audio and the corresponding singer recognition result.

[0114] In some optional implementations, the target model may include a pitch extraction sub-model and a sound detection sub-model; and the determining unit 803 may further be used to determine the output result based on the fusion features and the pre-trained target model in the following manner: the determining unit 803 may input the fusion features into the pitch extraction sub-model to obtain the pitch information of the vocal object; then, the fusion features may be input into the sound detection sub-model to obtain the sound detection result, wherein the sound detection result includes a first state probability value, a second state probability value and a third state probability value, the first state probability value is used to characterize the probability that there is a sound that matches the timbre features of the vocal object, the second state probability value is used to characterize the probability that there is no sound with extractable pitch, and the third state probability value is used to characterize the probability that the sound with extractable pitch does not match the timbre features of the vocal object.

[0115] In some optional implementations, the pitch extraction sub-model may include at least two convolutional layers; and the determining unit 803 may be further configured to input the fusion features into the pitch extraction sub-model to obtain the pitch information of the vocal object in the following manner: during the process of inputting the fusion features into the pitch extraction sub-model, the determining unit 803 may perform instance normalization and batch normalization on the output features of each target convolutional layer in the at least two convolutional layers, wherein the target convolutional layer is the first preset number of convolutional layers in the at least two convolutional layers; subsequently, batch normalization may be performed on the output features of each convolutional layer other than the target convolutional layer in the at least two convolutional layers.

[0116] In some optional implementations, the aforementioned sound detection sub-model may include at least two convolutional layers; and the aforementioned determining unit 803 may be further configured to input the aforementioned fused features into the aforementioned sound detection sub-model in the following manner to obtain the aforementioned sound detection result: the aforementioned determining unit 803 may, during the process of inputting the aforementioned fused features into the aforementioned sound detection sub-model, perform instance normalization and batch normalization processing on the output features of each of the aforementioned at least two convolutional layers, wherein the aforementioned target convolutional layer is the first preset number of convolutional layers among the aforementioned at least two convolutional layers; subsequently, batch normalization processing may be performed on the output features of each of the aforementioned at least two convolutional layers other than the aforementioned target convolutional layer.

[0117] In some alternative implementations, the target model can be trained as follows: First, a first training sample set and a second training sample set can be obtained, wherein the first training sample set is used to train the pitch extraction sub-model, and the second training sample set is used to train the sound detection sub-model; then, the following first training step can be performed: using the first training sample set, train the first initial model; determine whether the trained first initial model has converged; if so, determine the converged first initial model as the pitch extraction sub-model, and perform the second training step; finally, the following second training step can be performed: fixing the parameters of the pitch extraction sub-model, using the second training sample set, train the second initial model; determine whether the trained second initial model has converged; if so, determine the converged second initial model as the sound detection sub-model.

[0118] In some optional implementations, the training steps of the target model may further include: if the first initial model does not converge, adjusting the parameters of the first initial model, forming a first training sample set using unused first training samples, using the adjusted first initial model as the first initial model, and continuing to execute the first training steps.

[0119] In some optional implementations, the training steps of the target model may further include: if the second initial model does not converge, adjusting the parameters of the second initial model, forming a second training sample set using unused second training samples, using the adjusted second initial model as the second initial model, and continuing to execute the second training steps.

[0120] Please refer to Figure 9 , Figure 9 An exemplary system architecture in which a pitch extraction method according to an embodiment of this disclosure can be applied is illustrated.

[0121] likeFigure 9 As shown, the system architecture may include terminal devices 901, 902, and 903, a network 904, and a server 905. Network 904 serves as the medium for providing a communication link between terminal devices 901, 902, and 903 and server 905. Network 904 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0122] Terminal devices 901, 902, and 903 can interact with server 905 via network 904 to receive or send messages, etc. Various client applications can be installed on terminal devices 901, 902, and 903, such as music playback applications, voice recognition applications, and pitch extraction applications.

[0123] Terminal devices 901, 902, and 903 can first acquire audio segments and the timbre features of the sound-producing objects in the audio segments; then, they can convert the audio segments into Mel spectra and connect the timbre features with the Mel spectra to obtain fused features; finally, based on the fused features and the pre-trained target model, they can determine the output result including the pitch information of the sound-producing objects.

[0124] Terminal devices 901, 902, and 903 can be either hardware or software. When terminal devices 901, 902, and 903 are hardware, they can be various electronic devices with a display screen and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc. When terminal devices 901, 902, and 903 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0125] Server 905 can be a server that provides various services, such as receiving audio segments and timbre features of the sound-producing objects in the audio segments sent by terminal devices 901, 902, and 903; then, the audio segments can be converted into Mel spectra, and the timbre features can be concatenated with the Mel spectra to obtain fused features; finally, based on the fused features and the pre-trained target model, the output result including the pitch information of the sound-producing objects can be determined.

[0126] It should be noted that the pitch extraction method provided in this embodiment can be executed by a terminal device, and correspondingly, the pitch extraction device can be installed in terminal devices 901, 902, and 903. Furthermore, the pitch extraction method provided in this embodiment can also be executed by a server 905, and correspondingly, the pitch extraction device can be installed in server 905.

[0127] It should be understood that Figure 9 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0128] The following is for reference. Figure 10 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 9 The diagram shows the structure of the terminal device or server in this disclosure. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (e.g., vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0129] like Figure 10 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0130] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0131] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0132] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0133] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0134] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0135] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire an audio segment and the timbre features of the sound-producing object in the audio segment; convert the audio segment into a Mel spectrum and concatenate the timbre features with the Mel spectrum to obtain fused features; and determine an output result based on the fused features and a pre-trained target model, wherein the output result includes pitch information of the sound-producing object.

[0136] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, an acquisition unit can also be described as "a unit that acquires the timbre characteristics of an audio segment and the sound-producing object within the audio segment".

[0139] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0140] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0141] According to one or more embodiments of this disclosure, a pitch extraction method is provided, the method comprising: acquiring an audio segment and timbre features of a vocal object in the audio segment; converting the audio segment into a Mel spectrum, and concatenating the timbre features with the Mel spectrum to obtain fused features; and determining an output result based on the fused features and a pre-trained target model, wherein the output result includes pitch information of the vocal object.

[0142] According to one or more embodiments of this disclosure, the timbre features are determined by inputting an audio segment into a pre-trained speech object recognition model, and determining the feature vector output by the target convolutional layer in the speech object recognition model as the timbre features, wherein the speech object recognition model is used to characterize the correspondence between the input audio and the speech object recognition result corresponding to the input audio.

[0143] According to one or more embodiments of this disclosure, the vocal object recognition model includes a singer recognition model, which is used to characterize the correspondence between the input audio and the corresponding singer recognition result.

[0144] According to one or more embodiments of this disclosure, the target model includes a pitch extraction sub-model and a sound detection sub-model; and based on fusion features and a pre-trained target model, determining the output result includes: inputting fusion features into the pitch extraction sub-model to obtain pitch information of the vocal object; inputting fusion features into the sound detection sub-model to obtain a sound detection result, wherein the sound detection result includes a first state probability value, a second state probability value, and a third state probability value, the first state probability value being used to characterize the probability that there is a sound that matches the timbre features of the vocal object, the second state probability value being used to characterize the probability that there is no sound with extractable pitch, and the third state probability value being used to characterize the probability that the sound with extractable pitch does not match the timbre features of the vocal object.

[0145] According to one or more embodiments of this disclosure, a pitch extraction sub-model includes at least two convolutional layers; and inputting fused features into the pitch extraction sub-model to obtain pitch information of the vocal object includes: during the process of inputting fused features into the pitch extraction sub-model, for each target convolutional layer in the at least two convolutional layers, performing instance normalization processing and batch normalization processing on the output features of the target convolutional layer, wherein the target convolutional layer is the first preset number of convolutional layers in the at least two convolutional layers; and for each convolutional layer other than the target convolutional layer in the at least two convolutional layers, performing batch normalization processing on the output features of the convolutional layer.

[0146] According to one or more embodiments of this disclosure, a sound detection sub-model includes at least two convolutional layers; and inputting fused features into the sound detection sub-model to obtain a sound detection result includes: during the process of inputting fused features into the sound detection sub-model, for each target convolutional layer in the at least two convolutional layers, performing instance normalization processing and batch normalization processing on the output features of the target convolutional layer, wherein the target convolutional layer is the first preset number of convolutional layers in the at least two convolutional layers; and for each convolutional layer other than the target convolutional layer in the at least two convolutional layers, performing batch normalization processing on the output features of the convolutional layer.

[0147] According to one or more embodiments of this disclosure, the target model is trained as follows: A first training sample set and a second training sample set are obtained, wherein the first training sample set is used to train a pitch extraction sub-model, and the second training sample set is used to train a sound detection sub-model; a first training step is performed: training a first initial model using the first training sample set; determining whether the trained first initial model has converged; if so, determining the converged first initial model as the pitch extraction sub-model, and performing a second training step; a second training step is performed: fixing the parameters of the pitch extraction sub-model, training a second initial model using the second training sample set; determining whether the trained second initial model has converged; if so, determining the converged second initial model as the sound detection sub-model.

[0148] According to one or more embodiments of this disclosure, the method further includes: if the first initial model fails to converge, adjusting the parameters of the first initial model, forming a first training sample set using unused first training samples, using the adjusted first initial model as the first initial model, and continuing to execute the first training step.

[0149] According to one or more embodiments of this disclosure, the method further includes: if the second initial model fails to converge, adjusting the parameters of the second initial model, forming a second training sample set using unused second training samples, using the adjusted second initial model as the second initial model, and continuing to perform the second training step.

[0150] According to one or more embodiments of the present disclosure, a pitch extraction apparatus is provided, the apparatus comprising: an acquisition unit for acquiring an audio segment and timbre features of a sound-producing object in the audio segment; a conversion unit for converting the audio segment into a Mel spectrum and concatenating the timbre features with the Mel spectrum to obtain fused features; and a determination unit for determining an output result based on the fused features and a pre-trained target model, wherein the output result includes pitch information of the sound-producing object.

[0151] According to one or more embodiments of this disclosure, the timbre features are determined by inputting an audio segment into a pre-trained speech object recognition model, and determining the feature vector output by the target convolutional layer in the speech object recognition model as the timbre features, wherein the speech object recognition model is used to characterize the correspondence between the input audio and the speech object recognition result corresponding to the input audio.

[0152] According to one or more embodiments of this disclosure, the vocal object recognition model includes a singer recognition model, which is used to characterize the correspondence between the input audio and the corresponding singer recognition result.

[0153] According to one or more embodiments of this disclosure, the target model includes a pitch extraction sub-model and a sound detection sub-model; and the determining unit is further configured to determine the output result based on the fusion features and the pre-trained target model in the following manner: inputting the fusion features into the pitch extraction sub-model to obtain the pitch information of the vocal object; inputting the fusion features into the sound detection sub-model to obtain the sound detection result, wherein the sound detection result includes a first state probability value, a second state probability value and a third state probability value, the first state probability value being used to characterize the probability that there is a sound that matches the timbre features of the vocal object, the second state probability value being used to characterize the probability that there is no sound with extractable pitch, and the third state probability value being used to characterize the probability that the sound with extractable pitch does not match the timbre features of the vocal object.

[0154] According to one or more embodiments of this disclosure, a pitch extraction sub-model includes at least two convolutional layers; and a determining unit is further configured to input fused features into the pitch extraction sub-model to obtain pitch information of the vocal object in the following manner: during the process of inputting fused features into the pitch extraction sub-model, for each target convolutional layer in the at least two convolutional layers, instance normalization and batch normalization are performed on the output features of the target convolutional layer, wherein the target convolutional layer is the first preset number of convolutional layers in the at least two convolutional layers; for each convolutional layer other than the target convolutional layer in the at least two convolutional layers, batch normalization is performed on the output features of the convolutional layer.

[0155] According to one or more embodiments of this disclosure, a sound detection sub-model includes at least two convolutional layers; and a determining unit is further configured to input fused features into the sound detection sub-model to obtain a sound detection result by: during the process of inputting fused features into the sound detection sub-model, for each target convolutional layer in the at least two convolutional layers, performing instance normalization and batch normalization processing on the output features of the target convolutional layer, wherein the target convolutional layer is the first preset number of convolutional layers in the at least two convolutional layers; and for each convolutional layer other than the target convolutional layer in the at least two convolutional layers, performing batch normalization processing on the output features of the convolutional layer.

[0156] According to one or more embodiments of this disclosure, the target model is trained as follows: A first training sample set and a second training sample set are obtained, wherein the first training sample set is used to train a pitch extraction sub-model, and the second training sample set is used to train a sound detection sub-model; a first training step is performed: training a first initial model using the first training sample set; determining whether the trained first initial model has converged; if so, determining the converged first initial model as the pitch extraction sub-model, and performing a second training step; a second training step is performed: fixing the parameters of the pitch extraction sub-model, training a second initial model using the second training sample set; determining whether the trained second initial model has converged; if so, determining the converged second initial model as the sound detection sub-model.

[0157] According to one or more embodiments of this disclosure, the step of training the target model further includes: if the first initial model does not converge, adjusting the parameters of the first initial model, forming a first training sample set using unused first training samples, using the adjusted first initial model as the first initial model, and continuing to execute the first training step.

[0158] According to one or more embodiments of this disclosure, the step of training the target model further includes: if the second initial model does not converge, adjusting the parameters of the second initial model, forming a second training sample set using unused second training samples, using the adjusted second initial model as the second initial model, and continuing to perform the second training step.

[0159] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0160] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0161] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A pitch extraction method, characterized in that, include: Acquire the timbre characteristics of the audio segment and the sound-producing objects within the audio segment; The audio segment is converted into a Mel spectrum, and the timbre features are concatenated with the Mel spectrum to obtain the fusion features; Based on the fusion features and the pre-trained target model, the output result is determined, wherein the output result includes the pitch information of the vocal object; The target model includes a pitch extraction sub-model and a sound detection sub-model; and The process of determining the output result based on the fused features and the pre-trained target model includes: The fusion features are input into the pitch extraction sub-model to obtain the pitch information of the vocal object; The fusion features are input into the sound detection sub-model to obtain sound detection results, wherein the sound detection results include a first state probability value, a second state probability value, and a third state probability value. The first state probability value is used to characterize the probability that there is a sound that matches the timbre features of the vocal object. The second state probability value is used to characterize the probability that there is no sound with extractable pitch. The third state probability value is used to characterize the probability that the sound with extractable pitch does not match the timbre features of the vocal object.

2. The method according to claim 1, characterized in that, The timbre characteristics were determined in the following way: The audio segment is input into a pre-trained voice object recognition model, and the feature vector output by the target convolutional layer in the voice object recognition model is determined as the timbre feature. The voice object recognition model is used to characterize the correspondence between the input audio and the corresponding voice object recognition result.

3. The method according to claim 2, characterized in that, The vocal object recognition model includes a singer recognition model, which is used to represent the correspondence between the input audio and the corresponding singer recognition result.

4. The method according to claim 1, characterized in that, The pitch extraction sub-model includes at least two convolutional layers; as well as The step of inputting the fused features into the pitch extraction sub-model to obtain the pitch information of the vocal object includes: During the process of inputting the fused features into the pitch extraction sub-model, for each target convolutional layer in the at least two convolutional layers, the output features of the target convolutional layer are subjected to instance normalization and batch normalization, wherein the target convolutional layer is the first preset number of convolutional layers in the at least two convolutional layers. For each of the at least two convolutional layers other than the target convolutional layer, batch normalization is performed on the output features of that convolutional layer.

5. The method according to claim 1, characterized in that, The sound detection sub-model includes at least two convolutional layers; as well as The step of inputting the fused features into the sound detection sub-model to obtain the sound detection result includes: During the process of inputting the fused features into the sound detection sub-model, for each target convolutional layer in the at least two convolutional layers, the output features of the target convolutional layer are subjected to instance normalization and batch normalization, wherein the target convolutional layer is the first preset number of convolutional layers in the at least two convolutional layers. For each of the at least two convolutional layers other than the target convolutional layer, batch normalization is performed on the output features of that convolutional layer.

6. The method according to claim 1, characterized in that, The target model was trained in the following manner: Obtain a first training sample set and a second training sample set, wherein the first training sample set is used to train the pitch extraction sub-model, and the second training sample set is used to train the sound detection sub-model. The following first training step is performed: the first initial model is trained using the first training sample set; it is determined whether the first initial model obtained by training has reached convergence; if so, the first initial model that has reached convergence is determined as the pitch extraction sub-model, and the second training step is performed. The following second training step is performed: fix the parameters of the pitch extraction sub-model, and train the second initial model using the second training sample set; determine whether the trained second initial model has converged; if so, determine the converged second initial model as the sound detection sub-model.

7. The method according to claim 6, characterized in that, The method further includes: If the first initial model does not converge, the parameters of the first initial model are adjusted, and a first training sample set is formed using unused first training samples. The adjusted first initial model is then used as the first initial model, and the first training step is continued.

8. The method according to claim 6, characterized in that, The method further includes: If the second initial model does not converge, the parameters of the second initial model are adjusted, and a second training sample set is formed using unused second training samples. The adjusted second initial model is then used as the second initial model, and the second training step is continued.

9. A pitch extraction device, characterized in that, include: The acquisition unit is used to acquire audio segments and the timbre characteristics of the sound-producing objects in the audio segments; A conversion unit is used to convert the audio segment into a Mel spectrum and to connect the timbre features with the Mel spectrum to obtain a fusion feature; A determining unit is configured to determine an output result based on the fused features and a pre-trained target model, wherein the output result includes the pitch information of the vocal object; The target model includes a pitch extraction sub-model and a sound detection sub-model; and The determining unit is further configured to determine the output result based on the fused features and the pre-trained target model in the following manner: The fusion features are input into the pitch extraction sub-model to obtain the pitch information of the vocal object; The fusion features are input into the sound detection sub-model to obtain sound detection results, wherein the sound detection results include a first state probability value, a second state probability value, and a third state probability value. The first state probability value is used to characterize the probability that there is a sound that matches the timbre features of the vocal object. The second state probability value is used to characterize the probability that there is no sound with extractable pitch. The third state probability value is used to characterize the probability that the sound with extractable pitch does not match the timbre features of the vocal object.

10. An electronic device, characterized in that, include: At least one processor; Storage device having at least one program stored thereon When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images