Audio data processing method, device, terminal device and storage medium

Abstract

The present application discloses an audio data processing method, apparatus, terminal device, and storage medium. The method includes: obtaining a first pitch feature sequence corresponding to first audio data, where the first pitch feature sequence may be a pitch MIDI symbol sequence; encoding the first pitch feature sequence using an encoder, and obtaining a first encoded feature vector obtained by the encoder, where the encoder is trained based on multiple sample audio pairs; inputting the first encoded feature vector into a pitch recognition network, and determining a pitch recognition result for the first audio data using the pitch recognition network. Using the present application, the cost of pitch recognition can be reduced and the accuracy of pitch recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an audio data processing method, apparatus, terminal device, and storage medium. Background Art

[0002] With the development of artificial intelligence technology, mobile phones, tablet computers and other terminal devices have become an indispensable part of people's lives. Users can install various applications on the terminal to realize various application functions, thereby meeting their different needs in daily life, such as music software or karaoke software. In addition to providing the common song download function or music playback function, existing music software or karaoke software can also provide a song singing function and perform pitch determination on the user's singing songs to help users adjust the singing effect. In the related art pitch determination system, a pitch recognition method based on dry sound spectrum using convolutional network classification modeling is proposed. That is, by training the pitch recognition model, the spectrum corresponding to the audio data can be input to the pitch recognition model, and the pitch recognition model can output the pitch recognition result, that is, output the result of accurate pitch or inaccurate pitch. However, the pitch recognition model usually requires a large amount of audio data and manually labeled pitch labels corresponding to each audio data as training samples to train the model. This method, which relies on experienced teachers to determine the pitch of each audio data and label the pitch labels, is costly and the pitch recognition accuracy of the model is not very high. Summary of the Invention

[0003] The embodiments of the present application provide an audio data processing method, apparatus, terminal device, and storage medium, which can reduce the cost of pitch recognition and improve the accuracy of pitch recognition.

[0004] In a first aspect, an embodiment of the present application provides an audio data processing method, the method comprising:

[0005] Obtaining a first pitch feature sequence corresponding to the first audio data;

[0006] encoding the first pitch feature sequence by an encoder and obtaining a first encoding feature vector obtained by the encoder, wherein the encoder is trained based on a plurality of sample audio pairs;

[0007] The first encoded feature vector is input into a pitch recognition network, and a pitch recognition result of the first audio data is determined by the pitch recognition network.

[0008] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0009] Obtaining a masked pitch feature sequence associated with a pitch feature sequence corresponding to a target sample audio pair, wherein the target sample audio pair includes a first audio segment and a second audio segment, and the target sample audio pair is any one of the multiple sample audio pairs;

[0010] Determining an encoding feature vector corresponding to the masked pitch feature sequence through an initial encoder, performing pitch character prediction processing on the encoding feature vector according to a pitch character classification network to obtain pitch character prediction results corresponding to k masked characters in the masked pitch feature sequence, and performing audio classification prediction processing on the encoding feature vector according to an audio segment recognition network to obtain an audio classification prediction result of whether the first audio segment and the second audio segment in the target sample audio pair are derived from the same audio, where k is an integer greater than 0;

[0011] The network parameters of the initial encoder are adjusted according to the pitch character prediction result and the audio classification prediction result to obtain the encoder.

[0012] In conjunction with the first aspect, in one possible implementation, the encoded feature vector is an m×h dimensional feature matrix; and performing audio classification prediction processing on the encoded feature vector according to the audio segment recognition network to obtain an audio classification prediction result as to whether the first audio segment and the second audio segment in the target sample audio pair originate from the same audio includes:

[0013] Performing global average pooling on the encoded feature vector to obtain an audio feature vector, where the audio feature vector is a 1×h-dimensional feature vector;

[0014] The audio feature vector is input into the audio segment recognition network, and the audio segment recognition network is used to determine whether the first audio segment and the second audio segment are from the same audio, so as to obtain an audio classification prediction result.

[0015] In combination with the first aspect, in one possible implementation, adjusting the network parameters of the initial encoder according to the pitch character prediction result and the audio classification prediction result to obtain the encoder includes:

[0016] Determining a first loss function according to the pitch character prediction results corresponding to the k mask characters and the k pitch characters corresponding to the character positions of the k mask characters before masking;

[0017] A second loss function is determined according to the audio classification prediction result and the audio classification label corresponding to the target audio pair, so as to adjust the network parameters of the initial encoder according to the first loss function and the second loss function to obtain the encoder.

[0018] In combination with the first aspect, in a possible implementation, adjusting the network parameters of the initial encoder according to the first loss function and the second loss function includes:

[0019] Determine a joint loss function according to the first loss function and the second loss function;

[0020] The network parameters of the initial encoder are adjusted according to the joint loss function.

[0021] In conjunction with the first aspect, in one possible implementation, obtaining a masked pitch feature sequence associated with a pitch feature sequence corresponding to the target sample audio pair includes:

[0022] Obtaining a pitch feature sequence corresponding to the target sample audio pair, wherein the pitch feature sequence includes m pitch characters;

[0023] Masking is performed on the pitch characters in the pitch feature sequence according to a first masking rule to obtain a masked pitch feature sequence, wherein the masked pitch feature sequence includes k mask characters, where k is less than m.

[0024] In conjunction with the first aspect, in a possible implementation, obtaining the pitch feature sequence corresponding to the target sample audio pair includes:

[0025] Obtaining a pitch feature sequence corresponding to the first audio segment in the target sample audio pair, and obtaining a pitch feature sequence corresponding to the second audio segment in the target sample audio pair;

[0026] The pitch feature sequence corresponding to the first audio segment and the pitch feature sequence corresponding to the second audio segment are concatenated to obtain a pitch feature sequence corresponding to the target sample audio pair.

[0027] In combination with the first aspect, in a possible implementation, masking the pitch characters in the pitch feature sequence according to the first masking rule to obtain a masked pitch feature sequence includes:

[0028] replacing m1 pitch characters among the m pitch characters included in the pitch feature sequence with a first preset character; and / or,

[0029] The m2 pitch characters among the m pitch characters are replaced by adjacent pitch characters of the m2 pitch characters among the m pitch characters, and the sum of the m1 and m2 is equal to the k.

[0030] In conjunction with the first aspect, in one possible implementation, obtaining a first pitch feature sequence corresponding to the first audio data includes:

[0031] Determining frequency spectrum feature data corresponding to the first audio data;

[0032] A first pitch feature sequence corresponding to the first audio data is obtained according to the frequency spectrum feature data.

[0033] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0034] Acquire a sample audio data set, the sample audio data set comprising a plurality of sample audio data and a pitch label corresponding to each of the plurality of sample audio data;

[0035] For each sample audio data in the sample audio data set, obtaining, by the encoder, a coding feature vector corresponding to each sample audio data;

[0036] Inputting the encoded feature vector corresponding to each sample audio data into an initial pitch recognition network to obtain a pitch recognition result corresponding to each sample audio data output by the initial pitch recognition network;

[0037] The network parameters of the initial pitch recognition network are adjusted according to the pitch label corresponding to each sample audio data and the pitch recognition result corresponding to each sample audio data to obtain the pitch recognition network.

[0038] In a second aspect, an embodiment of the present application provides an audio data processing device, the device comprising:

[0039] a processing unit, configured to obtain a first pitch feature sequence corresponding to the first audio data;

[0040] The processing unit is configured to encode the first pitch feature sequence through an encoder and obtain a first encoding feature vector obtained by the encoder, wherein the encoder is trained based on a plurality of sample audio pairs;

[0041] The processing unit is configured to input the first encoded feature vector into a pitch recognition network, and determine a pitch recognition result of the first audio data through the pitch recognition network.

[0042] In conjunction with the second aspect, in one possible implementation, the processing unit is configured to:

[0043] Obtaining a masked pitch feature sequence associated with a pitch feature sequence corresponding to a target sample audio pair, wherein the target sample audio pair includes a first audio segment and a second audio segment, and the target sample audio pair is any one of the multiple sample audio pairs;

[0044] Determining an encoding feature vector corresponding to the masked pitch feature sequence through an initial encoder, performing pitch character prediction processing on the encoding feature vector according to a pitch character classification network to obtain pitch character prediction results corresponding to k masked characters in the masked pitch feature sequence, and performing audio classification prediction processing on the encoding feature vector according to an audio segment recognition network to obtain an audio classification prediction result of whether the first audio segment and the second audio segment in the target sample audio pair are derived from the same audio, where k is an integer greater than 0;

[0045] The network parameters of the initial encoder are adjusted according to the pitch character prediction result and the audio classification prediction result to obtain the encoder.

[0046] In conjunction with the second aspect, in one possible implementation, the encoded feature vector is an m×h dimensional feature matrix; and the processing unit is configured to:

[0047] Performing global average pooling on the encoded feature vector to obtain an audio feature vector, where the audio feature vector is a 1×h-dimensional feature vector;

[0048] The audio feature vector is input into the audio segment recognition network, and the audio segment recognition network is used to determine whether the first audio segment and the second audio segment are from the same audio, so as to obtain an audio classification prediction result.

[0049] In conjunction with the second aspect, in one possible implementation, the processing unit is configured to:

[0050] Determining a first loss function according to the pitch character prediction results corresponding to the k mask characters and the k pitch characters corresponding to the character positions of the k mask characters before masking;

[0051] A second loss function is determined according to the audio classification prediction result and the audio classification label corresponding to the target audio pair, so as to adjust the network parameters of the initial encoder according to the first loss function and the second loss function to obtain the encoder.

[0052] In conjunction with the second aspect, in one possible implementation, the processing unit is configured to:

[0053] Determine a joint loss function according to the first loss function and the second loss function;

[0054] The network parameters of the initial encoder are adjusted according to the joint loss function.

[0055] In conjunction with the second aspect, in one possible implementation, the processing unit is configured to:

[0056] Obtaining a pitch feature sequence corresponding to the target sample audio pair, wherein the pitch feature sequence includes m pitch characters;

[0057] Masking is performed on the pitch characters in the pitch feature sequence according to a first masking rule to obtain a masked pitch feature sequence, wherein the masked pitch feature sequence includes k mask characters, where k is less than m.

[0058] In conjunction with the second aspect, in one possible implementation, the processing unit is configured to:

[0059] Obtaining a pitch feature sequence corresponding to the first audio segment in the target sample audio pair, and obtaining a pitch feature sequence corresponding to the second audio segment in the target sample audio pair;

[0060] The pitch feature sequence corresponding to the first audio segment and the pitch feature sequence corresponding to the second audio segment are concatenated to obtain a pitch feature sequence corresponding to the target sample audio pair.

[0061] In conjunction with the second aspect, in one possible implementation, the processing unit is configured to:

[0062] replacing m1 pitch characters among the m pitch characters included in the pitch feature sequence with a first preset character; and / or,

[0063] The m2 pitch characters among the m pitch characters are replaced by adjacent pitch characters of the m2 pitch characters among the m pitch characters, and the sum of the m1 and m2 is equal to the k.

[0064] In conjunction with the second aspect, in one possible implementation, the processing unit is configured to:

[0065] Determining frequency spectrum feature data corresponding to the first audio data;

[0066] A first pitch feature sequence corresponding to the first audio data is obtained according to the frequency spectrum feature data.

[0067] In conjunction with the second aspect, in a possible implementation manner, the device further includes:

[0068] A transceiver unit is configured to obtain a sample audio data set, wherein the sample audio data set includes a plurality of sample audio data and a pitch label corresponding to each of the plurality of sample audio data;

[0069] The processing unit is configured to obtain, for each sample audio data in the sample audio data set, a coding feature vector corresponding to each sample audio data through the encoder;

[0070] The processing unit is configured to input the encoded feature vector corresponding to each sample audio data into an initial pitch recognition network to obtain a pitch recognition result corresponding to each sample audio data output by the initial pitch recognition network;

[0071] The processing unit is used to adjust the network parameters of the initial pitch recognition network according to the pitch label corresponding to each sample audio data and the pitch recognition result corresponding to each sample audio data to obtain the pitch recognition network.

[0072] In a third aspect, an embodiment of the present application provides a network device, comprising a processor, a memory, and a transceiver, the processor, the memory, and the transceiver being interconnected. The memory is configured to store a computer program that supports the network device in executing the method provided by the first aspect and / or any possible implementation of the first aspect, the computer program comprising program instructions, and the processor and the transceiver are configured to invoke the program instructions to execute the method provided by the first aspect and / or any possible implementation of the first aspect.

[0073] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program includes program instructions, which, when executed by a processor, enable the processor to execute the method provided by the above-mentioned first aspect and / or any possible implementation method of the first aspect.

[0074] In an embodiment of the present application, a first pitch feature sequence corresponding to the first audio data is obtained; the first pitch feature sequence is encoded by an encoder, and a first encoding feature vector obtained by the encoder processing is obtained, and the encoder is trained based on multiple sample audio pairs; the first encoding feature vector is input into the pitch recognition network, and the pitch recognition result of the first audio data is determined by the pitch recognition network. The embodiment of the present application obtains an encoder based on a large number of sample audio pairs for pre-training, which breaks through the limitation of manually labeled data (that is, the audio classification labels of the sample audio pairs involved in the embodiment of the present application are added according to the interception method when the terminal device automatically intercepts the sample audio pairs from the audio, so there is no need to manually mark them one by one), thereby reducing the cost of pitch recognition. Among them, by pre-training the encoder on a large number of sample audio pairs, the encoder can learn prior knowledge related to pitch, so that the encoder obtained by pre-training can have stronger characterization capabilities. Therefore, it is beneficial that in the subsequent use stage, the encoder obtained based on the pre-training can obtain a more accurate representation of the encoding feature vector of the audio data to be processed, thereby improving the accuracy of pitch recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0076] Figure 1 This is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0077] Figure 2 This is a flowchart of an audio data processing method provided by an embodiment of the present application;

[0078] Figure 3 1 is a flow chart of encoder training according to an embodiment of the present application;

[0079] Figure 4 Schematic diagram of a training process of an encoder according to an embodiment of the present application;

[0080] Figure 5 1 is a flow chart of a pitch recognition network training process provided by an embodiment of the present application;

[0081] Figure 6 is a structural diagram of an audio data processing device provided in an embodiment of the present application;

[0082] Figure 7 It is a structural diagram of the network device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0084] The embodiments of the present application relate to artificial intelligence (AI) and machine learning (ML). Among them, AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It mainly produces a new intelligent machine that can respond in a similar way to human intelligence by understanding the essence of intelligence, so that the intelligent machine has multiple functions such as perception, reasoning and decision-making.

[0085] AI technology is a comprehensive discipline encompassing several key areas, including computer vision (CV), speech processing, natural language processing, and machine learning (ML) / deep learning. Computer vision is the study of how machines can "see." Specifically, it involves using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. This is followed by further image processing, which transforms the computer's image into an image more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. This discipline typically encompasses techniques such as image processing, video processing, video semantic understanding, and video content / behavior recognition.

[0086] Machine learning is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of AI and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning / deep learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0087] Based on computer vision technology and machine learning technology in AI technology, an embodiment of the present application provides an audio data processing method, which can be used in scenarios such as real-time pitch score calculation for karaoke, overall pitch evaluation tasks, and various pitch detection. Specifically, the method includes: obtaining a first pitch feature sequence corresponding to the first audio data; encoding the first pitch feature sequence through an encoder, and obtaining a first encoding feature vector obtained by the encoder, the encoder being trained based on multiple sample audio pairs; inputting the first encoding feature vector into a pitch recognition network, and determining the pitch recognition result of the first audio data through the pitch recognition network.

[0088] It should be noted that the method in the embodiment of the present application can be used in scenarios such as pitch scoring in karaoke, pitch detection in content review, and singing dimension analysis. The pitch recognition model composed of the trained encoder and pitch recognition network can be accessed as an API service to the review backend. For high real-time requirements, such as real-time karaoke mobile terminal scoring, the encoder and pitch recognition network can be quantized and packaged and implanted into the mobile terminal as an SDK.

[0089] See Figure 1 , Figure 1 This is a schematic diagram of a scene of image processing provided by an embodiment of the present application. Figure 1 As shown, the image processing scenario includes a terminal device 101 and a server 102. The terminal device 101 is a device used by a user, and may include, but is not limited to, a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a portable personal computer, a mobile Internet device (MID), and the like; the terminal device is equipped with a display device, which may be a display, a display screen, a touch screen, etc., and the touch screen may be a touch screen, a touch panel, etc., which is not limited in the present embodiment.

[0090] The server 102 refers to a background device that can process the first audio data provided by the terminal device 101. After obtaining the pitch recognition result of the first audio data based on the first audio data, the server 102 can return the pitch recognition result to the terminal device 101. The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In addition, multiple servers can be grouped into a blockchain network, and each server is a node in the blockchain network. The terminal device 101 and the server 102 can be directly or indirectly connected through wired communication or wireless communication, and this application does not limit this.

[0091] It should be noted that Figure 1 The number of terminal devices and servers in the model processing scenario shown is for example only. For example, there can be multiple terminal devices and servers, and this application does not limit the number of terminal devices and servers. The methods provided in the embodiments of this application can be applied to servers as well as terminal devices, and are not limited here. For ease of description, the embodiments of this application may collectively refer to terminal devices and servers as network devices, and the following description will take network devices as an example.

[0092] The following will be combined Figures 2 to 7 The methods and related devices provided in the embodiments of the present application are described in detail respectively.

[0093] See Figure 2 , Figure 2 1 is a flow chart of an audio data processing method provided by an embodiment of the present application. The method provided by an embodiment of the present application may include the following steps S201 to S204:

[0094] S201. Obtain a first pitch feature sequence corresponding to first audio data.

[0095] In some feasible implementations, the network device obtains a first pitch feature sequence corresponding to the first audio data. Specifically, the pitch feature processing of the input first audio data can be performed through a Midi Note converter to obtain a corresponding pitch feature sequence, i.e., a first pitch feature sequence, or described as a pitch Midi symbol sequence. In other words, the first audio data can be directly input into the Midi Note converter to obtain the first pitch feature sequence corresponding to the first audio data output by the Midi Note converter.

[0096] Optionally, the above-mentioned acquisition of the first pitch feature sequence corresponding to the first audio data can also be understood as: first determining the spectral feature data corresponding to the first audio data, and then obtaining the first pitch feature sequence corresponding to the first audio data based on the spectral feature data. In other words, the spectral feature data corresponding to the first audio data can be input into the Midi Note converter to obtain the first pitch feature sequence corresponding to the first audio data. Among them, the first pitch feature sequence is a discrete representation of the continuous pitch of the first audio data. It should be noted that the above-mentioned spectral feature data can be a linear spectrum obtained after Fourier transforming the first audio data. Optionally, the above-mentioned spectral feature data can also be a Mel spectrum obtained by converting the linear spectrum obtained by Fourier transform after Fourier transforming the first audio data, etc., which is not limited here. Among them, the first audio data can be a dry sound, that is, a pure singing voice without accompaniment.

[0097] S202: Encode the first pitch feature sequence through an encoder, and obtain a first encoded feature vector obtained by the encoder.

[0098] In some feasible implementations, the network device encodes the first pitch feature sequence through an encoder and obtains a first encoded feature vector processed by the encoder, where the encoder is trained based on multiple sample audio pairs.

[0099] The following is a detailed description of the encoder training process. For example, see Figure 3 , Figure 3 : is a flow chart of encoder training provided in an embodiment of the present application. It includes the following steps S301 to S306:

[0100] S301. Obtain a pitch feature sequence corresponding to a target sample audio pair.

[0101] In some feasible implementations, the network device obtains a pitch feature sequence corresponding to a target sample audio pair, the target sample audio pair includes a first audio clip and a second audio clip, and the pitch feature sequence includes m pitch characters. The target sample audio pair is any one of a plurality of sample audio pairs, that is, each of the plurality of sample audio pairs can be used as a target sample audio pair in turn, and then the processing steps for the target sample audio pair described in the embodiment of the present application are performed on each sample audio pair. It should be noted that the first audio clip and the second audio clip included in the sample audio pair can be audio clips from the same user singing the same song, and the first audio clip and the second audio clip included in the sample audio pair can also be audio clips from different songs. Among them, when the first audio clip and the second audio clip are audio clips from the same user singing the same song, the two audio clips are usually two continuous audio segments. For example, the audio from the 10th to the 20th second of a song can be intercepted as the first audio clip, and the audio from the 20th to the 30th second of the song can be intercepted as the second audio clip.

[0102] It should be noted that the audio clips involved in the embodiments of this application are generally dry sounds. Optionally, the audio clips can also be singing voices with accompaniment, which is not a limitation here. For ease of understanding, the embodiments of this application are all illustrated by taking the audio clips as dry sounds as an example. Therefore, when an acquired audio clip is singing voice with accompaniment, the singing voice with accompaniment can be separated from the human voice to obtain the pure singing voice as one of the audio clips in the sample audio pair.

[0103] Specifically, the above-mentioned acquisition of the pitch feature sequence corresponding to the target sample audio pair can be understood as: acquiring the pitch feature sequence corresponding to the first audio segment in the target sample audio pair, and acquiring the pitch feature sequence corresponding to the second audio segment in the target sample audio pair. The pitch feature sequence corresponding to the first audio segment and the pitch feature sequence corresponding to the second audio segment are spliced ​​to obtain the pitch feature sequence corresponding to the target sample audio pair. Among them, for the two audio segments in the target sample audio pair (i.e., the first audio segment or the second audio segment), the spectral feature data of each audio segment can be calculated separately, and the pitch feature sequences corresponding to the two audio segments can be generated by inputting the calculated spectral feature data into the Midi Note converter. Furthermore, the pitch feature sequences corresponding to the two audio segments are spliced, and the sequence obtained after splicing can be used as the pitch feature sequence corresponding to the target sample audio pair. Exemplarily, it is assumed that the pitch feature sequence corresponding to the first audio segment is (mp1, mp2,…, mp n ), the pitch feature sequence corresponding to the second audio segment is (ms1, ms2, ..., ms l), the pitch feature sequences of the two audio clips are spliced ​​together to obtain (mp1, mp2, ..., mp n ,ms1,ms2,…,ms l ). It should be noted that Midi Note is a symbolic representation of Midi, where Midi values ​​use real numbers from 0 to 127 to represent pitches from C-2 to G8 (or are described as pitch symbols), which is a way of representing a pitch set. In the embodiment of the present application, the pitch feature sequence obtained by processing the audio data through the Midi Note converter is generally a pitch Midi symbol sequence, that is, the pitch feature sequence includes multiple pitch symbols, such as symbols such as C-2 or G8.

[0104] S302: Mask the pitch characters in the pitch feature sequence according to a first masking rule to obtain a masked pitch feature sequence.

[0105] In some feasible implementations, the network device masks the pitch characters in the pitch feature sequence according to the first masking rule to obtain a masked pitch feature sequence, wherein the masked pitch feature sequence includes k masked characters, and k is less than m. Exemplarily, the m1 pitch characters in the m pitch characters included in the pitch feature sequence can be replaced with the first preset character, and / or the m2 pitch characters in the m pitch characters can be replaced with adjacent pitch characters of the m2 pitch characters in the m pitch characters, and the sum of m1 and m2 is equal to k. The m1 pitch characters and / or m2 pitch characters can be a plurality of characters randomly selected from the m pitch characters, which are specifically determined according to the actual application scenario and are not limited here. For example, 20% of the pitch characters can be randomly selected from the m pitch characters for masking, that is, (m×20%) pitch characters are selected for masking. For example, for the 20% of pitch characters, half of the pitch characters in the 20% of pitch characters can be replaced with the first preset character (such as the [mask] character), and the other half of the pitch characters in the 20% of pitch characters can be replaced with their adjacent pitch characters. For another example, for the 20% of pitch characters, 80% of the pitch characters can be randomly selected from the 20% of pitch characters and replaced with the first preset character, and 10% of the pitch characters in the 20% of pitch characters can be replaced with their respective adjacent characters. Finally, the remaining 10% of the pitch characters in the 20% of pitch characters remain unchanged, that is, no processing is performed. The specific process is determined according to the actual scenario and is not limited here.

[0106] Among them, the first preset character can be a character [mask], etc., which is not limited here. Among them, in the embodiment of the present application, the adjacent pitch character of a certain pitch character can be the pitch character before the pitch character, or it can also be the pitch character after the pitch character, which is not limited here. For example, for the pitch feature sequence (mp1, mp2, ..., mp n ,ms1,ms2,…,ms l ), 20% of the pitch characters can be randomly selected from the pitch feature sequence for masking to obtain the masked pitch feature sequence (mp1, mask, ..., mp n ,mask,mask,…,ms l ).

[0107] S303 , encoding the masked pitch feature sequence through an initial encoder, and obtaining an encoded feature vector obtained by the initial encoder.

[0108] In some feasible implementations, the network device encodes the masked pitch feature sequence through an initial encoder, and can obtain the encoded feature vector obtained by the initial encoder. The above-mentioned initial encoder may include multiple layers of self-attention layers. Optionally, in addition to including multiple layers of self-attention layers, the above-mentioned initial encoder may also include a recurrent neural network layer, that is, the initial encoder may be composed of multiple layers of self-attention layers and recurrent neural network layers, wherein the output of the multiple layers of self-attention layers is the input of the recurrent neural network layer, and therefore, the output of the recurrent neural network layer can be used as the output result of the entire encoder. For ease of understanding, the embodiments of the present application are all schematically illustrated by taking an encoder based on a multi-layer self-attention layer as an example. It should be noted that the encoded feature vector obtained by the encoder processing in the embodiments of the present application is generally an m×h dimensional feature matrix. Wherein, m represents the length of the pitch feature sequence.

[0109] S304. Process the encoded feature vector through the pitch character classification network to obtain the pitch character prediction results of k masked characters, and determine the first loss function based on the pitch character prediction results of the k masked characters and the k pitch characters corresponding to the character positions of the k masked characters before masking processing.

[0110] In some feasible implementations, the network device processes the encoded feature vector through a pitch character classification network, and can obtain pitch character prediction results of k masked characters, and determine the first loss function based on the pitch character prediction results of the k masked characters and the k pitch characters corresponding to the character positions of the k masked characters before the masking process. That is to say, the pitch symbols corresponding to all masked pitch symbol positions can be classified and predicted by the pitch character classification network to obtain the pitch character prediction results of the k masked characters. Furthermore, the first loss function can be obtained by calculating the degree of proximity between the pitch character prediction results of the k masked characters and the actual corresponding pitch symbols at the k positions. It is understandable that the first loss function can be calculated using the cross entropy method. It should be noted that the pitch character classification network in the embodiment of the present application can adopt a fully connected classification layer, etc., which is not limited here. Among them, the pitch character classification network is a multi-classification network, and its classification result is related to the number of pitch symbols, which is not limited here. For example, the embodiment of the present application can be schematically explained by taking the 128 pitches including C-2 to G8 as an example.

[0111] S305. Process the encoded feature vector through an audio segment recognition network to obtain an audio classification prediction result as to whether the first audio segment and the second audio segment are from the same audio, and determine a second loss function based on the audio classification prediction result and the audio classification label of the target audio pair.

[0112] In some feasible implementations, the network device processes the encoded feature vector through an audio segment recognition network to obtain an audio classification prediction result of whether the first audio segment and the second audio segment are derived from the same audio, and determines the second loss function based on the audio classification prediction result and the audio classification label of the target audio pair. Specifically, processing the encoded feature vector through the audio segment recognition network to obtain an audio classification prediction result of whether the first audio segment and the second audio segment are derived from the same audio can be understood as: performing global average pooling processing on the encoded feature vector to obtain an audio feature vector, which is a 1×h-dimensional feature vector. Further, the audio feature vector is input into the audio segment recognition network, and the audio segment recognition network can be used to determine whether the first audio segment and the second audio segment are derived from the same audio to obtain an audio classification prediction result. In other words, the audio segment recognition network can be understood as a binary classification network, and the audio classification prediction result includes whether the first audio segment and the second audio segment are derived from the same audio, or whether the first audio segment and the second audio segment are not derived from the same audio. Furthermore, by calculating the loss function between the audio classification prediction result and the actual audio classification result corresponding to the target audio pair (i.e., the audio classification labels of the first audio segment and the second audio segment in the target audio pair), a second loss function can be obtained, where the second loss function can be calculated using a cross-entropy method. It should be noted that the audio segment recognition network in the embodiment of the present application can adopt a fully connected classification layer, etc., which is not limited here.

[0113] S306 : Adjust the network parameters of the initial encoder according to the first loss function and the second loss function to obtain an encoder.

[0114] In some feasible implementations, the network device adjusts the network parameters of the initial encoder according to the first loss function and the second loss function to obtain an encoder. Specifically, adjusting the network parameters of the encoder according to the first loss function and the second loss function can be understood as: determining the joint loss function according to the first loss function and the second loss function, and then adjusting the network parameters of the encoder according to the joint loss function. Wherein. The joint loss function can be equal to the sum of the first loss function and the second loss function. Optionally, the fusion loss function can also be equal to the product of the first loss function and the first weight value, and the product of the second loss function and the second weight value, wherein the sum of the first weight value and the second weight value is equal to 1. It should be noted that, in the embodiment of the present application, adjusting the network parameters of the encoder according to the joint loss function is actually adjusting the network parameters of the initial encoder, the pitch character classification network, and the audio segment recognition network according to the joint loss function, and then obtaining the adjusted initial encoder as the trained encoder.

[0115] For example, see Figure 4 , Figure 4 Schematic diagram of the training process of the encoder provided in the embodiment of the present application. Figure 4 As shown, it is assumed that the target sample audio pair includes a first audio segment x p and the second audio clip x s , respectively input the spectral feature data of the two audio clips into the Midi Note converter (Converter), and the pitch feature sequences corresponding to the two audio clips can be generated, such as Figure 4 First audio clip x p The corresponding pitch feature sequence (mp1, mp2, ..., mp n ), second audio clip x s The corresponding pitch feature sequence (ms1, ms2, ..., ms l ), where n+l=m. Then, the pitch feature sequences of the two audio clips are concatenated together to obtain (mp1, mp2, ..., mp n ,ms1,ms2,...,ms l ), and randomly mask out 20% (ie k = m * 20%) of the pitch characters, and the masked sequence can be obtained, that is, the masked pitch feature sequence (mp1, mask, ..., mp n ,mask,mask,...,ms l ), and then input the masked pitch feature sequence into the multi-layer self-attention mechanism layer (Multi-Layer Self Attention), which is used to perform deep feature encoding on the masked pitch feature sequence. After multiple layers of changes, it outputs a coded representation of the same sequence length, that is, a coded feature vector.

[0116] Finally, the pitch character classification network is used to classify and predict the pitch characters corresponding to all mask character positions, and the average classification prediction loss Loss1 (i.e., the first loss function) is calculated for the actual pitch characters. Here, L1 can be calculated using the cross entropy method. At the same time, the encoded feature vector is processed by global average pooling (AveragePool), or described as performing global average projection to obtain the audio feature vector as the overall representation of the two sequences, and then the audio segment recognition network is used to predict whether the two audio segments are the same song, that is, to predict that the target sample audio pair includes the first audio segment x p and the second audio clip x sWhether they come from the same audio, the classification loss function Loos2 of the audio classification prediction result and the audio classification label of the target audio pair is generated here. Finally, the two parts of the loss function are summarized to obtain the joint loss function Loss, and then combined with the random batch gradient descent method, the optimal solution is approximated using numerical iteration to train the network parameters of the encoder, pitch character classification network and audio segment recognition network. It should be noted that the pitch character classification network and the audio segment recognition network in the embodiment of the present application can both adopt a fully connected classification layer.

[0117] It should be noted that the pitch character classification network is a joint probability model P(x p , x s )=P(mp1,mp2,...,mp n ,ms1,ms2,...,ms l ), the audio segment recognition network for determining whether it is the same song is modeled as conditional probability P(Y|x p , x s )=P(Y|mp1,mp2,...,mp n ,ms1,ms2,...,ms l ).

[0118] In order to more clearly express the calculation process of the entire model, assume that x p , x s are the first and second audio segments in the target sample audio pair, respectively. The entire model calculation can be abstracted as follows:

[0119] X=MidiConvert( <x p , x s >)

[0120] X′=RandomMask(X)

[0121] R=MultiLayer(SelfAttention(X′))

[0122] L1=CrossEntropy(FC(R masked ), X masked )

[0123] L2=CrossEntropy(FC(AveragePool(R)),Y)

[0124] L=L1+L2

[0125] Among them, X represents the pitch feature sequence corresponding to the target sample audio pair, X′ represents the mask pitch feature sequence, R represents the encoded feature vector, L1 represents the first loss function, L2 represents the second loss function, L represents the joint loss function, FC represents the fully connected classification layer, AveragePool represents the global average pooling, and T represents the audio classification label of whether the two audio clips belong to the same song. If "yes", it is 1, and "no" is 0.

[0126] Generally speaking, during the construction phase of the sample audio pairs, that is, the data preparation phase of the training data, there is basically no limit on the amount of data (for example, more than 100,000 training data). This is because the audio classification labels of the sample audio pairs do not need to be manually labeled, but can be directly determined when the sample audio pairs are constructed. In other words, the audio classification labels of the sample audio pairs involved in the embodiments of the present application are added by the terminal device according to the interception method when the sample audio pairs are automatically intercepted from the audio, so there is no need to manually label them one by one. For example, if two clips of the same song are intercepted as sample audio pairs, then the audio classification label "derived from the same audio" is added; if two clips of different songs are intercepted as sample audio pairs, then the audio classification label "not derived from the same audio" is added. In the embodiments of the present application, during the data preparation phase of the sample audio pairs, songs with a playback volume greater than the playback volume threshold and / or a certain degree of good reviews can be selected as training data. The purpose of this is to eliminate some songs of very poor quality and to ensure that the pitch feature sequence used for pre-training is generally a relatively reasonable sequence, that is, a sequence with relatively accurate pitch, thereby reducing noise during pre-training.

[0127] S203: Input the first encoded feature vector into a pitch recognition network, and determine a pitch recognition result of the first audio data through the pitch recognition network.

[0128] In some feasible implementations, the network device inputs the first coded feature vector into the pitch recognition network, and the pitch recognition result of the first audio data can be determined by the pitch recognition network. Among them, the pitch recognition network is a two-category network, that is, the pitch recognition result includes two recognition results: accurate pitch and inaccurate pitch. It should be noted that, benefiting from the above-mentioned large-scale pre-training of the encoder based on a large amount of sample audio data, the encoder after pre-training can already better represent the coded representation of the pitch feature sequence, and can have a certain discrimination ability for the rationality of the pitch feature sequence. Based on this, for the pitch recognition network, since the pitch recognition network is a two-category network, its network is relatively simple, so only a small amount of labeled pitch classification data is needed to fine-tune the network parameters of the pitch recognition network to obtain better results. It should be noted that the "fine-tuning" in the embodiments of the present application can be understood as using a smaller learning rate during model training.

[0129] The following is a detailed description of the training process of the pitch recognition network. For example, see Figure 5 , Figure 5 : This is a flow chart of the pitch recognition network training process provided by the embodiment of the present application. It includes the following steps S501 to S504:

[0130] S501: Obtain a sample audio data set.

[0131] In some feasible implementations, a network device obtains a sample audio data set, wherein the sample audio data set includes a plurality of sample audio data, and a pitch label corresponding to each sample audio data in the plurality of sample audio data. It should be noted that the pitch label in the embodiment of the present application can be a classification result of whether the marked / annotated pitch is accurate after manual review. Since the encoder including the multi-layer attention mechanism layer has been pre-trained based on a large number of sample audio pairs, only a small amount of annotated sample audio data is needed to adjust the network parameters of the pitch recognition network. In other words, the number of sample audio data included in the sample audio data set can be only a few hundred, or a few thousand, etc., thereby reducing the cost of manual review.

[0132] It should be noted that the sample audio data involved in the embodiments of this application is generally dry sound. Optionally, the sample audio data may also be singing with accompaniment, without limitation. For ease of understanding, the embodiments of this application are all illustrated using the sample audio data as dry sound as an example. Therefore, when obtaining singing with accompaniment, it is necessary to first perform vocal separation on the singing with accompaniment to obtain pure singing as the sample audio data.

[0133] S502: For each sample audio data in the sample audio data set, obtain, through an encoder, a coding feature vector corresponding to each sample audio data.

[0134] In some feasible implementations, for each sample audio data in the sample audio data set, the encoder obtains an encoding feature vector corresponding to each sample audio data.

[0135] Specifically, the input sample audio data can first be processed for pitch features using a Midi Note converter to obtain a corresponding pitch feature sequence. The pitch feature sequence corresponding to each sample audio data is then input into an encoder to obtain an encoded feature vector for the pitch feature sequence corresponding to each sample audio data output by the encoder. Alternatively, the spectral feature data corresponding to the sample audio data can be first determined, and then the pitch feature sequence corresponding to the sample audio data can be obtained based on the spectral feature data. Specifically, the spectral feature data corresponding to the sample audio data can be input into a Midi Note converter to obtain a pitch feature sequence corresponding to the sample audio data. The pitch feature sequence corresponding to each sample audio data is then input into an encoder to obtain an encoded feature vector for the pitch feature sequence corresponding to each sample audio data output by the encoder. The pitch feature sequence is a discretized representation of the continuous pitch of the sample audio data. It should be noted that the spectral feature data can be a linear spectrum obtained by Fourier transforming the sample audio data. Alternatively, the spectral feature data can be a Mel spectrum obtained by Fourier transforming the linear spectrum obtained by Fourier transforming the sample audio data, etc., without limitation herein.

[0136] S503: Input the encoded feature vector corresponding to each sample audio data into the initial pitch recognition network to obtain the pitch recognition result corresponding to each sample audio data output by the initial pitch recognition network.

[0137] In some feasible implementations, the network device inputs the encoded feature vector corresponding to each audio sample into an initial pitch recognition network, and can obtain a pitch recognition result corresponding to each audio sample output by the initial pitch recognition network. The initial pitch recognition network can employ a fully connected classification layer, etc., without limitation herein.

[0138] S504. Adjust the network parameters of the initial pitch recognition network according to the pitch label corresponding to each sample audio data and the pitch recognition result corresponding to each sample audio data to obtain the pitch recognition network.

[0139] In some feasible implementations, the network device adjusts the network parameters of the initial pitch recognition network according to the pitch label corresponding to each sample audio data and the pitch recognition result corresponding to each sample audio data to obtain the pitch recognition network. Specifically, the third loss function can be determined by the pitch label corresponding to each sample audio data and the pitch recognition result corresponding to each sample audio data, and then the optimal solution is approximated by numerical iteration based on the third loss function in combination with the random batch gradient descent method. Generally speaking, fine-tuning can be understood as a model training in which the learning rate used is relatively small. Optionally, the embodiment of the present application can also adjust the network parameters of the initial pitch recognition network according to the pitch label corresponding to each sample audio data and the pitch recognition result corresponding to each sample audio data, while performing adjustment of the network parameters in the encoder, that is, the embodiment of the present application can also optimize the network parameters of the initial pitch recognition network and the network parameters of the encoder according to the pitch label corresponding to each sample audio data and the pitch recognition result corresponding to each sample audio data, and finally obtain a network model for judging pitch composed of an encoder and a pitch recognition network.

[0140] In an embodiment of the present application, a network device obtains a first pitch feature sequence corresponding to first audio data; encodes the first pitch feature sequence through an encoder and obtains a first encoded feature vector obtained by the encoder, the encoder being trained based on multiple sample audio pairs; the first encoded feature vector is input into a pitch recognition network, and the pitch recognition network determines the pitch recognition result of the first audio data. Using the embodiment of the present application, the cost of pitch recognition can be reduced and the accuracy of pitch recognition can be improved. It should be noted that the present application breaks through the limitation of labeled data through pre-training modeling, and can learn prior knowledge related to pitch on a large scale. Better performance can be achieved by simply adjusting the network parameters of the pitch recognition network on a small amount of data with pitch annotation, which greatly reduces the problem of difficulty in pitch annotation and the limited amount of annotated data, and greatly improves the model learning space. At the same time, based on the modeling method of the present application, the model learns the overall pitch semantics (this is because the encoder is pre-trained based on a large number of sample audio pairs), and the model will not be affected by the jitter of individual pitches on the evaluation, and the model has higher pitch recognition accuracy and robustness.

[0141] See Figure 6 , Figure 6 : is a structural diagram of an audio data processing device provided in an embodiment of the present application. The audio data processing device provided in an embodiment of the present application includes:

[0142] The processing unit 601 is configured to obtain a first pitch feature sequence corresponding to the first audio data;

[0143] The processing unit 601 is configured to encode the first pitch feature sequence through an encoder and obtain a first encoding feature vector obtained by the encoder, wherein the encoder is trained based on a plurality of sample audio pairs;

[0144] The processing unit 601 is configured to input the first encoded feature vector into a pitch recognition network, and determine a pitch recognition result of the first audio data through the pitch recognition network.

[0145] In a possible implementation, the processing unit 601 is configured to:

[0146] Obtaining a masked pitch feature sequence associated with a pitch feature sequence corresponding to a target sample audio pair, wherein the target sample audio pair includes a first audio segment and a second audio segment, and the target sample audio pair is any one of the multiple sample audio pairs;

[0147] Determining an encoding feature vector corresponding to the masked pitch feature sequence through an initial encoder, performing pitch character prediction processing on the encoding feature vector according to a pitch character classification network to obtain pitch character prediction results corresponding to k masked characters in the masked pitch feature sequence, and performing audio classification prediction processing on the encoding feature vector according to an audio segment recognition network to obtain an audio classification prediction result of whether the first audio segment and the second audio segment in the target sample audio pair are derived from the same audio, where k is an integer greater than 0;

[0148] The network parameters of the initial encoder are adjusted according to the pitch character prediction result and the audio classification prediction result to obtain the encoder.

[0149] In a possible implementation, the encoding feature vector is an m×h dimensional feature matrix; the processing unit 601 is configured to:

[0150] Performing global average pooling on the encoded feature vector to obtain an audio feature vector, where the audio feature vector is a 1×h-dimensional feature vector;

[0151] The audio feature vector is input into the audio segment recognition network, and the audio segment recognition network is used to determine whether the first audio segment and the second audio segment are from the same audio, so as to obtain an audio classification prediction result.

[0152] In a possible implementation, the processing unit 601 is configured to:

[0153] Determining a first loss function according to the pitch character prediction results corresponding to the k mask characters and the k pitch characters corresponding to the character positions of the k mask characters before masking;

[0154] A second loss function is determined according to the audio classification prediction result and the audio classification label corresponding to the target audio pair, so as to adjust the network parameters of the initial encoder according to the first loss function and the second loss function to obtain the encoder.

[0155] In a possible implementation, the processing unit 601 is configured to:

[0156] Determine a joint loss function according to the first loss function and the second loss function;

[0157] The network parameters of the initial encoder are adjusted according to the joint loss function.

[0158] In a possible implementation, the processing unit 601 is configured to:

[0159] Obtaining a pitch feature sequence corresponding to the target sample audio pair, wherein the pitch feature sequence includes m pitch characters;

[0160] Masking is performed on the pitch characters in the pitch feature sequence according to a first masking rule to obtain a masked pitch feature sequence, wherein the masked pitch feature sequence includes k mask characters, where k is less than m.

[0161] In a possible implementation, the processing unit 601 is configured to:

[0162] Obtaining a pitch feature sequence corresponding to the first audio segment in the target sample audio pair, and obtaining a pitch feature sequence corresponding to the second audio segment in the target sample audio pair;

[0163] The pitch feature sequence corresponding to the first audio segment and the pitch feature sequence corresponding to the second audio segment are concatenated to obtain a pitch feature sequence corresponding to the target sample audio pair.

[0164] In a possible implementation, the processing unit 601 is configured to:

[0165] replacing m1 pitch characters among the m pitch characters included in the pitch feature sequence with a first preset character; and / or,

[0166] The m2 pitch characters among the m pitch characters are replaced by adjacent pitch characters of the m2 pitch characters among the m pitch characters, and the sum of the m1 and m2 is equal to the k.

[0167] In a possible implementation, the processing unit 601 is configured to:

[0168] Determining frequency spectrum feature data corresponding to the first audio data;

[0169] A first pitch feature sequence corresponding to the first audio data is obtained according to the frequency spectrum feature data.

[0170] In a possible implementation, the device further includes:

[0171] The transceiver unit 602 is configured to obtain a sample audio data set, wherein the sample audio data set includes a plurality of sample audio data and a pitch label corresponding to each of the plurality of sample audio data;

[0172] The processing unit 601 is configured to obtain, for each sample audio data in the sample audio data set, a coding feature vector corresponding to each sample audio data through the encoder;

[0173] The processing unit 601 is configured to input the encoded feature vector corresponding to each sample audio data into an initial pitch recognition network to obtain a pitch recognition result corresponding to each sample audio data output by the initial pitch recognition network;

[0174] The processing unit 601 is configured to adjust the network parameters of the initial pitch recognition network according to the pitch label corresponding to each sample audio data and the pitch recognition result corresponding to each sample audio data, so as to obtain the pitch recognition network.

[0175] In an embodiment of the present application, an audio data processing device obtains a first pitch feature sequence corresponding to first audio data; encodes the first pitch feature sequence using an encoder and obtains a first encoded feature vector obtained by the encoder, the encoder being trained based on multiple sample audio pairs; inputs the first encoded feature vector into a pitch recognition network, and determines a pitch recognition result for the first audio data using the pitch recognition network. Using this embodiment of the present application, the cost of pitch recognition can be reduced and the accuracy of pitch recognition can be improved.

[0176] See Figure 7 , Figure 7 Schematic diagram of the structure of the network device provided in the embodiment of the present application. The network device can be a server, or a terminal device, etc., which is not limited here. Figure 7 As shown, the network device in this embodiment may include: one or more processors 701, memory 702, and transceiver 703. The processor 701, memory 702, and transceiver 703 are connected via a bus 704. The memory 702 is used to store a computer program, which includes program instructions. The processor 701 and transceiver 703 are used to execute the program instructions stored in the memory 702 to perform the following operations:

[0177] Processor 701, configured to obtain a first pitch feature sequence corresponding to first audio data;

[0178] The processor 701 is configured to encode the first pitch feature sequence through an encoder and obtain a first encoding feature vector obtained by the encoder, wherein the encoder is trained based on multiple sample audio pairs;

[0179] The processor 701 is configured to input the first encoded feature vector into a pitch recognition network, and determine a pitch recognition result of the first audio data through the pitch recognition network.

[0180] In a possible implementation, the device further includes:

[0181] A transceiver 703 is configured to obtain a pitch feature sequence corresponding to a target sample audio pair, wherein the target sample audio pair includes a first audio segment and a second audio segment, the pitch feature sequence includes m pitch characters, and the target sample audio pair is any one of the multiple sample audio pairs;

[0182] The processor 701 is configured to perform masking processing on the pitch characters in the pitch feature sequence according to a first masking rule to obtain a masked pitch feature sequence, wherein the masked pitch feature sequence includes k mask characters, where k is less than m;

[0183] The processor 701 is configured to encode the masked pitch feature sequence through an initial encoder and obtain an encoded feature vector obtained by the initial encoder;

[0184] The processor 701 is configured to process the encoded feature vector using a pitch character classification network to obtain pitch character prediction results of the k masked characters, and determine a first loss function based on the pitch character prediction results of the k masked characters and k pitch characters corresponding to the character positions of the k masked characters before masking;

[0185] The processor 701 is configured to process the encoded feature vector using an audio segment recognition network to obtain an audio classification prediction result as to whether the first audio segment and the second audio segment are from the same audio, and determine a second loss function based on the audio classification prediction result and the audio classification label of the target audio pair;

[0186] The processor 701 is configured to adjust network parameters of the initial encoder according to the first loss function and the second loss function to obtain the encoder.

[0187] In a possible implementation, the transceiver 703 is configured to obtain a sample audio data set, where the sample audio data set includes a plurality of sample audio data and a pitch label corresponding to each of the plurality of sample audio data;

[0188] The processor 701 is configured to obtain, for each sample audio data in the sample audio data set, a coding feature vector corresponding to each sample audio data through the encoder;

[0189] The processor 701 is configured to input the encoded feature vector corresponding to each sample audio data into the pitch recognition network to obtain the pitch recognition result corresponding to each sample audio data output by the pitch recognition network;

[0190] The processor 701 is configured to adjust network parameters of an initial pitch recognition network according to the pitch label corresponding to each audio sample data and the pitch recognition result corresponding to each audio sample data, so as to obtain the pitch recognition network.

[0191] In a possible implementation, the processor 701 is specifically configured to:

[0192] Obtaining a pitch feature sequence corresponding to the first audio segment in the target sample audio pair, and obtaining a pitch feature sequence corresponding to the second audio segment in the target sample audio pair;

[0193] The pitch feature sequence corresponding to the first audio segment and the pitch feature sequence corresponding to the second audio segment are concatenated to obtain a pitch feature sequence corresponding to the target sample audio pair.

[0194] In a possible implementation, the processor 701 is specifically configured to:

[0195] replacing m1 pitch characters among the m pitch characters included in the pitch feature sequence with a first preset character; and / or,

[0196] The m2 pitch characters among the m pitch characters are replaced by adjacent pitch characters of the m2 pitch characters among the m pitch characters, and the sum of the m1 and m2 is equal to the k.

[0197] In a possible implementation, the processor 701 is specifically configured to:

[0198] Determine a joint loss function according to the first loss function and the second loss function;

[0199] The network parameters of the encoder are adjusted according to the joint loss function.

[0200] In a possible implementation, the encoded feature vector is an m×h dimensional feature matrix; the processor 701 is specifically configured to:

[0201] Performing global average pooling on the encoded feature vector to obtain an audio feature vector, where the audio feature vector is a 1×h-dimensional feature vector;

[0202] The audio feature vector is input into the audio segment recognition network, and the audio segment recognition network is used to determine whether the first audio segment and the second audio segment are from the same audio, so as to obtain an audio classification prediction result.

[0203] In a possible implementation, the processor 701 is specifically configured to:

[0204] Determining frequency spectrum feature data corresponding to the first audio data;

[0205] A first pitch feature sequence corresponding to the first audio data is obtained based on the spectral feature data. It should be understood that in some feasible implementations, the above-mentioned processor 701 can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The memory 702 can include a read-only memory and a random access memory, and provide instructions and data to the processor 701. A portion of the memory 702 can also include a non-volatile random access memory. For example, the memory 702 can also store information about the device type.

[0206] In a specific implementation, the above network equipment can execute the above-mentioned functions through its built-in functional modules. Figures 2 to 5 For the implementation methods provided in each step, please refer to the implementation methods provided in the above steps for details, which will not be repeated here.

[0207] In an embodiment of the present application, a network device obtains a first pitch feature sequence corresponding to first audio data; encodes the first pitch feature sequence using an encoder and obtains a first encoded feature vector obtained by the encoder, the encoder being trained based on multiple sample audio pairs; inputs the first encoded feature vector into a pitch recognition network, and the pitch recognition network determines a pitch recognition result for the first audio data. Using this embodiment of the present application, the cost of pitch recognition can be reduced and the accuracy of pitch recognition can be improved.

[0208] The present invention also provides a computer-readable storage medium that stores a computer program. The computer program includes program instructions that are executed by a processor to implement Figures 2 to 5 For the audio data processing methods provided in each step, please refer to the implementation methods provided in the above steps, which will not be repeated here.

[0209] The above-mentioned computer-readable storage medium can be the internal storage unit of the audio data processing device or the above-mentioned terminal device provided in any of the aforementioned embodiments, such as the hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Furthermore, the computer-readable storage medium can also include both the internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0210] The terms "first," "second," "third," "fourth," and the like in the claims, specification, and drawings of this application are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0211] References to "embodiments" herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the present application. The appearance of such a phrase in various locations in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, that the embodiments described herein may be combined with other embodiments. The term "and / or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, including such combinations. Those skilled in the art will appreciate that the elements and algorithmic steps of the various examples described in connection with the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0212] The methods and related devices provided by the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided by the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

Claims

1. A method for processing audio data, characterized in that: The method comprises: Obtaining a first pitch feature sequence corresponding to the first audio data; encoding the first pitch feature sequence by an encoder and obtaining a first encoding feature vector obtained by the encoder, wherein the encoder is trained based on a plurality of sample audio pairs; Inputting the first encoded feature vector into a pitch recognition network, and determining a pitch recognition result of the first audio data through the pitch recognition network; The method further comprises: Obtaining a masked pitch feature sequence associated with a pitch feature sequence corresponding to a target sample audio pair, wherein the target sample audio pair includes a first audio segment and a second audio segment, and the target sample audio pair is any one of the multiple sample audio pairs; Determining an encoding feature vector corresponding to the masked pitch feature sequence through an initial encoder, performing pitch character prediction processing on the encoding feature vector according to a pitch character classification network to obtain pitch character prediction results corresponding to k masked characters in the masked pitch feature sequence, and performing audio classification prediction processing on the encoding feature vector according to an audio segment recognition network to obtain an audio classification prediction result of whether the first audio segment and the second audio segment in the target sample audio pair are derived from the same audio, where k is an integer greater than 0; The network parameters of the initial encoder are adjusted according to the pitch character prediction result and the audio classification prediction result to obtain the encoder.

2. The method according to claim 1, characterized in that The encoded feature vector is an m×h dimensional feature matrix; performing audio classification prediction processing on the encoded feature vector according to the audio segment recognition network to obtain an audio classification prediction result of whether the first audio segment and the second audio segment in the target sample audio pair are from the same audio, includes: Performing global average pooling on the encoded feature vector to obtain an audio feature vector, where the audio feature vector is a 1×h-dimensional feature vector; The audio feature vector is input into the audio segment recognition network, and the audio segment recognition network is used to determine whether the first audio segment and the second audio segment are from the same audio, so as to obtain an audio classification prediction result.

3. The method according to claim 1, characterized in that The step of adjusting the network parameters of the initial encoder according to the pitch character prediction result and the audio classification prediction result to obtain the encoder comprises: Determining a first loss function according to the pitch character prediction results corresponding to the k mask characters and the k pitch characters corresponding to the character positions of the k mask characters before masking; A second loss function is determined according to the audio classification prediction result and the audio classification label corresponding to the target audio pair, so as to adjust the network parameters of the initial encoder according to the first loss function and the second loss function to obtain the encoder.

4. The method according to claim 3, characterized in that The adjusting the network parameters of the initial encoder according to the first loss function and the second loss function includes: Determine a joint loss function according to the first loss function and the second loss function; The network parameters of the initial encoder are adjusted according to the joint loss function.

5. The method according to any one of claims 1 to 4, characterized in that The step of obtaining a masked pitch feature sequence associated with a pitch feature sequence corresponding to a target sample audio pair includes: Obtaining a pitch feature sequence corresponding to the target sample audio pair, wherein the pitch feature sequence includes m pitch characters; Masking is performed on the pitch characters in the pitch feature sequence according to a first masking rule to obtain a masked pitch feature sequence, wherein the masked pitch feature sequence includes k mask characters, where k is less than m.

6. The method according to claim 5, characterized in that The obtaining of the pitch feature sequence corresponding to the target sample audio pair includes: Obtaining a pitch feature sequence corresponding to the first audio segment in the target sample audio pair, and obtaining a pitch feature sequence corresponding to the second audio segment in the target sample audio pair; The pitch feature sequence corresponding to the first audio segment and the pitch feature sequence corresponding to the second audio segment are concatenated to obtain a pitch feature sequence corresponding to the target sample audio pair.

7. The method according to claim 5, characterized in that The step of performing masking processing on the pitch characters in the pitch feature sequence according to the first masking rule to obtain a masked pitch feature sequence includes: replacing m1 pitch characters among the m pitch characters included in the pitch feature sequence with a first preset character; and / or, The m2 pitch characters among the m pitch characters are replaced by adjacent pitch characters of the m2 pitch characters among the m pitch characters, and the sum of the m1 and m2 is equal to the k.

8. The method according to any one of claims 1 to 4, characterized in that The obtaining of a first pitch feature sequence corresponding to the first audio data includes: Determining frequency spectrum feature data corresponding to the first audio data; A first pitch feature sequence corresponding to the first audio data is obtained according to the frequency spectrum feature data.

9. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Acquire a sample audio data set, the sample audio data set comprising a plurality of sample audio data and a pitch label corresponding to each of the plurality of sample audio data; For each sample audio data in the sample audio data set, obtaining, by the encoder, a coding feature vector corresponding to each sample audio data; Inputting the encoded feature vector corresponding to each sample audio data into an initial pitch recognition network to obtain a pitch recognition result corresponding to each sample audio data output by the initial pitch recognition network; The network parameters of the initial pitch recognition network are adjusted according to the pitch label corresponding to each sample audio data and the pitch recognition result corresponding to each sample audio data to obtain the pitch recognition network.

10. A network device, characterized in that: comprising a processor, a memory and a transceiver, wherein the processor, the memory and the transceiver are connected to each other; The memory is used to store a computer program, the computer program includes program instructions, and the processor and the transceiver are configured to call the program instructions to execute the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 9.

12. A computer program product, characterized in that The computer program product includes computer program code, and when the computer program code is run on a computer, the method according to any one of claims 1 to 9 is implemented.

Images