Audio spectrum conversion method and device, computer equipment, storage medium and program product

Abstract

The invention relates to an audio spectrum conversion method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: determining note prediction probability distribution of each audio frame according to spectrum characteristics of each audio frame in song audio; according to the note prediction probability distribution of each audio frame, determining a candidate note state corresponding to each audio frame in the plurality of note states; splicing each candidate note state corresponding to each audio frame and each candidate note state corresponding to the adjacent audio frame in sequence to obtain a plurality of candidate note state chains; according to a state transition probability among the plurality of note states, screening out a target note boundary of each note of the song audio from candidate note boundaries of each note identified in each candidate note state chain; and according to the pitch sequence of the song audio and the target note boundary of each note of the song audio, generating a music score conversion result corresponding to the song to be subjected to music score conversion. By adopting the method, the accuracy of audio spectrum conversion can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio spectrum transposition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Audio transcription refers to the process of analyzing information such as pitch and note boundaries in an audio signal and converting it into corresponding musical notation or musical notation. With the widespread application of artificial intelligence generated content (AIGC), audio transcription technology has greatly advanced and can be applied in a variety of practical scenarios. For example, audio transcription can provide sheet music for vocal synthesis, reducing the cost of manual transcription. It can also provide a basis for pitch scoring in singing applications. However, these practical applications require very high accuracy in audio transcription. If the audio transcription is not accurate, the vocal synthesis will be off-tune and the pronunciation will not align. In addition, users may receive incorrect scores even if they sing correctly.

[0003] Existing audio-to-spectrum conversion algorithms typically use a multi-task learning approach, where the model simultaneously learns pitch information and note boundary information, and superimposes the two losses for learning. However, it is difficult to accurately balance the weights of the two losses during training, and it is easy for one of the losses to be poorly learned while the other is over-learned, resulting in low audio-to-spectrum conversion accuracy. Summary of the Invention

[0004] Based on this, it is necessary to provide an audio transcription method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the accuracy of audio transcription in order to address the above technical problems.

[0005] In a first aspect, the present application provides an audio spectrum transcoding method, comprising:

[0006] Obtaining the song audio of the song to be transcribed and the pitch sequence of the song audio;

[0007] Determine a note prediction probability distribution for each audio frame based on the spectral features of each audio frame in the song audio; wherein the note prediction probability distribution for the audio frame includes a prediction probability that the audio frame belongs to each of a plurality of note states; each of the plurality of note states is a note transition state or a note duration state described by a hidden Markov model;

[0008] Determining at least one candidate note state corresponding to each audio frame from the plurality of note states based on the note prediction probability distribution of each audio frame; and sequentially concatenating each candidate note state corresponding to each audio frame with each candidate note state corresponding to an adjacent audio frame according to the order of the audio frames in the song audio to obtain a plurality of candidate note state chains;

[0009] Identifying candidate note boundaries of each note of the song audio from each of the candidate note state chains; and screening target note boundaries of each note of the song audio from the candidate note boundaries of each note of the song audio identified from each of the candidate note state chains based on the state transition probabilities between the plurality of note states;

[0010] A transcription result corresponding to the song to be transcribed is generated based on the pitch sequence of the song audio and the target note boundaries of each note of the song audio.

[0011] In one embodiment, at least one candidate note state corresponding to each audio frame is determined from the plurality of note states based on the note prediction probability distribution of each audio frame; and each candidate note state corresponding to each audio frame is sequentially concatenated with each candidate note state corresponding to an adjacent audio frame according to the order of the audio frames in the song audio, to obtain a plurality of candidate note state chains, including:

[0012] Constructing a note prediction probability matrix based on the note prediction probability distribution of each audio frame; the element in the i-th row and j-th column of the note prediction probability matrix is used to represent the predicted probability of the j-th note state of the i-th audio frame, and the order of the audio frames in each row corresponds to the order of each audio frame in the song audio;

[0013] For each row of the note prediction probability matrix, based on whether the predicted probabilities of the respective note states in the row meet a preset condition, selecting at least one note state from the note states in each column as at least one candidate note state for the audio frame corresponding to the row;

[0014] According to the order of the audio frames in each row, each candidate note state selected in each row is sequentially spliced front to back with each candidate note state selected in the adjacent row to obtain multiple candidate note state chains.

[0015] In one embodiment, the step of selecting target note boundaries for each note of the song audio from candidate note boundaries for each note of the song audio identified in each of the candidate note state chains based on the state transition probabilities between the multiple note states includes:

[0016] For each candidate note state chain in the note prediction probability matrix, extracting the state transition probability between adjacent candidate note states in the candidate note state chain from the state transition probabilities between the multiple note states; calculating the path probability of the candidate note state chain based on the prediction probabilities of the respective candidate note states included in the candidate note state chain and the state transition probabilities between the respective adjacent candidate note states in the candidate note state chain;

[0017] From the multiple candidate note state chains, the candidate note state chain with the highest path probability is selected as the optimal state path; and the candidate note boundaries of each note of the song audio in the optimal state path are used as the target note boundaries of each note of the song audio.

[0018] In one embodiment, the calculating of the path probability of the candidate note state chain based on the predicted probability of each candidate note state included in the candidate note state chain and the state transition probability between each adjacent candidate note state in the candidate note state chain includes:

[0019] Calculating the product of the predicted probabilities of the candidate note states included in the candidate note state chain to obtain a predicted probability product value;

[0020] Calculating the product of the state transition probabilities between adjacent candidate note states in the candidate note state chain to obtain a transition probability product value;

[0021] The product of the predicted probability product value and the transition probability product value is determined as the path probability of the candidate note state chain.

[0022] In one embodiment, before selecting target note boundaries of each note of the song audio from the candidate note boundaries of each note of the song audio identified in each candidate note state chain based on the state transition probabilities between the multiple note states, the method further includes:

[0023] Converting a note sequence formed by sorting the notes of the song audio into a note state sequence;

[0024] The state transition probability between the plurality of note states is calculated according to the note state sequence by using a hidden Markov model.

[0025] In one embodiment, converting the note sequence formed by sorting the notes of the song audio into a note state sequence includes:

[0026] Based on the mapping relationship between the notes and the note states recorded in the character dictionary, each note in the note sequence is mapped to a note state to obtain the note state sequence.

[0027] In one embodiment, obtaining the song audio of the song to be transcribed and the pitch sequence of the song audio includes:

[0028] Slicing the song audio of the song to be transcribed according to the notes to obtain original audio segments corresponding to each of the notes; wherein the song audio in each of the original audio segments has the same pitch;

[0029] Adjusting the duration of each original audio segment to the target segment duration to obtain multiple note segments;

[0030] A pitch sequence of the song audio is obtained according to the frequency spectrum features of the multiple note segments.

[0031] In one embodiment, adjusting the duration of each original audio segment to a target segment duration to obtain multiple note segments includes:

[0032] If the duration of the original audio segment is shorter than the duration of the target segment, supplementing the original audio segment with blank audio to obtain supplemented audio having a duration equal to the duration of the target segment as the note segment;

[0033] When the duration of the original audio segment is longer than the target segment duration, the original audio segment is truncated to obtain a truncated audio with a duration equal to the target segment duration as the note segment.

[0034] In a second aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0035] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0036] In a fourth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0037] The above-mentioned audio transcription method, apparatus, computer device, computer-readable storage medium and computer program product obtain the song audio of the song to be transcribed and the pitch sequence of the song audio; determine the note prediction probability distribution of each audio frame based on the spectral characteristics of each audio frame in the song audio, wherein the note prediction probability distribution of the audio frame includes the prediction probability of the audio frame belonging to each note state in a plurality of note states, each note state in the plurality of note states being a note transition state or a note duration state described by a hidden Markov model; determine at least one candidate note corresponding to each audio frame in the plurality of note states based on the note prediction probability distribution of each audio frame state, and according to the order of each audio frame in the song audio, each candidate note state corresponding to each audio frame is sequentially spliced with each candidate note state corresponding to the adjacent audio frame to obtain multiple candidate note state chains; the candidate note boundaries of each note of the song audio are respectively identified from each candidate note state chain, and according to the state transition probability between the multiple note states, the target note boundaries of each note of the song audio are screened out from the candidate note boundaries of each note of the song audio identified in each candidate note state chain; according to the pitch sequence of the song audio and the target note boundaries of each note of the song audio, the transcription result corresponding to the song to be transcribed is generated. The note state is modeled through the hidden Markov model, and based on the independent extraction of the pitch sequence, the multi-state probability distribution of each audio frame is obtained to improve the accuracy and flexibility of identifying the note transition state and the note continuous state. Then, the candidate note state chain is constructed by dynamic splicing of the candidate note states, and the target note boundary is obtained by global screening based on the state transition probability, so that the note boundary conforms to the statistical law of state transition, thereby achieving accurate identification of the note boundary. Finally, the pitch sequence is matched with the note boundary to obtain an accurate quasi-spectrum result, thereby improving the accuracy of audio spectrum conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A diagram illustrating an application environment of an audio spectrum conversion method according to an embodiment;

[0040] Figure 2 1 is a flow chart of an audio spectrum conversion method according to an embodiment;

[0041] Figure 3 is a schematic diagram of an optimal state path in one embodiment;

[0042] Figure 4 A logic diagram for note boundary recognition in one embodiment;

[0043] Figure 5 is a logic diagram of a pitch classification task in one embodiment;

[0044] Figure 6 1 is a flow chart of an audio spectrum conversion method according to another embodiment;

[0045] Figure 7 is a structural block diagram of an audio spectrum transcoding device in one embodiment;

[0046] Figure 8 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0048] The audio spectrum conversion method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 obtains the song audio of the song to be transcribed and the pitch sequence of the song audio; the terminal 102 determines the note prediction probability distribution of each audio frame based on the spectral characteristics of each audio frame in the song audio; wherein the note prediction probability distribution of the audio frame includes the prediction probability of the audio frame belonging to each note state in a plurality of note states; each note state in the plurality of note states is a note transition state or a note duration state described by a hidden Markov model; the terminal 102 determines at least one candidate note state corresponding to each audio frame in the plurality of note states based on the note prediction probability distribution of each audio frame; according to the pitch sequence of each audio frame in the song The audio is sorted by sequentially concatenating each candidate note state corresponding to each audio frame with each candidate note state corresponding to the adjacent audio frame to obtain multiple candidate note state chains. Terminal 102 then identifies candidate note boundaries for each note in the song audio from each candidate note state chain. Based on the state transition probabilities between the multiple note states, terminal 102 selects target note boundaries for the song audio from the candidate note boundaries for each note in the song audio identified in each candidate note state chain. Terminal 102 generates a transcription result corresponding to the song to be transcribed based on the pitch sequence of the song audio and the target note boundaries for each note in the song audio. Terminal 102 may be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, projectors, and the like. Portable wearable devices may include smart watches, smart bracelets, head-mounted devices, and the like. Head-mounted devices may include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like. The server 104 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0049] In an exemplary embodiment, Figure 2 As shown, a method for audio spectrum conversion is provided, which is applied to Figure 1 The terminal 102 in FIG. 1 is used as an example for explanation, including:

[0050] Step S202: Acquire the song audio and the pitch sequence of the song audio to be transcribed.

[0051] The song to be transcribed may be a song that needs to be converted into a musical score, or may be a musical work containing audio elements such as vocals and musical instruments. The song audio of the song to be transcribed may refer to the audio signal of the song to be transcribed, and the song audio may be vocal audio or musical instrument audio. For example, if the song to be transcribed is a purely vocal song, the song audio is purely vocal audio; if the song to be transcribed is a purely instrumental song, the song audio is purely instrumental audio; if the song to be transcribed is a sung song with accompaniment, the vocal audio and the accompaniment audio may be separated from the song to be transcribed, and the vocal audio or the accompaniment audio may be used as the song audio to be transcribed.

[0052] In a specific implementation, the terminal obtains the song audio of the song to be transcribed, which can be audio loaded from a storage medium (such as a hard disk, cloud storage, etc.), or audio recorded in real time through a microphone, such as audio generated by human humming or audio generated by musical instrument performance.

[0053] In one embodiment, the terminal may obtain the song audio of the song to be transcribed, and obtain the pitch sequence of the song audio based on the spectrum characteristics of each note segment in the song audio.

[0054] A note segment can refer to a segment of a song audio that represents a single note, where the song audio within the segment has the same pitch. A note segment can correspond to the singing or playing of a single note, and the duration of the note segment is the duration of the note.

[0055] In a specific implementation, the song audio may be sliced and processed according to the audio to obtain note segments, so that the audio signals within the note segments have the same pitch.

[0056] Spectral features can be information extracted from an audio signal that describes the frequency components of the audio signal. Spectral features can be obtained through time-frequency analysis of the audio signal and can be used to describe the two-dimensional representation of the audio signal in frequency and time. Spectral features contain the intensity distribution of different frequency components in the audio signal and can be used to distinguish different notes and timbres.

[0057] Optionally, the spectral feature may be a Mel Spectrogram feature, a CQT (Constant-Q Transform) spectral feature, etc. The Mel Spectrum feature is obtained by applying a short-time Fourier transform (STFT) to the audio signal and performing Mel-scale filtering on the spectrogram.

[0058] Pitch is a key attribute in audio signals, representing the highness or lowness of a sound. A pitch sequence can be a chronological sequence of pitch values extracted from a song. Each pitch value corresponds to a note in a note segment. These pitch values can be used to recover the pitch information of the song audio, facilitating subsequent generation of the corresponding musical score.

[0059] In a specific implementation, the pitch corresponding to each note segment is determined based on the spectral features of each note segment in the song audio. The spectral features of each note segment in the song audio can be input into a pitch classification model to obtain the pitch corresponding to each note segment.

[0060] Optionally, the pitch classification model can be a neural network model based on a residual network (ResNet), which uses residual learning to extract pitch information from spectral features. The pitch corresponding to a typical song audio can be quantized into 128 intervals. The pitch classification model can map the spectral features of each audio clip into 128 pitch categories, representing different pitch values of the note. Alternatively, the pitch classification model can be a CRNN (Convolutional Recurrent Neural Network) model.

[0061] Step S204: Determine the note prediction probability distribution of each audio frame based on the spectrum characteristics of each audio frame in the song audio.

[0062] An audio frame can be a small segment of an audio signal segmented in the time dimension. Song audio can be divided into several audio frames, with the granularity of an audio frame being several tens of milliseconds, such as 20 milliseconds. The method for determining the spectral features of an audio frame can refer to the method for determining spectral features in the above embodiment and will not be described in detail here.

[0063] The note prediction probability distribution for an audio frame includes the predicted probability of the audio frame belonging to each of the multiple note states. For example, for any audio frame, the predicted probability of the audio frame belonging to note state S1 can be determined as P1, the predicted probability of the audio frame belonging to note state S2 can be determined as P2, and the predicted probability of the audio frame belonging to note state S3 can be determined as P3. These P1, P2, and P3 constitute the note prediction probability distribution for the audio frame.

[0064] The note state is used by the Hidden Markov Model to describe the variation of a note. The Hidden Markov Model is a statistical model used to describe random processes with hidden states. In audio spectrum conversion technology, the note state in the Hidden Markov Model is implicit and cannot be directly observed, but can be inferred by analyzing the spectral characteristics.

[0065] Each of the multiple note states is a note transition state or a note duration state described by a hidden Markov model. The hidden Markov model is associated with multiple preset note states. Specifically, a note state can be a state in the hidden Markov model used to represent a note duration or a note transition. Exemplarily, a note state can include a transition state between notes or a discrete state (also called a duration state) of a note. For example, a note state can include a state transitioning from note A to note B or a duration state of note A.

[0066] In a specific implementation, the spectral features of each audio frame in the song audio can be input into the acoustic model to obtain a note prediction probability distribution. Optionally, the acoustic model can be a Time Series Convolutional Neural Network (TDNN) model. The TDNN model is a neural network structure suitable for processing time series data and can learn the temporal relationship of notes from continuous spectral features. The acoustic model outputs the predicted probability of each audio frame belonging to each note state. In other words, for each audio frame, a probability distribution corresponding to multiple note states can be output, indicating the predicted probability of the audio frame belonging to different note states.

[0067] Step S206: Determine at least one candidate note state corresponding to each audio frame from multiple note states based on the note prediction probability distribution of each audio frame; and according to the order of each audio frame in the song audio, sequentially splice each candidate note state corresponding to each audio frame with each candidate note state corresponding to the adjacent audio frame to obtain multiple candidate note state chains.

[0068] It's important to note that in actual song audio, the start and end points of notes are often unclear, especially when there are no noticeable pauses between notes or when background noise is present. Note transitions are gradual, and there may not be a clear demarcation between the start and end points of a note. This means that a single note may start or end at multiple possible points in time. Therefore, multiple candidate note boundaries can be determined based on the note prediction probability distribution. Note boundaries can refer to the start and end points of a note, i.e., the start and end times of the note.

[0069] In a specific implementation, based on the note prediction probability distribution of each audio frame, at least one candidate note state for each audio frame can be retained to form at least one possible local state selection. Each of the at least one candidate note state can be a note state with a probability value greater than a threshold or the top N high-probability note states. Then, by sequentially connecting the candidate note states of adjacent note frames, for example, by concatenating the candidate note states of each note frame with the candidate note states of the previous note frame in timeline order, multiple candidate note state chains are obtained to generate multiple candidate note boundaries covering different note boundary possibilities.

[0070] A candidate note state chain can refer to a state sequence formed by sequentially connecting candidate note states from adjacent audio frames, with each chain representing a possible note segmentation path. Therefore, candidate note boundaries for each note in the song audio can be identified from each candidate note state chain. For example, assuming audio frames F1, F2, and F3 are three consecutive audio frames, and assuming the candidate note state corresponding to audio frame F1 is note A changing to note B, the candidate note state corresponding to audio frame F2 is sustained note B, and the candidate note state corresponding to audio frame F3 is note B changing to note C, then each candidate note state is concatenated to form a candidate note state chain. From the concatenated candidate note state chain, it can be determined that the starting point of note B falls within the time period corresponding to audio frame F1 (i.e., the moment when note A changes to note B), and the ending point of note B falls within the time period corresponding to audio frame F3 (i.e., the moment when note B changes to note C). Thus, candidate note boundaries for note B can be identified. Similarly, candidate note boundaries for each note in the song audio can be identified from this candidate note state chain. Different candidate note state chains may correspond to different candidate note boundaries. For example, in candidate note boundary 1, the boundary of a note A is located between time T1 and time T2, and in candidate note boundary 2, the boundary of the note A is located between time T1 and time T3.

[0071] Step S208, respectively identify the candidate note boundaries of each note of the song audio from each candidate note state chain; based on the state transition probability between multiple note states, filter out the target note boundaries of each note of the song audio from the candidate note boundaries of each note of the song audio identified in each candidate note state chain.

[0072] The state transition probability may include the probability of transitioning between multiple note states. The multiple note states are predefined by the Hidden Markov Model. Therefore, the state transition probability may be a fixed value precalculated by the Hidden Markov Model, and the state transition probability may be calculated based on predefined musical logic. For example, to determine the state transition probability, large-scale annotated music data may be acquired, and then the state transition frequencies between the multiple note states may be statistically determined based on predefined musical logic (rules determined by music theory).

[0073] Then, based on the state transition probabilities between the multiple note states, the terminal can filter out target note boundaries for each note in the song audio from the candidate note boundaries of each note identified in each candidate note state chain. The target note boundaries are the exact note boundaries determined in steps S204 through S208.

[0074] In a specific implementation, the terminal can calculate the corresponding global probability for each candidate note state chain based on the state transition probability between multiple note states, and then select the candidate note state chain with the highest global probability from each candidate note state chain, and identify the candidate note boundaries of each note of the song audio from the candidate note state chain with the highest global probability as the target note boundaries of each note of the song audio.

[0075] As an example, the global probability may be determined according to the product of the state transition probabilities between adjacent candidate note states in the candidate note state chain.

[0076] As another example, the global probability can be determined based on the state transition probabilities between adjacent candidate note states in the candidate note state chain and the predicted probabilities of each candidate note state included in the candidate note state chain. For example, a first product is determined based on the product of the state transition probabilities between any two adjacent candidate note states in the candidate note state chain, and a second product is determined based on the product of the predicted probabilities of each candidate note state. The result of multiplying the first and second products is then used as the global probability.

[0077] Step S210 , generating a transcription result corresponding to the song to be transcribed based on the pitch sequence of the song audio and the target note boundaries of each note of the song audio.

[0078] The pitch sequence includes the pitch of each note in the song audio. The pitch sequence does not directly include the start time and end time of the note, but provides the temporal arrangement of the pitch of the note.

[0079] Note boundaries identify the start and end times of a note and provide a duration for the note, making them crucial for aligning note pitches with the timeline. Therefore, a start and end time can be assigned to each note in the pitch sequence based on the target note boundaries.

[0080] In a specific implementation, a MIDI file or musical notation representation can be generated based on the pitch of each note in the song audio and the target note boundaries, serving as the transcription result for the song to be transcribed. A MIDI (Musical Instrument Digital Interface) file contains information such as the pitch, start time, and end time of each note, enabling the MIDI file to accurately reproduce the timing and pitch changes of the notes in the audio.

[0081] In the above-mentioned audio spectrum conversion method, the note state is modeled by the hidden Markov model, and on the basis of independent extraction of the pitch sequence, the multi-state probability distribution of each audio frame is obtained to improve the accuracy and flexibility of identifying the note transition state and the note continuous state. Then, the candidate note state chain is constructed by dynamic splicing of the candidate note states, and the target note boundary is obtained by global screening based on the state transition probability, so that the note boundary conforms to the statistical law of state transition, thereby realizing accurate identification of the note boundary. Finally, the pitch sequence is matched with the note boundary to obtain an accurate quasi-spectrum result, thereby improving the accuracy of audio spectrum conversion.

[0082] In another embodiment, based on the note prediction probability distribution of each audio frame, at least one candidate note state corresponding to each audio frame is determined from multiple note states; according to the order of each audio frame in the song audio, each candidate note state corresponding to each audio frame is spliced front to back with each candidate note state corresponding to the adjacent audio frame in turn to obtain multiple candidate note state chains, including: constructing a note prediction probability matrix based on the note prediction probability distribution of each audio frame; for each row in the note prediction probability matrix, according to whether the prediction probability of each note state in the row meets a preset condition, selecting at least one note state from the note states in each column as at least one candidate note state of the audio frame corresponding to the row; according to the order of the audio frames in each row, each candidate note state selected from each row is spliced front to back with each candidate note state selected from the adjacent row in turn to obtain multiple candidate note state chains.

[0083] The element in the i-th row and j-th column of the note prediction probability matrix represents the predicted probability of the j-th note state in the i-th audio frame. The order of the audio frames in each row corresponds to the order of the audio frames in the song audio. The note prediction probability matrix is a two-dimensional matrix, where each row represents an audio frame and each column represents a note state. The order of the audio frames in each row of the note prediction probability matrix corresponds to the order of the audio frames in the song audio.

[0084] After constructing the note prediction probability matrix, for each row of the note prediction probability matrix, at least one note state is selected from the note states in each column as at least one candidate note state for the audio frame corresponding to that row, based on whether the predicted probabilities of the various note states in that row meet a preset condition. The preset condition may include the predicted probability of the note state being greater than a threshold, or the predicted probability of the note state being among the top N highest predicted probabilities. If the predicted probability of a note state meets the preset condition, it is selected as a candidate note state.

[0085] For example, the note prediction probability matrix may be shown in Table 1:

[0086] Table 1

[0087]

[0088] Where P1-P9 are prediction probabilities, S1-S3 are different note states, and F1-F3 are different audio frames. For example, for each row in the note prediction probability matrix, the note states with prediction probabilities above a threshold are determined as candidate note states from the note states in each column corresponding to the audio frame in each row, or the top N note states with the highest prediction probabilities are determined as candidate note states. Assume that the candidate note states of note frame F1 are S1 and S2, the candidate note states of note frame F2 are S2 and S3, and the candidate note states of note frame F3 are S1 and S3. Then, the candidate note states of adjacent audio frames are spliced front to back in chronological order to generate possible audio state transition paths. For example, path 1 can be F1 (S1) -> F2 (S2) -> F3 (S1), path 2 can be F1 (S2) -> F2 (S2) -> F3 (S1), path 3 can be F1 (S1) -> F2 (S2) -> F3 (S3)...path 8 can be F1 (S2) -> F2 (S3) -> F3 (S3). Each audio state transition path can correspond to a set of candidate note boundaries, and a set of candidate note boundaries includes the note boundaries of each note in the song audio.

[0089] The technical solution of the above embodiment significantly improves the robustness of note boundary generation while ensuring computational efficiency through the multi-candidate mechanism of the note prediction probability matrix and the dynamic splicing strategy of the note state.

[0090] In another embodiment, based on the state transition probabilities between multiple note states, the target note boundaries of each note of the song audio are screened out from the candidate note boundaries of each note of the song audio identified in each candidate note state chain, including: for each candidate note state chain in the note prediction probability matrix, extracting the state transition probability between adjacent candidate note states in the candidate note state chain from the state transition probabilities between multiple note states; calculating the path probability of the candidate note state chain based on the prediction probability of each candidate note state included in the candidate note state chain and the state transition probability between each adjacent candidate note state in the candidate note state chain; selecting the candidate note state chain with the highest path probability from multiple candidate note state chains as the optimal state path; and using the candidate note boundaries of each note of the song audio in the optimal state path as the target note boundaries of each note of the song audio.

[0091] The state transition probabilities between the multiple note states are pre-calculated fixed values that fully cover the state transition scenarios and corresponding state transition probabilities between the pre-defined multiple note states. From the state transition probabilities between the multiple note states, the terminal can extract the state transition probabilities between adjacent candidate note states in each candidate note state chain.

[0092] In one embodiment, calculating the path probability of the candidate note state chain based on the predicted probabilities of each candidate note state included in the candidate note state chain and the state transition probabilities between each adjacent candidate note state in the candidate note state chain may include: calculating the product of the predicted probabilities of each candidate note state included in the candidate note state chain to obtain a predicted probability product value; calculating the product of the state transition probabilities between each adjacent candidate note state in the candidate note state chain to obtain a transfer probability product value; and determining the product of the predicted probability product value and the transfer probability product value as the path probability of the candidate note state chain.

[0093] As another example, Viterbi decoding can be performed based on the state transition probabilities between multiple note states and the note prediction probability matrix to calculate the path probabilities of each candidate note state chain. The candidate note state chain with the highest path probability is then selected as the optimal state path from the multiple candidate note state chains. Viterbi decoding can be a dynamic programming algorithm for hidden Markov models, which traverses all possible paths to ensure that the path ultimately selected has the highest global probability, i.e., the highest path probability.

[0094] The technical solution of this embodiment combines the note prediction probability matrix and the state transition probability for path selection, and can calculate the optimal state path, so as to effectively screen out the optimal target note boundary from multiple candidate note boundaries, ensuring the accuracy of the start and end time of the note, thereby improving the accuracy of the note boundary and further improving the accuracy of subsequent transcription.

[0095] In another embodiment, before filtering out the target note boundaries of each note of the song audio from the candidate note boundaries of each note of the song audio identified in each candidate note state chain based on the state transition probability between multiple note states, it also includes: converting the note sequence formed by sorting the notes of the song audio into a note state sequence; and calculating the state transition probability between multiple note states based on the note state sequence through a hidden Markov model.

[0096] The note sequence can include the order of individual notes in the song audio and their relative timing. It should be noted that in actual song audio, the start and end points of notes are often unclear, especially when there are no noticeable pauses or background noise between notes. Because note transitions are gradual, the start and end points of a note sequence may not have a clear demarcation, necessitating more refined processing of the note sequence.

[0097] In practice, converting a note sequence into a note state sequence involves expanding discrete note symbols into a finer-grained state description for subsequent time series modeling within a hidden Markov model. For example, assuming the note sequence is: note C -> note D -> note E -> note F, the corresponding note state sequence might be: note state S1 (note C -> note C -> note C -> note C -> note D), note state S2 (note D -> note D -> note E), note state S3 (note E -> note E -> note E), and note state S4 (note E -> note F -> note F).

[0098] Based on a sequence of note states, a hidden Markov model can learn the transition patterns of note states and determine the state transition probabilities between multiple note states. For example, the hidden Markov model can traverse the note state sequence and calculate the state transition probabilities between multiple note states based on the transition frequencies between adjacent note states, thereby quantifying the dynamic transition patterns of hidden states in time series data.

[0099] Optionally, the terminal can also map the note sequence into multiple different note state sequences based on different mapping rules, and calculate the state transition probability between multiple note states according to the transition frequency between each adjacent note state in the multiple note state sequences through the hidden Markov model to improve the accuracy of the state transition probability.

[0100] The technical solution of this embodiment converts the note sequence into a note state sequence, and learns the state transition rules in the note state sequence through a hidden Markov model to calculate the state transition probability, which provides an important basis for note timing alignment, boundary recognition and subsequent note processing, thereby improving the accuracy of the transcription.

[0101] In another embodiment, a note sequence formed by sorting the notes of the song audio is converted into a note state sequence, including: mapping each note in the note sequence to a note state based on the mapping relationship between the notes and the note states recorded in the character dictionary to obtain a note state sequence.

[0102] The character dictionary records the state decomposition rules corresponding to each note symbol, converting discrete note symbols into a continuous sequence of note states, providing a foundation for subsequent time series modeling using a hidden Markov model (HMM). For example, the character dictionary can be a mapping table containing the mapping relationships between notes and note states. The character dictionary maps each note to a corresponding note state. For example, the character dictionary may specify that note C -> note state S1, note D -> note state S2, note E -> note state S3, and note F -> note state S4. Using the mappings in the character dictionary, each note in the note sequence is replaced with its corresponding note state, resulting in a note state sequence. For example, given the note sequence "C, D, E, F," the corresponding note state sequence is obtained according to the character dictionary mapping: "Note state S1, note state S2, note state S3, note state S4." For example, the notes in the character dictionary can include 128 notes, which can refer to the note range defined by the MIDI standard, covering all possible pitch and octave combinations. In the MIDI protocol, note numbers range from 0 to 127, a total of 128, corresponding to different pitches.

[0103] For example, the character dictionary can be manually designed based on music theory, or the acoustic features (duration, energy, spectrum) of the notes can be clustered through cluster analysis, with each class being assigned a note state.

[0104] The technical solution provided in this embodiment converts the note sequence into a note state sequence in a unified format through standardized mapping of a character dictionary, accurately describes the note dynamics through fine-grained state decomposition, and provides reliable optimization for the subsequent global optimization of the hidden Markov model.

[0105] To facilitate understanding by those skilled in the art, Figure 3 A schematic diagram of an optimal state path is provided as an example. Figure 3 For example, a three-row, three-column note prediction probability matrix is provided. The element in the i-th row and j-th column of the note prediction probability matrix represents the predicted probability that the i-th audio frame belongs to the j-th note state. P1 through P9 represent the corresponding predicted probabilities. Viterbi decoding can select the optimal state path within this note prediction probability matrix. For example, the optimal state path can be selected from P2 to P6 to P8, meaning that the first audio frame corresponds to the second audio state, the second audio frame to the third audio state, and the third audio frame to the second audio state.

[0106] To facilitate understanding by those skilled in the art, Figure 4A logic diagram for note boundary identification is provided as an example. In a specific implementation, mel-spectrogram features are extracted from song audio and input into an acoustic model, which can be a time-series convolutional neural network (TDNN) model. The acoustic model outputs note prediction probabilities for each audio frame. Simultaneously, the note sequence corresponding to the song audio is mapped into a sequence of note states in a hidden Markov model using a note dictionary. State transition probabilities are calculated using a hidden Markov model (HMM). Viterbi decoding is then performed based on the state transition probabilities and the probability matrix of the audio frame belonging to each note state, as predicted by the acoustic model. This forces alignment between the audio space and the note space, resulting in the alignment result, i.e., the note boundary. As can be seen, note boundary identification can be based on the HMM-TDNN framework, where the TDNN is used to acoustically model the note states, while the HMM is used to describe the transition relationships between notes. The TDNN predicts the note state probabilities based on the mel-spectrogram features, while the HMM models the transitions between note states using state transition probabilities. The note prediction probabilities are output by the TDNN and further inferred and aligned using the HMM model. The transition probability of HMM describes the transition rules between note states. Combined with the output probability of TDNN, it can help accurately identify the boundaries and timing of notes.

[0107] It can be seen that the present application can separate note pitch recognition and note boundary recognition, and use two different models to achieve pitch recognition and note boundary recognition in a targeted manner. Pitch recognition is a classification task, and a neural network model based on a residual network can be used for pitch classification. Note boundary recognition is a task of automatic alignment of notes and audio, and can be identified using the HMM-TDNN framework. The results of the two can be fused to obtain the transcribed result. Using the idea of ​​split model for transcription can achieve better transcription effect.

[0108] In another embodiment, obtaining song audio of a song to be transcribed and a pitch sequence of the song audio includes: slicing the song audio of the song to be transcribed according to notes to obtain original audio segments corresponding to each note; wherein the song audio in each original audio segment has the same pitch; adjusting the duration of each original audio segment to the target segment duration to obtain multiple note segments; and obtaining the pitch sequence of the song audio based on the spectral characteristics of the multiple note segments.

[0109] In the specific implementation, the song audio is sliced according to notes to obtain multiple original audio clips, each original audio clip corresponds to a note in the song audio, and the song audio in the original audio clips has the same pitch, and each original audio clip represents the duration of a note.

[0110] The duration of multiple original audio segments is adjusted to a target segment duration. The lengths of the original audio segments for different notes can be normalized to obtain multiple note segments, each of which has the same duration and pitch. The pitch sequence of the song audio is then obtained based on the spectral characteristics of the multiple note segments. For example, the target segment duration can be 2 seconds or 3 seconds.

[0111] The technical solution of this embodiment ensures the fragmentation and standardization of notes, facilitating subsequent accurate pitch recognition, note alignment and transcription tasks.

[0112] In one embodiment, the duration of each original audio segment is adjusted to the target segment duration to obtain multiple note segments, including: when the duration of the original audio segment is less than the target segment duration, blank audio is supplemented to the original audio segment to obtain supplemented audio with the target segment duration as the note segment; when the duration of the original audio segment is greater than the target segment duration, the original audio segment is truncated to obtain truncated audio with the target segment duration as the note segment.

[0113] Blank audio refers to sections of audio with zero amplitude inserted into an audio signal. This blank audio does not affect the pitch or other audio characteristics of the note, but it can fill in the gaps in the original audio clip to make it fit within the target clip duration. For example, if the target clip duration is 1 second and the original audio clip is only 0.5 seconds long, 0.5 seconds of blank audio will be inserted to make the final note clip duration 1 second.

[0114] Truncation can mean shortening the duration of the original audio segment and discarding the portion that exceeds the target segment duration. The truncated note segment retains the first portion and discards the portion that exceeds the target segment duration. For example, if the target segment duration is 1 second and the original audio segment duration is 1.5 seconds, the first 1 second of the original audio segment will be retained and the last 0.5 seconds will be truncated to make it fit the target segment duration.

[0115] After padding or truncation, the resulting note segments will match the target segment duration. These processed note segments can be used for subsequent tasks such as pitch analysis, feature extraction, and note alignment.

[0116] The technical solution of this embodiment ensures the uniformity of the duration of the note segments, facilitates subsequent pitch analysis and feature extraction, and provides standardized input for audio processing.

[0117] To facilitate understanding by those skilled in the art, Figure 5A logic diagram of a pitch classification task is provided as an example. In the training stage, the song audio is sliced according to the notes so that the audio in the segment all belongs to the same pitch. Since the song audio corresponding to the notes belongs to segments of unequal length, in order to facilitate training, the sliced audio is uniformly lengthened to obtain note segments: for segments less than 3 seconds, 0 is padded to 3 seconds; for segments longer than 3 seconds, the audio is truncated and only the first 3 seconds are retained. Mel spectrum features are extracted from the processed note segments and input into a pitch classification model for training. The pitch classification model can be a neural network model based on a residual network. Since the pitch corresponding to general singing can be quantized into 128 intervals, the number of categories of the pitch classification model can be determined to be 128. The Softmax loss function is used for classification training, and the training is carried out until the model converges and the model parameters are saved. In the inference stage, the song audio segment is padded with zeros / truncated into 3-second segments and then input into the trained pitch classification model. The pitch classification model outputs the pitch corresponding to the note segment. For a song, after slice inference, we can get a pitch sequence such as 0, 0, 0, 63, 65, 65, 68, 72, 75, …, 0, 0, 0. 0 represents a segment without singing.

[0118] The audio spectrum conversion method based on pitch recognition and note boundary recognition in this application divides pitch recognition and note boundary recognition into two subtasks. For the pitch classification task, the training audio can be unified into 3-second segments, and a 128-category pitch classification model can be trained. For the note boundary recognition task, the dictionary HMM model can be combined with the acoustic model TDNN to force the audio to be aligned in the note space. In the result fusion stage, the pitch classification result of the first sub-model and the note boundary recognition result of the second model can be used to obtain the midi spectrum as the spectrum conversion result. The two models are used to specifically identify the pitch and note boundaries, thereby improving the accuracy of the spectrum conversion.

[0119] In another embodiment, Figure 6 As shown, a method for audio spectrum conversion is provided, which is applied to Figure 1 Taking the terminal 102 in FIG. 1 as an example, the method includes the following steps:

[0120] S602: Slice the audio of the song to be transcribed according to the notes to obtain the original audio segments corresponding to each note.

[0121] The song audio in each original audio segment has the same pitch.

[0122] S604: Adjust the duration of each original audio segment to the target segment duration to obtain multiple note segments.

[0123] In one embodiment, the duration of each original audio segment is adjusted to the target segment duration to obtain multiple note segments, including: when the duration of the original audio segment is less than the target segment duration, blank audio is supplemented to the original audio segment to obtain supplemented audio with the target segment duration as the note segment; when the duration of the original audio segment is greater than the target segment duration, the original audio segment is truncated to obtain truncated audio with the target segment duration as the note segment.

[0124] S606: Obtain a pitch sequence of the song audio according to the frequency spectrum features of the plurality of note segments.

[0125] S608: Determine the note prediction probability distribution of each audio frame based on the spectrum characteristics of each audio frame in the song audio.

[0126] Among them, the note prediction probability distribution of the audio frame includes the prediction probability that the audio frame belongs to each note state in multiple note states; each note state in the multiple note states is a note transition state or a note duration state described by a hidden Markov model.

[0127] S610: Construct a note prediction probability matrix according to the note prediction probability distribution of each audio frame.

[0128] Among them, the element in the i-th row and j-th column of the note prediction probability matrix is used to represent the predicted probability of the j-th note state of the i-th audio frame, and the order of the audio frames in each row corresponds to the order of each audio frame in the song audio.

[0129] S612, for each row in the note prediction probability matrix, select at least one note state from the note states in each column as at least one candidate note state of the audio frame corresponding to the row according to whether the predicted probability of each note state in the row meets the preset conditions; according to the order of the audio frames in each row, each candidate note state selected from each row is spliced back and forth with each candidate note state selected from the adjacent row in turn to obtain multiple candidate note state chains.

[0130] S614: Convert the note sequence formed by sorting the notes of the song audio into a note state sequence.

[0131] Among them, the note sequence formed by sorting the notes of the song audio is converted into a note state sequence, including: mapping each note in the note sequence to a note state based on the mapping relationship between the notes and the note states recorded in the character dictionary to obtain the note state sequence.

[0132] S616, calculating the state transition probability between multiple note states according to the note state sequence through a hidden Markov model.

[0133] S618, respectively identify the candidate note boundaries of each note of the song audio from each candidate note state chain; based on the state transition probability between multiple note states, filter out the target note boundaries of each note of the song audio from the candidate note boundaries of each note of the song audio identified in each candidate note state chain.

[0134] In one embodiment, based on the state transition probabilities between multiple note states, the target note boundaries of each note of the song audio are screened out from the candidate note boundaries of each note of the song audio identified in each candidate note state chain, including: for each candidate note state chain in the note prediction probability matrix, extracting the state transition probability between adjacent candidate note states in the candidate note state chain from the state transition probabilities between multiple note states; calculating the path probability of the candidate note state chain based on the prediction probability of each candidate note state included in the candidate note state chain and the state transition probability between each adjacent candidate note state in the candidate note state chain; selecting the candidate note state chain with the highest path probability from multiple candidate note state chains as the optimal state path; and using the candidate note boundaries of each note of the song audio in the optimal state path as the target note boundaries of each note of the song audio.

[0135] S620: Generate a transcription result corresponding to the song to be transcribed based on the pitch sequence of the song audio and the target note boundaries of each note of the song audio.

[0136] It should be noted that the specific limitations of the above steps can refer to the specific limitations of an audio spectrum conversion method above.

[0137] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0138] Based on the same inventive concept, embodiments of the present application also provide an audio transcoding device for implementing the aforementioned audio transcoding method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more of the following embodiments of the audio transcoding device can be found in the aforementioned limitations of the audio transcoding method and will not be further elaborated here.

[0139] In an exemplary embodiment, Figure 7 As shown, an audio spectrum transcoding device is provided, comprising:

[0140] The acquisition module 710 is used to obtain the song audio of the song to be transcribed and the pitch sequence of the song audio.

[0141] The probability prediction module 720 is used to determine the note prediction probability distribution of each audio frame in the song audio based on the spectral characteristics of each audio frame; wherein the note prediction probability distribution of the audio frame includes the prediction probability that the audio frame belongs to each of the multiple note states; each of the multiple note states is a note transition state or a note duration state described by a hidden Markov model.

[0142] The state determination module 730 is used to determine at least one candidate note state corresponding to each audio frame among the multiple note states based on the note prediction probability distribution of each audio frame; and according to the order of each audio frame in the song audio, each candidate note state corresponding to each audio frame is sequentially spliced with each candidate note state corresponding to the adjacent audio frame to obtain multiple candidate note state chains.

[0143] The boundary determination module 740 is used to identify the candidate note boundaries of each note of the song audio from each candidate note state chain respectively; according to the state transition probability between the multiple note states, the target note boundaries of each note of the song audio are screened out from the candidate note boundaries of each note of the song audio identified in each candidate note state chain.

[0144] The generation module 750 is used to generate a transcription result corresponding to the song to be transcribed based on the pitch sequence of the song audio and the target note boundaries of each note of the song audio.

[0145] In one embodiment, the state determination module 730 is specifically used to construct a note prediction probability matrix based on the note prediction probability distribution of each audio frame; the elements of the i-th row and j-th column of the note prediction probability matrix are used to represent the prediction probability of the j-th note state of the i-th audio frame, and the order of the audio frames in each row corresponds to the order of each audio frame in the song audio; for each row in the note prediction probability matrix, according to whether the prediction probability of each note state in the row meets a preset condition, at least one note state is selected from the note states in each column as at least one candidate note state of the audio frame corresponding to the row; according to the order of the audio frames in each row, each candidate note state selected from each row is sequentially spliced with each candidate note state selected from the adjacent row to obtain multiple candidate note state chains.

[0146] In one embodiment, the boundary determination module 740 is specifically used to extract, for each candidate note state chain in the note prediction probability matrix, the state transition probability between adjacent candidate note states in the candidate note state chain from the state transition probabilities between the multiple note states; calculate the path probability of the candidate note state chain based on the prediction probability of each candidate note state included in the candidate note state chain and the state transition probability between each adjacent candidate note state in the candidate note state chain; select the candidate note state chain with the highest path probability from the multiple candidate note state chains as the optimal state path; and use the candidate note boundaries of each note of the song audio in the optimal state path as the target note boundaries of each note of the song audio.

[0147] In one embodiment, the boundary determination module 740 is specifically used to calculate the product of the predicted probabilities of each candidate note state included in the candidate note state chain to obtain a predicted probability product value; calculate the product of the state transition probabilities between each adjacent candidate note state in the candidate note state chain to obtain a transition probability product value; and determine the product of the predicted probability product value and the transition probability product value as the path probability of the candidate note state chain.

[0148] In one embodiment, the boundary determination module 740 is specifically used to convert the note sequence formed by sorting the notes of the song audio into a note state sequence; and calculate the state transition probability between the multiple note states according to the note state sequence through a hidden Markov model.

[0149] In one embodiment, the boundary determination module 740 is specifically configured to map each note in the note sequence to a note state based on a mapping relationship between notes and note states recorded in a character dictionary to obtain the note state sequence.

[0150] In one embodiment, the acquisition module 710 is specifically used to slice the song audio of the song to be transcribed according to notes to obtain original audio segments corresponding to each note; wherein the song audio in each original audio segment has the same pitch; adjust the duration of each original audio segment to the target segment duration to obtain multiple note segments; and obtain the pitch sequence of the song audio based on the spectral characteristics of the multiple note segments.

[0151] Each module in the aforementioned audio-to-spectrum transcoding device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0152] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements an audio spectrum transcoding method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0153] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0154] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0155] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0156] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0157] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0158] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0159] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0160] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An audio spectrum conversion method, characterized in that: The method comprises: Obtaining the song audio of the song to be transcribed and the pitch sequence of the song audio; Determine a note prediction probability distribution for each audio frame based on the spectral features of each audio frame in the song audio; wherein the note prediction probability distribution for the audio frame includes a prediction probability that the audio frame belongs to each of a plurality of note states; each of the plurality of note states is a note transition state or a note duration state described by a hidden Markov model; Determining at least one candidate note state corresponding to each audio frame from the plurality of note states based on the note prediction probability distribution of each audio frame; and sequentially concatenating each candidate note state corresponding to each audio frame with each candidate note state corresponding to an adjacent audio frame according to the order of the audio frames in the song audio to obtain a plurality of candidate note state chains; Identifying candidate note boundaries of each note of the song audio from each of the candidate note state chains; and screening target note boundaries of each note of the song audio from the candidate note boundaries of each note of the song audio identified from each of the candidate note state chains based on the state transition probabilities between the plurality of note states; A transcription result corresponding to the song to be transcribed is generated based on the pitch sequence of the song audio and the target note boundaries of each note of the song audio.

2. The method according to claim 1, characterized in that The method further comprises: determining at least one candidate note state corresponding to each audio frame from the plurality of note states according to the note prediction probability distribution of each audio frame; and sequentially concatenating each candidate note state corresponding to each audio frame with each candidate note state corresponding to an adjacent audio frame according to the order of the audio frames in the song audio, to obtain a plurality of candidate note state chains, including: Constructing a note prediction probability matrix based on the note prediction probability distribution of each audio frame; the element in the i-th row and j-th column of the note prediction probability matrix is used to represent the predicted probability of the j-th note state of the i-th audio frame, and the order of the audio frames in each row corresponds to the order of each audio frame in the song audio; For each row of the note prediction probability matrix, based on whether the predicted probabilities of the respective note states in the row meet a preset condition, selecting at least one note state from the note states in each column as at least one candidate note state for the audio frame corresponding to the row; According to the order of the audio frames in each row, each candidate note state selected in each row is sequentially spliced front to back with each candidate note state selected in the adjacent row to obtain multiple candidate note state chains.

3. The method according to claim 2, characterized in that The step of screening out target note boundaries of each note of the song audio from the candidate note boundaries of each note of the song audio identified in each of the candidate note state chains based on the state transition probabilities between the multiple note states includes: For each candidate note state chain in the note prediction probability matrix, extracting the state transition probability between adjacent candidate note states in the candidate note state chain from the state transition probabilities between the multiple note states; calculating the path probability of the candidate note state chain based on the prediction probabilities of the respective candidate note states included in the candidate note state chain and the state transition probabilities between the respective adjacent candidate note states in the candidate note state chain; From the multiple candidate note state chains, the candidate note state chain with the highest path probability is selected as the optimal state path; and the candidate note boundaries of each note of the song audio in the optimal state path are used as the target note boundaries of each note of the song audio.

4. The method according to claim 3, characterized in that The step of calculating the path probability of the candidate note state chain based on the predicted probability of each candidate note state included in the candidate note state chain and the state transition probability between each adjacent candidate note state in the candidate note state chain includes: Calculating the product of the predicted probabilities of the candidate note states included in the candidate note state chain to obtain a predicted probability product value; Calculating the product of the state transition probabilities between adjacent candidate note states in the candidate note state chain to obtain a transition probability product value; The product of the predicted probability product value and the transition probability product value is determined as the path probability of the candidate note state chain.

5. The method according to claim 1, wherein Before selecting target note boundaries of each note of the song audio from the candidate note boundaries of each note of the song audio identified in each of the candidate note state chains based on the state transition probabilities between the multiple note states, the method further includes: Converting a note sequence formed by sorting the notes of the song audio into a note state sequence; The state transition probability between the plurality of note states is calculated according to the note state sequence by using a hidden Markov model.

6. The method according to claim 5, characterized in that The step of converting the note sequence formed by sorting the notes of the song audio into a note state sequence includes: Based on the mapping relationship between the notes and the note states recorded in the character dictionary, each note in the note sequence is mapped to a note state to obtain the note state sequence.

7. The method according to claim 1, characterized in that The step of obtaining the song audio of the song to be transcribed and the pitch sequence of the song audio includes: Slicing the song audio of the song to be transcribed according to the notes to obtain original audio segments corresponding to each of the notes; wherein the song audio in each of the original audio segments has the same pitch; Adjusting the duration of each original audio segment to the target segment duration to obtain multiple note segments; A pitch sequence of the song audio is obtained according to the frequency spectrum features of the multiple note segments.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images