Audio transcription method and apparatus, computer device, and storage medium

Abstract

The present application relates to the field of speech analysis, and particularly relates to an audio transcription method and device, computer equipment and storage medium. The method comprises: obtaining Fourier spectrum by performing Fourier transform on the audio to be transcribed; obtaining multi-task features by performing multi-task feature extraction on the Fourier spectrum based on a multi-task feature fusion model based on an attention mechanism; obtaining multi-task prediction results by performing multi-task prediction on the Fourier spectrum according to the multi-task features; and generating a transcription file corresponding to the audio to be transcribed according to the prediction results. The multi-task prediction results obtained by the present application take into account the hidden information between multiple task features, making the multi-task prediction results more accurate. Furthermore, the transcription file generated based on the multi-task prediction results is more accurate, improving the accuracy of audio transcription.

Description

Technical Field

[0001] This invention relates to the field of speech analysis, and more particularly to an audio transcription method, apparatus, computer device, and storage medium. Background Technology

[0002] In the music field, music is mainly stored and disseminated in audio form, with only a very small portion presented in sheet music form, allowing users to further study and learn the music corresponding to that sheet music.

[0003] Currently, existing methods for converting audio into sheet music include template methods and deep learning methods.

[0004] Template-based methods determine the pitch category of an input sound by comparing the similarity between the input audio data and corresponding templates in a database, ultimately producing a transcribed score. However, template-based methods perform poorly with music containing many chords and at high tempos, resulting in low audio transcription accuracy. Deep learning methods convert the input audio data into a Short-Time Fourier Transform (STFT) spectrum or a Mel spectrum, then use the resulting spectrum as an image for feature extraction, performing frame-by-frame pitch classification to obtain the transcribed score. However, deep learning methods exhibit varying accuracy for music segments with different timbres and recording environments. Furthermore, the biggest drawback of deep learning methods is their inability to accurately detect the start and end points of notes, rendering the transcribed score unsuitable for user learning and research. Therefore, existing methods for converting audio to score all suffer from low transcription accuracy and poor overall performance. Summary of the Invention

[0005] Therefore, it is necessary to provide an audio transcription method, apparatus, computer equipment, and storage medium to address the aforementioned technical problems, thereby resolving the issues of low transcription accuracy and poor performance in existing methods for converting audio into sheet music.

[0006] An audio transcription method, comprising: Perform a Fourier transform on the audio to be transcribed to obtain the Fourier spectrum; Multi-task features are extracted from the Fourier spectrum using an attention-based multi-task feature fusion model. Based on the multi-task characteristics, multi-task prediction is performed on the Fourier spectrum to obtain the multi-task prediction result; Based on the prediction results, a transcription file corresponding to the audio to be transcribed is generated.

[0007] An audio transcription device, comprising: The Fourier spectrum module is used to perform Fourier transform on the audio to be transcribed to obtain the Fourier spectrum. The multi-task feature module is used to extract multi-task features from the Fourier spectrum using an attention-based multi-task feature fusion model to obtain multi-task features. The multi-task prediction result module is used to perform multi-task prediction on the Fourier spectrum based on the multi-task features to obtain multi-task prediction results. The transcription file module is used to generate a transcription file corresponding to the audio to be transcribed based on the prediction results.

[0008] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-described audio transcription method when executing the computer-readable instructions.

[0009] One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the audio transcription method described above.

[0010] The aforementioned audio transcription method, apparatus, computer equipment, and storage medium obtain a Fourier spectrum by performing a Fourier transform on the audio to be transcribed; extract multi-task features from the Fourier spectrum using a multi-task feature fusion model based on an attention mechanism to obtain multi-task features; perform multi-task prediction on the Fourier spectrum based on the multi-task features to obtain multi-task prediction results; and generate a transcription file corresponding to the audio to be transcribed based on the prediction results. This invention uses a multi-task feature fusion model based on an attention mechanism to perform multi-task fusion task prediction on the audio to be transcribed, making the obtained multi-task prediction results more accurate by considering the hidden information between multiple task features. Consequently, the transcription file generated based on the multi-task prediction results is more accurate, improving the accuracy of audio transcription. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for an audio transcription method according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of an audio transcription method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an audio transcription device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] The audio transcription method provided in this embodiment can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0015] In one embodiment, such as Figure 2 As shown, an audio transcription method is provided, which is applied to... Figure 1 Taking the server-side as an example, the explanation includes the following steps: S10. Perform Fourier transform on the audio to be transcribed to obtain the Fourier spectrum.

[0016] Understandably, the audio to be transcribed refers to audio that needs to be transcribed into sheet music. For example, music data obtained from a music platform. The Fourier transform can convert the audio to be transcribed from a non-periodic continuous signal in the time domain to a non-periodic continuous signal in the frequency domain, obtaining the Fourier spectrum. In other words, the Fourier transform converts the audio to be transcribed from a time-domain signal to a frequency-domain signal.

[0017] S20. Multi-task features are extracted from the Fourier spectrum using a multi-task feature fusion model based on an attention mechanism to obtain multi-task features.

[0018] Understandably, the attention-based multi-task feature fusion model is a trained neural network model. This attention-based multi-task feature fusion model utilizes an attention mechanism and includes multiple task prediction models. These multiple task prediction models include, but are not limited to, a start-point prediction model, an end-point prediction model, a pitch prediction model, and a dynamics prediction model. Each task prediction model corresponds to one task. Multiple task prediction models are used for prediction of multiple different tasks. Here, the number of models and tasks included in the multiple task prediction models are not limited and can be set according to actual needs. Specifically, the Fourier spectrum is input into the multiple task prediction models to extract different task features, resulting in multiple features corresponding to each task, i.e., multi-task features. These multi-task features contain multiple task features. That is, different task models correspond to different task features.

[0019] S30. Based on the multi-task characteristics, perform multi-task prediction on the Fourier spectrum to obtain the multi-task prediction result.

[0020] Understandably, after obtaining the multi-task features of the recording to be transcribed, a multi-task feature fusion model based on an attention mechanism is used to perform feature fusion and classification prediction processing on the multi-task features to obtain multi-task prediction results. Specifically, the attention mechanism in the multi-task feature fusion model based on an attention mechanism is used to learn and fuse the hidden information between multiple task features to obtain features that have fused the hidden information. Based on the features that have fused the hidden information, a multi-task prediction model is then used to perform classification prediction processing on the features that have fused the hidden information and their corresponding task features to obtain task prediction results. These prediction results include multiple prediction results corresponding to the multi-task features. For example, they include pitch prediction results, start point prediction results, end point prediction results, and velocity prediction results.

[0021] S40. Based on the prediction results, generate a transcription file corresponding to the audio to be transcribed.

[0022] Understandably, after the prediction result, a transcription file corresponding to the audio to be transcribed can be generated based on the information contained in the prediction result. This transcription file is the sheet music corresponding to the audio to be transcribed. Generally, sheet music is a regular combination of various symbols recording musical pitch and rhythm, such as common simplified musical notation, staff notation, guitar tablature, guqin tablature, piano tablature, etc. Here, the transcription file includes pitch information, start point information, end point information, and dynamics prediction information. The start point information includes the time and note of the start point. The end point information includes the time and note of the end point. Dynamics information refers to the playing force corresponding to the note. For example, in piano music performance, dynamics information could be the degree of pressure applied by the performer when pressing the keys.

[0023] In steps S10-S40, a Fourier spectrum is obtained by performing a Fourier transform on the audio to be transcribed; multi-task features are extracted from the Fourier spectrum using a multi-task feature fusion model based on an attention mechanism to obtain multi-task features; multi-task prediction is performed on the Fourier spectrum based on the multi-task features to obtain multi-task prediction results; and a transcription file corresponding to the audio to be transcribed is generated based on the prediction results. In this embodiment, the multi-task feature fusion model based on an attention mechanism performs multi-task fusion task prediction on the audio to be transcribed, so that the obtained multi-task prediction results take into account the hidden information between multiple task features, making the multi-task prediction results more accurate. Therefore, the transcription file generated based on the multi-task prediction results is more accurate, improving the accuracy of audio transcription.

[0024] Optionally, the multi-task feature fusion model includes a start point prediction model, an end point prediction model, a pitch prediction model, and a dynamics prediction model; the multi-task features include start point features, end point features, pitch features, and dynamics features.

[0025] In step S20, the multi-task features are extracted from the Fourier spectrum using an attention-based multi-task feature fusion model to obtain multi-task features, including: S201. Extract the starting point features from the Fourier spectrum using the starting point prediction model to obtain the starting point features; S202. Extract the end-point features from the Fourier spectrum using the end-point prediction model to obtain the end-point features; S203. Extract pitch features from the Fourier spectrum using the pitch prediction model to obtain the pitch features; S204. The Fourier spectrum is subjected to force feature extraction using the force prediction model to obtain the force feature.

[0026] Understandably, the start point prediction model, end point prediction model, pitch prediction model, and pitch prediction model are all task prediction models. The end point prediction model and the start point prediction model are structurally identical task prediction models. All three models contain structurally identical convolutional neural networks. Specifically, the start point prediction model includes a convolutional neural network for detecting and identifying start points in the Fourier spectrum. The end point prediction model includes a convolutional neural network for detecting and identifying end points in the Fourier spectrum. The pitch prediction model includes a convolutional neural network for detecting and identifying pitch information in the Fourier spectrum. The pitch prediction model includes a convolutional neural network for detecting and identifying velocity information in the Fourier spectrum. The start point features include information on the start point of the entire audio file to be transcribed, as well as the start point information of each sentence within the audio file. The end point features include information on the end point of the entire audio file to be transcribed, as well as the end point information of each sentence within the audio file. The pitch features include all pitch information of the audio file to be transcribed. The velocity signature includes all velocity information of the audio to be transcribed.

[0027] In steps S201-S204, the Fourier spectrum is detected and identified for different tasks by multiple task prediction models, which can obtain multiple features corresponding to the tasks. The tasks are independent of each other and the tasks are simple, making the obtained features more complete and accurate.

[0028] Optionally, the task prediction results include pitch prediction results.

[0029] In step S30, namely, performing multi-task prediction on the audio to be transcribed based on the multi-task features to obtain multi-task prediction results, the following steps are included: S301. Extract the hidden features of the starting point features and the ending point features through the attention mechanism; S302. The hidden feature, the pitch feature, and the dynamic feature are fused to obtain a fused feature; S303. The pitch prediction result is obtained by performing pitch prediction processing on the fused features through the pitch long short-term neural network and the pitch fully connected classifier of the pitch prediction model.

[0030] Understandably, the hidden information of the start and end point features is learned through an attention mechanism to extract useful hidden features. These hidden features are then fused with pitch and dynamics features to obtain a fused feature that combines useful information from the start and end point features, as well as pitch and dynamics information. The pitch prediction model includes a convolutional neural network for detecting and recognizing dynamics information in the Fourier spectrum, a pitch long short-term neural network, and a pitch fully connected classifier. The pitch long short-term neural network is a special type of recurrent neural network capable of learning long sequences and predicting time series. Here, the pitch long short-term neural network can extract more sufficient and effective pitch information from the fused features, improving the accuracy and sufficiency of pitch information extraction. The pitch fully connected classifier is used to classify pitch based on the more sufficient and effective pitch information extracted by the pitch long short-term neural network, obtaining the pitch prediction result.

[0031] In this embodiment, pitch prediction is performed based on fused features that integrate start and end point features, as well as pitch and dynamics information, which improves the accuracy of pitch prediction results. Furthermore, the fused features are processed by a pitch long short-term neural network, allowing for the extraction of more comprehensive and effective pitch information, further ensuring the accuracy of pitch prediction results.

[0032] Optionally, the task prediction results include start point prediction results, end point prediction results, and intensity prediction results; In step S30, namely, performing multi-task prediction on the audio to be transcribed based on the multi-task features to obtain the multi-task prediction result, the method further includes: S304. The starting point features are processed by the starting point prediction model using the starting point long short-term neural network and the starting point fully connected classifier to obtain the starting point prediction result. S305. The endpoint features are processed by the endpoint long short-term neural network and the endpoint fully connected classifier of the endpoint prediction model to obtain the endpoint prediction result. S306. The force features are processed by the force prediction model through the force prediction fully connected classifier to obtain the force prediction result.

[0033] Understandably, the start point prediction model includes a convolutional neural network for detecting and identifying start points in the Fourier spectrum, a start point long short-term neural network (LSN), and a start point fully connected classifier. The LSN extracts more comprehensive and effective start point information from the start point features. The fully connected classifier classifies the start point prediction based on the more comprehensive and effective pitch information extracted by the LSN, yielding the start point prediction result. The end point prediction model has the same structure as the start point prediction model and will not be described further. The velocity prediction model includes a convolutional neural network for detecting and identifying velocity information in the Fourier spectrum, and a velocity fully connected classifier. The velocity fully connected classifier classifies the velocity prediction based on velocity features, yielding the velocity prediction result. The velocity prediction result includes the velocity information corresponding to each note in the audio to be transcribed. The start point prediction result includes the start point information of each sentence in the audio to be transcribed. The end point prediction result includes the end information of each sentence in the audio to be transcribed.

[0034] In steps S304-306, different task predictions are performed for different task features, which can improve the accuracy of the prediction results. In particular, the start point features and end point features are further processed by a long short-term neural network, which can fully extract the effective information of the start point and end point in the audio to be transcribed, further ensuring the accuracy of the prediction results.

[0035] Optionally, in step S10, i.e., performing a Fourier transform on the audio to be transcribed to obtain a Fourier spectrum, the following steps are included: S101. Input the audio to be transcribed into the audio encoder to obtain the audio code; S102. Quantize the audio encoding to obtain Fourier spectrum encoding; S103. Input the Fourier spectrum encoding into the Fourier spectrum generator for decoding to obtain the Fourier spectrum.

[0036] In essence, audio coding refers to the encoding obtained by encoding the audio to be transcribed using an audio encoder. Quantization refers to the process of querying a codebook using a quantization function to find the corresponding Fourier spectrum code for the audio code. The codebook is an existing database containing several mappings between audio codes and Fourier spectrum codes. Fourier spectrum coding refers to the spectrum code found in the codebook using a quantization function.

[0037] Optionally, before step S10, i.e. before performing a Fourier transform on the audio to be transcribed to obtain the Fourier spectrum, the following steps are included: S104. Obtain the initial audio to be transcribed; S105. Perform noise reduction preprocessing on the initial audio to be transcribed to obtain the audio to be transcribed.

[0038] Understandably, the initial audio to be transcribed refers to the unprocessed audio initially acquired for transcription. When noise is present in the initial audio to be transcribed, it can easily lead to errors in the transcription file. Therefore, to improve the quality of audio transcription, noise reduction preprocessing can be performed on the acquired unprocessed audio to remove the influence of noise and invalid information on the transcription effect, thereby improving transcription accuracy and quality.

[0039] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0040] In one embodiment, an audio transcription device is provided, which corresponds one-to-one with the audio transcription methods described in the above embodiments. For example... Figure 3 As shown, the audio transcription device includes a Fourier spectrum module 10, a multi-task feature module 20, a multi-task prediction result module 30, and a transcription file module 40. Detailed descriptions of each functional module are as follows: Fourier spectrum module 10 is used to perform Fourier transform on the audio to be transcribed to obtain the Fourier spectrum; Multi-task feature module 20 is used to extract multi-task features from the Fourier spectrum through an attention-based multi-task feature fusion model to obtain multi-task features; The multi-task prediction result module 30 is used to perform multi-task prediction on the Fourier spectrum based on the multi-task features to obtain multi-task prediction results. The transcription file module 40 is used to generate a transcription file corresponding to the audio to be transcribed based on the prediction result.

[0041] Optionally, the multi-task feature fusion model includes a start point prediction model, an end point prediction model, a pitch prediction model, and a dynamics prediction model; the multi-task features include start point features, end point features, pitch features, and dynamics features. Multi-task feature module 20, including: The starting point feature unit is used to extract the starting point features from the Fourier spectrum through the starting point prediction model to obtain the starting point features; The termination point feature unit is used to extract termination point features from the Fourier spectrum using the termination point prediction model to obtain the termination point features; The pitch feature unit is used to extract pitch features from the Fourier spectrum using the pitch prediction model to obtain the pitch features; The force feature unit is used to extract the force features from the Fourier spectrum through the force prediction model to obtain the force features.

[0042] Optionally, the task prediction results include pitch prediction results; The multi-task prediction results module 30 includes: A hidden feature unit is used to extract hidden features of the starting point features and the ending point features through the attention mechanism; A feature fusion unit is used to fuse the hidden feature, the pitch feature, and the dynamic feature to obtain a fused feature; The pitch prediction result unit is used to perform pitch prediction processing on the fused features through the pitch long short-term neural network and the pitch fully connected classifier of the pitch prediction model to obtain the pitch prediction result.

[0043] Optionally, the task prediction results include start point prediction results, end point prediction results, and intensity prediction results; The multi-task prediction results module 30 also includes: The starting point prediction result unit is used to perform starting point prediction processing on the starting point features through the starting point long short-term neural network and the starting point fully connected classifier of the starting point prediction model to obtain the starting point prediction result. The endpoint prediction result unit is used to perform endpoint prediction processing on the endpoint features through the endpoint long short-term neural network and the endpoint fully connected classifier of the endpoint prediction model to obtain the endpoint prediction result. The force prediction result unit is used to perform force prediction processing on the force features through the force fully connected classifier of the force prediction model to obtain the force prediction result.

[0044] Optionally, the Fourier spectrum module 10 includes: An audio encoding unit is used to input the audio to be transcribed into an audio encoder to obtain audio encoding; A quantization processing unit is used to quantize the audio code to obtain a Fourier spectrum code; The Fourier spectrum unit is used to input the Fourier spectrum encoding into the Fourier spectrum generator for decoding processing to obtain the Fourier spectrum.

[0045] Optionally, prior to the Fourier spectrum module 10, the following is included: Initial audio unit to be transcribed, used to obtain the initial audio to be transcribed; The audio unit to be transcribed is used to perform noise reduction preprocessing on the initial audio to be transcribed to obtain the audio to be transcribed.

[0046] Specific limitations regarding the audio transcription device can be found in the limitations of the audio transcription method described above, and will not be repeated here. Each module in the aforementioned audio transcription device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the corresponding operations of each module.

[0047] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium and internal memory. The readable storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database stores data related to the audio transcription method. The network interface communicates with external terminals via a network connection. When the computer-readable instructions are executed by the processor, they implement an audio transcription method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0048] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions: Retrieve piano performance event data; The piano performance event data is input into the event encoder for event encoding to obtain the piano performance event code. The piano performance event encoding is converted using an audio code conversion model to obtain the Mel spectrum encoding corresponding to the piano performance event encoding. The Mel spectrum code is decoded by a Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code; The Mel spectrum data is input into a vocoder for reconstruction to obtain the piano performance audio.

[0049] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps: Retrieve piano performance event data; The piano performance event data is input into the event encoder for event encoding to obtain the piano performance event code. The piano performance event encoding is converted using an audio code conversion model to obtain the Mel spectrum encoding corresponding to the piano performance event encoding. The Mel spectrum code is decoded by a Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code; The Mel spectrum data is input into a vocoder for reconstruction to obtain the piano performance audio.

[0050] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0051] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0052] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An audio transcription method, characterized in that, include: Perform a Fourier transform on the audio to be transcribed to obtain the Fourier spectrum; Before performing a Fourier transform on the audio to be transcribed to obtain the Fourier spectrum, the following steps are included: Obtain the initial audio to be transcribed; The initial audio to be transcribed is subjected to noise reduction preprocessing to obtain the audio to be transcribed; Multi-task features are extracted from the Fourier spectrum using an attention-based multi-task feature fusion model. The multi-task feature fusion model includes a start-point prediction model, an end-point prediction model, a pitch prediction model, and a velocity prediction model. The multi-task features include start-point features, end-point features, pitch features, and velocity features. Specifically, the start-point prediction model extracts start-point features from the Fourier spectrum to obtain start-point features; the end-point prediction model extracts end-point features from the Fourier spectrum to obtain end-point features; the pitch prediction model extracts pitch features from the Fourier spectrum to obtain pitch features; and the velocity prediction model extracts velocity features from the Fourier spectrum to obtain velocity features. Features; the start point prediction model includes a convolutional neural network for detecting and identifying start points in the Fourier spectrum, the end point prediction model includes a convolutional neural network for detecting and identifying end points in the Fourier spectrum, the pitch prediction model includes a convolutional neural network for detecting and identifying pitch information in the Fourier spectrum, the velocity prediction model includes a convolutional neural network for detecting and identifying velocity information in the Fourier spectrum, the start point feature includes information on the start point of the entire audio to be transcribed and information on the start point of each sentence in the audio to be transcribed, the end point feature includes information on the end point of the entire audio to be transcribed and information on the end point of each sentence in the audio to be transcribed, the pitch feature includes all pitch information of the audio to be transcribed, and the velocity feature includes all velocity information of the audio to be transcribed. Based on the multi-task features, multi-task prediction is performed on the Fourier spectrum to obtain multi-task prediction results; wherein, the task prediction results include pitch prediction results; specifically, hidden features of the starting point features and the ending point features are extracted through the attention mechanism, and the hidden features, the pitch features, and the dynamic features are fused to obtain fused features; the fused features are then processed for pitch prediction using the pitch long short-term neural network and the pitch fully connected classifier of the pitch prediction model to obtain the pitch prediction results; Based on the prediction results, a transcription file corresponding to the audio to be transcribed is generated.

2. The audio transcription method as described in claim 1, characterized in that, The task prediction results include the starting point prediction results, the ending point prediction results, and the intensity prediction results; The step of performing multi-task prediction on the Fourier spectrum based on the multi-task features to obtain multi-task prediction results includes: The starting point prediction result is obtained by performing starting point prediction processing on the starting point features using the starting point long short-term neural network and the starting point fully connected classifier of the starting point prediction model. The endpoint prediction result is obtained by performing endpoint prediction processing on the endpoint features using the endpoint long short-term neural network and the endpoint fully connected classifier of the endpoint prediction model. The force prediction result is obtained by performing force prediction processing on the force features using the force fully connected classifier of the force prediction model.

3. The audio transcription method as described in claim 1, characterized in that, The process of performing a Fourier transform on the audio to be transcribed to obtain a Fourier spectrum includes: The audio to be transcribed is input into an audio encoder to obtain the audio encoding; The audio code is quantized to obtain Fourier spectrum code; The Fourier spectrum encoding is input into the Fourier spectrum generator for decoding to obtain the Fourier spectrum.

4. An audio transcription device, characterized in that, include: The Fourier spectrum module is used to perform Fourier transform on the audio to be transcribed to obtain the Fourier spectrum. The audio transcription device further includes: Initial audio unit to be transcribed, used to obtain the initial audio to be transcribed; The audio unit to be transcribed is used to perform noise reduction preprocessing on the initial audio to be transcribed to obtain the audio to be transcribed. A multi-task feature module is used to extract multi-task features from the Fourier spectrum using an attention-based multi-task feature fusion model to obtain multi-task features. The multi-task feature fusion model includes a start-point prediction model, an end-point prediction model, a pitch prediction model, and a velocity prediction model. The multi-task features include start-point features, end-point features, pitch features, and velocity features. Specifically, the start-point prediction model extracts start-point features from the Fourier spectrum to obtain start-point features; the end-point prediction model extracts end-point features from the Fourier spectrum to obtain end-point features; the pitch prediction model extracts pitch features from the Fourier spectrum to obtain pitch features; and the velocity prediction model extracts velocity features from the Fourier spectrum to obtain... The dynamics feature includes the following: the start point prediction model includes a convolutional neural network for detecting and identifying start points in the Fourier spectrum; the end point prediction model includes a convolutional neural network for detecting and identifying end points in the Fourier spectrum; the pitch prediction model includes a convolutional neural network for detecting and identifying pitch information in the Fourier spectrum; the dynamics prediction model includes a convolutional neural network for detecting and identifying dynamics information in the Fourier spectrum; the start point feature includes information on the start point of the entire audio to be transcribed and information on the start point of each sentence in the audio to be transcribed; the end point feature includes information on the end point of the entire audio to be transcribed and information on the end point of each sentence in the audio to be transcribed; the pitch feature includes all pitch information of the audio to be transcribed; and the dynamics feature includes all dynamics information of the audio to be transcribed. The multi-task prediction result module is used to perform multi-task prediction on the Fourier spectrum based on the multi-task features to obtain multi-task prediction results; wherein, the task prediction results include pitch prediction results; specifically, the hidden features of the starting point features and the ending point features are extracted through the attention mechanism, and the hidden features, the pitch features, and the dynamic features are fused to obtain fused features; the fused features are processed by the pitch long short-term neural network and the pitch fully connected classifier of the pitch prediction model to obtain the pitch prediction results; The transcription file module is used to generate a transcription file corresponding to the audio to be transcribed based on the prediction results.

5. The audio transcription device as described in claim 4, characterized in that, The multi-task feature fusion model includes a start point prediction model, an end point prediction model, a pitch prediction model, and a dynamics prediction model; the multi-task features include start point features, end point features, pitch features, and dynamics features. The multi-task feature module includes: The starting point feature unit is used to extract the starting point features from the Fourier spectrum using the starting point prediction model to obtain the starting point features. The termination point feature unit is used to extract termination point features from the Fourier spectrum using the termination point prediction model to obtain the termination point features; The pitch feature unit is used to extract pitch features from the Fourier spectrum using the pitch prediction model to obtain the pitch features; The force feature unit is used to extract the force features from the Fourier spectrum through the force prediction model to obtain the force features.

6. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the audio transcription method as described in any one of claims 1 to 3.

7. One or more readable storage media storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors cause the audio transcription method as described in any one of claims 1 to 3 to be performed.

Images