Audio processing model training method, audio information processing method, device and apparatus

Abstract

The application provides an audio processing model training method and device, electronic equipment and a storage medium. The application can be applied to scenarios including but not limited to map, navigation, automatic driving, vehicle networking, vehicle-road cooperation and the like. The method comprises: processing audio information in a training sample set by a discrete encoder network to obtain an audio feature encoding vector; determining a multi-task loss function of an audio processing model; classifying the audio feature encoding vector by a transformer network of the audio processing model to obtain a beat category probability corresponding to the audio feature encoding vector; obtaining beat label information corresponding to the audio feature encoding vector; and adjusting model parameters of the audio processing model according to the beat category probability, the beat label information and the multi-task loss function. Thus, the audio information can be accurately identified by the audio information processing model, and the user can have a better use experience.

Description

Technical Field

[0001] This invention relates to audio information processing technology, and more particularly to audio processing model training methods, audio information processing methods, devices, electronic devices, and storage media. Background Technology

[0002] In related technologies, audio information takes many forms, and the demand for audio information processing is experiencing explosive growth. Beat, in music, is an organizational form representing fixed unit time values ​​and dynamic patterns; beat detection is one of the important tasks in the field of music information retrieval. Beat detection algorithms automatically identify beat positions by analyzing audio data. Accurate beat detection algorithms can be applied to many areas, such as video production, music recommendation, and music style recognition, among other applications. Therefore, accurately identifying the beat in an audio clip will significantly impact the user experience. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an audio processing model training method, an audio information processing method, an apparatus, an electronic device, and a storage medium, which can accurately identify the beat and next beat in audio information through the audio information processing model, so as to provide users with a better user experience. At the same time, the trained audio processing model has better generalization ability and can process various types of audio information, reducing the probability of overfitting.

[0004] The technical solution of this invention is implemented as follows: This invention provides an audio processing model training method, comprising: The audio information in the training sample set is processed by the discrete encoder network of the audio processing model to obtain the audio feature encoding vector. Based on the usage environment of the audio processing model, determine the multi-task loss function of the audio processing model; The audio feature encoding vector is classified using the transformer network of the audio processing model to obtain the beat type probability corresponding to the audio feature encoding vector. Obtain the beat annotation information corresponding to the audio feature encoding vector; The audio processing model is trained based on the beat type probability, the beat annotation information, and the multi-task loss function. The model parameters of the audio processing model are adjusted to achieve the effect of predicting the beat type of audio information through the audio processing model.

[0005] This invention also provides an audio information processing method, including: Acquire audio information to be identified, wherein the audio information to be identified includes at least one beat type; The audio information to be identified is processed by a discrete encoder network of an audio processing model to obtain the audio feature encoding vector to be identified. The audio processing model uses a transformer network to classify the audio feature encoding vector to be identified, thereby obtaining the probability of the beat type corresponding to the audio feature encoding vector. The probability of the beat type corresponding to all the audio feature encoding vectors to be identified is combined to form the beat likelihood curve corresponding to the audio information to be identified. The local maximum value is selected from the beat likelihood curve as the beat recognition result of the audio information to be recognized. This invention also provides an audio processing model training device, comprising: The information transmission module is used to process the audio information in the training sample set through the discrete encoder network of the audio processing model to obtain the audio feature encoding vector. The information processing module is used to determine the multi-task loss function of the audio processing model based on the usage environment of the audio processing model. The information processing module is used to classify the audio feature encoding vector through the transformer network of the audio processing model to obtain the beat type probability corresponding to the audio feature encoding vector. The information processing module is used to obtain the beat annotation information corresponding to the audio feature encoding vector; The information processing module is used to train the audio processing model based on the beat type probability, the beat annotation information, and the multi-task loss function, and adjust the model parameters of the audio processing model to achieve the effect of predicting the beat type of audio information through the audio processing model.

[0006] In the above scheme, The information processing module is used to determine the multi-task loss function of the audio processing model as the beat recognition loss function when the audio processing model is used in the environment of music beat recognition. The information processing module is used to determine the multi-task loss function of the audio processing model as the transcription beat judgment loss function when the audio processing model is used in the music transcription environment. The information processing module is used to determine that the multi-task loss function of the audio processing model is the music recommendation loss function when the usage environment of the audio processing model is music recommendation.

[0007] In the above scheme, The information processing module is used when the audio information in the training sample set is mono audio data; The information processing module is used to perform a short-time Fourier transform on the mono audio data based on a windowing function corresponding to the audio processing model, thereby generating a corresponding Mel spectrogram. The information processing module is used to determine the corresponding input triplet samples based on the Mel spectrogram, and input the input triplet samples into the audio processing model; The information processing module is used to process the input triplet samples by cross-processing through the convolutional layer and max pooling layer of the audio processing model to obtain downsampling results of different input triplet samples; The information processing module is used to normalize the downsampling results of the different input triplet samples through the fully connected layer of the audio processing model to obtain the audio feature encoding vector.

[0008] In the above scheme, The information processing module is used to determine a dynamic noise threshold that matches the usage environment of the audio processing model. The information processing module is used to denoise the training sample set according to the dynamic noise threshold to form a training sample set that matches the dynamic noise threshold; or, The information processing module is used to determine a fixed noise threshold corresponding to the audio processing model, and to perform denoising processing on the training sample set according to the fixed noise threshold to form a training sample set that matches the fixed noise threshold.

[0009] In the above scheme, The information processing module is used to substitute the probability of the beat type corresponding to all audio feature encoding vectors and all beat annotation information into the multi-task loss function to obtain the calculation result of the multi-task loss function; The information processing module is used to adjust the network parameters of the audio processing model based on the calculation results of the multi-task loss function. The information processing module is used to determine the network parameters of the audio processing model until the calculation result of the multi-task loss function of the audio processing model reaches the corresponding convergence condition, so as to achieve the effect of forming a beat likelihood curve.

[0010] This invention also provides an electronic device, the electronic device comprising: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements a preceding audio processing model training method or an audio information processing method.

[0011] This invention also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement a preceding audio processing model training method or an audio information processing method.

[0012] The embodiments of the present invention have the following beneficial effects: This invention employs a discrete encoder network of an audio processing model to process audio information in a training sample set, obtaining audio feature encoding vectors. Based on the usage environment of the audio processing model, a multi-task loss function is determined. The audio feature encoding vectors are then classified using a transformer network of the audio processing model to obtain the beat type probability corresponding to each vector. Beat annotation information corresponding to each audio feature encoding vector is acquired. The audio processing model is trained based on the beat type probability, the beat annotation information, and the multi-task loss function, adjusting its model parameters. This allows the audio processing model to accurately identify beats and the next beat in audio information, providing a better user experience. Furthermore, the trained audio processing model exhibits better generalization ability, enabling it to process various types of audio information and reducing the likelihood of overfitting. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the usage environment of an audio processing model training method provided in an embodiment of the present invention; Figure 2 A schematic diagram of the composition structure of an electronic device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the working process of beat recognition in the related technology of this invention. Figure 4 This is a schematic diagram illustrating the working process of beat recognition in the related technology of this invention. Figure 5 A schematic diagram of an optional process for training an audio processing model provided in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the audio processing process of the audio information processing model in this embodiment of the invention. Figure 7 This is a schematic diagram illustrating the working process of the discrete encoder network in an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating the process of adjusting the model parameters of the audio processing model in an embodiment of the present invention; Figure 9 This is a schematic diagram of the audio processing method in an embodiment of the present invention; Figure 10 This is a schematic diagram of the audio information processing method in an embodiment of the present invention; Figure 11 This is a schematic diagram of the audio information processing method in an embodiment of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0016] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0017] 1) Client: The carrier that implements specific functions in a terminal. For example, a mobile client (APP) is a carrier of specific functions in a mobile terminal, such as an application that performs music playback or online instant messaging.

[0018] 2) Audio information: including but not limited to: background music in long videos (film and television works), music in short videos (videos uploaded by users with a length of less than 1 minute), audio (such as music videos or records with fixed scenes), and audio of sound effects in animations.

[0019] 3) Mel-Bank Features (MBF): Since the obtained spectrogram is large, in order to obtain sound features of appropriate size, it is usually passed through Mel-scale filter banks to become the Mel-Bank Features.

[0020] 4) Audio feature encoding vector, i.e. audio 01 vector, is a binary feature vector generated based on audio.

[0021] 5) Supervised fine-tuning: The unsupervised pre-trained network is further trained on manually labeled data to learn the ability to solve specific tasks.

[0022] 6) Model Training: Multi-class classification learning on the audio dataset. This model can be built using deep learning frameworks such as TensorFlow and Torch, employing multiple layers of neural networks like CNNs to form a multi-class classification model. The model's input is the audio feature vector, and algorithms such as softmax are used to output the probability of the beat type. During training, the model approximates the correct trend using objective functions such as cross-entropy.

[0023] 6) Neural Network (NN): Artificial Neural Network (ANN), also known as neural network or neural network-like network, is a mathematical or computational model in the fields of machine learning and cognitive science that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) and is used to estimate or approximate functions.

[0024] 7) Multi-task learning: In the field of machine learning, by simultaneously learning and optimizing multiple related tasks, better model accuracy can be achieved than that of a single task. Multiple tasks help each other by sharing a representation layer. This training method is called multi-task learning, also known as joint learning.

[0025] Figure 1 This is a schematic diagram illustrating the usage environment of the audio processing model training method provided in this embodiment of the invention. (See attached diagram.) Figure 1The terminals (including terminals 10-1 and 10-2) are equipped with clients capable of performing different functions. Each terminal (including terminals 10-1 and 10-2) obtains different audio information (e.g., music audio, audio track information from videos, and vocal songs) from a corresponding server 200 via a network 300 using different service processes. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both, and uses a wireless link for data transmission. The types of audio information obtained by the terminals (including terminals 10-1 and 10-2) from the corresponding server 200 via the network 300 are not the same. The audio information includes, but is not limited to, [specific examples of audio information]. This includes, but is not limited to: long audio (e.g., audio information in user-uploaded audio), short audio (e.g., audio information in user-uploaded audio less than 1 minute long), music audio (e.g., music videos or records with fixed visuals), and audio information in voice communication of instant messaging clients. For example, terminals (including terminals 10-1 and 10-2) can either obtain live audio information of vocal songs (i.e., audio information or corresponding audio links carried in the audio) from the corresponding server 200 via network 300, or obtain and listen to live audio from the corresponding server 400 via the same audio client or instant messaging client's mini-program using network 300. Servers 200 and 400 can store different types of audio. This application does not distinguish between the playback environments of different types of audio information. Taking short audio live streaming as an example, the audio information processing model provided by this invention can be applied to the playback of audio information during live streaming of short audio. In short audio playback, different live audio from different data sources is usually processed, and finally presented on the user interface (UI) with corresponding beat information, which can be used by users to edit audio information. Since network resources are limited during audio information transmission, audio information can be encoded to improve the encoding quality of audio information, save network resources during audio information transmission, and provide users with a better user experience.

[0026] The audio processing model training method provided in this application is based on artificial intelligence (AI). AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions.

[0027] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0028] In the embodiments of this application, the main artificial intelligence software technologies involved include the aforementioned speech processing technologies and machine learning. For example, it may involve Automatic Speech Recognition (ASR) technology in speech technology, including speech signal preprocessing, speech signal frequency analyzing, speech signal feature extraction, speech signal feature matching / recognition, and speech training.

[0029] For example, this could involve machine learning (ML), a multidisciplinary field encompassing probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning typically includes techniques such as deep learning, which includes artificial neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and deep neural networks (DNNs).

[0030] The structure of the electronic device according to embodiments of the present invention will be described in detail below. The electronic device can be implemented in various forms, such as a terminal with audio information processing function, such as a mobile phone running an audio client. The trained audio information processing model can be encapsulated in the terminal's storage medium, or it can be a server or server group with audio information processing function. The trained audio information processing model can be deployed in a server, as described above. Figure 1 Server 200 in the middle. Figure 2 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of the present invention. It can be understood that... Figure 2 The diagram shows only an exemplary structure of the electronic device, not all of it; implementation is possible as needed. Figure 2 The structure shown may be part or all of the structure.

[0031] The electronic device provided in this embodiment of the invention may include at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. The various components in the electronic device 20 are coupled together via a bus system 205. It is understood that the bus system 205 is used to implement communication between these components. In addition to a data bus, the bus system 205 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 205.

[0032] The user interface 203 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0033] It is understood that memory 202 can be volatile memory or non-volatile memory, or both. In this embodiment of the invention, memory 202 is capable of storing data to support the operation of a terminal (such as 10-1). Examples of this data include any computer programs used to operate on the terminal (such as 10-1), such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0034] In some embodiments, the audio information processing apparatus provided in this invention can be implemented using a combination of hardware and software. For example, the audio information processing apparatus provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the audio processing model training method provided in this invention. For instance, the hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0035] As an example of the audio information processing device provided in this embodiment of the invention, which is implemented by combining software and hardware, the audio information processing device provided in this embodiment of the invention can be directly embodied as a combination of software modules executed by processor 201. The software modules can be located in a storage medium, which is located in memory 202. Processor 201 reads the executable instructions included in the software modules in memory 202 and combines them with necessary hardware (e.g., including processor 201 and other components connected to bus system 205) to complete the audio processing model training method provided in this embodiment of the invention.

[0036] As an example, processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0037] As an example of the hardware implementation of the audio information processing device provided in the embodiments of the present invention, the device provided in the embodiments of the present invention can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, it can be executed by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the audio processing model training method provided in the embodiments of the present invention.

[0038] In this embodiment of the invention, the memory 202 is used to store various types of data to support the operation of the electronic device 20. Examples of such data include: any executable instructions for operation on the electronic device 20, such as executable instructions that can be included in the program implementing the audio processing model training method of this embodiment of the invention.

[0039] In other embodiments, the audio information processing apparatus provided in this invention can be implemented in software. Figure 2 An audio processing model training device 2020 stored in memory 202 is shown. This device can be software in the form of programs and plugins, and includes a series of modules. As an example of a program stored in memory 202, it may include the audio processing model training device 2020. The audio processing model training device 2020 includes the following software modules: an information transmission module 2081 and an information processing module 2082. When the software modules in the audio processing model training device 2020 are read into RAM and executed by processor 201, the audio processing model training method provided in this embodiment of the invention will be implemented. The functions of each software module in the audio processing model training device 2020 are described below: The information transmission module 2081 is used to process the audio information in the training sample set through the discrete encoder network of the audio processing model to obtain the audio feature encoding vector.

[0040] The information processing module 2082 is used to determine the multi-task loss function of the audio processing model based on the usage environment of the audio processing model.

[0041] The information processing module 2082 is used to classify the audio feature encoding vector through the transformer network of the audio processing model to obtain the beat type probability corresponding to the audio feature encoding vector.

[0042] The information processing module 2082 is used to obtain the beat annotation information corresponding to the audio feature encoding vector.

[0043] The information processing module is used to train the audio processing model based on the beat type probability, the beat annotation information, and the multi-task loss function, and adjust the model parameters of the audio processing model to achieve the effect of predicting the beat type of audio information through the audio processing model.

[0044] Once the audio processing model is trained, the audio processing device can also be deployed. Figure 2 Specifically, the electronic devices shown may include: The data transmission module is used to acquire audio information to be identified, wherein the audio information to be identified includes at least one beat type.

[0045] The data processing module is used to process the audio information to be identified through the discrete encoder network of the audio processing model to obtain the audio feature encoding vector to be identified.

[0046] The data processing module is used to classify the audio feature encoding vector to be identified through the transformer network of the audio processing model to obtain the beat type probability corresponding to the audio feature encoding vector.

[0047] The data processing module is used to combine the probability of the beat type corresponding to all the audio feature encoding vectors to be identified, and form the beat likelihood curve corresponding to the audio information to be identified.

[0048] The data processing module is used to select the local maximum value in the beat likelihood curve as the beat recognition result of the audio information to be recognized.

[0049] according to Figure 2 The electronic device shown, in one aspect of this application, also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform various embodiments and combinations of embodiments provided in the various optional implementations of the above-described audio processing model training method.

[0050] Combination Figure 2 The illustrated electronic device 20 demonstrates the audio processing model training method provided by an embodiment of the present invention. Before introducing the audio processing model training method provided by the present invention, the shortcomings of related technologies are first described, with reference to... Figure 3 and Figure 4, Figure 3 This is a schematic diagram illustrating the working process of beat recognition in the related technology of this invention. Figure 4 This is a schematic diagram illustrating the working process of beat recognition in the related technology of this invention. Figure 3 The method shown directly uses the audio signal as input, enabling end-to-end training and learning to extract more representative features from the audio signal, thus achieving better detection performance. However, in the likelihood estimation stage, music beat detection requires the use of contextual information from longer music segments (e.g., 30 seconds to 1 minute), but this is problematic due to the high sampling rate of music (CD quality is 44.1kHz, i.e., 44,100 samples per second). Figure 3 The neural network shown is difficult to train directly on such high-dimensional data. Therefore, a large dilation factor is used to allow the deep network to acquire information from longer musical segments. For example, if a piece of music has 120 segments, the optimal solution would be to arbitrarily interact the information of all 120 segments, but due to… Figure 3 The method shown uses a large expansion coefficient, and the t-th segment (any segment in the 120 segments) interacts only with the segments t-32, t-16, t, t+16, t+32, ... This results in an overly sparse temporal relationship of musical information and a decrease in the accuracy of beat detection.

[0051] exist Figure 4 In the method shown, if manually designed features are used for feature extraction, some audio details will be lost, which is detrimental to the accuracy of subsequent beat detection. Training a neural network requires a large amount of data, but due to the difficulty of annotating music beats, currently available music beat datasets are relatively small, containing only a few hundred songs. The limited number and types of music make it easy for deep learning models to overfit, hindering practical applications.

[0052] To overcome the above-mentioned shortcomings, see Figure 5 , Figure 5 This is an optional flowchart illustrating the audio processing model training method provided in an embodiment of the present invention. It can be understood that... Figure 5 The steps shown can be performed by various electronic devices running audio information processing devices, such as terminals, servers, or server clusters with audio information processing functions. When it is necessary to deploy an audio processing model, the audio processing model is first trained to determine the network parameters of the audio processing model. The following section focuses on... Figure 5 The steps shown are explained.

[0053] Step 501: The audio processing model training device processes the audio information in the training sample set through the discrete encoder network of the audio processing model to obtain the audio feature encoding vector.

[0054] In some embodiments of the present invention, the method for obtaining the audio feature encoding vector code further includes: When the audio information in the training sample set is mono audio data; based on the windowing function corresponding to the audio processing model, a short-time Fourier transform is performed on the mono audio data to form a corresponding Mel spectrogram; based on the Mel spectrogram, the corresponding input triplet samples are determined and input into the audio processing model; the input triplet samples are cross-processed through the convolutional layer and max pooling layer of the audio processing model to obtain downsampling results of different input triplet samples; the downsampling results of different input triplet samples are normalized through the fully connected layer of the audio processing model to obtain the audio feature encoding vector. Figure 6This diagram illustrates the audio processing process of the audio information processing model in this embodiment of the invention. Feature extraction can be performed using a VGGish network. Specifically, feature extraction in the audio processing model can be achieved through a Visual Geometry Group (VGGish) network. For example, for video information with a resolution lower than 480p, audio files can be extracted. For each audio file, a corresponding Mel spectrogram is obtained. Then, based on the Mel spectrogram, audio features are extracted using the VGGish network. The extracted vectors are then clustered and encoded using NetVlad (Net Vector of locally aggregated descriptors) to obtain audio feature vectors. NetVlad can store the distance between each feature point and its nearest cluster center and use this distance as a new feature. Since the acquired audio information may contain environmental noise, to better calculate the audio feature encoding vector, the audio can first be resampled to a 16kHz single-channel audio. Then, using a 25ms Hann window with a 10ms frame shift, a short-time Fourier transform is performed on the audio using a periodic Hann window to obtain the corresponding spectrogram. The spectrogram is then mapped to a 64th-order mel filter bank to calculate the mel spectrum, where the mel bins range from 125 to 7500Hz. The stable mel spectrum is obtained by calculating log(mel-spectrum+0.01), with the added bias of 0.01 to avoid taking the logarithm of 0. These obtained features are framed with 0.96s features, with no frame overlap. Each frame contains 64 mel bands and has a duration of 10ms (96 frames in total), thereby extracting the corresponding mel spectrogram. Finally, a clear audio feature encoding vector is obtained by processing the mel spectrogram. Therefore, compared to the various types of clear audio samples in the training sample set, normalization can expand the sources of sample information, enabling the trained audio processing model to process audio information with poorer sound quality and improve the accuracy of audio processing.

[0055] In some embodiments of the present invention, reference is made to Figure 7 , Figure 7 This is a schematic diagram of the working process of the discrete encoder network in this embodiment of the invention. Since the audio signals in the samples have a very high sampling rate, typically 44,100 samples per second, a one-minute audio file would require a time series representation of tens of millions of times in length, making it difficult to directly input into the audio processing model. Therefore, a discrete encoder network can be used to divide the audio information into different segments, i.e., 344 segments per second, and encode each segment into a feature vector. In this way, a one-minute piece of music is compressed to 344 × 60 = 20,640 segments, significantly shortening the sequence length compared to the previous method.

[0056] like Figure 7 As shown, the audio signal is segmented into 344 segments per second, and each audio segment is fed into an encoder network consisting of 12 layers of convolutional neural networks, outputting a 1024-dimensional feature vector. Then, the current vector is replaced with the closest vector (the encoder itself contains 2048 1024-dimensional vectors), achieving discretization of the audio encoding. Thus, any audio in the sample can be represented by a combination of 2048 encoded vectors. Finally, another 12-layer convolutional neural network is used to decode the encoded vector, recovering it as an audio signal. The difference between this signal and the input audio signal is calculated as a supervision signal, simultaneously training the encoder and decoder. In this way, audio segments can be encoded by the encoder and then recovered into an audio signal by the decoder. Compared to the manually coded audio segments input in related technologies, this application utilizes a massive dataset of 1.2 million music tracks in the training sample set for unsupervised pre-training, enabling the encoder to possess powerful representational and generalization capabilities, reducing the probability of failing to recognize the beat of the user-input audio.

[0057] Step 502: The audio processing model training device determines the multi-task loss function of the audio processing model based on the usage environment of the audio processing model.

[0058] In some embodiments of the present invention, the multi-task loss function of the audio processing model is determined according to the usage environment of the audio processing model, which can be achieved in the following ways: 1) When the audio processing model is used for music beat recognition, the multi-task loss function of the audio processing model is determined to be the beat recognition loss function. In modern music, the next beat usually coincides with chord changes or harmonic changes. Therefore, the joint detection of the beat and the next beat is crucial for music analysis, music structure delineation, automatic music composition, and even some creative beat-synced video production, beat-driven light shows, and other practical applications. For example, the purpose of chord recognition is to identify all chords played in an audio segment. This usually requires first detecting the beat and the next beat in the audio signal, and then identifying the notes or chords at each beat. The beat recognition loss function in this case can be represented by Formula 1: Formula 1 in, For the prediction result of a certain music segment, y is the corresponding label. They are all 3-dimensional vectors, representing the probability of beat, next beat, and not a beat or next beat, respectively, and are represented by the subscripts i=0, 1, 2.

[0059] 2) When the audio processing model is used in music transcription, the multi-task loss function of the audio processing model is determined to be the transcription beat judgment loss function. Music transcription is the task of converting the acoustic signal of music into a certain form of musical notation, such as converting MP3, WAV, and other audio files into symbolic representations in MIDI, MusicXML, ABC, and other formats for music modification. Music transcription typically requires first detecting the beat and next beat of the audio signal, then dividing the audio signal according to the detection results, and finally identifying notes in segments. Therefore, the accuracy of beat / next beat detection is crucial for music transcription. The transcription beat judgment loss function in this case can be represented by Formula 2: Formula 2 in, For the prediction result of a certain music segment, x is the corresponding label. They are all 3-dimensional vectors, representing the probability of a beat, the next beat, and not a beat or the next beat, respectively, and are represented by the subscripts i=0, 1, 2.

[0060] 3) When the audio processing model is used in music recommendation, the multi-task loss function of the audio processing model is determined to be the music recommendation loss function. Music recommendation requires measuring the similarity between music tracks to recommend similar music based on the user's historical playback records. Rhythm, as an important component of music, is also a crucial indicator for measuring music similarity in the music recommendation system. Therefore, it is necessary to accurately obtain strong and weak beats and compare them pairwise with a large number of sheet music tracks in the database to find music with the required similarity. Simultaneously, the music recommendation loss function can also detect the similarity of uploaded audio information during copyright detection to determine whether the copyright information of the audio information is subject to compliance checks. In this case, the music recommendation loss function can be represented by Formula 3: Formula 3 in, For the prediction result of a certain music segment, z is the corresponding label. They are all 3-dimensional vectors, representing the probability of a strong beat, a weak beat, and neither a strong beat nor a weak beat, respectively, and are represented by the subscripts i=0, 1, 2.

[0061] Step 503: The audio processing model training device classifies the audio feature encoding vector through the transformer network of the audio processing model to obtain the beat type probability corresponding to the audio feature encoding vector.

[0062] Step 504: The audio processing model training device obtains the beat annotation information corresponding to the audio feature encoding vector.

[0063] In some embodiments of the present invention, since the training sample set includes 1.2 million music tracks of various types, unsupervised pre-training on this massive dataset of 1.2 million music tracks enables the discrete encoder network of the audio processing model to possess powerful representation and generalization capabilities. However, for service providers of beat recognition functions, the model training time is too long, increasing the cost of hardware equipment. Therefore, a dynamic noise threshold that matches the usage environment of the audio processing model can be determined. The training sample set is then denoised according to the dynamic noise threshold to form a training sample set that matches the dynamic noise threshold. Thus, the training time of the audio processing model is further compressed using the training sample set that matches the dynamic noise threshold, reducing the cost of hardware.

[0064] In some embodiments of the present invention, since the training sample set includes 1.2 million music tracks of various types, unsupervised pre-training on this massive dataset of 1.2 million tracks enables the discrete encoder network of the audio processing model to possess powerful representation and generalization capabilities. However, for users who need to edit single tracks, it is necessary to further improve the beat recognition accuracy of the audio processing model while reducing the model training waiting time and compressing the hardware cost of the model. Therefore, for users editing single tracks such as jazz, blues, and Peking Opera, a fixed noise threshold corresponding to the audio processing model can be determined, and the training sample set can be denoised according to the fixed noise threshold to form a training sample set that matches the fixed noise threshold. Thus, while ensuring that the model does not overfit, the beat recognition accuracy of the audio processing model can be further improved, and the hardware cost of the model can be reduced.

[0065] Step 505: The audio processing model training device trains the audio processing model based on the beat type probability, the beat annotation information, and the multi-task loss function, and adjusts the model parameters of the audio processing model to achieve the effect of predicting the beat type of audio information through the audio processing model.

[0066] refer to Figure 8 , Figure 8 This is a schematic diagram illustrating the process of adjusting the model parameters of the audio processing model in an embodiment of the present invention, specifically including the following steps: Step 801: Substitute the beat type probabilities corresponding to all audio feature encoding vectors and all beat annotation information into the multi-task loss function to obtain the calculation result of the multi-task loss function.

[0067] Step 802: Adjust the network parameters of the audio processing model based on the calculation results of the multi-task loss function.

[0068] In some embodiments of the present invention, when adjusting the model parameters of the audio processing model using an unsupervised training method, for a piece of music containing n segments, firstly, the encoder extracts the feature vector of each segment. Then, the feature vectors of the first n-1 segments are input into the transformer network, the output is decoded, and compared with the audio signal of the last n-1 segments. For example, if a piece of music has 10 segments, the training method is to input the first segment x1 and predict the second segment using the audio processing model. Compare the result with the actual second segment X2, and calculate the L2 loss. And so on, input the first segment, predict the second, input the first two, predict the third, and so on, until all 10 segments have been predicted. However, since the training sample set in this application includes 1.2 million music tracks, in order to improve training speed, supervised parameter fine-tuning can be used for training. Specifically, all n feature vectors are input into the transformer network, and the output results are compared with the existing beat labels to achieve supervised fine-tuning, saving training time for the audio processing model.

[0069] Step 803: When the calculation result of the multi-task loss function of the audio processing model reaches the corresponding convergence condition, determine the network parameters of the audio processing model to achieve the effect of forming a beat likelihood curve.

[0070] The parameters of the model are solved using maximum likelihood estimation, which involves finding a set of parameters such that the likelihood (probability) of the data is maximized under this set of parameters. In the logistic regression model, the likelihood can be expressed as:

[0071] Taking the logarithm yields the log-likelihood:

[0072] Using the above equation as the objective function, the gradient descent method is used to solve for the parameters, resulting in a beat likelihood curve.

[0073] Once the model parameters of the audio processing model are determined, they can be uploaded to a cloud server network for storage. This invention can be implemented using cloud technology or blockchain network technology. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. It can also be understood as a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on cloud computing business models. The backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites; therefore, cloud technology needs cloud computing as its support.

[0074] It's important to note that cloud computing is a computing model that distributes computing tasks across a resource pool comprised of numerous computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" are infinitely scalable, readily available, and can be used on demand, expanded at any time, and paid for based on usage. As the foundational providers of cloud computing capabilities, they establish cloud resource pool platforms, often referred to as cloud platforms or Infrastructure as a Service (IaaS). These platforms deploy various types of virtual resources within the resource pool for external customers to choose from. The cloud resource pool primarily includes: computing devices (which can be virtualized machines containing operating systems), storage devices, and network devices.

[0075] Once the audio processing model is trained, it can be deployed on a server for application. (See reference...) Figure 9 , Figure 9 This is a schematic diagram of the audio processing method in an embodiment of the present invention, specifically including the following steps: Step 901: Obtain the audio information to be identified, wherein the audio information to be identified includes at least one beat type.

[0076] Step 902: Process the audio information to be identified using a discrete encoder network of the audio processing model to obtain the audio feature encoding vector to be identified.

[0077] Step 903: Classify the audio feature encoding vector to be identified using the transformer network of the audio processing model to obtain the beat type probability corresponding to the audio feature encoding vector.

[0078] Step 904: Combine the probability of the beat type corresponding to all the audio feature encoding vectors to be identified to form the beat likelihood curve corresponding to the audio information to be identified.

[0079] Step 905: Select the local maximum value from the beat likelihood curve as the beat recognition result of the audio information to be recognized.

[0080] In some embodiments of the present invention, when the audio processing model is used in a music beat recognition environment, the audio information to be recognized is marked with beats based on the beat recognition results; when the audio processing model is used in a music transcription environment, the starting position of the audio information to be recognized is marked with beats based on the beat recognition results; when the audio processing model is used in a music recommendation environment, audio similar to the audio information to be recognized is found based on the beat recognition results.

[0081] To better illustrate the use of the audio information processing method provided in this application, the following explanation uses the automatic production of montage videos as an example. (Refer to...) Figure 10 , Figure 10 This is a schematic diagram of the audio information processing method in an embodiment of the present invention. Using the audio processing model provided in this application, the beat position of the music to be edited is detected, and an indicator indicating the beat position of the music is output on the user interface. In this way, the user can intuitively see the position of the music beat (the time point of the beat in the music), and insert different pictures and / or video segments at the corresponding positions by making selections on the user interface, thereby realizing the function of changing pictures and / or videos according to the rhythm of the music.

[0082] refer to Figure 11 , Figure 11 This is a schematic diagram of the audio information processing method in an embodiment of the present invention. After obtaining the beats and next beats of multiple audio clips, the editing points can be selected from the detected beats according to the length of the video material to obtain an editing template. Finally, the video material is cut and spliced ​​according to the editing template to output a mixed video, which saves the time of beat detection when making mixed videos.

[0083] Beneficial technical effects: This invention employs a discrete encoder network of an audio processing model to process audio information in a training sample set, obtaining audio feature encoding vectors. Based on the usage environment of the audio processing model, a multi-task loss function is determined. The audio feature encoding vectors are then classified using a transformer network of the audio processing model to obtain the beat type probability corresponding to each vector. Beat annotation information corresponding to each audio feature encoding vector is acquired. The audio processing model is trained based on the beat type probability, the beat annotation information, and the multi-task loss function, adjusting its model parameters. This allows the audio processing model to accurately identify beats and the next beat in audio information, providing a better user experience. Furthermore, the trained audio processing model exhibits better generalization ability, enabling it to process various types of audio information and reducing the likelihood of overfitting.

[0084] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training an audio processing model, characterized in that, The method includes: The audio information in the training sample set is processed by the discrete encoder network of the audio processing model to obtain the audio feature encoding vector. Based on the usage environment of the audio processing model, the beat type of the audio information is determined; wherein, when the usage environment is music recommendation, the beat type is strong beat and weak beat; when the usage environment is music transcription or music beat recognition, the beat type is next beat and beat. Based on the beat type and the usage environment, determine the multi-task loss function of the audio processing model; The audio feature encoding vector is classified using the transformer network of the audio processing model to obtain the beat type probability corresponding to the audio feature encoding vector. Obtain the beat annotation information corresponding to the audio feature encoding vector; The audio processing model is trained based on the beat type probability, the beat annotation information, and the multi-task loss function. The model parameters of the audio processing model are adjusted to achieve the effect of predicting the beat type of audio information through the audio processing model.

2. The method according to claim 1, characterized in that, The step of determining the multi-task loss function of the audio processing model based on the beat type and the usage environment includes: When the audio processing model is used in a music beat recognition environment, and the beat type is next beat and beat, the multi-task loss function of the audio processing model is determined to be the beat recognition loss function; wherein, the beat recognition loss function is used to enable the audio processing model to detect beats and next beats in the audio information in the music beat recognition environment; When the audio processing model is used in a music transcription environment and the beat type is a next beat and a beat, the multi-task loss function of the audio processing model is determined to be the transcription beat judgment loss function; wherein, the transcription beat judgment loss function is used to enable the audio processing model to detect beats and next beats in the audio information in the music transcription environment; When the audio processing model is used in a music recommendation environment, and the beat types are strong beats and weak beats, the multi-task loss function of the audio processing model is determined to be the music recommendation loss function; wherein, the music recommendation loss function is used to enable the audio processing model to detect strong beats and weak beats in the audio information in the music recommendation environment.

3. The method according to claim 1, characterized in that, The method further includes: When the audio information in the training sample set is mono audio data; Based on the windowing function corresponding to the audio processing model, a short-time Fourier transform is performed on the mono audio data to form the corresponding Mel spectrogram. Based on the Mel spectrogram, the corresponding input triplet samples are determined, and the input triplet samples are input into the audio processing model; The input triplet samples are processed by the convolutional layer and max pooling layer of the audio processing model to obtain downsampling results of different input triplet samples; The downsampling results of the different input triplet samples are normalized through the fully connected layer of the audio processing model to obtain the audio feature encoding vector.

4. The method according to claim 1, characterized in that, The method further includes: Determine a dynamic noise threshold that matches the usage environment of the audio processing model; The training sample set is denoised according to the dynamic noise threshold to form a training sample set that matches the dynamic noise threshold; or, A fixed noise threshold corresponding to the audio processing model is determined, and the training sample set is denoised according to the fixed noise threshold to form a training sample set that matches the fixed noise threshold.

5. The method according to claim 1, characterized in that, The step of training the audio processing model based on the beat type probability, the beat annotation information, and the multi-task loss function, and adjusting the model parameters of the audio processing model, includes: Substitute the probability of each beat type corresponding to all audio feature encoding vectors and all beat annotation information into the multi-task loss function to obtain the calculation result of the multi-task loss function; Based on the calculation results of the multi-task loss function, adjust the network parameters of the audio processing model; The network parameters of the audio processing model are determined when the calculation result of the multi-task loss function of the audio processing model reaches the corresponding convergence condition, so as to achieve the effect of forming a beat likelihood curve.

6. An audio information processing method, characterized in that, The method includes: Acquire audio information to be identified, wherein the audio information to be identified includes at least one beat type; The audio information to be identified is processed by a discrete encoder network of an audio processing model to obtain the audio feature encoding vector to be identified. The audio processing model uses a transformer network to classify the audio feature encoding vector to be identified, thereby obtaining the probability of the beat type corresponding to the audio feature encoding vector. The probability of the beat type corresponding to all the audio feature encoding vectors to be identified is combined to form the beat likelihood curve corresponding to the audio information to be identified. The local maximum value is selected from the beat likelihood curve as the beat recognition result of the audio information to be recognized, wherein the audio processing model is trained by the method described in any one of claims 1 to 5.

7. The method according to claim 6, characterized in that, The method further includes: When the audio processing model is used in the context of music beat recognition, the beat labeling is performed on the audio information to be recognized based on the beat recognition result; When the audio processing model is used in the context of music transcription, the starting position of the audio information to be identified is marked with a beat according to the beat recognition result. When the audio processing model is used in a music recommendation environment, it searches for audio similar to the audio information to be identified based on the beat recognition result.

8. An audio processing model training device, characterized in that, The device includes: The information transmission module is used to process the audio information in the training sample set through the discrete encoder network of the audio processing model to obtain the audio feature encoding vector. An information processing module is used to determine the beat type of the audio information based on the usage environment of the audio processing model; wherein, when the usage environment is music recommendation, the beat type is strong beat and weak beat; when the usage environment is music transcription or music beat recognition, the beat type is next beat and beat; and to determine the multi-task loss function of the audio processing model based on the beat type and the usage environment. The information processing module is used to classify the audio feature encoding vector through the transformer network of the audio processing model to obtain the beat type probability corresponding to the audio feature encoding vector. The information processing module is used to obtain the beat annotation information corresponding to the audio feature encoding vector; The information processing module is used to train the audio processing model based on the beat type probability, the beat annotation information, and the multi-task loss function, and adjust the model parameters of the audio processing model to achieve the effect of predicting the beat type of audio information through the audio processing model.

9. The apparatus according to claim 8, characterized in that, The information processing module is used to determine the multi-task loss function of the audio processing model as a beat recognition loss function when the audio processing model is used in a music beat recognition environment and the beat type is next beat and beat; wherein, the beat recognition loss function is used to enable the audio processing model to detect beats and next beats in the audio information in the music beat recognition environment; The information processing module is used to determine the multi-task loss function of the audio processing model as the transcription beat judgment loss function when the audio processing model is used in a music transcription environment and the beat type is the next beat and the beat; wherein, the transcription beat judgment loss function is used to enable the audio processing model to detect the beat and the next beat of the audio information in the music transcription environment; The information processing module is used to determine the multi-task loss function of the audio processing model as a music recommendation loss function when the usage environment of the audio processing model is music recommendation and the beat types are strong beat and weak beat; wherein, the music recommendation loss function is used to enable the audio processing model to detect strong beat and weak beat in the audio information in the music recommendation environment.

10. The apparatus according to claim 8, characterized in that, The information processing module is used when the audio information in the training sample set is mono audio data; The information processing module is used to perform a short-time Fourier transform on the mono audio data based on a windowing function corresponding to the audio processing model, thereby generating a corresponding Mel spectrogram. The information processing module is used to determine the corresponding input triplet samples based on the Mel spectrogram, and input the input triplet samples into the audio processing model; The information processing module is used to process the input triplet samples by cross-processing through the convolutional layer and max pooling layer of the audio processing model to obtain downsampling results of different input triplet samples; The information processing module is used to normalize the downsampling results of the different input triplet samples through the fully connected layer of the audio processing model to obtain the audio feature encoding vector.

11. The apparatus according to claim 8, characterized in that, The information processing module is used to determine a dynamic noise threshold that matches the usage environment of the audio processing model. The information processing module is used to denoise the training sample set according to the dynamic noise threshold to form a training sample set that matches the dynamic noise threshold; or, The information processing module is used to determine a fixed noise threshold corresponding to the audio processing model, and to perform denoising processing on the training sample set according to the fixed noise threshold to form a training sample set that matches the fixed noise threshold.

12. The apparatus according to claim 8, characterized in that, The information processing module is used to substitute the probability of the beat type corresponding to all audio feature encoding vectors and all beat annotation information into the multi-task loss function to obtain the calculation result of the multi-task loss function; The information processing module is used to adjust the network parameters of the audio processing model based on the calculation results of the multi-task loss function. The information processing module is used to determine the network parameters of the audio processing model until the calculation result of the multi-task loss function of the audio processing model reaches the corresponding convergence condition, so as to achieve the effect of forming a beat likelihood curve.

13. An audio information processing device, characterized in that, The device includes: A data transmission module is used to acquire audio information to be identified, wherein the audio information to be identified includes at least one beat type; The data processing module is used to process the audio information to be identified through the discrete encoder network of the audio processing model to obtain the audio feature encoding vector to be identified. The data processing module is used to classify the audio feature encoding vector to be identified through the transformer network of the audio processing model to obtain the probability of the beat type corresponding to the audio feature encoding vector. The data processing module is used to combine the probability of the beat type corresponding to all the audio feature encoding vectors to be identified, and form the beat likelihood curve corresponding to the audio information to be identified. The data processing module is used to select the local maximum value in the beat likelihood curve as the beat recognition result of the audio information to be recognized, wherein the audio processing model is trained by the method described in any one of claims 1 to 5.

14. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, configured to implement the audio processing model training method according to any one of claims 1 to 5, or the audio information processing method according to any one of claims 6 to 7, when executing executable instructions stored in the memory.

15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the audio processing model training method according to any one of claims 1 to 5, or the audio information processing method according to any one of claims 6 to 7.

16. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the audio processing model training method according to any one of claims 1 to 5, or the audio information processing method according to any one of claims 6 to 7.

Images