Audio track data labeling method and device, equipment and storage medium

Abstract

The invention discloses an audio track data labeling method and device, equipment and a storage medium, and belongs to the field of audio processing. The method comprises the following steps: calling an audio track division model to separate a predicted track signal corresponding to at least one audio track from to-be-processed audio; under the condition that the energy of the at least one predicted track signal meets a threshold condition, determining the audio to be processed as an effective audio; when determining that the to-be-processed audio is valid based on the energy of the predicted track signal corresponding to the at least one audio track, acquiring a track labeling result of the to-be-processed audio; and training an audio track division model based on the to-be-processed audio and the track labeling result. According to the method, the annotation efficiency of the audio track data can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing, and in particular to a method, device, equipment and storage medium for labeling audio track data. Background Art

[0002] Audio usually includes track signals of multiple tracks. For example, song audio includes vocal audio and accompaniment audio, or accompaniment audio includes track audios of different instruments such as drum audio, bass audio, piano audio, etc.

[0003] In the related art, it is necessary to manually annotate some track signals in the audio so as to use the annotated track signals to perform corresponding functions. For example, in a song recording scenario, it is necessary to play the accompaniment track audio in the song audio separately so as to generate the recorded audio based on the recorded user vocal audio and the accompaniment track audio.

[0004] However, relying on manual audition and labeling of track signals has high labor costs and low labeling efficiency. Summary of the invention

[0005] The present application provides a method, device, equipment and storage medium for annotating audio track data, which can improve the efficiency of annotating audio track data. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for labeling audio track data is provided, the method comprising:

[0007] Calling the audio track separation model to separate the audio to be processed to obtain a predicted track signal corresponding to at least one audio track;

[0008] In a case where the energy of the at least one predicted track signal satisfies a threshold condition, determining the audio to be processed as valid audio;

[0009] When it is determined that the audio to be processed is valid based on the energy of the predicted track signal corresponding to the at least one audio track, obtaining a track labeling result of the audio to be processed;

[0010] The audio track segmentation model is trained based on the audio to be processed and the track labeling result.

[0011] According to another aspect of the present application, a device for labeling audio track data is provided, the device comprising:

[0012] A separation module, used for calling the audio track separation model to separate the audio to be processed to obtain a predicted track signal corresponding to at least one audio track;

[0013] A determination module, configured to determine the audio to be processed as valid audio if the energy of the at least one predicted track signal meets a threshold condition;

[0014] a labeling module, configured to obtain a track labeling result of the audio to be processed when it is determined that the audio to be processed is valid based on the energy of the predicted track signal corresponding to the at least one audio track;

[0015] A training module is used to train the audio track segmentation model based on the audio to be processed and the track labeling result.

[0016] According to another aspect of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the audio track data labeling method as described above.

[0017] According to another aspect of the present application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the computer-readable storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the audio track data labeling method as described above.

[0018] According to another aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the audio track data labeling method provided in various optional implementations of the above aspects.

[0019] The beneficial effects of the technical solution provided by this application include at least:

[0020] Use the audio track separation model to perform audio track separation on the audio to be processed, and obtain at least one predicted track signal corresponding to at least one audio track. Subsequently, calculate the energy value of each predicted track signal, and determine whether the audio to be processed contains the track signal of the audio track that needs to be marked according to the threshold condition corresponding to the audio track. If it does, the audio to be processed is determined as valid data, and the original track signal of the valid data is manually listened to to mark the audio track. If it does not contain, it means that the audio to be processed does not contain the track signal of the audio track that needs to be marked, and there is no need to manually listen to the original track signal of the audio to be processed, thereby improving the efficiency of track marking. In addition, after obtaining the track marking results, the track marking results can be used to train the audio track separation model, and the track marking results can be used to improve the separation accuracy of the audio track separation model, more accurately screen valid data, and form a virtuous cycle of track data marking. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a schematic diagram of the structure of a computer system provided by an exemplary embodiment of the present application;

[0023] Figure 2 is a flowchart of a method for labeling audio track data provided by an exemplary embodiment of the present application;

[0024] Figure 3 is a flowchart of a method for labeling audio track data provided by an exemplary embodiment of the present application;

[0025] Figure 4 is a schematic diagram of a method for labeling audio track data provided by an exemplary embodiment of the present application;

[0026] Figure 5 is a schematic diagram of a method for labeling audio track data provided by an exemplary embodiment of the present application;

[0027] Figure 6 is a schematic diagram of a method for labeling audio track data provided by an exemplary embodiment of the present application;

[0028] Figure 7 is a schematic diagram of a method for labeling audio track data provided by an exemplary embodiment of the present application;

[0029] Figure 8is a schematic diagram of a method for labeling audio track data provided by an exemplary embodiment of the present application;

[0030] Fig. 9 is a flowchart of a method for labeling audio track data provided by an exemplary embodiment of the present application;

[0031] Fig.10 is a schematic diagram of a method for labeling audio track data provided by an exemplary embodiment of the present application;

[0032] Fig.11 is a schematic diagram of the structure of an audio track data labeling device provided by an exemplary embodiment of the present application;

[0033] Fig.12 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application.

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0036] Figure 1 A schematic diagram of a computer system provided by an exemplary embodiment of the present application is shown, and the computer system may include a terminal device 101 and a server 103 .

[0037] Exemplarily, the method for annotating audio track data shown in the embodiment of the present application can be applied in a terminal device, and the terminal device 101 runs an application 102 with an annotation requirement for audio track data. The terminal device may include a mobile phone, a tablet computer, a notebook computer, a laptop computer, a desktop computer, an all-in-one computer, an Internet of Things device, an intelligent robot workstation, a television, a set-top box, smart glasses, a smart watch, a digital camera, an MP4 player, an MP5 player, a learning machine, a point-reading machine, an electronic paper book, an electronic dictionary, a vehicle-mounted device, a virtual reality (VR) player or an augmented reality (AR) player, etc.

[0038] Exemplarily, the method for annotating audio track data provided by the present application can be executed by a client in a terminal device. The client is a client of an application that has an annotation requirement for audio track data. For example, the application may include at least one of the following: an audio player, an audio application, a video application, a social application, a shopping application, a live broadcast application, an in-car audio playback application, an information application, a browser, a game application, and a recording application.

[0039] For example, an audio player provides a singing function, which requires separating the original audio into vocal audio and accompaniment audio, and marking the accompaniment audio. The audio player can use the audio track data marking method provided in this application to separate the accompaniment audio from the original audio and mark it for use by the singing function.

[0040] The terminal device 101 includes a first memory and a first processor. The first memory stores a labeling program for audio track data; the labeling program for the audio track data is called and executed by the first processor to implement the labeling method for audio track data provided in the present application. The first memory may include but is not limited to the following: Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electric Erasable Programmable Read-Only Memory (EEPROM).

[0041] The first processor may be composed of one or more integrated circuit chips. Optionally, the first processor may be a general-purpose processor, such as a central processing unit (CPU) or a network processor (NP). Optionally, the first processor may implement the audio track data annotation method provided in the present application by running a program or code.

[0042] In an optional embodiment, the terminal device 101 and the server 103 may be connected to each other via a wired or wireless network.

[0043] Exemplarily, the method for annotating audio track data provided in the present application may also be executed by a server. The server 103 is used to provide background services for the target use case system of the terminal device 101. Optionally, the server 103 undertakes the main computing work, and the terminal device 101 undertakes the secondary computing work; or, the server 103 undertakes the secondary computing work, and the terminal device 101 undertakes the main computing work; or, the server 103 and the terminal device 101 adopt a distributed computing architecture for collaborative computing.

[0044] The server 103 may be a single server, or a server cluster consisting of multiple servers, or a cloud computing service center.

[0045] Optionally, the server 103 includes a second memory and a second processor. The second memory stores a labeling program for audio track data; the labeling program for audio track data is called by the second processor to implement the labeling method for audio track data provided in the present application. Optionally, the second memory may include but is not limited to the following: RAM, ROM, PROM, EPROM, EEPROM. Optionally, the second processor may be a general-purpose processor, such as a CPU or NP.

[0046] Figure 2 FIG. 1 is a flow chart of a method for labeling audio track data provided by an exemplary embodiment of the present application. The method can be used for Figure 1 The terminal device or server shown in the figure. The method comprises the following steps.

[0047] Step 210: Calling the audio track separation model to separate the audio to be processed to obtain a predicted track signal corresponding to at least one audio track.

[0048] Exemplarily, the audio to be processed is composed of at least one original track signal, and the audio track to which the at least one original track signal belongs is unknown. The method provided in the embodiment of the present application can mark the original track signal of the target audio track that needs to be marked.

[0049] For example, the requirement is to mark the accompaniment track audio in all the audio to be processed, but the audio to be processed may include at least one of the vocal track audio, accompaniment track audio, and noise track audio. Using the method in the related art, it is necessary to manually audition each original track audio of each audio to be processed to identify the accompaniment track audio therein, and the annotation efficiency is too low. However, using the method provided in the embodiment of the present application, the audio track separation model can be used to first separate the predicted accompaniment track audio from each audio to be processed. Then, the energy threshold of the accompaniment track audio is used to roughly screen the predicted accompaniment track audio to identify whether the energy of the predicted accompaniment track audio reaches the energy threshold of the accompaniment track audio. If it reaches, the audio to be processed is likely to contain the accompaniment track audio, and the audio to be processed is marked as valid data; if it does not reach, the audio to be processed is likely not to contain the accompaniment track audio, and the audio to be processed is marked as invalid data. Then, the human only needs to audition the original track audio of the valid data and mark the accompaniment track audio from it; there is no need to audition the original track audio of the invalid data, which greatly improves the annotation efficiency of the audio track data.

[0050] The audio track separation model is a neural network model used to separate the predicted track signals corresponding to at least one audio track from the input audio to be processed. At least one audio track is an audio track that needs to be labeled (i.e., the target audio track). For example, the audio track separation model is trained to separate the predicted track signals of the first audio track, the second audio track, and the third audio track. After the audio to be processed is input into the audio track separation model, the first predicted track signal corresponding to the first audio track, the second predicted track signal corresponding to the second audio track, and the third predicted track signal corresponding to the third audio track can be obtained.

[0051] Exemplarily, the audio track splitting model can be trained using a small number of training samples. For example, the training samples can use the open source data set musdbhq. Each training sample includes: a sample audio signal (mixture), and the original track signals of four audio tracks of vocals, drums, bass, and other contained in the sample audio signal. During the training process, the sample audio signal is input into the audio track splitting model to obtain the predicted track signals of the four audio tracks, and then the audio track splitting model is trained based on the loss of the predicted track signal and the original track signal in the training sample.

[0052] Step 220: When the energy of at least one predicted track signal meets a threshold condition, the audio to be processed is determined as valid audio.

[0053] Exemplarily, the track signal of each audio track has its own energy value threshold. If the predicted track signal reaches the energy value threshold, the predicted track signal may be the track signal of the audio track. Therefore, the predicted track signal may be roughly screened based on the energy of the predicted track signal to determine whether it is the track signal of the audio track.

[0054] If at least one predicted track signal contains a track signal of an audio track that needs to be marked, the audio to be processed is determined to be valid audio, so that the original track audio of the valid audio can be manually auditioned to mark the track signal of the audio track that needs to be marked.

[0055] Exemplarily, the at least one predicted track signal includes a first predicted track signal corresponding to a first audio track separated by the audio track separation model. Then, valid data may be determined in one of the following ways:

[0056] 1) When the energy of the first prediction track signal meets the first threshold condition corresponding to the first audio track, the audio to be processed is determined as valid data.

[0057] That is, when the energy of one predicted track signal in at least one predicted track signal satisfies the threshold condition, the audio to be processed can be marked as valid data. Exemplarily, the threshold in the threshold condition corresponding to an audio track can be set according to the energy average value of the historical audio signal of the audio track, or according to the energy minimum value of the historical audio signal of the audio track, or the threshold value can be obtained by subtracting a preset error from the energy average value of the historical audio signal.

[0058] 2) When the energy of at least one predicted track signal satisfies the threshold conditions corresponding to the respective audio tracks, the audio to be processed is determined as valid data.

[0059] That is, only when the energy of at least one prediction track signal meets the corresponding threshold conditions, the audio to be processed will be marked as valid data. If the energy of at least one prediction track signal does not meet the threshold condition of the corresponding audio track, the audio to be processed will be marked as invalid data.

[0060] Valid data is the audio to be processed that needs to be manually auditioned. The valid data may contain the original track signal of the target audio track that needs to be labeled. Correspondingly, invalid data is the audio to be processed that does not need to be manually auditioned. The invalid data may not contain the original track signal of the target audio track that needs to be labeled.

[0061] Since the audio signals of different audio tracks have different energies (the energy of the audio signal is proportional to the square of the amplitude, and different audio tracks have different energies due to different amplitudes), for example, the audio energy of the suona track is usually higher than the first threshold, and the audio energy of the vocal track is usually higher than the second threshold, and the first threshold is higher than the second threshold (the amplitude of the suona audio is usually higher than the vocal audio). Therefore, the audio to be processed can be preliminarily screened according to the thresholds corresponding to the respective audio tracks, and the predicted track signals that may belong to the audio track can be screened out, and the predicted track signals with low energy and a high probability of not belonging to the audio track can be screened out. Subsequently, only the audio to be processed that contains the predicted track signal of the audio track is manually auditioned and labeled, to avoid manually auditioning a large amount of invalid data, waste manpower costs, and improve the annotation efficiency of audio track data.

[0062] Step 230: When it is determined that the audio to be processed is valid based on the energy of the predicted track signal corresponding to at least one audio track, a track labeling result of the audio to be processed is obtained.

[0063] Exemplarily, track labeling is performed on valid data to obtain a track labeling result, which includes the audio track type to which the original track signal belongs.

[0064] For example, you can manually audition the valid data and mark the track of the valid data. After obtaining the valid data, you can manually audition the original track signal of the valid data. The original track signal is likely to contain the track signal of the target audio track that needs to be marked. If the track signal of the target audio track is manually auditioned, it is marked to obtain the track marking result; if the track signal of the target audio track is not heard during the manual audition, it means that the valid data does not contain the track signal of the target audio track and does not need to be marked.

[0065] Alternatively, a track labeling model (neural network model) may be used to label the audio track to which each original track signal in the valid data belongs, and obtain a track labeling result. The track labeling model is trained to output the audio track to which the original track signal belongs based on the input original track signal.

[0066] The track labeling result includes at least one original track signal respectively describing the audio track. Alternatively, the track labeling result includes the original track signal corresponding to the target audio track. For example, at least one original track signal includes a first original track signal, a second original track signal, and a third original track signal. Then the track labeling result may include: the first original track signal corresponds to the first audio track, the second original track signal corresponds to the second audio track, and the third original track signal corresponds to the third audio track. For another example, at least one original track signal includes a first original track signal, a second original track signal, and a third original track signal. And the requirement is to label the track signal of the second audio track. Then the track labeling result may include: the second original track signal corresponds to the second audio track; and the first original track signal and the third original track signal do not need to be labeled.

[0067] Step 240: Train an audio track segmentation model based on the audio to be processed and the track annotation results.

[0068] Exemplarily, when the audio to be processed is valid data and there is a track labeling result corresponding to the audio to be processed, the audio to be processed is used as sample input data, and the track labeling result is used as a sample label to train the audio track division model. For example, the audio to be processed is input into the audio track division model to obtain the predicted track signal corresponding to the target audio track; then, the original track signal corresponding to the target audio track in the track labeling result is obtained, the loss between the predicted track signal and the original track signal is calculated, and the audio track division model is trained based on the loss. For example, the valid data includes the first original track signal, and the track labeling result includes that the first original track signal belongs to the first audio track. Then, the audio to be processed is input into the audio track division model to obtain the first predicted track signal corresponding to the first audio track, and then the first loss between the first predicted track signal and the first original track signal is calculated, and the audio track division model is trained based on the first loss.

[0069] Exemplarily, the audio track separation model includes a first network corresponding to a first audio track, the first network is used to separate the predicted track signal of the first audio track from the input audio to be processed; the track labeling result includes the first original track signal corresponding to the first audio track. The audio to be processed is used as sample input data of the first network, and the first original track signal is used as a sample label of the first network to train the audio track separation model.

[0070] This embodiment does not limit the network structure of the audio track splitting model. Any neural network model used to perform the audio track splitting task can be applied to the audio track splitting data labeling method provided in this embodiment.

[0071] In summary, the method provided in this embodiment uses an audio track separation model to perform audio track separation on the audio to be processed, and obtains at least one predicted track signal corresponding to at least one audio track. Subsequently, the energy value of each predicted track signal is calculated, and according to the threshold condition corresponding to the audio track, it is determined whether the audio to be processed contains the track signal of the audio track that needs to be marked. If it is contained, the audio to be processed is determined as valid data, and the original track signal of the valid data is manually listened to to mark the audio track. If it is not contained, it means that the audio to be processed does not contain the track signal of the audio track that needs to be marked, and there is no need to manually listen to the original track signal of the audio to be processed, thereby improving the efficiency of track marking. In addition, after obtaining the track marking result, the track marking result can be used to train the audio track separation model, and the track marking result can be used to improve the separation accuracy of the audio track separation model, and the valid data can be screened more accurately to form a virtuous circle of track data marking.

[0072] An exemplary embodiment of determining valid data is given.

[0073] Figure 3 FIG. 1 is a flow chart of a method for labeling audio track data provided by an exemplary embodiment of the present application. The method can be used for Figure 1 The terminal device or server shown. Figure 2 In the illustrated embodiment, step 220 includes step 221 .

[0074] Step 210: calling the audio track separation model to separate at least one predicted track signal corresponding to an audio track from the audio to be processed; the audio to be processed is composed of at least one original track signal, and the audio track corresponding to the at least one original track signal is unknown.

[0075] The model structure of the audio track model can be arbitrary. Figure 4 The audio track splitting model provided by this embodiment is shown. The sub-network corresponding to the target audio track in the audio track splitting model includes a sub-band segmentation module, a time-frequency modeling module, and a masking value prediction module. When the audio track splitting model is used to separate the predicted track signals of multiple audio tracks, the audio track splitting model may include sub-networks corresponding to the multiple audio tracks, that is, the audio track splitting model includes Figure 4 Multiple subnetworks are shown.

[0076] For example, Figure 4 As shown, the audio to be processed X is input into the sub-band segmentation module to obtain the sub-band result Z, the sub-band result Z is input into the time-frequency modeling module to obtain the modeling result Q, and the modeling result is input into the masking value prediction module to obtain the masking value M corresponding to the target audio track. The masking value M is multiplied by the audio to be processed X to obtain the predicted track signal S corresponding to the target audio track.

[0077] like Figure 5 As shown, a sub-band segmentation module provided by this embodiment. The sub-band segmentation module includes a segmentation network, a normalization and convolution network 501, and a merging network. The audio to be processed X is input into the segmentation network, and the segmentation network performs a short-time Fourier transform on the audio to be processed X to obtain a frequency domain signal, and divides the frequency point of the frequency domain signal into at least one sub-band to obtain a frequency domain signal of each sub-band. Subsequently, the frequency domain signal of each sub-band is respectively input into the normalization and convolution network 501 corresponding to the sub-band to obtain the convolution result corresponding to each sub-band. The normalization and convolution network 501 includes a normalization layer and at least one convolution layer. Subsequently, the convolution result corresponding to each sub-band is input into the merging network, and data splicing is performed in the frequency domain, and the convolution result of each sub-band is merged into a band division result Z.

[0078] like Figure 6 As shown, a time-frequency modeling module is provided in this embodiment. The time-frequency modeling module can be implemented using RNN (Recurrent Neural Network). For example, the time-frequency modeling module may include a time-domain RNN network and a frequency-domain RNN network; the band division result Z is input into the time-domain RNN network according to the time-domain sequence to obtain an intermediate result Z'. The time-domain RNN network includes a normalization layer, a BLSTM (Bidirectional Long Short-Term Memory) and a convolutional network. The intermediate result Z' is input into the frequency-domain RNN network according to the frequency-domain sequence to obtain a modeling result Q. The frequency-domain RNN network includes a normalization layer, a BLSTM and a convolutional network.

[0079] like Figure 6 As shown, a masking value prediction module provided by the present embodiment. The masking value prediction module includes a segmentation network, a normalization and MLP (Multilayer Perceptron) network 502 and a merging network. The modeling result Q is input into the segmentation network, and the segmentation network divides the modeling result Q into at least one sub-band to obtain the modeling result of each sub-band. Subsequently, the modeling result of each sub-band is respectively input into the normalization and MLP network 502 corresponding to the sub-band to obtain the prediction result corresponding to each sub-band. The normalization and MLP network 502 includes a normalization layer and an MLP network. Subsequently, the prediction result corresponding to each sub-band is input into the merging network, data splicing is performed in the frequency domain, and the prediction result of each sub-band is merged into a masking value M. Subsequently, the masking value M is multiplied by the audio to be processed X to obtain the first predicted track signal of the first audio track.

[0080] Step 221: When the energy of at least one predicted track signal satisfies the threshold conditions corresponding to the respective audio tracks, the audio to be processed is determined as valid data.

[0081] Taking the first predicted track signal corresponding to the first audio track as an example, calculate the first energy value of the first predicted track signal; and calculate the sum of the energy values ​​of at least one predicted track signal; calculate the ratio of the first energy value to the sum of the energy values; when the ratio is not less than the first threshold corresponding to the first audio track, determine that the energy of the first predicted track signal meets the threshold condition corresponding to the first track.

[0082] For example, the predicted track signal includes four audio tracks: vocals, drums, bass, and other. The threshold conditions corresponding to the four audio tracks can be:

[0083] Vocals:

[0084] drum:

[0085] Bass:

[0086] other:

[0087] Among them, y v (t) is the predicted track signal of the vocal track, y d (t) is the predicted track signal of the drum track, y b (t) is the predicted orbit signal of the bass track, y o (t) is the predicted orbit signal of other orbits. v is the threshold for the vocal track, α d is the threshold of the drum track, α b is the threshold of the bass track, α o is the threshold for other tracks.

[0088] Step 230: When it is determined that the audio to be processed is valid based on the energy of the predicted track signal corresponding to at least one audio track, a track labeling result of the audio to be processed is obtained, where the track labeling result includes an audio track corresponding to at least one original track signal.

[0089] Step 240: Train an audio track segmentation model based on the audio to be processed and the track annotation results.

[0090] For example, Figure 8As shown, the audio to be processed 301 is input into the audio track model 302 to obtain four predicted track signals on four audio tracks, and then the four predicted track signals are energy determined 304. When the energy of the four predicted track signals meets the threshold of the respective audio tracks, the audio to be processed 301 is determined as valid data. The valid data is provided to manual labeling. The result of manual labeling can be used to train the audio track model 302, and can also be used to synthesize the audio to be processed 301.

[0091] In summary, the method provided in this embodiment uses an audio track separation model to perform audio track separation on the audio to be processed, and obtains at least one predicted track signal corresponding to at least one audio track. Subsequently, for the predicted track signal of an audio track, the ratio of the energy value of the predicted track signal to the total energy value of all predicted track signals is calculated. When the ratio is greater than the capacity threshold corresponding to the audio track, it is determined that the predicted track signal of the audio track meets the threshold condition. When at least one predicted bone carving signal meets the threshold condition, the audio to be processed is marked as valid data. Invalid data of the track signal that does not contain the target audio track is screened out, and track marking is performed only on valid data to improve the efficiency of track marking.

[0092] An exemplary embodiment of training an audio track segmentation model is given below.

[0093] Fig. 9 FIG. 1 is a flow chart of a method for labeling audio track data provided by an exemplary embodiment of the present application. The method can be used for Figure 1 The terminal device or server shown. Figure 2 In the illustrated embodiment, step 240 of the method includes steps 241 to 245 .

[0094] Step 241: Input the audio to be processed into an audio track segmentation model to obtain a first number of predicted track signals.

[0095] The first number is the number of audio tracks that can be separated by the audio track splitting model. For example, if the audio track splitting model is used to separate track signals of three audio tracks, the first number is 3.

[0096] For another example, the audio to be processed is input into the audio track splitting model to obtain four predicted track signals on four audio tracks: vocals, drums, bass, and other.

[0097] Then, the loss between the predicted track signal and the original track signal is calculated.

[0098] For example, the audio to be processed is input into the audio track splitting model to obtain a first number of predicted track signals, and then the loss of the first number of predicted track signals and the first number of original track signals is calculated according to the manual labeling results. The loss calculation method is as follows.

[0099] The loss function is:

[0100] Among them, L loss For loss, is the time domain loss of the audio signal of J audio tracks (ie, J is the first number), is the frequency domain loss of the audio signal of J audio tracks. J is the number of audio tracks. α is the proportional factor between the time domain loss and the frequency domain loss, which is a preset value.

[0101] The time domain loss function is:

[0102] Among them, y j (t) is the original track signal (label data or real data) of the jth audio track, is the predicted track signal of the jth audio track, and ‖‖1 is the first-order norm.

[0103] The frequency domain loss function is:

[0104]

[0105] in, is the resolution loss of the i-th resolution, and M is the total number of resolutions. is the amplitude loss of the jth audio track. is the logarithmic loss of the j audio tracks. STFT(y(t)) is the frequency domain signal obtained by short-time Fourier transform of the original track signal. is the predicted frequency domain signal obtained by performing short-time Fourier transform on the predicted orbit signal. ‖‖ F To take the second-order norm.

[0106] According to the calculation formula of the above loss function, it can be obtained that the loss calculation method includes the following steps 242 to 244.

[0107] Step 242: Calculate a first number of predicted trajectory signals The original track signal y of the corresponding audio track j (t) time domain loss

[0108] Exemplarily, for one of the first number of audio tracks, a first-order norm of the amplitude difference between the predicted track signal and the original track signal in the time domain is calculated. The sum of the sub-track time domain loss of an audio track is obtained. Calculate the sum of the time domain losses of the first number of audio tracks Get the time domain loss

[0109] Step 243: Calculate a first number of predicted trajectory signals The original track signal y of the corresponding audio track j Frequency domain loss of (t)

[0110] Exemplarily, for one of the first number of audio tracks, the amplitude loss between the predicted track signal and the original track signal in the frequency domain is calculated. and log loss Calculate the sum of amplitude loss and logarithmic loss to get the sub-track frequency domain loss of an audio track Calculate the sum of the sub-track frequency domain losses of the first number of audio tracks to obtain the frequency domain loss

[0111] Exemplarily, the frequency domain loss of a sub-track of an audio track may be calculated by: performing a short-time Fourier transform on the predicted track signal according to at least one set of resolutions to obtain at least one set of predicted frequency domain signals; one set of resolutions in the at least one set of resolutions includes a window length and an offset step size; performing a short-time Fourier transform on the original track signal according to the at least one set of resolutions to obtain at least one set of frequency domain signals; for one set of resolutions in the at least one set of resolutions, calculating the amplitude loss and logarithmic loss of the predicted frequency domain signal and the frequency domain signal, and determining the sum of the amplitude loss and the logarithmic loss as the resolution loss corresponding to one set of resolutions Calculate the sum of the resolution loss of at least one set of resolutions to obtain the sub-track frequency domain loss of an audio track

[0112] The amplitude loss can be calculated by calculating the difference between the absolute values ​​of the amplitudes of the frequency domain signal and the predicted frequency domain signal to obtain the amplitude difference. Calculate the second norm of the amplitude difference and get the first second norm Calculate the second norm of the absolute value of the amplitude of the frequency domain signal and get the second second norm ‖|STFT(y(t))|‖ F ; Calculate the ratio of the first and second norms to get the amplitude loss

[0113] The logarithmic loss can be calculated by: calculating the common logarithm of the absolute value of the amplitude of the frequency domain signal to obtain the first logarithm log|STFT(y(t))|; calculating the common logarithm of the absolute value of the amplitude of the predicted frequency domain signal to obtain the second logarithm Calculate the first norm of the difference between the first and second logarithms Get the logarithmic loss

[0114] Step 244: Calculate the weighted sum of the time domain loss and the frequency domain loss The ratio of the weighted sum to the first quantity Determined as the loss value L loss .

[0115] Step 245: Train the audio track segmentation model based on the loss value.

[0116] For example, Fig.10 As shown, after the audio to be processed 301 is input into the audio track segmentation model 302, four predicted track signals of four audio tracks are obtained, and then the loss value is calculated according to the label, and the audio track segmentation model is trained based on the loss value.

[0117] In summary, the method provided in this embodiment uses the audio track separation model to perform audio track separation on the audio to be processed, and obtains at least one predicted bone carving signal corresponding to at least one audio track. According to the energy of the predicted track signal, it is determined whether the audio to be processed is valid data, and the valid data is manually labeled. According to the manual labeling results, the audio to be processed is used as a training sample to train the audio track separation model, improve the separation accuracy of the audio track separation model, more accurately screen valid data, and form a virtuous cycle of track data labeling.

[0118] It should be noted that the sequence of the method steps provided in the embodiments of the present application can be appropriately adjusted, and the steps can be increased or decreased accordingly according to the circumstances. Any technician familiar with the technical field can easily think of the changed methods within the technical scope disclosed in the present application, and they should be included in the protection scope of the present application, so they will not be repeated here.

[0119] Fig.11 : is a schematic diagram of the structure of an audio track data annotation device provided by an exemplary embodiment of the present application. The device includes:

[0120] The separation module 601 is used to call the audio track separation model to separate the audio to be processed to obtain a predicted track signal corresponding to at least one audio track; the audio to be processed is composed of at least one original track signal, and the audio track corresponding to the at least one original track signal is unknown;

[0121] A determination module 602, configured to determine the audio to be processed as valid audio if the energy of the at least one predicted track signal meets a threshold condition;

[0122] A labeling module 603 is used to obtain a track labeling result of the audio to be processed when it is determined that the audio to be processed is valid based on the energy of the predicted track signal corresponding to the at least one audio track, wherein the track labeling result includes the audio track corresponding to the at least one original track signal;

[0123] The training module 604 is used to train the audio track segmentation model based on the audio to be processed and the track labeling result.

[0124] In an optional embodiment, the at least one predicted track signal includes a first predicted track signal corresponding to a first audio track separated by the audio track separation model;

[0125] The determination module 602 is configured to perform one of the following:

[0126] When the energy of the first predicted track signal meets a first threshold condition corresponding to the first audio track, determining the audio to be processed as the valid data;

[0127] When the energy of the at least one predicted track signal satisfies the threshold conditions corresponding to the respective audio tracks, the audio to be processed is determined as the valid data.

[0128] In an optional embodiment, the determination module 602 is configured to calculate a first energy value of the first predicted track signal; and calculate a sum of energy values ​​of the at least one predicted track signal;

[0129] The determination module 602 is used to calculate the ratio of the first energy value to the sum of the energy values;

[0130] The determination module 602 is configured to determine, when the ratio is not less than a first threshold corresponding to the first audio track, whether the energy of the first predicted track signal satisfies a threshold condition corresponding to the first track.

[0131] In an optional embodiment, the audio track separation model includes a first network corresponding to a first audio track, and the first network is used to separate the predicted track signal of the first audio track from the input audio to be processed;

[0132] The track marking result includes that the first original track signal in the audio to be processed corresponds to the first audio track;

[0133] The training module 604 is used to train the audio track segmentation model by using the audio to be processed as sample input data of the first network and the first original track signal as a sample label of the first network.

[0134] In an optional embodiment, the training module 604 is used to input the audio to be processed into the audio track model to obtain a first number of predicted track signals;

[0135] The training module 604 is configured to calculate the time domain loss between the first number of predicted track signals and the original track signals of the corresponding audio tracks; and calculate the frequency domain loss between the first number of predicted track signals and the original track signals of the corresponding audio tracks;

[0136] The training module 604 is used to calculate the weighted sum of the time domain loss and the frequency domain loss; and determine the ratio of the weighted sum to the first number as the loss value;

[0137] The training module 604 is used to train the audio track segmentation model based on the loss value.

[0138] In an optional embodiment, the training module 604 is configured to calculate, for one audio track among the first number of audio tracks, a sum of first-order norms of amplitude differences between the predicted track signal and the original track signal in the time domain, to obtain a time domain loss of the audio track;

[0139] The training module 604 is used to calculate the sum of the time domain losses of the first number of audio tracks to obtain the time domain loss.

[0140] In an optional embodiment, the training module 604 is used to calculate, for one audio track among the first number of audio tracks, an amplitude loss and a logarithmic loss between the predicted track signal and the original track signal in the frequency domain, and calculate the sum of the amplitude loss and the logarithmic loss to obtain a sub-track frequency domain loss of the audio track;

[0141] The training module 604 is used to calculate the sum of the frequency domain losses of the first number of audio tracks to obtain the frequency domain loss.

[0142] In an optional embodiment, the training module 604 is used to perform short-time Fourier transform on the predicted orbit signal according to at least one set of resolutions to obtain at least one set of predicted frequency domain signals; one set of resolutions in the at least one set of resolutions includes a window length and an offset step size;

[0143] The training module 604 is configured to perform short-time Fourier transform on the original track signal according to the at least one set of resolutions to obtain at least one set of frequency domain signals;

[0144] The training module 604 is used to calculate the amplitude loss and logarithmic loss of the predicted frequency domain signal and the frequency domain signal for one set of resolutions in the at least one set of resolutions, and determine the sum of the amplitude loss and the logarithmic loss as the resolution loss corresponding to one set of resolutions;

[0145] The training module 604 is used to calculate the sum of the resolution losses of the at least one set of resolutions to obtain the sub-track frequency domain loss of an audio track.

[0146] In an optional embodiment, the training module 604 is used to calculate the difference between the absolute values ​​of the amplitudes of the frequency domain signal and the predicted frequency domain signal to obtain an amplitude difference;

[0147] The training module 604 is used to calculate the second norm of the amplitude difference to obtain a first second norm; and calculate the second norm of the absolute value of the amplitude of the frequency domain signal to obtain a second second norm;

[0148] The training module 604 is used to calculate the ratio of the first second norm to the second second norm to obtain the amplitude loss.

[0149] In an optional embodiment, the training module 604 is used to calculate the common logarithm of the absolute value of the amplitude of the frequency domain signal to obtain a first logarithm; and calculate the common logarithm of the absolute value of the amplitude of the predicted frequency domain signal to obtain a second logarithm;

[0150] The training module 604 is used to calculate the first-order norm of the difference between the first logarithm and the second logarithm to obtain the logarithmic loss.

[0151] It should be noted that the audio track data annotation device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the audio track data annotation device provided in the above embodiment and the audio track data annotation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0152] The embodiment of the present application further provides a computer device, which includes: a processor and a memory, wherein at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the audio track data labeling method provided by the above-mentioned method embodiments. The computer device can be implemented as a terminal device.

[0153] For example, Fig.12It is a structural diagram of a computer device provided by an exemplary embodiment of the present application.

[0154] Typically, the computer device 1700 includes a processor 1701 and a memory 1702 .

[0155] The processor 1701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0156] The memory 1702 may include one or more computer-readable storage media, which may be non-transitory. The memory 1702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1702 is used to store at least one instruction, which is used to be executed by the processor 1701 to implement the audio track data labeling method provided in the method embodiment of the present application.

[0157] In some embodiments, the computer device 1700 may also optionally include: a peripheral device interface 1703 and at least one peripheral device. The processor 1701, the memory 1702 and the peripheral device interface 1703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1704, a display screen 1705, a camera assembly 1706, an audio circuit 1707 and a power supply 1708.

[0158] The peripheral device interface 1703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1701 and the memory 1702. In some embodiments, the processor 1701, the memory 1702, and the peripheral device interface 1703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1701, the memory 1702, and the peripheral device interface 1703 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.

[0159] The radio frequency circuit 1704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1704 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1704 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1704 can communicate with other computer devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1704 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0160] The display screen 1705 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1705 is a touch display screen, the display screen 1705 also has the ability to collect touch signals on the surface or above the surface of the display screen 1705. The touch signal can be input to the processor 1701 as a control signal for processing. At this time, the display screen 1705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1705 can be one, and the front panel of the computer device 1700 is set; in other embodiments, the display screen 1705 can be at least two, which are respectively set on different surfaces of the computer device 1700 or are folded; in some other embodiments, the display screen 1705 can be a flexible display screen, which is set on the curved surface or folded surface of the computer device 1700. Even, the display screen 1705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1705 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0161] The camera assembly 1706 is used to capture images or videos. Optionally, the camera assembly 1706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the computer device 1700, and the rear camera is arranged on the back of the computer device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0162] The audio circuit 1707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1701 for processing, or input them into the radio frequency circuit 1704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the computer device 1700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1701 or the radio frequency circuit 1704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1707 may also include a headphone jack.

[0163] The power supply 1708 is used to power various components in the computer device 1700. The power supply 1708 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1708 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0164] In some embodiments, the computer device 1700 further includes one or more sensors 1709 , including but not limited to: an acceleration sensor 1710 , a gyroscope sensor 1711 , a pressure sensor 1712 , an optical sensor 1713 , and a proximity sensor 1714 .

[0165] The acceleration sensor 1710 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the computer device 1700. For example, the acceleration sensor 1710 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1701 can control the touch display screen 1705 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1710. The acceleration sensor 1710 can also be used to collect game or user motion data.

[0166] The gyro sensor 1711 can detect the body direction and rotation angle of the computer device 1700, and the gyro sensor 1711 can cooperate with the acceleration sensor 1710 to collect the user's 3D actions on the computer device 1700. The processor 1701 can implement the following functions based on the data collected by the gyro sensor 1711: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0167] The pressure sensor 1712 can be set on the side frame of the computer device 1700 and / or the lower layer of the touch display screen 1705. When the pressure sensor 1712 is set on the side frame of the computer device 1700, it can detect the user's grip signal of the computer device 1700, and the processor 1701 performs left and right hand recognition or shortcut operations according to the grip signal collected by the pressure sensor 1712. When the pressure sensor 1712 is set on the lower layer of the touch display screen 1705, the processor 1701 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 1705. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0168] The optical sensor 1713 is used to collect the ambient light intensity. In one embodiment, the processor 1701 can control the display brightness of the touch display screen 1705 according to the ambient light intensity collected by the optical sensor 1713. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1705 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1705 is reduced. In another embodiment, the processor 1701 can also dynamically adjust the shooting parameters of the camera assembly 1706 according to the ambient light intensity collected by the optical sensor 1713.

[0169] The proximity sensor 1714, also called a distance sensor, is usually arranged on the front panel of the computer device 1700. The proximity sensor 1714 is used to collect the distance between the user and the front of the computer device 1700. In one embodiment, when the proximity sensor 1714 detects that the distance between the user and the front of the computer device 1700 is gradually decreasing, the processor 1701 controls the touch display screen 1705 to switch from the screen-on state to the screen-off state; when the proximity sensor 1714 detects that the distance between the user and the front of the computer device 1700 is gradually increasing, the processor 1701 controls the touch display screen 1705 to switch from the screen-off state to the screen-on state.

[0170] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the computer device 1700, and the computer device 1700 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0171] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set. When the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor of a computer device, the audio track data marking method provided by the above-mentioned method embodiments is implemented.

[0172] The present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the audio track data labeling method provided by each of the above method embodiments.

[0173] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0174] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switching, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for labeling audio track data, characterized in that: The method comprises: Calling the audio track separation model to separate the audio to be processed to obtain a predicted track signal corresponding to at least one audio track; In a case where the energy of the at least one predicted track signal satisfies a threshold condition, determining the audio to be processed as valid audio; When it is determined that the audio to be processed is valid based on the energy of the predicted track signal corresponding to the at least one audio track, obtaining a track labeling result of the audio to be processed; The audio track segmentation model is trained based on the audio to be processed and the track labeling result.

2. The method according to claim 1, characterized in that The at least one predicted track signal includes a first predicted track signal corresponding to the first audio track separated by the audio track separation model; The step of determining the to-be-processed audio as valid data when the energy of the at least one predicted track signal satisfies a threshold condition comprises one of the following: When the energy of the first predicted track signal meets a first threshold condition corresponding to the first audio track, determining the audio to be processed as the valid data; When the energy of the at least one predicted track signal satisfies the threshold conditions corresponding to the respective audio tracks, the audio to be processed is determined as the valid data.

3. The method according to claim 2, characterized in that The energy of the first predicted track signal satisfies a threshold condition corresponding to the first audio track, including: Calculating a first energy value of the first predicted track signal; and calculating a sum of energy values ​​of the at least one predicted track signal; calculating a ratio of the first energy value to the sum of the energy values; When the ratio is not less than a first threshold corresponding to the first audio track, it is determined that the energy of the first predicted track signal meets the threshold condition corresponding to the first track.

4. The method according to any one of claims 1 to 3, characterized in that: The audio track separation model includes a first network corresponding to a first audio track, and the first network is used to separate the predicted track signal of the first audio track from the input audio to be processed; The track marking result includes that the first original track signal in the audio to be processed corresponds to the first audio track; The step of training the audio track segmentation model based on the audio to be processed and the track labeling result includes: The audio to be processed is used as sample input data of the first network, and the first original track signal is used as a sample label of the first network to train the audio track segmentation model.

5. The method according to any one of claims 1 to 3, characterized in that: The step of training the audio track segmentation model based on the audio to be processed and the track labeling result includes: Inputting the audio to be processed into the audio track model to obtain a first number of predicted track signals; Calculating the time domain loss between the first number of predicted track signals and the original track signals of the corresponding audio tracks; and calculating the frequency domain loss between the first number of predicted track signals and the original track signals of the corresponding audio tracks; Calculating a weighted sum of the time domain loss and the frequency domain loss; determining a ratio of the weighted sum to the first number as a loss value; The audio track segmentation model is trained based on the loss value.

6. The method according to claim 5, characterized in that The calculating the time domain loss between the first number of predicted track signals and the original track signals of the corresponding audio tracks includes: For one audio track among the first number of audio tracks, calculating the sum of the first-order norms of the amplitude differences between the predicted track signal and the original track signal in the time domain, to obtain a time domain loss of the audio track; The sum of the time domain losses of the sub-tracks of the first number of audio tracks is calculated to obtain the time domain loss.

7. The method according to claim 5, characterized in that The calculating the frequency domain loss between the first number of predicted track signals and the original track signal of the corresponding audio track includes: For one audio track among the first number of audio tracks, calculating the amplitude loss and logarithmic loss in the frequency domain between the predicted track signal and the original track signal, and calculating the sum of the amplitude loss and the logarithmic loss to obtain a sub-track frequency domain loss of the audio track; The sum of the sub-track frequency domain losses of the first number of audio tracks is calculated to obtain the frequency domain loss.

8. The method according to claim 7, characterized in that The calculating the amplitude loss and logarithmic loss of the predicted track signal and the original track signal in the frequency domain, calculating the sum of the amplitude loss and the logarithmic loss, and obtaining the sub-track frequency domain loss of an audio track includes: Performing a short-time Fourier transform on the predicted track signal according to at least one set of resolutions to obtain at least one set of predicted frequency domain signals; one set of resolutions in the at least one set of resolutions includes a window length and an offset step length; Performing a short-time Fourier transform on the original track signal according to the at least one set of resolutions to obtain at least one set of frequency domain signals; For a set of resolutions among the at least one set of resolutions, calculating the amplitude loss and the logarithmic loss of the predicted frequency domain signal and the frequency domain signal, and determining the sum of the amplitude loss and the logarithmic loss as the resolution loss corresponding to a set of resolutions; The sum of the resolution losses of the at least one set of resolutions is calculated to obtain the sub-track frequency domain loss of an audio track.

9. The method according to claim 8, characterized in that The calculating the amplitude loss between the predicted frequency domain signal and the frequency domain signal comprises: Calculating the difference between the absolute values ​​of the amplitudes of the frequency domain signal and the predicted frequency domain signal to obtain an amplitude difference; Calculating the second norm of the amplitude difference to obtain a first second norm; and calculating the second norm of the absolute value of the amplitude of the frequency domain signal to obtain a second second norm; The ratio of the first second norm to the second second norm is calculated to obtain the amplitude loss.

10. The method according to claim 8, characterized in that The calculating the logarithmic loss of the predicted frequency domain signal and the frequency domain signal comprises: Calculating the common logarithm of the absolute value of the amplitude of the frequency domain signal to obtain a first logarithm; and calculating the common logarithm of the absolute value of the amplitude of the predicted frequency domain signal to obtain a second logarithm; A first-order norm of a difference between the first logarithm and the second logarithm is calculated to obtain the logarithmic loss.

11. A device for labeling audio track data, characterized in that: The device comprises: A separation module, used for calling the audio track separation model to separate the audio to be processed to obtain a predicted track signal corresponding to at least one audio track; A determination module, configured to determine the audio to be processed as valid audio if the energy of the at least one predicted track signal meets a threshold condition; a labeling module, configured to obtain a track labeling result of the audio to be processed when it is determined that the audio to be processed is valid based on the energy of the predicted track signal corresponding to the at least one audio track; A training module is used to train the audio track segmentation model based on the audio to be processed and the track labeling result.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the audio track data labeling method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one program, and the at least one program is loaded and executed by a processor to implement the audio track data marking method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the audio track data labeling method according to any one of claims 1 to 10.

Images