Method of processing an audio track signal, computer device, storage medium and program product

Abstract

The solution disclosed in this application belongs to the field of artificial intelligence technology, and specifically relates to a method, computer device, storage medium, and program product for processing audio track signals. The method acquires multiple audio track signals from a target song, including audio track signals corresponding to multiple instruments contained in the target song; determines a first audio track signal to be processed from the multiple audio track signals; inputs the first audio track signal and a second audio track signal into a trajectory generation model, which determines the spatial motion trajectory corresponding to the first audio track signal based on the second audio track signal. The second audio track signal comprises the remaining audio track signals besides the first audio track signal, and the spatial motion trajectory corresponding to the first audio track signal includes the first spatial position of each audio frame included in the first audio track signal within the sound field. Using this application can improve the efficiency of generating spatial motion trajectories.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for processing audio track signals, a computer device, a storage medium, and a program product. Background Technology

[0002] With the development of computer technology, there are increasingly more ways to encode song audio, resulting in richer playback effects. For example, to improve the quality of song playback, the audio data can include multiple audio tracks, each corresponding to vocal audio or an instrument audio. Based on this, spatial motion trajectories can be set for one or more audio tracks in a three-dimensional sound field, allowing users to experience dynamic auditory sensations in three-dimensional space.

[0003] The spatial motion trajectory of an audio track signal within a sound field can be achieved by setting the spatial position of each audio frame within the three-dimensional sound field. Currently, high-quality spatial motion trajectories for audio track signals are set by professionals using specialized music editing software. Therefore, setting spatial motion trajectories for audio track signals within a song's audio is relatively inefficient. Summary of the Invention

[0004] This application provides a method, computer device, storage medium, and program product for processing audio track signals, which can improve the efficiency of generating spatial motion trajectories for generating audio track signals. The technical solution is as follows: In a first aspect, a method for processing audio track signals is provided. The method includes: acquiring multiple audio track signals included in a target song, wherein the multiple audio track signals include audio track signals corresponding to multiple sound objects included in the target song; determining a first audio track signal to be processed among the multiple audio track signals, and determining at least one second audio track signal that has an association relationship with the first audio track signal; inputting the first audio track signal and the second audio track signal into a trajectory generation model, wherein the trajectory generation model generates a spatial motion trajectory corresponding to the first audio track signal based on the association relationship between the second audio track signal and the first audio track signal, wherein the spatial motion trajectory includes a first spatial position of each audio frame included in the first audio track signal in a sound field, and the first spatial position is used to characterize the change of the position of the first audio track signal in the sound field over time.

[0005] Optionally, the trajectory generation model includes a feature extraction unit and a trajectory generation unit; the step of inputting the first audio track signal and the second audio track signal into the trajectory generation model, and having the trajectory generation model generate the spatial motion trajectory corresponding to the first audio track signal based on the correlation between the second audio track signal and the first audio track signal, includes: inputting the first audio track signal and the second audio track signal into the feature extraction unit respectively, to obtain first feature data after the feature extraction unit performs harmonic short-time Fourier transform on the first audio track signal, and second feature data after performing harmonic short-time Fourier transform on the second audio track signal; inputting the first feature data and the second feature data into the trajectory generation unit, to obtain the spatial motion trajectory corresponding to the first audio track signal output by the trajectory generation unit according to the correlation between the first feature data and the second feature data.

[0006] Optionally, the trajectory generation unit includes multiple feature processing sub-units, a temporal convolutional layer, and a linear layer. The multiple feature processing sub-units are connected in series, and the temporal convolutional layer is connected in parallel with the last feature processing sub-unit among the multiple feature processing sub-units. The output data of the temporal convolutional layer and the last feature processing sub-unit are the input data of the linear layer.

[0007] Optionally, the trajectory generation model further includes a bridging unit. The step of inputting the first feature data and the second feature data into the trajectory generation unit includes: inputting the first feature data and the second feature data into the bridging unit to obtain third feature data after the bridging unit processes the first feature data and the second feature data; and inputting the third feature data into the trajectory generation unit.

[0008] Optionally, before inputting the first audio track signal and the second audio track signal into the trajectory generation model, the method includes: acquiring training samples, wherein the training samples include a first sample song, a second sample song, and a third sample song, wherein the first sample song is a song in which the included original audio track signal has a spatial motion trajectory, the second sample song is a song in which a spatial motion trajectory is manually added to the included audio track signal, and the third sample song is a song in which a spatial motion trajectory is added to the included audio track signal according to a template of spatial motion trajectory; and training the trajectory generation model based on the training samples.

[0009] Optionally, before training the trajectory generation model based on the training samples, the method includes: for each audio track signal included in the training samples, obtaining the second spatial position corresponding to each sample audio frame in the audio track signal, determining the maximum displacement corresponding to the audio track signal based on the second spatial position corresponding to each sample audio frame, and deleting the audio track signal if the maximum displacement is less than the displacement threshold.

[0010] Optionally, before training the trajectory generation model based on the training samples, the method includes: for each audio track signal included in the training samples, if a second spatial position is set for a portion of the sample audio frames included in the audio track signal, setting the second spatial position for other sample audio frames in the audio track signal excluding the portion of sample audio frames, wherein the second spatial position set for the other sample audio frames is the same as the second spatial position of the most recent portion of sample audio frames.

[0011] Secondly, an apparatus for processing audio track signals is provided, the apparatus comprising: The acquisition module is used to acquire multiple audio track signals included in the target song, wherein the multiple audio track signals include audio track signals corresponding to multiple sound objects included in the target song; The determining module is used to determine the first audio track signal to be processed among the plurality of audio track signals, and to determine at least one second audio track signal that is associated with the first audio track signal; The generation module is used to input the first audio track signal and the second audio track signal into the trajectory generation model. The trajectory generation model generates a spatial motion trajectory corresponding to the first audio track signal based on the correlation between the second audio track signal and the first audio track signal. The spatial motion trajectory includes the first spatial position of each audio frame included in the first audio track signal in the sound field. The first spatial position is used to characterize the change of the position of the first audio track signal in the sound field over time.

[0012] Optionally, the trajectory generation model includes a feature extraction unit and a trajectory generation unit. The generation module is configured to: input the first audio track signal and the second audio track signal to the feature extraction unit respectively, to obtain first feature data after the feature extraction unit performs a harmonic short-time Fourier transform on the first audio track signal, and second feature data after performing a harmonic short-time Fourier transform on the second audio track signal; input the first feature data and the second feature data to the trajectory generation unit, to obtain the spatial motion trajectory corresponding to the first audio track signal output by the trajectory generation unit based on the correlation between the first feature data and the second feature data.

[0013] Optionally, the trajectory generation unit includes multiple feature processing sub-units, a temporal convolutional layer, and a linear layer. The multiple feature processing sub-units are connected in series; the temporal convolutional layer is connected in parallel with the last feature processing sub-unit among the multiple feature processing sub-units; the output data of the temporal convolutional layer and the last feature processing sub-unit are the input data of the linear layer.

[0014] Optionally, the trajectory generation model further includes a bridging unit; the generation module is configured to: input the first feature data and the second feature data into the bridging unit to obtain third feature data after the bridging unit processes the first feature data and the second feature data; and input the third feature data into the trajectory generation unit.

[0015] Optionally, the device further includes a training module for: acquiring training samples, the training samples including a first sample song, a second sample song, and a third sample song, wherein the first sample song is a song in which the included original audio track signal has a spatial motion trajectory, the second sample song is a song in which a spatial motion trajectory is manually added to the included audio track signal, and the third sample song is a song in which a spatial motion trajectory is added to the included audio track signal according to a template of spatial motion trajectory; and training the trajectory generation model based on the training samples.

[0016] Optionally, the acquisition module is further configured to: for each audio track signal included in the training samples, acquire the second spatial position corresponding to each sample audio frame in the audio track signal, determine the maximum displacement corresponding to the audio track signal based on the second spatial position corresponding to each sample audio frame, and delete the audio track signal if the maximum displacement is less than the displacement threshold.

[0017] Optionally, the device further includes a filtering module for setting the second spatial position for other sample audio frames in the audio track signal excluding the partial sample audio frames, when the second spatial position is set for a partial sample audio frame included in the audio track signal for each audio track signal included in the training samples, wherein the second spatial position set for the other sample audio frames is the same as the second spatial position of the most recent partial sample audio frame.

[0018] Thirdly, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to perform the operations performed by the method for processing audio track signals as described in the first aspect above.

[0019] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the instruction being loaded and executed by a processor to perform the operations performed by the method for processing audio track signals as described in the first aspect above.

[0020] Fifthly, a computer program product is provided, the computer program product including program code, the program code being loaded and executed by a processor to perform the operations performed by the method for processing audio track signals as described in the first aspect above.

[0021] The beneficial effects of the technical solution provided in this application include at least the following: The technical solution provided in this application can acquire multiple audio track signals of a target song, and then split these signals into two parts: one part is a single audio signal for generating a spatial motion trajectory (i.e., the first audio signal), and the other part is the remaining reference audio signal (i.e., the second audio signal). In this way, the trajectory generation model can refer to the characteristics of the audio signals to specifically generate dynamic trajectories in 3D generation for each individual audio signal. This eliminates the need for manual settings by professionals and improves the efficiency of generating high-quality dynamic trajectories corresponding to audio signals. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the audio track signal in the three-dimensional sound field provided in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the effect of playing an audio track signal according to an embodiment of this application; Figure 3 This is a flowchart of a method for processing audio track signals provided in an embodiment of this application; Figure 4 This is a schematic diagram of a trajectory generation model provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a trajectory generation unit provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a trajectory generation model provided in an embodiment of this application; Figure 7 This is a flowchart of a method for processing audio track signals provided in an embodiment of this application; Figure 8This is a structural diagram of a device for processing audio track signals provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0025] 3D sound field is a technology that can create and reproduce a sound experience with a sense of height, depth, and immersion. It goes beyond traditional "stereo" (left and right channels) and "surround sound" (planar surround sound) and can simulate the way humans hear sound in the real world, allowing sound to appear precisely at any location in three-dimensional space, including above, below, in front, behind, and in any diagonal direction.

[0026] An audio track signal is an independent data stream used for recording, editing, and playing audio. To improve the playback quality of a song's audio, a corresponding audio track signal can be set for each sound source included in the song's audio, such as vocals, each instrument, and each sound effect. By recording the sound information of each sound source through separate audio tracks, more sound details can be captured, thereby improving the playback quality of the song.

[0027] In some embodiments, dynamic trajectories can be set for one or more audio tracks in a song within a three-dimensional sound field. These dynamic trajectories represent the changing positions of the audio track sources within the three-dimensional sound field as the song plays. Thus, when a user plays a song using a player that supports three-dimensional sound field playback (such as headphones or speakers), the user perceives the dynamic changes in the position of the same sound source (such as a human voice) in three-dimensional space. This allows the user to experience a dynamic auditory sensation in three-dimensional space. For example, it's as if a human voice is singing around the user.

[0028] The spatial motion trajectory of the audio track signal in the three-dimensional sound field can be achieved by setting the spatial position of each audio frame in the three-dimensional sound field. Figure 1 This is a schematic diagram of the audio track signal in a three-dimensional sound field provided in an embodiment of this application. For example... Figure 1 As shown, the audio track signal includes multiple audio frames. Each audio frame can be set to a spatial position in the three-dimensional sound field, such as the spatial position corresponding to video frame 1 being (X1, Y1, Z1). In this way, when playing the audio track signal, each audio frame can be played according to its corresponding spatial position, thus achieving the effect of dynamic playback of the audio track signal in the three-dimensional generation. Figure 2 This is a schematic diagram illustrating the effect of playing an audio track signal according to an embodiment of this application. Figure 2As shown, taking the audio track signal corresponding to human voice as an example, the audio track signal corresponding to human voice can be played around the user in a three-dimensional sound field to achieve a dynamic playback effect. Other instruments (such as guitar, piano, drums, etc.) can be played at fixed positions in the three-dimensional sound field, which can enrich the playback effect of the song.

[0029] Currently, high-quality spatial motion trajectories for audio tracks are set by professionals using specialized music editing software. For example, in a professional digital audio workstation, professionals can set spatial motion trajectories for the audio tracks in a song based on their experience. However, generating spatial motion trajectories for a large number of songs requires a huge investment of manpower, resulting in extremely high costs.

[0030] Figure 3 This is a flowchart illustrating a method for processing audio track signals according to an embodiment of this application. In this embodiment, an artificial intelligence model can be used to learn from the experience of professionals in setting spatial motion trajectories for audio track signals. By referencing other audio track signals in the song, spatial motion trajectories can be set for the audio track signals in the song that require spatial motion trajectories. In one possible implementation, this embodiment can be executed by a computer device, which can be a backend server for a music playback application. This server can be a virtual machine or a physical machine. Figure 3 As shown, the method includes: Step 301: Obtain multiple audio track signals included in the target song. The multiple audio track signals include the audio track signals corresponding to multiple sound objects contained in the target song.

[0031] The target song can be any song selected from the music library of a music player application. One or more audio tracks included in the target song need to generate spatial motion trajectories. The sound objects include the vocal elements of the target song, such as instruments, vocals, and sound effects.

[0032] In one example, if the original audio of the target song includes multiple audio tracks that have been segmented, then all the audio tracks included in the song audio can be obtained. In another example, if the audio of the target song has not been segmented, then the multiple audio tracks included in the song audio can be obtained by segmenting the audio into tracks. Optionally, the track segmentation of the song audio can be achieved using a trained artificial intelligence (AI) model.

[0033] Step 302: Determine the first audio track signal to be processed from among multiple audio track signals, and determine at least one second audio track signal that is related to the first audio track signal.

[0034] After acquiring multiple audio track signals from the target song, the audio track signal for the spatial motion trajectory to be generated, as specified by the technician, can be determined. For example, the technician can pre-specify the sound object corresponding to the audio track signal for the spatial motion trajectory to be generated; this sound object can be a piano, guitar, drums, bass, vocals, etc. After acquiring multiple audio track signals from the target song, the server can determine the audio track signal for the spatial motion trajectory to be generated, i.e., determine the first audio track signal, based on the sound object corresponding to each audio track signal.

[0035] After determining the first audio track signal, a second audio track signal that is related to the first audio track signal can be determined. This second audio track signal can be any other audio track signal in the target song besides the first audio track signal. Alternatively, the second audio track signal that is related to the first audio track signal can be pre-set by technicians, such as setting a correspondence based on the type of sound object. For example, in a correspondence, piano corresponds to guitar, drums, and bass; thus, if the first audio track signal is piano, the corresponding second audio track signal would be the audio track signal corresponding to guitar, drums, and bass.

[0036] It is understandable that the correlation between the first audio track signal and the second audio track signal refers to the relationship between the spatial position change of the first audio track signal in the sound field over time and the spatial position change of the first audio track signal in the sound field over time. Optionally, this correlation does not need to be recorded with specific data; it can be specifically reflected through the data of the first audio track signal and the data of the second audio track signal.

[0037] In cases where the target song's audio comprises multiple audio tracks, the metadata for each track can include a corresponding sound object. Furthermore, for the multiple audio tracks of the target song obtained through an AI model, the AI ​​model can further identify the sound object corresponding to each output audio track.

[0038] It is understood that a target song may include one or more audio track signals for which a spatial motion trajectory is to be generated, and the audio track signals for which spatial motion trajectories need to be generated can be specified by a technician. When a spatial motion trajectory needs to be generated for one audio track signal in the target song, it can be achieved by executing steps 301 to 303 once. When multiple audio track signals in the target song need to have spatial motion trajectories generated, it can be achieved by executing steps 301 to 303 multiple times. In each step 301 to 303, the audio track signal for which a spatial motion trajectory needs to be generated can be selected as the first audio track signal from among the multiple audio track signals.

[0039] Step 303: Input the first audio track signal and the second audio track signal into the trajectory generation model. The trajectory generation model generates the spatial motion trajectory corresponding to the first audio track signal according to the correlation between the second audio track signal and the first audio track signal. The second audio track signal is the other audio track signals besides the first audio track signal. The spatial motion trajectory corresponding to the first audio track signal includes the first spatial position of each audio frame included in the first audio track signal in the sound field. The first spatial position is used to characterize the change of the position of the first audio track signal in the sound field over time.

[0040] The trajectory generation model is an AI model proposed in this application to generate a spatial motion trajectory for a first audio track signal. The trajectory generation model can include two inputs: one is the first audio track signal to be generated as the spatial motion trajectory, and the other is the remaining second audio track signal in the target song.

[0041] After determining the first audio track signal of the target song, the remaining audio track signals can be used as the second audio track signal. Then, the first and second audio track signals can be input into a trajectory generation model. The trajectory generation model can refer to the characteristic data of the second audio track signal and combine it with the feature data of the first audio track signal to output the spatial motion trajectory of the first audio track signal. For example, the input is as follows... Figure 1 The image shows the first spatial position of each audio frame in the three-dimensional sound field.

[0042] Alternatively, the trajectory generation model can also output the audio track type corresponding to the first audio track signal. In the server, a correspondence between audio track type and spatial motion trajectory can be set. Then, the spatial motion trajectory corresponding to the first audio track signal can be determined by the audio track type corresponding to the first audio track signal and the correspondence.

[0043] As can be seen, in this embodiment, the spatial motion trajectory of the first audio track signal of the target song can be obtained based on the trajectory generation model. The input to the trajectory generation model only includes the audio track signal of the target song, thus eliminating the need to refer to other information or manually set dynamic trajectories by professionals. Therefore, this embodiment can improve the efficiency of generating the spatial motion trajectory of the audio track signal.

[0044] In practice, the background server of the music application can sequentially execute the method for processing audio track signals provided in this application embodiment for songs in the music library that have not yet generated dynamic tracks within a set time period, thereby efficiently and cost-effectively generating dynamic tracks for the songs included in the music library and improving the playback effect of each song in the music library.

[0045] Figure 4 This is a schematic diagram of a trajectory generation model provided in an embodiment of this application. For example... Figure 4As shown, the trajectory generation model includes a feature extraction unit and a trajectory generation unit. The feature extraction unit extracts features from the audio track signal, while the trajectory generation unit predicts and outputs the dynamic trajectory. The trajectory generation model's processing of the audio track signal to generate a spatial motion trajectory includes: Step 401: Input the first audio track signal and the second audio track signal into the feature extraction unit respectively to obtain the first feature data after the feature extraction unit performs harmonic short-time Fourier transform on the first audio track signal and the second feature data after the feature extraction unit performs harmonic short-time Fourier transform on the second audio track signal.

[0046] In implementation, the feature extraction unit can extract features from each audio track signal included in the first and second audio track signals respectively. For the first audio track signal, the feature extraction unit can perform a harmonic short-time Fourier transform on the first audio track signal to obtain the first feature data. For the second audio track signal, the feature extraction unit can perform a harmonic short-time Fourier transform on the second audio track signal to obtain the second feature data.

[0047] In one example, taking the feature extraction of the first audio track signal as an example, the first audio signal undergoes a harmonic short-time Fourier transform, and the resulting data is the first spectrum data. In this embodiment, the first spectrum data can also be filtered out using a set harmonic filter to remove frequencies imperceptible to the human ear within a specified frequency band (such as the high-frequency band, mid-frequency band, and / or low-frequency band). The second spectrum data of the second audio track signal can be processed based on the same harmonic filter. In this way, using the first and second spectrum data after harmonic filtering as feature data extracted by the feature extraction unit is more consistent with the characteristics perceived by the human ear.

[0048] In addition, the feature extraction unit may also include a convolutional processing unit, such as a convolutional neural network (CNN), to further extract features from the filtered spectrum data (including the first spectrum data and the second spectrum data) output by the harmonic filter, so as to standardize the data format of the spectrum data.

[0049] Step 402: Input the first feature data and the second feature data into the trajectory generation unit to obtain the spatial motion trajectory corresponding to the first audio track signal output by the trajectory generation unit according to the correlation between the first feature data and the second feature data.

[0050] After obtaining the first and second feature data extracted by the feature extraction unit, the first and second feature data can be input into the trajectory generation unit. The trajectory generation unit then outputs the spatial motion trajectory corresponding to the first audio track signal based on the first and second feature data. It can be understood that the correlation between the first and second feature data is also the correlation between the first and second audio track signals.

[0051] Figure 5 This is a schematic diagram of the structure of a trajectory generation unit provided in an embodiment of this application, as shown below. Figure 5 As shown, the trajectory generation unit comprises multiple feature processing sub-units, a temporal convolutional layer, and a linear layer. The multiple feature processing sub-units are connected sequentially. The temporal convolutional layer is connected in parallel to the last feature processing sub-unit. The output data of the temporal convolutional layer and the last feature processing sub-unit serves as the input data for the linear layer.

[0052] The first feature processing subunit in a series of feature processing subunits can extract the time-domain and frequency-domain features of the input first and second feature data. Subunits other than the first feature processing subunit take the output of the previous feature processing subunit as input, and similarly extract time-domain and frequency-domain features from the input data. The output time-domain and frequency-domain features are then input into the next feature processing subunit, and so on, until the last feature processing subunit.

[0053] In one example, a temporal convolutional layer can be set up parallel to the last feature processing unit, with the input of this temporal convolutional layer being the output of the last feature processing unit. The outputs of the temporal convolutional layer and the last feature processing unit can be added together, and the resulting activation data can be used as the input to a linear layer. Finally, the linear layer can be configured to process the first audio track signal output and... Figure 1 Similar data that records dynamic trajectories.

[0054] In this embodiment, a temporal convolutional layer can be added to assist the linear layer in completing the output, resulting in a more accurate spatial motion trajectory output by the generated linear layer. In one example, the trajectory generation unit can be implemented based on a Spectral-Temporal Transformer in Transformer (SpecTNT) model. The feature processing subunit can be a Spec-TNT block. The temporal convolutional layer can be a Temporal Convolutional Neural Network (TCN).

[0055] In one possible implementation, the trajectory generation model also includes a bridging unit, which can be used to convert the feature data extracted by the feature extraction unit into a unified format and input it to the trajectory generation unit. Figure 6 This is a schematic diagram of the structure of a trajectory generation model including a bridging unit provided in an embodiment of this application.

[0056] like Figure 6 As shown, the feature extraction unit may include multiple layers of output during the feature extraction process of feature data (including first feature data and second feature data). Figure 6 As shown, the output (Spec) after performing a harmonic short-time Fourier transform on the audio signal (Wav), the result after harmonic filtering on the output of the harmonic short-time Fourier transform (Harmonic tensor), and the result after neural network processing on the harmonic filtered result (prediction), etc. The bridging unit can take the output of any layer as input and output it to the trajectory generation unit below. In one example, the bridging module can consist of a convolutional network with residual connections and a max-pooling layer. When the output of any layer of the feature extraction unit is used as input to the bridging unit, the output of that layer becomes the first feature data and the second feature data. After processing by the bridging unit, the first and second feature data can be normalized into the third feature data input to the trajectory generation unit. Thus, in practical processing, due to the existence of the bridging unit, technicians can flexibly set the output of the feature extraction unit to achieve the best effect in outputting the spatial motion trajectory of the final trajectory generation model.

[0057] The training process of the trajectory generation model is described below: Figure 7 This is a schematic diagram illustrating a method for training a trajectory generation model provided in an embodiment of this application. Figure 7 As shown, the method includes: Step 701: Obtain training samples. The training samples include a first sample song, a second sample song, and a third sample song. The first sample song is a song with a spatial motion trajectory in the original audio track signal. The second sample song is a song with a spatial motion trajectory manually added to the included audio track signal. The third sample song is a song with a spatial motion trajectory added to the included audio track signal based on a template of spatial motion trajectory.

[0058] In practical scenarios, since there are relatively few songs in the music library that correspond to multiple audio track signals and whose audio track signals have spatial motion trajectories (i.e., the first sample songs), this application also includes a number of sample songs for which spatial motion trajectories have been added by professionals (i.e., the second sample songs). Furthermore, considering that manually generated sample songs also require a certain amount of manpower, this application also includes a number of sample songs whose corresponding spatial motion trajectories are generated based on trajectory templates (i.e., the third sample songs).

[0059] It's understandable that a trajectory template refers to the pre-defined positional information of a spatial motion trajectory within a three-dimensional sound field. The spatial motion trajectory generated based on the trajectory template refers to setting the spatial motion trajectory of a specified audio track signal in a sample song according to the positional information included in the trajectory template. In implementation, multiple relatively matching trajectory templates can be set for audio track signals of different source types. For example, the spatial motion trajectory of the trajectory template corresponding to a vocal track signal can be surround, the spatial motion trajectory of the trajectory template corresponding to a piano track signal can be surround oscillating on the left, and the spatial motion trajectory of the trajectory template corresponding to a guitar track signal can be surround oscillating on the right, etc. In this way, the third sample song generated based on the trajectory template can also have a certain representativeness, which can improve the training effect of the trajectory generation model.

[0060] In one feasible approach, for each audio track signal included in the training samples, the second spatial position corresponding to each sample audio frame in the audio track signal is obtained, and the maximum displacement corresponding to the audio track signal is determined based on the second spatial position corresponding to each sample audio frame. If the maximum displacement is less than the displacement threshold, the audio track signal is deleted.

[0061] In practice, the spatial motion trajectories of the audio tracks corresponding to the sample songs in the training samples may be manually created by professionals using music editing software. Therefore, there may be some errors in the spatial position of the spatial motion trajectories. For example, some motion signals without set spatial motion trajectories may have their positions in three-dimensional space changed due to improper operation. Therefore, in this application, the maximum displacement corresponding to the spatial position of the audio track signal in the three-dimensional sound field can be calculated, and audio track signals with a maximum displacement less than a set displacement threshold can be filtered out, thereby avoiding interference with the training of the trajectory generation model.

[0062] In one example, for each audio track signal, the maximum and minimum values ​​of its second spatial position during generation can be determined in the x, y, and z directions. The sum of the differences between the maximum and minimum values ​​in the x, y, and z directions is determined as the maximum displacement. Then, by comparing the maximum displacement with a displacement threshold, the audio track signal can be filtered out. For example, the audio track signal can be filtered out based on the following formula.

[0063] x_max-x_min+y_max-y_min+z_max-z_min≤0.1 Where x_max is the maximum value in the X direction, x_min is the minimum value in the X direction, y_max is the maximum value in the Y direction, y_min is the minimum value in the Y direction, z_max is the maximum value in the Z direction, and z_min is the minimum value in the Z direction. 0.1 is the set displacement threshold.

[0064] In one feasible approach, for each audio track signal included in the training samples, if a second spatial position is set for a portion of the sample audio frames included in the audio track signal, a second spatial position is set for the other sample audio frames in the audio track signal excluding the portion of the sample audio frames, wherein the second spatial position set for the other sample audio frames is the same as the second spatial position of the most recent portion of the sample audio frames.

[0065] Since most audio signals do not play for the entire duration of a song, their corresponding spatial motion trajectory data is also discrete along with the playback time. This means that some silent audio frames may not have a designated second spatial position within the three-dimensional sound field. Furthermore, due to limitations in the spatial motion trajectory data format, extracting the positional information from some audio frames that do not undergo positional changes may not yield their positional information, meaning the second spatial position of these audio frames cannot be obtained.

[0066] Therefore, to avoid increasing the difficulty of training the trajectory prediction model due to discrete spatial motion trajectories, in this embodiment, the discrete spatial motion trajectories can be completed. The corresponding processing could be that for audio frames for which a second spatial position has not been obtained (excluding some sample audio frames), the corresponding spatial position can be set to the spatial position corresponding to the nearest audio frame with a spatial position (the nearest sample audio frame). This ensures the continuity of the spatial motion trajectory, thereby reducing the training difficulty of the trajectory generation model.

[0067] Step 702: Train the trajectory generation model based on the training samples.

[0068] After obtaining training samples, the trajectory generation model can be trained based on the training samples, for example, for... Figure 5 The trajectory generation model shown is trained until it converges. Thus, in this embodiment, the trained trajectory generation model can learn from the experience of professionals in setting dynamic sound effects, generating dynamic sound effects for the audio track signals included in the song, thereby improving the efficiency of generating high-quality dynamic sound effects.

[0069] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0070] Based on the same inventive concept, embodiments of this application provide an apparatus for processing audio track signals, which can be configured on the computer device described above for processing audio track signals. Figure 8 This is a schematic diagram of a device structure for processing audio track signals provided in an embodiment of this application, as shown below. Figure 8 As shown, the device includes: The acquisition module 810 is used to acquire multiple audio track signals included in the target song, wherein the multiple audio track signals include audio track signals corresponding to multiple sound objects included in the target song; The determining module 820 is used to determine the first audio track signal to be processed among the plurality of audio track signals, and to determine at least one second audio track signal that is associated with the first audio track signal; The generation module 830 is used to input the first audio track signal and the second audio track signal into the trajectory generation model. The trajectory generation model generates a spatial motion trajectory corresponding to the first audio track signal based on the correlation between the second audio track signal and the first audio track signal. The spatial motion trajectory includes the first spatial position of each audio frame included in the first audio track signal in the sound field. The first spatial position is used to characterize the change of the position of the first audio track signal in the sound field over time.

[0071] Optionally, the trajectory generation model includes a feature extraction unit and a trajectory generation unit. The generation module 830 is configured to: input the first audio track signal and the second audio track signal to the feature extraction unit respectively, to obtain first feature data after the feature extraction unit performs harmonic short-time Fourier transform on the first audio track signal and second feature data after performing harmonic short-time Fourier transform on the second audio track signal; input the first feature data and the second feature data to the trajectory generation unit, to obtain the spatial motion trajectory corresponding to the first audio track signal output by the trajectory generation unit based on the first feature data and the second feature data.

[0072] Optionally, the trajectory generation unit includes multiple feature processing sub-units, a temporal convolutional layer, and a linear layer. The multiple feature processing sub-units are connected in series; the temporal convolutional layer is connected in parallel with the last feature processing sub-unit among the multiple feature processing sub-units; the output data of the temporal convolutional layer and the last feature processing sub-unit are the input data of the linear layer.

[0073] Optionally, the trajectory generation model further includes a bridging unit; the generation module 830 is configured to: input the first feature data and the second feature data into the bridging unit to obtain third feature data after the bridging unit processes the first feature data and the second feature data; and input the third feature data into the trajectory generation unit.

[0074] Optionally, the device further includes a training module for: acquiring training samples, the training samples including a first sample song, a second sample song, and a third sample song, wherein the first sample song is a song in which the included original audio track signal has a spatial motion trajectory, the second sample song is a song in which a spatial motion trajectory is manually added to the included audio track signal, and the third sample song is a song in which a spatial motion trajectory is added to the included audio track signal according to a template of spatial motion trajectory; and training the trajectory generation model based on the training samples.

[0075] Optionally, the acquisition module 810 is further configured to: for each audio track signal included in the training samples, acquire the second spatial position corresponding to each sample audio frame in the audio track signal, determine the maximum displacement corresponding to the audio track signal based on the second spatial position corresponding to each sample audio frame, and delete the audio track signal if the maximum displacement is less than the displacement threshold.

[0076] Optionally, the device further includes a filtering module for setting the second spatial position for other sample audio frames in the audio track signal excluding the partial sample audio frames, when the second spatial position is set for a partial sample audio frame included in the audio track signal for each audio track signal included in the training samples, wherein the second spatial position set for the other sample audio frames is the same as the second spatial position of the most recent partial sample audio frame.

[0077] It should be noted that the above embodiments of the device for processing audio track signals are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments for processing audio track signals provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0078] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device can be the server described in the above embodiments. The server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 901 and one or more memories 902. The memory 902 stores at least one instruction, which is loaded and executed by the processor 901 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0079] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to perform the method for processing facial images in the above embodiments. This computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (CompactDisc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0080] In an exemplary embodiment, a computer program product is also provided, which includes program code that can be executed by a processor in a computer device to perform the method for processing audio track signals described above.

[0081] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the songs involved in this application were all obtained with full authorization.

[0082] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0083] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0084] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing audio track signals, characterized in that, The method includes: Acquire multiple audio track signals included in the target song, wherein the multiple audio track signals include audio track signals corresponding to multiple sound objects included in the target song; Determine the first audio track signal to be processed from among the plurality of audio track signals, and determine at least one second audio track signal that is associated with the first audio track signal; The first audio track signal and the second audio track signal are input into the trajectory generation model. The trajectory generation model generates a spatial motion trajectory corresponding to the first audio track signal based on the correlation between the second audio track signal and the first audio track signal. The spatial motion trajectory includes the first spatial position of each audio frame included in the first audio track signal in the sound field. The first spatial position is used to characterize the change of the position of the first audio track signal in the sound field over time.

2. The method according to claim 1, characterized in that, The trajectory generation model includes a feature extraction unit and a trajectory generation unit; The step of inputting the first audio track signal and the second audio track signal into the trajectory generation model, and having the trajectory generation model generate the spatial motion trajectory corresponding to the first audio track signal based on the correlation between the second audio track signal and the first audio track signal, includes: The first audio track signal and the second audio track signal are respectively input to the feature extraction unit to obtain the first feature data after the feature extraction unit performs harmonic short-time Fourier transform on the first audio track signal and the second feature data after the feature extraction unit performs harmonic short-time Fourier transform on the second audio track signal. The first feature data and the second feature data are input into the trajectory generation unit to obtain the spatial motion trajectory corresponding to the first audio track signal output by the trajectory generation unit based on the correlation between the first feature data and the second feature data.

3. The method according to claim 2, characterized in that, The trajectory generation unit includes multiple feature processing sub-units, a temporal convolutional layer, and a linear layer; The multiple feature processing subunits are connected in series; The temporal convolutional layer is connected in parallel to the last feature processing subunit among the plurality of feature processing subunits; The output data of the temporal convolutional layer and the last feature processing subunit are the input data of the linear layer.

4. The method according to claim 2, characterized in that, The trajectory generation model also includes a bridging unit; The step of inputting the first feature data and the second feature data into the trajectory generation unit includes: The first feature data and the second feature data are input to the bridging unit to obtain the third feature data after the bridging unit has processed the first feature data and the second feature data; The third feature data is input into the trajectory generation unit.

5. The method according to claim 1, characterized in that, Before inputting the first audio track signal and the second audio track signal into the trajectory generation model, the following steps are included: Obtain training samples, which include a first sample song, a second sample song, and a third sample song. The first sample song is a song in which the original audio track signal has a spatial motion trajectory. The second sample song is a song in which a spatial motion trajectory is manually added to the audio track signal. The third sample song is a song in which a spatial motion trajectory is added to the audio track signal according to a template of spatial motion trajectory. The trajectory generation model is trained based on the training samples.

6. The method according to claim 5, characterized in that, Before training the trajectory generation model based on the training samples, the following steps are included: For each audio track signal included in the training samples, the second spatial position corresponding to each sample audio frame in the audio track signal is obtained. Based on the second spatial position corresponding to each sample audio frame, the maximum displacement corresponding to the audio track signal is determined. If the maximum displacement is less than the displacement threshold, the audio track signal is deleted.

7. The method according to claim 5, characterized in that, Before training the trajectory generation model based on the training samples, the following steps are included: For each audio track signal included in the training samples, if a portion of the sample audio frames included in the audio track signal is provided with the second spatial position, the second spatial position is provided for the other sample audio frames in the audio track signal excluding the portion of sample audio frames, wherein the second spatial position provided for the other sample audio frames is the same as the second spatial position of the most recent portion of sample audio frames.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction that is loaded and executed by the processor to perform the operation performed by the method for processing audio track signals as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to perform the operations performed by the method for processing audio track signals as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes computer program code, which, when executed by a computer device, performs the operations performed by the method for processing audio track signals as described in any one of claims 1 to 7.

Images