A method for identifying audio match cut candidates and performing audio match cutting

Abstract

An aspect of the present disclosure relates to a method for creating an audio match cut between a first audio clip and a second audio clip, each audio clip comprising a plurality of audio content samples. The method comprises obtaining a transition point for crossfading from the first audio clip into the second audio clip and determining a plurality of similarity metrics. Each similarity metric indicating a similarity between the audio content of a sample of the first audio clip and the audio content of a sample of the second audio clip, wherein the plurality of similarity metrics are determined in a transition context window. The method further comprises determining a variance of the plurality of similarity metrics, determining a crossfading length based on the variance and generating a match cut audio clip by crossfading between the first and second audio clip at the transition point.

Description

A METHOD FOR IDENTIFYING AUDIO MATCH CUT CANDIDATES AND PERFORMING AUDIO MATCH CUTTING CROSS-REFERNECE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Application No.63 / 632,972 filed April 11, 2024 and U.S. Provisional Application No.63 / 579,906 filed August 31, 2023, each of which is hereby incorporate by reference in their entireties. TECHNICAL FIELD

[0002] The present invention relates to a method and a system for generating audio match cut clips. BACKGROUND

[0003] In the field of video editing, the “cut” is a foundational editing technique that is used to transition from one scene or shot to the next. The precise use of cuts often crafts the story being portrayed, whether it controls pacing, highlights emotions, or connects disparate scenes into a cohesive story. There are many variations of cuts that are used across the film industry, including smash cuts, reaction cuts, J-cuts, L-cuts, and others. Commonly, cuts are considered for visual content, but the same or similar cuts may also be used for audio content.

[0004] A specific type of cut is the “match cut” where a pair of video clips that have a similar composition transition fluidly from one to another. A match cut is a transition between a pair of video clips that uses similar framing, composition, or action to fluidly bring the viewer from one clip (depicting the first scene) to the next clip (depicting the second scene). Match cuts often match visuals across each scene, either through similar objects and their placement, colors, or camera movements.

[0005] However, there are also match cuts that involve the fluid transition of audio, where sounds from different audio clips merge into one indistinguishable transition between two audio clips. For example, two clips depicting different scenes can be match cut such that sound of the two scenes transition seamlessly between each other. These audio match cuts (also referred to as “sound bridges”) either blend together sounds or carry similar sound across scenes, often from different sound sources, to create a fluid audio transition between potentially very different visual scenes.

[0006] That is, using audio match cuts for video or audio content editors can create acoustically seamless transitions between different audio or visual scenes. For example, a firstvideo clip shows a mixer and the audio track includes the sound of the mixer and a second video clip shows a motorcycle and the audio track includes the sound of the motorcycle. An audio match cut can be created between the two (visually very different scenes) due to the audio content of the scenes being similar.

[0007] To identify and use video or audio match cuts, the process typically involves often involves a team of expert editors that manual search across a large collection of recorded content to find strong candidates to transition to. SUMMARY

[0008] While audio match cuts are a powerful tool available to professional editors that edit video and audio content, audio match cuts are notoriously difficult to identify since the video editors typically work with large amount of video and audio content. Since identification of audio match cuts is challenging and time consuming, professional editors may turn to other types of cutting techniques leading to audio match cuts being an underutilized cutting technique. For the same reasons, audio match cuts are very seldom seen in user generated content which often is unedited or only undergoes a minimal amount of editing.

[0009] It is a purpose of the present disclosure to present a method for generating an audio match cut clip which overcomes at least some of the drawbacks of the previous methods. The method of the present disclosure has the benefit of being completely automatic and capable of generating audio match cut clips without human interaction, allowing this type of cutting technique to be used more efficiently.

[0010] According to a first aspect of the present invention, a method for creating an audio match cut between a first audio clip and a second audio clip, each audio clip comprising a plurality of audio content samples. The method comprises obtaining a transition point for crossfading from the first audio clip into the second audio clip and determining a plurality of similarity metrics. Each similarity metric indicating a similarity between the audio content of a sample of the first audio clip and the audio content of a sample of the second audio clip, wherein the plurality of similarity metrics are determined at least for samples of the first and second audio clip in a transition context window associated with the transition point. The method further comprises determining a variance of the similarity metrics of a plurality of samples, determining a crossfading length based on the variance and generating a match cut audio clip by crossfading between the first audio clip and the second audio clip at the transition point using the crossfading length.

[0011] By determining the crossfading length adaptively, based on a variance of the similarity metrics, high quality audio match cuts between two audio clips can be formedautomatically. Since the variance of the similarity metrics can be determined analytically a high quality audio match may be generated without human interaction and without a professional editor going through the labor intensive process of forming match audio cuts manually.

[0012] In some implementations, the crossfading length increases with a decreasing variance and the crossfading length decreases with an increasing variance. For example, the crossfading length is inversely proportional to the variance.

[0013] In some implementations, the transition point indicates a transition sample in the first audio clip and a transition sample in the second audio clip.

[0014] The transition point hereby indicates a point in time (a sample) in each of the first and second audio clip where the transition from the first audio clip to the second is to be performed by means of a crossfade to form the match cut audio clip. The transition point may define the timing of the crossfade, and e.g. the transition point indicates the temporal start of the crossfade, the temporal center point of the crossfade or the temporal end of the crossfade. The transition point may be a predetermined transition point or the method may further comprise determining the transition point, as described below.

[0015] In some implementations, the transition context window comprises one or more samples temporally surrounding the transition sample of the first audio clip and one or more samples temporally surrounding the transition sample of the second audio clip.

[0016] By determining the plurality of similarity metrics for sample pairs falling inside a transition context window covering samples in the vicinity of the transition point the variance will be based on samples in the vicinity of the transition point. Since the variance of the plurality of similarity metrics temporally closer to the transition point are most important for determining an appropriate crossfading length the method can be applied to arbitrarily long audio clips.

[0017] Alternatively, the transition context window covers all samples of at least one of the first and second clip.

[0018] In some implementations, the method further comprises obtaining the first audio clip, obtaining a set of audio clips comprising at least two audio clips, and selecting the second audio clip as the audio clip of the set of audio clips that has the most similar audio content to the first audio clip.

[0019] Accordingly, the method may be expanded to further identify the second audio clip among a set of second audio clips, further automating the process of generating the match cut audio clip. The method is therefore capable of automatically identifying a second audio clip which is similar to the first audio clip and then generate a high quality match cut audio clip with the first audio clip and the similar second audio clip. Hereby, it is not necessary to perform any manual clip selection of two clips are to be processed with the method since it suffices to providea first audio clip and a set of second audio clip whereby the method involves automatically identifying a similar second audio clip. The first audio clip may be referred to as a query audio clip whereby the set of second audio clips are a set of candidate audio clips. The second audio clip which is identified as being most similar to the first audio clip, may be referred to as a matching candidate audio clip.

[0020] In some implementations, the method comprises extracting feature data for each audio clip in the set of audio clips, extracting feature data for the first audio clip and wherein selecting the second audio clip comprises selecting the second audio clip as the audio clip of the set of audio clips associated with feature data that is most similar to the feature data of the first audio clip.

[0021] By extracting feature data from each audio clip and identifying similar audio clips based on similarity of the feature data the process of finding similar audio clips can be made more efficient and accurate. The feature data may comprise one or more of Mel-frequency cepstral coefficients, MFCC, representing each audio clip, Mel-Spectrogram coefficients representing each clip, and a feature vector extracted by a neural network representing each clip. To determine a degree of similarity between different instances of feature data various methods could be used, such as a maximum inner product search (MIPS) process, cosine similarity measures or Euclidean distance measures.

[0022] In some implementations, the feature data comprises the feature vector extracted by a neural network, the neural network being the encoder part of an autoencoder neural network. The method further comprises extracting the feature vector for each audio clip in the set of audio clips using the encoder neural network and extracting the feature vector for the first audio clip using the encoder neural network. For example, the neural network comprises an ImageBind encoder neural network and / or a Contrastive Language-Audio Pretraining, CLAP, encoder neural network. These types of neural networks are capable of extracting feature vectors of reduced dimensionality (compared to the input data) that capture the perceptual properties of each clip, allowing for accurate audio clip matching.

[0023] In some implementations, the feature vector is obtained by processing the output of the neural network encoder with at least one linear prediction layer.

[0024] The linear projection layer modifies the feature vectors into modified feature vectors even better suited for identification of similar audio clips. The linear prediction layer may e.g. a trained neural network layer which has been trained with a contrastive learning loss function.

[0025] In some implementations, the method comprises determining a plurality of second similarity metrics, each second similarity metric indicating a similarity between the audiocontent of a sample of the first audio clip and the audio content of a sample of the second audio clip, wherein the plurality of second similarity metrics are determined at least for a subset of the samples of the first audio clip and at least for a subset of the samples of the second audio clip. The method further comprises identifying a sample pair associated with a second similarity metric indicating a highest degree of similarity among the plurality of second similarity metrics, the sample pair comprising a sample of the first audio clip and a sample of the second audio clip and determining the transition point based on the identified sample pair.

[0026] That is, the transition point can be dynamically determined by analyzing the plurality of second similarity metrics. Accordingly, both the point in time when to perform a crossfading transition between two clips, and the crossfading length for the transition, may be determined adaptively so as to automatically generate high quality audio match cuts.

[0027] Each similarity metric indicates a degree of similarity between two samples are (one sample from each audio clip) whereby the most similar pair of samples can be identified by finding the second similarity metric, among the plurality of second similarity metrics, which indicates a highest degree of similarity (e.g. a highest similarity metric). Each second similarity metric may be a distance metric (e.g. Euclidean distance) whereby a low second similarity metric value indicates similar samples (with smaller Euclidean distance) and a high similarity metric value indicates less similar samples (with larger Euclidean distance). Alternatively, each second similarity metric is e.g. an inner product whereby a low second similarity metric value indicates less similar samples and a high similarity metric value indicates more similar samples.

[0028] In some implementations, each second similarity metric is determined as a dot product between a sample of the first audio clip and a sample of the second audio clip. For example, each audio clip of the first and second audio clips is represented in a time-frequency domain with each sample of the first and second audio clips comprising at least one spectral coefficient. Determining each second similarity metric of the plurality of second similarity metrics may then comprise determining a dot product between the at least one spectral coefficient a sample of the first audio clip and the at least one spectral coefficient a sample of the second audio clip.

[0029] In some implementations, dot product for determining each second similarity metric is non-normalized. In this way, the transition point will be biased towards being temporally located where the audio content has more energy, which results in more high quality audio match cuts.

[0030] The plurality of (first) similarity metrics may also be determined as a dot product, such as a normalized dot product. In some implementations, each audio clip of the first and second audio clips is represented in a time-frequency domain with each sample of the first andsecond audio clips comprising at least one spectral coefficient. Wherein determining the plurality similarity metrics comprises calculating a normalized dot product between the at least one spectral coefficient of a sample in the first audio clip and the at least one spectral coefficient of a sample in the second audio clip in the transition context window.

[0031] By using a normalized dot product (e.g. cosine similarity) to determine each of the first similarity metrics, the first similarity metrics will be bounded on a defined range, which facilitates calculation of a more accurate variance. Hereby, determining the variance may comprise determining the variance of the plurality of normalized dot products corresponding to the plurality of first similarity metrics.

[0032] Accordingly, the first and second similarity metrics may be the different, e.g. the first similarity metric is a normalized dot product, and the second similarity metric is a non- normalized dot product. Alternatively, however, it is envisaged that the first and second similarity metrics may be the same, e.g. both being normalized or non-normalized.

[0033] According to a second aspect of the present invention there is provided an audio processing system comprising a processor and memory, wherein the processor is configured to perform the method according to the first aspect.

[0034] According to a third aspect of the present invention there is provided a non- transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform the method according to the first aspect.

[0035] According to a fourth aspect of the present invention there is provided a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the first aspect.

[0036] According to a fifth aspect of the present invention there is provided a method for training a neural network. The method comprises obtaining a number ^^^^ ≥ 2 of audio files. Each audio file comprising a sequence of ^^^^ ≥ 4 consecutive segments, generating, with a feature extraction neural network, a feature vector for each audio segment in each audio file and determining, for each audio file, an index for splitting the ^^^^ consecutive segments into a first consecutive segment sequence and a second consecutive segment sequence. Wherein the first segment sequence comprises at least one leading segment and a last segment, and the second segment sequence comprises a first segment and at least one trailing segment, wherein the last segment and first segment are subsequent segments in the audio file. Wherein the method further comprises inputting the feature vector of each segment of each audio file to the neural network; and training the neural network to generate a modified feature vector in a feature space for each feature vector of each audio file. Wherein the training comprises using a contrastive learning loss function such that a distance in the feature space between modified feature vectors associatedwith the first and last segment of each audio file is smaller than a distance in the feature space between modified feature vectors associated with a leading segment and a trailing segment of any of the ^^^^ audio files.

[0037] The trained neural network may be used to transform the feature vectors of a feature extraction neural network into a modified feature space. The modified feature space learned by the neural network during training with the contrastive learning loss function is better suited for finding matching audio clips.

[0038] In some implementations, the method of the fifth aspect of the present invention further comprises randomly determining, for each audio file, the index for splitting the ^^^^ consecutive audio segments into the first consecutive audio segment sequence and the second consecutive audio segment sequence.

[0039] The present invention according to the second, third and fourth aspect features the same or equivalent benefits as the present invention according to the first aspect. Any functions described in relation to a method, may have corresponding features in an audio processing system and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention will be described in more detail with reference to the appended drawings, showing currently preferred embodiments of the invention.

[0041] Figure 1 is a block diagram illustrating a system for automatically generating match cut audio clips, according to some implementations.

[0042] Figure 2 is a block diagram illustrating the details of an audio clip matching module according to some implementations.

[0043] Figure 3 is a block diagram illustrating the details of a transition parameter extractor module, according to some implementations.

[0044] Figure 4 is a graph schematically illustrating the maximum inner product search according to some implementations.

[0045] Figure 5 is a block diagram illustrating how the audio match cut generator generates a match cut audio clip in one example.

[0046] Figure 6 is a graph schematically illustrating how a match cut audio clip is generated by the audio match cut generator using a crossfade profile according to some implementations.

[0047] Figure 7A-B are flowcharts depicting a method for identifying a matched candidate audio clip and for calculating the crossfading length according to some implementations.

[0048] Figure 8A-C are block diagrams illustrating how a contrastive learning loss function is used to train a projection neural network according to some implementations.

[0049] Figure 9 is a flowchart describing a method for training the neural network projection layer according to some implementations.

[0050] Figure 10 illustrates a schematic block diagram of an example device or architecture that may be used to implement various aspects of the present invention. DETAILED DESCRIPTION

[0051] FIG.1 is a block diagram illustrating a system 10 for automatically generating one or more audio match cuts. The system 10 comprises a plurality of modules including a clip pair matcher module 1 and a transition parameter extractor module 2. The clip pair matcher module 1 and the transition parameter extractor module 2 extract information that allow an audio match cut generator module 3 to generate a high quality match cut audio clip ^^^^^^^^. While the modules 1, 2, 3 of the system 10 are depicted as blocks in FIG.1 it is understood that each module 1, 2, 3 may be implemented as hardware, firmware or software.

[0052] The clip pair matcher module 1 compares a query audio clip ^^^^^^^^to a set of ^^^^ ≥ 2 candidate audio clips ^^^^1, ^^^^2, …, ^^^^^^^^to identify one or more matching candidate audio clips ^^^^^^^^ ^^^^among the ^^^^ candidate audio clips ^^^^1, ^^^^2, …, ^^^^^^^^that are perceptually similar to the query clip ^^^^^^^^. The clip pair matcher module 1 may identify the one or more matching candidate audio clips ^^^^^^^^ ^^^^as a best matching candidate audio clip ^^^^^^^^ ^^^^out of the set of candidate audio clips ^^^^1, ^^^^2, …, ^^^^^^^^that is most similar to the query clip ^^^^^^^^. Alternatively, the clip pair matcher module 1 may identify a plurality of matching candidate audio clips ^^^^^^^^1, ^^^^^^^^2, … such as a predetermined number ^^^^ of most similar candidate audio clips ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^.

[0053] Each audio clip comprises a duration of audio content. The audio content may e.g. comprise speech content or music content. However, in the context of this disclosure the audio content is not limited to one or more specific categories of audio (like speech or music) but could be any type of audio content. That is, the audio content may be referred to as general audio content since the audio processing techniques described herein can be applied to any type of audio content. For example, the audio content comprises at least one of speech, music, effects, noise or a mixtures thereof.

[0054] The transition parameter extractor 2 obtains the query audio clip ^^^^^^^^and a matched candidate audio clip ^^^^^^^^ ^^^^and is configured to determine at least one parameter describing how a transition between the two clips ^^^^^^^^, ^^^^^^^^ ^^^^should be performed to achieve a match cut. The at least one parameter may be one or more of a crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^and a transition point. Theat least one parameter is determined by the transition parameter extractor 2 for each pair of the query audio clip ^^^^^^^^and a matched candidate audio clip ^^^^^^^^ ^^^^. For example, if the query audio clip ^^^^^^^^is matched with ^^^^ candidate audio clips ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^^^^^ instances of the at least one parameter is determined by the transition parameter extractor 2, with one instance of the at least one parameter per clip pair.

[0055] The clip pair matcher module 1 and the transition parameter extractor 2 may be said to perform a coarse and fine processing respectively, with the clip pair matcher 1 operating on the comparatively coarse audio clip level and the transition parameter extractor operating within a pair of audio clips, on a comparatively fine intra audio clip level. For example, the clip pair matcher 1 operates on audio clips, with each audio clip e.g. comprising at least 0.5 seconds of audio content or at least 1 second of audio content and identifies at least one matched candidate audio clip ^^^^^^^^ ^^^^which is generally similar to the query audio clip ^^^^^^^^. The transition parameter extractor 2, on the other hand, may determine a transition point within two matching clips by determining a similarity metric between individual samples of the audio clips, where each audio clip comprises tens, hundreds or thousands samples. That is, the clip pair matcher 1 identifies matching clips based on many samples of audio content whereas the transition parameter extractor 2, at least when determining the transition point, compares the similarity metrics of individual samples.

[0056] A matched audio clip pair comprising the query audio clip ^^^^^^^^and a matched candidate audio clip ^^^^^^^^ ^^^^of the clip pair matcher module 1 and the at least one transition parameter, determined by transition parameter extractor 2, are provided to the audio match cut generator 3 which forms a match cut audio clip VM comprising a portion of the audio content of each audio clip in the matched audio clip pair and a transition between the audio content of the matched audio clips of the matched audio clip pair. That is, given a query audio clip ^^^^^^^^and a set of matched candidate audio clips ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^the audio match cut generator 3 will generate a match cut transition audio clip ^^^^^^^^ ^^^^as ^^^^^^^^ ^^^^= ^^^^^^^^� ^^^^^^^^,^^^^wherein ^^^^^^^^is a transition function describing a transition (e.g. a crossfading) between the arguments and ^^^^^^^^ ^^^^is a matching clip from the set of candidate audio clips ^^^^1, ^^^^2, …, ^^^^^^^^.

[0057] It is the purpose of the clip pair matcher 1 to identify the best matching (or a plurality of best matching) audio clip ^^^^^^^^ ^^^^in the set of candidate audio clips ^^^^1, ^^^^2, …, ^^^^^^^^and it is the purpose of the transition parameter extractor 2 to determine the parameters for the transition function ^^^^^^^^.

[0058] While the clip pair matcher module 1 and the transition parameter extractor 2 are depicted together in the system 10 of fig.1 it is noted that provision of both of these modules,while beneficial, is not absolutely necessary. For example, if a matched pair of audio clips ^^^^^^^^, ^^^^^^^^ ^^^^is already available, the clip pair matcher module 1 can be omitted, and the matched pair can be provided directly to the transition parameter extractor 2 which extracts transition parameters for realizing a high quality audio match cut between the matched clips. Alternatively, if a static transition type is specified (e.g. by providing predetermined transition parameters) the transition parameter extractor 2 can be omitted whereby automatically matched audio clip pairs of the clip pair matcher module 1 are provided the audio match cut generator 3 which generates the audio match cut transition audio clip ^^^^^^^^.

[0059] Turning to FIG.2, and the flowchart in FIG.7A, a process performed by the clip pair matcher module 1 will now be described in the further detail.

[0060] At step S1, the clip pair matcher module 1 obtains a query audio clip ^^^^^^^^and at step S2 the clip pair matcher module obtains a set of ^^^^ candidate audio clips ^^^^1, ^^^^2, … , ^^^^^^^^. Each audio clip ^^^^^^^^, ^^^^1, ^^^^2, … , ^^^^^^^^comprises a duration of audio content (e.g. general audio content). In some implementations, each audio clip ^^^^^^^^, ^^^^1, ^^^^2, … , ^^^^^^^^comprises the same duration of audio content. For example, each audio clip comprises between 0.5 and 5 seconds of audio content, such as 1 second of audio content.

[0061] The domain in which each audio clip ^^^^^^^^, ^^^^1, ^^^^2, … , ^^^^^^^^is represented may for the purposes of this disclosure be any domain. For example, each audio clip ^^^^^^^^, ^^^^1, ^^^^2, … , ^^^^^^^^may be represented in the time domain, such as in a waveform domain, or in a time-frequency domain such as in a Modified Discrete Cosine Transform (MDCT) domain, a set of Mel-Frequency Cepstral Coeffecients (MFCCs), in a Mel-Spectrogram domain or in a QMF-domain.

[0062] Since audio match cuts often are used for video (e.g. in movies) each audio clip may optionally be associated with a video clip. For example, the query audio clip ^^^^^^^^is an audio clip associated with a first video clip from which a video editor wishes to transition from and the set of candidate audio clips ^^^^1, ^^^^2, … , ^^^^^^^^are audio clips associated with one or more other video clips which the video editor could transition to. By identifying one or more matching candidate clips ^^^^^^^^1, ^^^^^^^^2, … , ^^^^^^^^ ^^^^that are perceptually similar to the query audio clip ^^^^^^^^matching audio clip pairs can be formed that are promising clip pairs for realizing an audio match cut.

[0063] The task of the clip pair matcher module 1 is therefore to find among the set of ^^^^ candidate audio clips ^^^^1, ^^^^2, … , ^^^^^^^^one or more matching candidate audio clips ^^^^^^^^1, ^^^^^^^^2, … , ^^^^^^^^ ^^^^that are perceptually similar to the query audio clip ^^^^^^^^.

[0064] In some implementations, the number ^^^^ of matching candidate audio clips ^^^^^^^^1, ^^^^^^^^2, … , ^^^^^^^^ ^^^^that are identified by the clip pair matcher module 1 is smaller than thenumber ^^^^ of candidate audio clips ^^^^1, ^^^^2, … , ^^^^^^^^. For example, ^^^^ is at least two times, at least five times or at least ten times greater than ^^^^.

[0065] To find matching candidate audio clips, the query audio clip is at step S3 processed with a feature extractor 11 to extract feature data ^^^^^^^^ ^^^^associated with the query audio clip ^^^^^^^^.

[0066] Similarly, each of the ^^^^ candidate audio clips of the set ^^^^1, ^^^^2, …, ^^^^^^^^are at step S4 processed with a feature extractor 11 to extract feature data ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^associated with each candidate audio clip ^^^^1, ^^^^2, …, ^^^^^^^^, respectively.

[0067] The feature extractor 11 may be an analytical feature extractor configured to extract feature data describing a Time-Frequency (TF) feature representation of each audio ^^^^^^^2, …, ^^^^^^^^. In one implementation, the feature extractor 11 may be configured to Mel- frequency cepstral coefficients (MFCCs) of the audio clips ^^^^^^^^, ^^^^1, ^^^^2, …, ^^^^^^^^whereby the feature data ^^^^^^^^ ^^^^, ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^is in the form of MFCCs. In one implementation, the feature extractor 11, may be configured to extract the Mel-Frequency spectrum of the audio clips ^^^^^^^^, ^^^^1, ^^^^2, …, ^^^^^^^^whereby the feature data ^^^^^^^^ ^^^^, ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^is in the form of a Mel-Frequency spectrum. Other types of feature extractors, extracting other types of feature data may also be used.

[0068] The feature extractor 11 may be a neural network configured to extract a feature vector whereby the feature data comprises a neural network extracted feature vector. In general, there are many types of neural networks that can be used to extract features.

[0069] In some implementations, the neural network of the feature extractor 11 may be an embedding neural network trained to extract a feature vector from input data. For example, the embedding neural network may be the encoder network of a trained neural network autoencoder. A neural network autoencoder comprises an encoder part and a decoder part wherein the encoder part is trained to transform input data into a feature vector and the decoder part is trained to transform the feature vector into output data. The dimensionality of the feature vector may be smaller than that of the input data whereby the feature vector serves as a compressed representation (which may be referred to as a hidden or latent representation) of the input data. The neural network of the feature extractor 11 may be the encoder part of an audio autoencoder, such as an audio autoencoder used for audio compression, audio denoising or audio generator.

[0070] In one implementation, the feature extractor 11 is realized as the encoder block of the ImageBind neural network as described in R. Girdhar et al., “ImageBind One Embedding Space to Bind Them All,” 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023, pp.15180-15190. ImageBind is trained to generate feature vectors in a feature space which is shared across multiple types of input data(including audio), referred to as modalities. Other neural networks trained to generate feature vectors based on audio input data (and optionally input data of other modalities) can also be used.

[0071] In one implementation, the feature extractor 11 is the encoder block of the CLAP neural network as described in Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick and S. Dubnov, “Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation,” ICASSP 2023 – 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp.1-5. The CLAP neural network extracts feature vector in a feature space given input data in the form of audio and / or text.

[0072] Since these types of feature vectors are extracted by the encoding portion of a neural network (which may comprise a decoding portion, in addition to the encoding portion) the feature vectors may alternatively be referred to as encoding vectors.

[0073] The clip pair matcher module 1 of FIG.2 features two feature extractors 11, one for the query audio clip ^^^^^^^^and one for the candidate audio clips ^^^^1, ^^^^2, …, ^^^^^^^^, and while this allows the features data to be extracted in parallel the person skilled in the art will realize that the clip pair matcher 1 may be realized with a single feature extractor 11 which processes audio clips ^^^^^^^^, ^^^^1, ^^^^2, …, ^^^^^^^^.

[0074] To identify the one or more best matching candidate audio clips ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^out of the set of candidate audio clips ^^^^1, ^^^^2, …, ^^^^^^^^the similarity search module 14 compares the feature data extracted for each audio clip and identifies the ^^^^ candidate audio clips that have the most similar feature data to the query audio clip at step S5.

[0075] To accomplish this, the similarity search module 14 may perform a maximum inner-product search (MIPS) between the feature data ^^^^^^^^ ^^^^of the query audio clip ^^^^^^^^and the feature data instances ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^of the set of candidate audio clips ^^^^1, ^^^^2, …, ^^^^^^^^. The candidate audio clip which has the largest inner product with the query audio clip ^^^^^^^^is the best matching candidate audio clip, the candidate audio clip which as the second largest inner product with the with the query audio clip ^^^^^^^^is the second best matching candidate audio clip and so on.

[0076] Optionally, the feature data instances ^^^^^^^^ ^^^^, ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^are normalized prior to performing the MIPS. For example, the feature data instances ^^^^^^^^, ^^^^^^^^, ^^^^^^^^, …, ^^^^^are ^^^^2^^^^ 1 2 ^^^ ^^^^normalized prior to performing the MIPS.

[0077] As another example, the similarity search module 14 calculates the cosine similarity between the feature data ^^^^^^^^ ^^^^of the query audio clip ^^^^^^^^and the feature data instances ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^of the set of candidate audio clips ^^^^1, ^^^^2, …, ^^^^^^^^whereby the ^^^^ best matchingcandidate audio clips are identified as the ^^^^ candidate audio clips whose feature data features the highest cosine similarity with the feature data ^^^^^^^^ ^^^^of the query audio clip ^^^^^^^^.

[0078] Other measures may alternatively be used to identify the ^^^^ ≥ 1 best matching candidate audio clips, such as various forms of ^^^^1normalized distance measures or ^^^^2normalized distance measures. For example, the Euclidian distance (an ^^^^1normalized distance measure) or Manhattan distance (an ^^^^2normalized distance measure) may be used instead.

[0079] If each feature data instance ^^^^^^^^ ^^^^, ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^is in the form of a vector (e.g. as extracted by a neural network) the distance measures described above can be applied directly to find the ^^^^ best matching candidate audio clips. If each feature data instance ^^^^^^^^ ^^^^, ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^instead is in the form of a matrix (as is the case for MFCC representations or Mel-Frequency spectrum) each matrix can be flattened (e.g. in the time dimension) to form a vector which can be used with any of the above distance measures. Alternatively, it is envisaged that a matrix distance measure may be used.

[0080] In the case when the MIPS process used, the identification of the ^^^^ best matching candidate audio clips based on the feature data may be formulated as ^^^^^∗^^^wherein ^^^^^∗^^^ ^^^^is the feature data of candidate audio clip ^^^^^^^^which best matches the feature data of the query audio clip ^^^^^^^^and ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^is the dot product between the transposed (and optionally normalized) feature data of the query audio clip ^^^^^^^^and the feature data of the ^^^^-th candidate audio clip where ^^^^ = 1, 2, … ^^^^.

[0081] Additionally or alternatively, it is possible to obtain the top- ^^^^ candidate audio clips having the largest similarity to the query audio clip ^^^^^^^^based on feature data. For example, a group of similar feature data instances { ^^^^^^∗^^}^^^^^^^=^1 can be obtained using the MIPS process of equation 1 wherein each of the ^^^^ feature data instances are associated with a respective matching candidate audio clip ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^.

[0082] The similarity search module 14 of the clip pair matcher module 1 may output identifiers identifying the one or more candidate audio clips ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^which are good matches to the query audio ^^^^^^^^. In the embodiment of FIG.2, ^^^^ = 4 matching clips havebeen identified with indices ^^^^1, ^^^^2, ^^^^3, ^^^^4. The identifiers are provided to the transition parameter extractor 2 which can then access the at least one matching candidate audio clip ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^and the query audio clip ^^^^^^^^. Alternatively, the similarity search module 14 outputs the at least one matching candidate clip ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^and the query audio clip ^^^^^^^^directly to the transition parameter extractor 2.

[0083] Turning to FIG.3, the details of the transition parameter extractor 2 will now be described in detail with further reference to the flow chart of FIG.7B.

[0084] The transition parameter extractor 2 obtains two audio clips, the query audio clip ^^^^^^^^and a matching candidate audio clip ^^^^^^^^ ^^^^at step S6.

[0085] In some implementations the transition parameter extractor 2 may obtain multiple matching candidate audio clips ^^^^^^^^ ^^^^(e.g. the top ^^^^ best matching candidate audio clips) and one query audio clip ^^^^^^^^. In such cases, the transition parameter extractor 2 repeats the following process for all combinations of the query audio clip ^^^^^^^^with one of the ^^^^ matching candidate audio clips ^^^^^^^^1, ^^^^^^^^2, …, ^^^^^^^^ ^^^^.

[0086] In FIG.3, the query audio clip ^^^^^^^^and the matching candidate audio clip ^^^^^^^^ ^^^^are depicted schematically as a respective time-frequency representation 31, 32. The time-frequency representation 31, 32 of each audio clip ^^^^^^^^, ^^^^^^^^ ^^^^comprises a plurality of coefficients divided into a plurality of bins spanning at least one frequency band ^^^^1, ^^^^2, … ^^^^ and a plurality of samples (which may also be referred to as time slots) ^^^^1, ^^^^2, … ^^^^^^^^. For example, each bin comprises a single coefficient. That is, the query audio clip ^^^^^^^^is given by the representation 31 labeled ^^^^^^^^∈ ℝ^^^^, ^^^^ ^^^^and the matching candidate audio clip ^^^^^^^^ ^^^^is given by the representation 32 labeled ^^^^^^^^∈ ℝ

[0087] In some implementations, the coefficients in each bin of time-frequency representations 31, 32 are Mel-spectrogram coefficients, MFCCs, MDCT coefficients or QMF- bank coefficients whereby the representations 31, 32 are said to be a Mel-spectrogram representation, MFCC representation, MDCT representation or QMF representation respectively.

[0088] If the query audio clip ^^^^^^^^, 31 and the matching candidate audio clip ^^^^^^^^ ^^^^, 32 are not already available in a time-frequency representation 31, 32 the audio clips may first be transformed to this type of representation by the transition parameter extractor 2. For example, if the query audio clip ^^^^^^^^and the matching candidate audio clip ^^^^^^^^ ^^^^are originally available in a time domain representation (such as a waveform representation) they may be converted to the time-frequency representation 31, 32 with a plurality of bins spanning a plurality of samples in the time dimension and plurality of frequency bands in the frequency dimension.

[0089] The type of time-frequency representations 31, 32 used by the transition parameter extractor 2 may be the same type of time-frequency representation used by the clip pair matcher 1 operates or a different type. That is, the clip pair matcher 1 and the transition parameter extractor 2 may operate in the same domain or in different domains.

[0090] For example, both the clip pair matcher 1 and the transition parameter extractor 2 operate in the Mel-spectrogram domain with Mel-spectrogram representations. As a furtherexample, the clip pair matcher 1 identifies matching candidate clips using feature vectors extracted by a neural network and the transition parameter extractor module 2 obtains Mel- spectrogram representations 31, 32 of the matched candidate audio clips as an input.

[0091] At step S7 a transition point is obtained by the transition parameter extractor 2. Obtaining the transition point may comprise determining the transition point by the transition parameter extractor 2, as will be described below.

[0092] A transition point indicates a point in time (e.g. a sample index ^^^^^^^^) in the representations of the query audio clip ^^^^^^^^, 31 and the matching candidate audio clip ^^^^^^^^ ^^^^, 32, respectively, where the transition is to take place to form the match cut audio clip. For example, the transition point indicates a sample index ^^^^^^^^in the representations of the query audio clip ^^^^^^^^, 31 and a sample index ^^^^^^^^in the matching candidate audio clip ^^^^^^^^ ^^^^, 32 where the transition is to be performed. In the resulting representation of the match cut audio clip the audio content of samples ^^^^1, ^^^^2, ^^^^3, … , ^^^^ the query audio clip ^^^^^^, 31 are used to form a first portion of the^^samples of the match cut clip and the audio content of samples ^^^^^^^^, ^^^^^^^^+1, ^^^^^^^^+2, … ^^^^^^^^in the matching candidate audio clip ^^^^^^^^ ^^^^, 32 are used to form a second portion of the samples of the match cut audio clip. In addition, to form a smooth transition between the two clips, a number ^^^^ of samples temporally later than ^^^^^^^^of query audio clip ^^^^^^^^and a number ^^^^ of samples temporally earlier than ^^^^^^^^of the matching candidate audio clip ^^^^^^^^ ^^^^are used to gradually fade out the audio content of the query audio clip ^^^^^^^^and fade in the audio content of the matching candidate audio clip ^^^^^^^^ ^^^^.

[0093] In some examples, the transition point may be predetermined. For example, the transition point is a predetermined sample (e.g. a middle sample or first / last sample) of the representation of the query audio clip ^^^^^^^^, 31 and the matching candidate audio clip ^^^^^^^^ ^^^^, 32.

[0094] The method may then go to step S8 wherein the representations ^^^^^^^^, ^^^^^^^^of the query audio clip ^^^^^^^^, 31 and the matching candidate audio clip ^^^^^^^^ ^^^^, 32 are provided to a similarity metric calculator 21 which calculates a plurality of similarity metrics for the two representations ^^^^^^^^, ^^^^^^^^. Each individual similarity metric may be calculated for a pair of samples wherein the pair of samples comprises a sample from the query audio clip ^^^^^^^^, 31 and a sample from the matching candidate audio clip ^^^^^^^^ ^^^^, 32.

[0095] Similarity metrics are at least calculated for a plurality of sample pairs in a transition context window associated with the transition point. The transition context window comprises samples surrounding the transition point. That is, the transition context window covers samples [ ^^^^^^^^− ^^^^1, ^^^^^^^^+ ^^^^1] in the query clip ^^^^^^^^, 31 and samples [ ^^^^^^^^− ^^^^2, ^^^^^^^^+ ^^^^2] of the matching candidate audio clip ^^^^^^^^ ^^^^,32 wherein ^^^^1and ^^^^2are integers defining the size of the transitioncontext window in the query audio clip ^^^^^^^^, 31 and the matching candidate audio clip ^^^^^^^^ ^^^^, 32, respectively. ^^^^1and ^^^^2may be equal, meaning that an equal number of samples are considered in both the clip ^^^^^^^^, 31 and the matching candidate audio clip ^^^^^^^^ ^^^^, 32.

[0096] For example, the transition context window covers samples spanning at least 0.5 seconds in time or samples spanning at least 1 second in time.

[0097] Optionally, similarity metrics are calculated for all sample pairs of the query audio clip ^^^^^^^^, 31 and the matching candidate audio clip ^^^^^^^^ ^^^^, 32 respectively.

[0098] In one implementation, the similarity metric calculator 21 calculates the plurality of similarity metrics as the inner product between representations ^^^^^^^^, ^^^^^^^^yielding a similarity matrix ^^^^ = ^^^^^^^^^^^^^^^^^^^^which represents the similarity metrics, wherein the similarity matrix ^^^^ ∈ℝ ^^^^ ^^^^, ^^^^ ^^^^. Accordingly, the similarity matrix ^^^^ comprises a plurality of elements wherein element[ ^^^^, ^^^^] is the similarity metric indicating a similarity between sample ^^^^ ^^^^ in representation ^^^^ ^^^^(representing the query audio clip ^^^^^^^^, 31) and sample ^^^^^^^^in representation ^^^^^^^^(representing the matched candidate audio clip ^^^^^^^^ ^^^^, 32). Since each sample comprises one or more bins, wherein each bin is associated with a frequency band, each element in the similarity matrix ^^^^ is the sum of the elementwise product of one or more bins of a sample of the query audio clip ^^^^^^^^and a sample of the matching candidate audio clip ^^^^^^^^ ^^^^. That is, each element in the similarity matrix M is a similarity metric which is influenced by the signal properties of all frequency bands.

[0099] Optionally, if only similarity metrics of a transition context window are to be considered the representations ^^^^^^^^, ^^^^^^^^may first be processed so as to remove columns (i.e. samples) falling outside the transition context window.

[0100] The similarity matrix ^^^^ is provided to a variance calculator 22 at step S9 which calculates the variance of the similarity matrix ^^^^ as ^^^^ ^^^^ ^^^^( ^^^^). Optionally, the similarity metriccalculator 22 first calculates a normalized matrix ^^�^^ as^^�^^allowing the variance calculator 22 to determine the variance of the normalized similarity matrix^^�^^ as ^^^^ ^^^^ ^^^^(^^�^^).

[0101] The variance of the similarity matrix ^^^^, or the variance of the normalized similarity matrix ^^�^^ , is provided to the crossfading length calculator 23 which calculates acrossfading length ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ at step S10 based on the variance ^^^^ ^^^^ ^^^^(^^^^)or ^^^^ ^^^^ ^^^^(^^�^^).

[0102] In general, the crossfading length calculator 23 determines a crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^which increases with decreasing variance of the normalized or non-normalized similarity metric. With this relationship between the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^and thevariance, high quality audio match cuts can be generated automatically. For example, it has been found has audio clips with a very high variance in their similarity metrics do not benefit from a long crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^while audio clips with a low variance in their similarity benefit from a longer crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^due to there being multiple plausible samples where the cut could be performed with small perceptual difference.

[0103] In one example, the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^is determined using the varianceof the normalized similarity matrix ^^�^^ as^^^^1^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^=^^^^ ^^^^ ^^^^( ^^�^^) ^^^^(3) wherein ^^^^ is a tunable parameter controlling the scaling of the relationship of the normalized similarity matrix variance to the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^. In some implementations, ^^^^ is selected such that 1 ≤ ^^^^ ≤ 10, or such that 5 ≤ ^^^^ ≤ 10. For example, ^^^^ = 8.

[0104] The crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^determined by the crossfading length calculator 23 is outputted by the parameter extractor 2 to the audio match cut generator (see FIG.1) which generates an match cut audio clip with the calculated crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^at step S11.

[0105] Turning back to step S7 of obtaining the transition point, it is highlighted above that the transition point may be predetermined or determined adaptively. To determine the transition point adaptively a plurality of similarity metrics may be determined between a subset (e.g. all) samples of the query audio clip ^^^^^^^^and a subset (e.g. all) of the samples of the matching candidate audio clip ^^^^^^^^ ^^^^. The second similarity metrics may be the same as the similarity metrics used to determine the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^or different similarity metrics. For example, the second similarity metrics are non-normalized wherein the similarity metrics are normalized.

[0106] For example, the second similarity metrics are determined as the inner product between representations ^^^^^^^^, ^^^^^^^^yielding the similarity matrix ^^^^ = ^^^^^^^^^^^^^^^^^^^^which represents the similarity metrics.

[0107] In some implementations, there is no default transition point available whereby the similarity metric calculator 21 determines the plurality of second similarity metrics as the non- normalized similarity matrix ^^^^ = ^^^^^^^^^^^^^^^^^^^^which represents the second similarity metric for all sample pairs of the query audio clip ^^^^^^^^and the matched candidate audio clip ^^^^^^^^ ^^^^.

[0108] The similarity matrix ^^^^ representing the second similarity metrics may be provided to a maximum similarity metric extractor 24 which analyzes the similarity matrix ^^^^ and determines the matrix element being associated with a maximum second similarity metric. For example, the maximum similarity metric extractor 24 identifies the index[^^^^^^^^ ^^^^ ^^^^, ^^^^^^^^ ^^^^ ^^^^]of themaximum similarity metric as[^^^^ ^^^^ ^^^^ ^^^^, ^^^^ ^^^^ ^^^^ ^^^^]= ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^, ^^^^(^^^^). The sample pair associated with ofthe maximum second similarity metric indicates a transition point that is likely suitable for a high quality match cut since the representations ^^^^^^^^, ^^^^^^^^are aligned at samples� ^^^^^^^^ ^^^^ ^^^^ ^^^^, ^^^^^^^^ ^^^^ ^^^^ ^^^^�. The maximum similarity metric extractor 24 may then output the sample^^^^ ^^^^]to the audio match cut generator (see FIG.1) which uses this information to form the match cut audio clip. That is, the match cut generator will combine the query audio clip ^^^^^^^^, 31 and the candidate audio clip ^^^^^^^^ ^^^^, 32 such that the cut from one clip to the other is performed at sample ^^^^^^^^ ^^^^ ^^^^ ^^^^, for the query audio clip ^^^^^^^^and sample ^^^^^^^^ ^^^^ ^^^^ ^^^^for the matched candidate audio clip ^^^^^^^^ ^^^^.

[0109] FIG.4 schematically illustrates how the indexes [ ^^^^^^^^ ^^^^ ^^^^, ^^^^^^^^ ^^^^ ^^^^] defining the transitionpoint can be identified by performing a maximum element search in the similarity matrix ^^^^ as[ ^^^^ ^^^^ ^^^^ ^^^^, ^^^^ ^^^^ ^^^^ ^^^^] = ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^, ^^^^( ^^^^). Alternatively, the indexes [ ^^^^ ^^^^ ^^^^ ^^^^, ^^^^ ^^^^ ^^^^ ^^^^] defining the transitionpoint may be found by performing a MIPS on the representations ^^^^^^^^, ^^^^^^^^or calculating the cosine similarity between samples.

[0110] As a further example, each second similarity metric is determined by calculating any of the distance measures mentioned above (e.g. Euclidean distance, Manhattan distance) for sample pairs (i.e. columns in the time-frequency representations ^^^^^^^^, ^^^^^^^^) whereby a lower second similarity metric value indicates a higher degree of similarity. Accordingly, the second similarity metric associated with the highest degree of similarity will be the second similarity metric with the lowest second similarity metric value.

[0111] In some implementations, the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^determined at S10 is determined using normalized (first) similarity metrics (e.g. the normalized similarity matrix ^^�^^) and the transition point is determined at step S7 using a non-normalized (second) similarity metrics (e.g. the non-normalized similarity matrix ^^^^). A benefit of using these types of similarity metrics is that the non-normalized similarity metrics for the determination of transition point tends to place transition point at times with high acoustic energy in each clip.

[0112] In other implementations, it is envisaged that normalized or non-normalized similarity metrics are used for determining both the transition point and the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^.

[0113] It is understood that when the representations ^^^^^^^^, ^^^^^^^^are equally long in terms of number of samples the similarity matrix ^^^^ (or ^^�^^) is a square matrix. However the processes of the similarity metric calculator 21, variance calculator 22 and maximum similarity metric extractor 24 can be performed in a completely analogous manner if the representations ^^^^^^^^, ^^^^^^^^are of different length in number of samples. For example, if the representations ^^^^^^^^, ^^^^^^^^are of different length in number of samples the similarity matrix ^^^^ will not be a square matrix but amaximum element search, MIPS and variance of the similarity matrix may still be determined in manner analogous to as described above.

[0114] FIG.5 illustrates schematically how the audio match cut generator 3 combines the candidate audio clip ^^^^^^^^, 31 and the candidate audio clip ^^^^^^^^ ^^^^, 32 to form an match cut audio clip ^^^^^^^^, 32. While the clip pair matcher and the transition parameter extractor operate using feature data in a feature domain or representations of a time-frequency domain it is understood that the audio match cut generator 3 may operate in the same domains or in a different domain. For example, the match cut generator 3 operates using time domain representations of the audio clips or using time-frequency domain representations.

[0115] The audio match cut generator 3 obtains the query audio clip ^^^^^^^^, 31 and the matched candidate audio clip ^^^^^^^^ ^^^^, 32 (e.g. as time-frequency domain representations 31, 32 as shown ) alongside one or more parameters such as the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^and the transition point (e.g. indices [ ^^^^^^^^ ^^^^ ^^^^, ^^^^^^^^ ^^^^ ^^^^]) indicating the transition point for performing of the audio match cut. Using this information, the audio match cut generator 3 forms the match cut audio clip ^^^^^^^^by forming a crossfade between the candidate audio clip ^^^^^^^^and the candidate audio clip ^^^^^^^^ ^^^^.

[0116] The timing of the crossfade may be determined by the transition point. The crossfade may be centered around the transition point. For example, at the transition point the match cut audio clip ^^^^^^^^, 33 comprises a 50 / 50 mix of the audio content of the query audio clip ^^^^^^^^, 31 sample of the transition point and the matched candidate audio clip ^^^^^^^^ ^^^^, 32 sample of the transition point.

[0117] The audio content of the match cut audio clip ^^^^^^^^comprises a crossfade between the matched audio clips around the transition point. That is, in the match cut audio clip ^^^^^^^^, 33 the audio content of the query audio clip ^^^^^^^^, 31 gradually becomes dominant when moving away from the transition point towards temporally earlier samples and the audio content of the candidate audio clip ^^^^^^^^ ^^^^, 32 gradually becomes dominant when moving away from the transition point towards temporally later samples.

[0118] In the example of FIG.5 the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^is the length of three samples. At the transition point corresponding to e.g. samples ^^^^^^^^ ^^^^ ^^^^ ^^^^, ^^^^^^^^ ^^^^ ^^^^ ^^^^(or default samples) of the query audio clip ^^^^^^^^, 31 and the matched candidate audio clip ^^^^^^^^ ^^^^, 32 and sample ^^^^^^^^ ^^^^ ^^^^ ^^^^ℎof the matched cut audio clip ^^^^^^^^, 33 the audio content of the match cut audio clip ^^^^^^^^, 33 is a mix (e.g. a 50 / 50 mix) of the audio content at sample ^^^^^^^^ ^^^^ ^^^^ ^^^^, ^^^^^^^^ ^^^^ ^^^^ ^^^^of the query audio clip ^^^^^^^^, 31 and the matched candidate audio clip ^^^^^^^^ ^^^^, 32. Sample ^^^^^^^^of the match cut audio clip ^^^^^^^^, 33 which precedes the sample ^^^^^^^^ ^^^^ ^^^^ ^^^^ℎis still within the crossfading region defined by the crossfading length^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^meaning that sample ^^^^^^^^also contains a mix of the audio content of the query audio clip ^^^^^^^^, 31 and the candidate audio clip ^^^^^^^^ ^^^^, 32. However, at sample ^^^^^^^^of the match cut audio clip ^^^^^^^^, 33 audio content of samples ^^^^^^^^ ^^^^ ^^^^ ^^^^−1, ^^^^^^^^ ^^^^ ^^^^ ^^^^−1of the query audio clip ^^^^^^^^, 31 and the matched candidate audio clip ^^^^^^^^ ^^^^, 32mixture at sample ^^^^^^^^may be dominated by the audio content of samples ^^^^^^^^ ^^^^ ^^^^ ^^^^−1of the query audio clip ^^^^^^^^, 31 (e.g. the mixture is a 75 / 25 mixture favoring the audio content of the query audio clip ^^^^^^^^, 31). At sample ^^^^^^^^of the match cut audio clip ^^^^^^^^, 33, which is temporally preceding sample ^^^^^^^^and outside the crossfading region defined by the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^in this example the audio content may be a copy of the audio content at sample ^^^^^^^^ ^^^^ ^^^^ ^^^^−2of the query audio clip ^^^^^^^^, 31. The same may apply for samples preceding sample ^^^^^^^^in the match cut audio clip ^^^^^^^^, 33 where each sample preceding sample ^^^^^^^^corresponds to samples ^^^^ ^^^^ ^^^^ and so on, going towards temporallyearlier samples of the match cut audio clip ^^^^^^^^, 33.

[0119] An opposite transition may occur for samples ^^^^^^^^and ^^^^^^^^of the match cut audio clip ^^^^^^^^that temporally follow sample ^^^^^^^^ ^^^^ ^^^^ ^^^^ℎ. Here, at sample ^^^^^^^^the audio content of the samples ^^^^^^^^ ^^^^ ^^^^ ^^^^+1, ^^^^^^^^ ^^^^ ^^^^ ^^^^+1are mixed and preferably the mixture at sample ^^^^^^^^is dominated by the audio content of sample ^^^^^^^^ ^^^^ ^^^^ ^^^^+1of the matched candidate audio clip ^^^^^^^^ ^^^^, 32 (e.g. the mixture is a 25 / 75 mixture favoring the audio content of the matched candidate audio clip ^^^^^^^^ ^^^^, 32). At sample ^^^^^^^^, which is temporally later than slot ^^^^^^^^, and which in this example lies outside the crossfading region defined by the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^the audio content may be a copy of the audio content of sample ^^^^^^^^ ^^^^ ^^^^ ^^^^+2of the matched candidate audio clip ^^^^^^^^ ^^^^, 32. Similarly, the audio content of samples ^^^^^^^^, ^^^^^^^^, ^^^^^^^^, ^^^^ℎfollowing sample ^^^^^^^^may be copies of the audio content at sample ^^^^^^^^of the matched candidate audio ^^^^^^^^ ^^^^, 32, respectively.

[0120] In this way, when the match cut audio clip ^^^^^^^^, 33 is played back, the audio content will crossfade from the audio content of the query audio clip ^^^^^^^^, 31 to that of the matched candidate audio clip ^^^^^^^^ ^^^^, 32 in a seamless manner which has been dynamically adapted based on the similarity of the two clips ^^^^^^^^, 31, ^^^^^^^^ ^^^^, 32 (as described above) so as to create a high quality audio match cut ^^^^^^^^, 33.

[0121] As described above, the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^is dynamically determined based on the variance of the similarity metrics between the audio content of the query audio clip ^^^^^^^^, 31 and the matched candidate audio clip ^^^^^^^^ ^^^^, 32.

[0122] In general, the larger the variance of the similarity metrics, the shorter the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^. That is, the variance of the similarity metrics may be inversely proportional to the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^.

[0123] At the same time, there are many alternative crossfading profiles which may be used to establish how the audio content of one audio clip is faded-in and the other faded-out. A crossfading profile determines the type of crossfading that is to be applied whereas the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^specifies the duration of the crossfade. The crossfading profile may specify a fading-in profile which defines the gain profile according to which the temporally later audio clip is introduced and a fading-out profile which defines the attenuation profile according to which the temporally earlier audio clip is silenced. Throughout the crossfading region defined by the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^the audio content of both audio clips may be mixed at each sample.

[0124] In one implementation, the crossfading profile is given by root functions wherein the fading-in function is given by�√^^^^� ^^^^ ∈ [0,1]� and the fading-out function is given by �√1 − ^^^^� ^^^^ ∈ [0,1]� with an overlap corresponding to the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^. This is shown schematically in FIG.6 where fading-in function is denoted ^^^^ ^^^^ ^^^^ ^^^^^^^^ ^^^^and the fading-out function is denoted ^^^^ ^^^^ ^^^^ ^^^^^^^^ ^^^^ ^^^^.

[0125] The parameter ^^^^ is distributed uniformly across the interval [0,1] for all samples covered by the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^. For example, in FIG.6 where the crossfade length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^covers five sample the parameter ^^^^ may obtain values of 0, 0.25, 0.5, 0.75, 1 for the timeslots covered by crossfading time.

[0126] Other examples of crossfading profiles include linear crossfading ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ={ ^^^^| ^^^^ ∈ [0,1]} and ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ = {1 − ^^^^| ^^^^ ∈ [0,1]}.

[0127] Additionally it is noted that if the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^is long the transition from one clip to the other may occur gradually, over several samples. On the other hand, if the crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^is very short (e.g. smaller than or equal to the duration of a sample) the transition may be more abrupt. In an extreme case of a crossfading length ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^which is shorter than one sample, the match cut audio clip ^^^^^^^^, 33 is forming a mixed sampleat ^^^^^^^^ ^^^^ ^^^^ ^^^^ℎby mixing (e.g.50 / 50 mixing) the audio content of^^^^ ^^^^ ^^^^ ^^^^of the query audio clip ^^^^^^^^, 31 with the audio content of sample ^^^^^^^^candidate audio clip ^^^^^^^^ ^^^^, 32wherein the audio content of samples ^^^^1, ^^^^2, ... , ^^^^^^^^ ^^^^ ^^^^ ^^^^−1of the query audio clip ^^^^^^^^, 31 areconcatenated temporally before ^^^^^^^^ ^^^^ ^^^^ ^^^^ℎand the audio content of samples ^^^^^^^^ ^^^^ ^^^^ ^^^^+1, ^^^^^^^^ ^^^^ ^^^^ ^^^^+2, ... , ^^^^^^^^of the matched candidate audio clip ^^^^^^^^ ^^^^, 32 are concatenated temporally before the mixed sample.

[0128] Turning back to the clip pair matcher 1 of FIG.2 it has been found that usage of feature vectors extracted by neural networks facilitates more accurate audio clip pair matchings compared to usage of analytically determined features such as Mel-spectrogram or MFCCs. Moreover, the encoder of the CLAP neural network has proved to offer better performance than the ImageBind encoder.

[0129] It has been realized that a further improvement to the audio clip matching can be obtained by generating modified encoding vectors which facilitate more accurate identification of matching audio clips. The modification of the (original) feature vectors into modified feature vectors is done as a linear transformation implemented using a projection neural network 12. The projection neural network 12 is trained to output a modified feature vector of a feature space, given an (original) feature vector extracted by a neural network encoder acting as the feature extractor 11. Optionally, the projection neural network 12 is a single layer defining a linear transformation. However it is envisaged that the projection neural network 12 may be realized using more than one layer.

[0130] To train the projection neural network 12 a contrastive learning loss function is used. A contrastive learning loss function defines a loss in terms of how close or far away the generated modified feature vectors are to each other in the feature space. As will now be described with reference to figs.8a-c and the flow chart of FIG.9, the goal of the contrastive learning is to train the projection neural network 12 to both generate similar modified encoding vectors (i.e. modified encoding vectors which are comparatively close together according to some norm, such as the ^^^^2norm, in the feature space of the modified feature vectors) for audio content which is temporally closer together and generate non-similar encoding vectors (i.e. modified feature vectors which are comparatively far apart according to some norm in the feature space of the modified feature vectors) for audio content which is temporally further apart.

[0131] To train the projection neural network layer 12 a plurality of audio files ^^^^1, ^^^^2are provided at step S21. Each audio file ^^^^1, ^^^^2represents a duration of audio content and each audio file ^^^^1, ^^^^2is divided into a plurality of consecutive audio segments ^^^^1, …, ^^^^^^^^. Each audio segment may in turn span one or more discrete audio samples, such as time domain samples or time-frequency domain samples. For example, each audio file ^^^^1, ^^^^2is an audio signal in the time domain and each audio segment ^^^^1, …, ^^^^^^^^is a portion of that audio file ^^^^1, ^^^^2.

[0132] In FIG.8A each audio file ^^^^1, ^^^^2is divided into six segments ^^^^1, …, ^^^^6and ^^^^7, …, ^^^^12respectively. However this is merely exemplary and each audio file ^^^^1, ^^^^2may comprisefewer or more than six audio segments, such as ten segments. The audio segments ^^^^1, …, ^^^^6of each audio file ^^^^1, ^^^^2may be partially overlapping or non-overlapping.

[0133] As a further example, each audio file ^^^^1, ^^^^2comprises 10 seconds of audio content and each audio file ^^^^1, ^^^^2is divided into ten, non-overlapping, consecutive audio segments each comprising 1of audio content.

[0134] Each audio segment ^^^^1, …, ^^^^12is provided to a neural network encoder of the feature extractor 11 which extracts at step S22, for each audio segment ^^^^1, …, ^^^^12individually, an encoded feature vector ^^^^1, …, ^^^^12. For example, the neural network encoder in the feature extractor 11 is the encoder of the CLAP neural network.

[0135] A given audio file ^^^^1may accordingly be represented by a set of feature vectors ^^^^1, …, ^^^^6with each feature vector ^^^^1, …, ^^^^6being associated with an audio segment ^^^^1, …, ^^^^6of that audio file ^^^^1. Accordingly, for audio files ^^^^1and ^^^^2there is a consecutive sequence of feature vectors ^^^^1, …, ^^^^6and ^^^^7, …, ^^^^12corresponding to the consecutive sequences of audio segments ^^^^1, …, ^^^^6and ^^^^7, …, ^^^^12that constitute the audio files ^^^^1and ^^^^2.

[0136] Turning to FIG.8B each audio file ^^^^1, ^^^^2is divided into two consecutive segment sequences, a first segment sequence ^^^^11, ^^^^21and a second segment sequence ^^^^21, ^^^^22, each segment sequence comprising at least one audio segment…, ^^^^12. In some implementations, each audio file ^^^^1, ^^^^2is randomly divided into the two segment sequences. For example, this can be achieved by drawing a random number indicating between which segments of the data file division into two consecutive segment sequences should be made.

[0137] In general each audio file ^^^^1, ^^^^2comprises ^^^^ segments and dividing an audio file ^^^^1, ^^^^2into two consecutive segment sequences may involve (e.g. randomly) selecting two adjacent segments out of the ^^^^ segments. The two adjacent segments comprise a temporally earlier segment ^^^^4, ^^^^8which will constitute the temporally last segment in the first segment sequence ^^^^11, ^^^^21and a temporally later segment ^^^^5, ^^^^8which will constitute the temporally first segment in the second segment sequence ^^^^21, ^^^^22.

[0138] In other words, the first segment sequence ^^^^11, ^^^^21of each audio file ^^^^1, ^^^^2comprises ^^^^^^^^segments and the second sequence ^^^^21, ^^^^22of each audio file ^^^^1, ^^^^2comprises ^^^^^^^^segments wherein ^^^^^^^^+ ^^^^^^^^= ^^^^. In some implementations, ^^^^^^^^and ^^^^^^^^varies from file to file since the division into the first and second segment sequence is randomized.

[0139] The division of each audio file ^^^^1, ^^^^2into segment sequences ^^^^11, ^^^^12, ^^^^21, ^^^^22may equivalently be performed in the domain of the feature vectors as illustrated in FIG.8b. Since each segment ^^^^1, …, ^^^^6is associated with a feature vector ^^^^1, …, ^^^^6a division of the feature vectors associated with an audio file ^^^^1into to vector sequences, with ^^^^^^^^and ^^^^^^^^featurevectors in each vector sequence, may be done in an analogous manner as for the division of the segments into segment sequences ^^^^11, ^^^^12, ^^^^21, ^^^^22.

[0140] Each audio file ^^^^1, ^^^^2is associated with two segment sequences, and corresponding vector sequences, comprising a first segment sequence ^^^^11, ^^^^21and a second segment sequence ^^^^12, ^^^^22. The first segment sequence ^^^^11, ^^^^21comprises at least one leading segment and a last segment and the second segment sequence ^^^^12, ^^^^22comprises a first segment and at least one trailing segment. In the audio file, the last segment of the first segment sequence and the first segment of the last segment sequence are subsequent segments. In FIG.8b the first audio file ^^^^1comprises a first sequence wherein segments ^^^^1, ^^^^2, ^^^^3are leading segmentsand segment ^^^^4is the last segment. The segment sequence ^^^^12comprises a segment ^^^^5as the first segment and segment ^^^^6as trailing segment.

[0141] Turning to FIG.8C the feature vectors ^^^^1, …, ^^^^12of ^^^^ audio files are provided as input to the projection neural network 12 at step S23. For each input feature vector ^^^^1, …, ^^^^12the projection neural network outputs a modified feature ^^^^1∗, …, ^^^^1∗2 vector in a feature space.

[0142] The method may then go to step S24 comprising training the projection neural network 12 with a contrastive learning loss function to generate, for each feature vector, a modified encoding vector ^^^^1∗, …, ^^^^1∗2 in a feature space.

[0143] The contrastive learning loss function is formulated so as to promote generation of modified feature vectors ^^^^1∗, …, ^^^^1∗2 which are close together in the feature space for input (unmodified) feature vectors ^^^^1, …, ^^^^12associated with the last and first segment of each audio file. At the same time, the contrastive learning loss function is configured to promote generation of modified feature vectors which are far apart in the feature space for input feature vectors that are associated with a leading segment or a trailing segment.

[0144] In some implementations, the feature vectors ^^^^1, …, ^^^^12associated with a last segment and a first segment of an audio file are denoted ^^^^^^^^ ^^^^= ^^^^^^^^ ^^^^ ^^^^and ^^^^^^^^ ^^^^= ^^^^^^^^0whereby the contrastive loss function be defined as^^^^^^^^=1 ^^^^=1^^^^ ^ ^^^^ ^^^^wherein ^^^^^^^^^^^^^^^^^^^^denotes the inner product of ^^^^2normalized feature vectors and ^^^^ denotes a temperature parameter for SoftMax (e.g. ^^^^ is set to 0.1).

[0145] Using the contrastive learning loss function of equation 3 it is possible to maximize the similarity (i.e. minimize the distance in the feature space) between modified feature vectors ^^^^1∗, …, ^^^^1∗2 associated with adjacent segments in an audio file which has been split. Thesemodified feature vectors ^^^^1∗, …, ^^^^1∗2 provide a feature representation better suited for comparison to determine similar audio files.

[0146] The neural network projection layer 12 may be trained together with the neural network encoder 11 or it is envisaged that that the neural network projection layer 12 is trained with a pretrained and frozen encoder neural network 11.

[0147] After the neural network projection layer 12 has been trained, it can be used together with the audio encoder 11 in the clip pair matcher 1 shown in FIG.2. For example, audio clips ^^^^^^^^and ^^^^1, ^^^^2, …, ^^^^^^^^may be of the same format as the segments which the neural network encoder 11 and projection layer 12 encountered during training.

[0148] FIG.10 shows a schematic block diagram of an example electronic device or architecture 200 (e.g., an apparatus 200) suitable for implementing example embodiments of the present disclosure. Architecture 200 includes but is not limited to servers and client devices, systems, and methods as will be described in reference to FIGS.1-3, 5 and 8a-c. As shown, the architecture 200 includes central processing unit (CPU) 201 which is capable of performing various processes in accordance with a program stored in, for example, read only memory (ROM) 202 or a program loaded from, for example, storage unit 208 to random access memory (RAM) 203. The CPU 201 may be, for example, an electronic processor 201, which may include one or more processor cores, and in some examples the processor 201 may be multiple processors. In RAM 203, the data required when CPU 201 performs the various processes is also stored, as required. CPU 201, ROM 202 and RAM 203 are connected to one another via bus 204. Input / output (I / O) interface 205 is also connected to bus 204.

[0149] The following components are connected to I / O interface 205: input unit 206, that may include a keyboard, a mouse, or the like; output unit 207 that may include a display such as a liquid crystal display (LCD) and one or more speakers; storage unit 208 including a hard disk, or another suitable storage device; and communication unit 209 which may include a network interface card such as a network card (e.g., wired or wireless).

[0150] In some implementations, input unit 206 includes one or more microphones in different positions (depending on the host device) enabling capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0151] In some implementations, output unit 207 include systems with various number of speakers. Output unit 207 (depending on the capabilities of the host device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

[0152] In some embodiments, communication unit 209 is configured to communicate with other devices (e.g., via a network). Drive 210 is also connected to I / O interface 205, as required. Removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a flashdrive or another suitable removable medium is mounted on drive 210, so that a computer program read therefrom is installed into storage unit 208, as required. A person skilled in the art would understand that although apparatus 200 is described as including the above-described components, in real applications, it is possible to add, remove, and / or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.

[0153] In accordance with example embodiments of the present disclosure, the processes described above may be implemented as computer software programs or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods. In such embodiments, the computer program may be downloaded and mounted from the network via the communication unit 209, and / or installed from the removable medium 211, as shown in FIG.10.

[0154] Generally, various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits (e.g., control circuitry), software, logic or any combination thereof. For example, the units discussed above can be executed by control circuitry (e.g., CPU 201 in combination with other components of FIG.10), thus, the control circuitry may be performing the actions described in this disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, a processor and / or other computing device(s), which may include control circuitry. While various aspects of the example embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it will be appreciated that the blocks, apparatus, systems, techniques, or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0155] Additionally, various blocks shown in the flowcharts may be viewed as method steps, and / or as operations that result from operation of computer program code, and / or as a plurality of coupled logic circuit elements constructed to carry out the associated function(s). For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program containing program codes configured to carry out the methods as described above.

[0156] In the context of the disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to one or more processors of a general-purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by one or more processors of the computer or other programmable data processing apparatus, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and / or servers.

[0158] Various aspects of the present disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs):

[0159] EEE 1. A method for creating an audio match cut between a first audio clip and a second audio clip, each audio clip comprising a plurality of audio content samples, the method comprising: obtaining a transition point for crossfading from the first audio clip into the second audio clip; determining a plurality of similarity metrics, each similarity metric indicating a similarity between the audio content of a sample of the first audio clip and the audio content of a sample of the second audio clip, wherein the plurality of similarity metrics are determined at least for samples of the first and second audio clip in a transition context window associated with the transition point; determining a variance of the plurality of similarity metrics; determining a crossfading length based on the variance; and generating a match cut audio clip by crossfading between the first audio clip and the second audio clip at the transition point using the crossfading length.

[0160] EEE 2. A method according to EEE 1, wherein the crossfading length increases with decreasing variance and the crossfading length decreases with increasing variance.

[0161] EEE 3. A method according to EEE 2, wherein the crossfading length is inversely proportional to the variance.

[0162] EEE 4. The method according to any preceding EEE, wherein the transition point indicates a transition sample in the first audio clip and a transition sample in the second audio clip.

[0163] EEE 5. The method according to EEE 4, wherein the transition context window comprises one or more samples temporally surrounding the transition sample of the first audio clip and one or more samples temporally surrounding the transition sample of the second audio clip.

[0164] EEE 6. The method according to any of EEEs 1 to 4, wherein the transition context window covers all samples of at least one of the first and second clip.

[0165] EEE 7. The method according to any of the preceding EEEs, further comprising: obtaining the first audio clip; obtaining a set of audio clips comprising at least two audio clips; and selecting the second audio clip as the audio clip of the set of audio clips that has the most similar audio content to the first audio clip.

[0166] EEE 8. The method according to EEE 4, further comprising: extracting feature data for each audio clip in the set of audio clips; extracting feature data for the first audio clip; and wherein selecting the second audio clip comprises selecting the second audio clip as the audio clip of the set of audio clips associated with feature data that is most similar to the feature data of the first audio clip.

[0167] EEE 9. The method according to EEE 8, wherein the feature data comprises at least one of: Mel-frequency cepstral coefficients, MFCC, Mel-Spectrogram, and a feature vector extracted by a neural network encoder.

[0168] EEE 10. The method according to EEE 9, wherein the feature data comprises the feature vector extracted by the neural network encoder, the neural network encoder comprising an ImageBind encoder neural network or a Contrastive Language-Audio Pretraining, CLAP, encoder neural network, and the method further comprises: extracting the feature vector for each audio clip in the set of audio clips using the ImageBind encoder neural network or the CLAP encoder neural network; and extracting the feature vector for the first audio clip using the ImageBind encoder neural network or the CLAP encoder neural network.

[0169] EEE 11. The method according to EEE 9 or EEE 10, wherein the feature vector is obtained by processing the output of the neural network encoder with at least one linear prediction layer.

[0170] EEE 12. The method according to EEE 11, wherein the linear prediction layer is configured to be trained with a contrastive learning loss function.

[0171] EEE 13. The method according to any of the preceding EEEs, further comprising determining a plurality of second similarity metrics, each second similarity metric indicating a similarity between the audio content of a sample of the first audio clip and the audio content of a sample of the second audio clip, wherein the plurality of second similarity metrics are determined at least for a subset of the samples of the first audio clip and at least for a subset of the samples of the second audio clip, wherein the method further comprises: identifying a sample pair associated with the second similarity metric, among the plurality of second similarity metrics, indicating a maximum degree of similarity the sample pair comprising a sample of the first audio clip and a sample of the second audio clip; and determining the transition point based on the identified sample pair.

[0172] EEE 14. The method according to EEE 13, wherein each second similarity metric is a distance metric.

[0173] EEE 15. The method according to EEE 13, wherein each second similarity metric is an inner product.

[0174] EEE 16. The method according to EEE 15, wherein each audio clip of the first and second audio clips is represented in a time-frequency domain with each sample of the first and second audio clips comprising at least one spectral coefficient, and wherein determining each second similarity metric of the plurality of second similarity metrics comprises: determining a dot product between the at least one spectral coefficient a sample of the first audio clip and the at least one spectral coefficient a sample of the second audio clip.

[0175] EEE 17. The method according to claim 16, wherein the dot product for determining each second similarity metric is non-normalized.

[0176] EEE 18. The method according to any of the preceding EEEs, wherein each audio clip of the first and second audio clips is represented in a time-frequency domain with each sample of the first and second audio clips comprising at least one spectral coefficient, wherein determining each similarity metric comprises: calculating a normalized dot product between the at least one spectral coefficient of a sample in the first audio clip and the at least one spectral coefficient of a sample in the second audio clip in the transition context window.

[0177] EEE 19. The method according to EEE 18, further comprising determining the variance as the variance of the plurality of normalized dot products.

[0178] EEE 20. The method according to claim 18 or claim 19, wherein the normalized dot product is a cosine-similarity.

[0179] EEE 21. An audio processing system comprising a processor and memory, wherein the processor is configured to perform the method according to any of the preceding EEEs.

[0180] EEE 22. A non-transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform the method according to any of EEEs 1–20.

[0181] EEE 23. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any of EEEs 1-20.

[0182] EEE 24. A method for training a neural network comprising: obtaining a number N ≥ 2 of audio files, each audio file comprising a sequence of n ≥ 4 consecutive audio segments; generating, with a feature extraction neural network, a feature vector for each audio segment in each audio file; determining, for each audio file, an index for splitting the n consecutive audio segments into a first consecutive audio segment sequence and a second consecutive audio segment sequence, wherein the first audio segment sequence comprises at least one leading audio segment and a last audio segment, and the second audio segment sequence comprises a first audio segment and at least one trailing audio segment, wherein the last audio segment and first audio segment are subsequent audio segments in the audio file; inputting the feature vector of each audio segment of each audio file to the neural network; and training the neural network to generate a modified feature vector in a feature space for each feature vector of each audio file; wherein the training comprises using a contrastive learning loss function such that a distance in the feature space between modified feature vectors associated with the first and last audio segment of each audio file is smaller than a distance in the feature space between modified feature vectors associated with a leading audio segment and a trailing audio segment of any of the N audio files.

[0183] EEE 25. The method according to EEE 24, further comprising: randomly determining, for each audio file, the index for splitting the n consecutive audio segments into the first consecutive audio segment sequence and the second consecutive audio segment sequence.

Claims

CLAIMS 1. A method for creating an audio match cut between a first audio clip and a second audio clip, each audio clip comprising a plurality of audio content samples, the method comprising: obtaining a transition point for crossfading from the first audio clip into the second audio clip; determining a plurality of similarity metrics, each similarity metric indicating a similarity between the audio content of a sample of the first audio clip and the audio content of a sample of the second audio clip, wherein the plurality of similarity metrics are determined at least for samples of the first and second audio clip in a transition context window associated with the transition point; determining a variance of the plurality of similarity metrics; determining a crossfading length based on the variance; and generating a match cut audio clip by crossfading between the first audio clip and the second audio clip at the transition point using the crossfading length.

2. A method according to claim 1, wherein the crossfading length increases with decreasing variance and the crossfading length decreases with increasing variance.

3. A method according to claim 2, wherein the crossfading length is inversely proportional to the variance.

4. The method according to any preceding claim, wherein the transition point indicates a transition sample in the first audio clip and a transition sample in the second audio clip.

5. The method according to claim 4, wherein the transition context window comprises one or more samples temporally surrounding the transition sample of the first audio clip and one or more samples temporally surrounding the transition sample of the second audio clip.

6. The method according to any of claims 1 to 4, wherein the transition context window covers all samples of at least one of the first and second clip.

7. The method according to any of the preceding claims, further comprising: obtaining the first audio clip; obtaining a set of audio clips comprising at least two audio clips; andselecting the second audio clip as the audio clip of the set of audio clips that has the most similar audio content to the first audio clip.

8. The method according to claim 4, further comprising: extracting feature data for each audio clip in the set of audio clips; extracting feature data for the first audio clip; and wherein selecting the second audio clip comprises selecting the second audio clip as the audio clip of the set of audio clips associated with feature data that is most similar to the feature data of the first audio clip.

9. The method according to claim 8, wherein the feature data comprises at least one of: - Mel-frequency cepstral coefficients, MFCC, - Mel-Spectrogram, and - a feature vector extracted by a neural network encoder.

10. The method according to claim 9, wherein the feature data comprises the feature vector extracted by the neural network encoder, the neural network encoder comprising an ImageBind encoder neural network or a Contrastive Language-Audio Pretraining, CLAP, encoder neural network, and the method further comprises: extracting the feature vector for each audio clip in the set of audio clips using the ImageBind encoder neural network or the CLAP encoder neural network; and extracting the feature vector for the first audio clip using the ImageBind encoder neural network or the CLAP encoder neural network.

11. The method according to claim 9 or claim 10, wherein the feature vector is obtained by processing the output of the neural network encoder with at least one linear prediction layer.

12. The method according to claim 11, wherein the linear prediction layer is configured to be trained with a contrastive learning loss function.

13. The method according to any of the preceding claims, further comprising determining a plurality of second similarity metrics, each second similarity metric indicating a similarity between the audio content of a sample of the first audio clip and the audio content of a sample of the second audio clip, wherein the plurality of second similarity metricsare determined at least for a subset of the samples of the first audio clip and at least for a subset of the samples of the second audio clip, wherein the method further comprises: identifying a sample pair associated with the second similarity metric, among the plurality of second similarity metrics, indicating a maximum degree of similarity the sample pair comprising a sample of the first audio clip and a sample of the second audio clip; and determining the transition point based on the identified sample pair.

14. The method according to claim 13, wherein each second similarity metric is a distance metric.

15. The method according to claim 13, wherein each second similarity metric is an inner product.

16. The method according to claim 15, wherein each audio clip of the first and second audio clips is represented in a time-frequency domain with each sample of the first and second audio clips comprising at least one spectral coefficient, and wherein determining each second similarity metric of the plurality of second similarity metrics comprises: determining a dot product between the at least one spectral coefficient a sample of the first audio clip and the at least one spectral coefficient a sample of the second audio clip.

17. The method according to claim 16, wherein the dot product for determining each second similarity metric is non-normalized.

18. The method according to any of the preceding claims, wherein each audio clip of the first and second audio clips is represented in a time-frequency domain with each sample of the first and second audio clips comprising at least one spectral coefficient, wherein determining each similarity metric comprises: calculating a normalized dot product between the at least one spectral coefficient of a sample in the first audio clip and the at least one spectral coefficient of a sample in the second audio clip in the transition context window.

19. The method according to claim 18, further comprising: determining the variance as the variance of the plurality of normalized dot products.

20. The method according to claim 18 or claim 19, wherein the normalized dot product is a cosine-similarity.

21. An audio processing system comprising a processor and memory, wherein the processor is configured to perform the method according to any of the preceding claims.

22. A non-transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform the method according to any of claims 1–20.

23. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any of claims 1-20.

24. A method for training a neural network comprising: obtaining a number N ≥ 2 of audio files, each audio file comprising a sequence of n ≥ 4 consecutive audio segments; generating, with a feature extraction neural network, a feature vector for each audio segment in each audio file; determining, for each audio file, an index for splitting the n consecutive audio segments into a first consecutive audio segment sequence and a second consecutive audio segment sequence, wherein the first audio segment sequence comprises at least one leading audio segment and a last audio segment, and the second audio segment sequence comprises a first audio segment and at least one trailing audio segment, wherein the last audio segment and first audio segment are subsequent audio segments in the audio file; inputting the feature vector of each audio segment of each audio file to the neural network; and training the neural network to generate a modified feature vector in a feature space for each feature vector of each audio file; wherein the training comprises using a contrastive learning loss function such that a distance in the feature space between modified feature vectors associated with the first and last audio segment of each audio file is smaller than a distance in the feature space between modified feature vectors associated with a leading audio segment and a trailing audio segment of any of the N audio files.

25. The method according to claim 24, further comprising: randomly determining, for each audio file, the index for splitting the n consecutive audio segments into the first consecutive audio segment sequence and the second consecutive audio segment sequence.

Images