Method for generating an audio signal

Abstract

The invention relates to a method for generating an audio signal accompanying a slow-motion video signal, comprising: generating a first graphical representation of the input audio signal, dividing the first graphical representation into a set of grains, wherein each has a length and a temporal position, inserting the grains into a second graphical representation representing the output audio signal at a temporal position corresponding to the original temporal position of the grains, corrected by a factor dependent on the slow-motion speed, generating, by audio inpainting based on diffusion, the filling grains intended to fill the spaces between the grains of the second representation, inserting the filling grains into the spaces between the grains, converting the second representation by applying the inverse of the first step.

Description

Method of generating an audio signal

[0001] Description

[0002] The present invention relates to a method for generating an audio signal intended to accompany a slow motion video signal of an action.

[0003] This involves modifying the audio signal (in practice, an artificial audio signal is generated) to match the duration of the slowed-down video sequence. Such a method finds its application, for example, in the live, delayed or slightly delayed transmission of team or individual sporting events such as football, basketball or tennis matches. The present invention is particularly suitable for the representation of motor sport sequences. It should be noted, however, that the method can be applied to any video sequence representing an action.

[0004] Indication of prior art

[0005] The first solution that comes to mind to solve this problem is to change the playback speed of the audio file so that its playback time matches that of the slowed-down video signal. However, the sound produced by this solution is of poor quality since the frequency of the audio signal is altered by the reduction in its speed and is unrealistic and even unpleasant for the viewer.

[0006] We have already looked for more realistic solutions to get closer to the sound atmosphere of the original audio signal.

[0007] For example, in document EP-A1-2509073, a method for stretching the audio signal has been proposed, consisting of dividing the original audio signal into a series of grains, inserting these grains at a distance from each other in the generated audio signal and filling the spaces between the original grains of the restored signal with an audio texture. For the audio texture, it is proposed, in particular, to repeat grains of the audio signal which are close (arranged before or after) but not directly adjacent to the space to be filled. These grains must have been stored beforehand in a database, which does not allow rapid processing of the audio signal, so that the use of this solution is excluded for an event broadcast live or slightly delayed.

[0008] Another solution has been proposed in document EP-A1-2960904. Just like the previous document, document EP-A1-2960904 proposes to divide the original audio signal into a series of grains and to insert these grains at a distance from each other in the generated audio signal. Silences are then inserted in the spaces between the original grains of the reproduced signal. This method is not suitable for the retransmission of sporting events during which there is a fairly significant and permanent sound environment so that the silences inserted in the reproduced audio signal are easily identified by the spectator and the result is not satisfactory.

[0009] It would therefore be desirable to provide a solution for generating an audio signal intended to accompany a video signal broadcast at slow speed which approximates the sound ambiance of the original audio signal and which can be applied to any type of transmission (delayed, slightly delayed or live).

[0010] Statement of the invention

[0011] To achieve this, a method is proposed for generating an output audio signal (A 1 ) intended to accompany a video signal played at slow speed (V r ) comprising the following steps performed in order: a) generating a first graphical representation of the input audio signal (A), b) dividing the first graphical representation of the input audio signal (A) into a set of grains (GQ, GI, G2, ...) each having a length (LQ, LI, 1-2, ...) and a time position (PQ, Pl, P2, ■■■), c) inserting the grains (GQ, GI, G2, ...) into a second graphical representation representing the output audio signal (A 1) at a time position (P'o, P'1, P'2, ■■■) which corresponds to the original time position of the grains (GQ, GI, G2, ...) corrected by a factor depending on the speed of the slowed down sequence, d) generation by audio inpainting based on the diffusion of filler grains (G'o, G'i, ...) intended to fill the spaces between the grains (GQ, GI, G2, ...) of the second graphic representation representing the output audio signal (A 1 ) and e) inserting the filler grains (G'o, G'i, ...) into the spaces between the grains (GQ, GI, G2, ...) of the second graphical representation representing the output audio signal (A 1 ), and f) converting the second graphical representation representing the output audio signal (A 1 ) by the reverse operation of step a).

[0012] Note that in steps b) and c), the time position (PQ, Pl, P2, ■■■) or (P'o, P'1, P'2, ...) which corresponds to the time position of the grains (Go, Gi, G2, ...) respectively in the original audio signal (A) or generated (A 1 ) is based on the time position of the start of the grains (Go, Gi, G2, ...). In another embodiment, the time position of the grains (Go, G , G2, ...) is based on the centers of the grains (Go, Gi, G2, ...).

[0013] It should be noted that the method according to the invention does not use video signal processing. This makes it possible to speed up processing and provide a result compatible with the desired goals.

[0014] Inpainting is the name given to techniques for reconstructing or generating missing parts of an image. These techniques can be implemented manually or using digital tools, with or without user interaction. Digital models for implementing inpainting can use artificial intelligence, for example deep learning. The audio inpainting technique (or sometimes also audio interpolation) is a technique that is mainly used in restoration or reconstruction of missing or corrupted portions of an audio signal, for example, due to faulty signal transmission or recording errors. The goal of audio inpainting is therefore to fill in the gaps in the audio signal transparently, making the reconstructed portions indistinguishable from the original content and avoiding the introduction of audible distortions or alterations. Many techniques have been proposed to solve the problem of audio inpainting, generally by analyzing the temporal and spectral information surrounding each missing portion of the audio signal in question. Conventional methods use statistical models or digital signal processing algorithms to predict and synthesize the damaged sections.Recent solutions, on the other hand, leverage deep learning models, thanks to the growing trend of leveraging data-driven methods in the context of audio restoration. Diffusion-based audio inpainting is a category of inpainting in which learning is based on a diffusion model. Diffusion models can be very useful for processing images or audio signals, as they offer a flexible and efficient way to transform data. Diffusion models are based on variational inference, which is a form of machine learning that allows machines to learn not only the relationships between different datasets but also their hidden structure. The advantages of diffusion models for image generation are that they can be used to produce highly realistic images from textual descriptions.The article Eloi Moliner: "Diffusion-Based Audio Inpainting", arXiv.org, September 14, 2023 (2023-09-14), pages 1-15, DOI: 10.48550 / arxiv.2305.15266 (Retrieved from the Internet: URL:htps: / / arxiv.org / pdf / 2305.15266. pdf) describes a suitable example of broadcast-based audio inpainting.

[0015] In an advantageous embodiment, the first graphical representation of the input audio signal (A) and the second graphical representation of the output audio signal (A 1 ) are spectrograms, preferably mel-scale spectrograms.

[0016] In one embodiment, the filling grains (G'o, G'i, ...) are generated directly on the waveform or on an encoded version of the waveform, for example with codecs. Advantageously, the preferred audio inpainting technique consists of representing the audio signal in the form of a spectrogram, advantageously a spectrogram at the mel scale. According to an advantageous variant of the invention, the inpainting operations are carried out on a visual representation of the input audio signal (A).

[0017] A spectrogram is a visual representation of sound frequency versus time. It displays the intensity (or power) of different frequencies present in the sound signal over time. The Mel scale is an auditory perception scale that aims to mimic how the human ear perceives sounds. Unlike a scale linear frequency, the mel scale is constructed in such a way that equal distances on the scale represent equal perceptual distances for the human ear. The techniques for generating such a spectrogram are well known. For example, one can generate such a spectrogram of an audio extract by applying the well-known "stft" (Short-Time Fourier Transform) algorithm to this extract with a finite number of frequencies. Then, the power scale obtained for these frequencies is converted to the mel scale by a non-linear operation consisting of applying specific superimposed triangular filters on these frequencies. An important property of the mel scale spectrogram is that it is possible to reconstruct the original signal from it with differences practically imperceptible to the human ear.

[0018] Using the audio inpainting technique allows the generation in step d) of a filling grain which is consistent with the grains which surround it and which correspond to sequences of the original audio signal.

[0019] In reality, the signal may suffer some losses because on the one hand, there is a notion of audio phase which is lost during the inverse operation of the stft algorithm and on the other hand, the operation of applying the superimposed triangular filters is not totally reversible. Nevertheless, these losses, even combined, are relatively low and barely perceptible to a human ear so they are acceptable in view of the desired objective.

[0020] The parameters applied to obtain the mel-scale spectrogram (mel-scale resolution n_mel and number of frequencies chosen for the stft algorithm n_fft) can also help mitigate this problem. Thus, for audio extracts of 6 seconds or less, when the parameter n_mel > n_fft / 8, the reconstruction of the audio signal from the mel-scale spectrogram shows almost no anomalies. Preferably, the parameters n_mel > n_fft / 2 are used. Even more preferably, n_mel and n_fft are equal to 1024.

[0021] Advantageously, the slow motion rate is greater than 25%, and generally between 25 and 90% of the initial speed. Indeed, a slow motion rate of less than 25% would mean having to generate 75% of the audio signal and could cause perception problems.

[0022] The grains corresponding to the input signal can have the same or different lengths.

[0023] According to one embodiment, the method defined above is carried out on sequences of the input audio signal less than or equal to 15 seconds, preferably less than or equal to 6 seconds.

[0024] As indicated above, the inventors have found that the operations of generating the spectrogram of the input audio signal and converting the spectrogram corresponding to the output audio signal are not necessarily linear. As a consequence, amplitude differences can be observed between the input audio signals. and output so that the difference between the original grains and the generated grains can be perceived. In a particularly advantageous manner, the method according to the invention comprises a step in which the average amplitude of the grains corresponding to the input audio signal IJnit surrounding a generated grain is calculated and this average amplitude is compared with the average amplitude of the generated audio signal l_gen. The average amplitude of the generated grain is then weighted by a factor l_init / l_gen so that it corresponds to the average amplitude of the grains surrounding the generated grain. The average amplitude can be easily calculated by taking the average of the absolute value of the waveform.

[0025] Preferably, the input audio signal is cut into grains longer than 1 second, preferably longer than 1.5 seconds. Indeed, it has been observed that if the grains corresponding to the input audio signal are too short, there is a risk of generating filler grains whose sound ambiance is quite far from the ambiance of the original audio signal.

[0026] An audio signal corresponding to a sporting event typically includes two types of sounds: the first are so-called "ambient" sounds, which represent more or less random and chaotic noises, shouts or chants without direct relation to the action on the field. The second, which can be called "Specific Sound Elements" (ESS), are sounds linked to the course of the action such as whistles from the referee, or the sounds of kicks or rackets hitting the ball. In motor sports, this can be, for example, the start procedure buzzer. This sound signal accompanies each step of the start sequence, in particular when the red lights gradually come on and go off simultaneously to start the race. This buzzer is an FIA regulatory element, used to alert teams and drivers of the precise course of the start.During racing, it can be the whir of turbochargers, the squeal of tires accompanying sudden acceleration or braking. During pit stops, it can be the sudden roar of engines, the rapid clicking of pneumatic guns loosening and tightening wheel nuts. Contact between two cars can also cause a sharp sound of breaking carbon fiber, sometimes followed by the shrill sound of a yellow flag announced on the radio. Finally, the explosion of an engine at the end of its life is accompanied by a characteristic dull thud. Several problems can arise with these ESS. A first problem is that LESS could be misplaced temporally and therefore be offset from the video signal, instantly causing an unpleasant perception of temporal inconsistency.Furthermore, when dividing the original audio signal into grains, it may happen that an ESS ends up exactly at the interface between two successive grains. Applying the inpainting technique to this type of configuration produces noticeable inconsistencies. For example, the short whistle corresponding to a very specific game action can either be. duplicated (a first whistle at the end of the first grain and a second whistle at the beginning of the second grain) or be stretched across the entire grain generated between the two original grains.

[0027] So the inventors tried to find a solution to these additional problems.

[0028] To overcome the first problem, it is possible to train a sound detector / localizer system separately. Such a sound detector / localizer system is quite standard and can be easily trained using known methods. For example, the article "Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019" by Archontis Politis, Annamaria Mesaros, Sharath Adavanne, Toni Heittola, and Tuomas Virtanen in arXiv:2009.02792v2 (https: / / arxiv.org / abs / 2009.Q2792) describes many methods to achieve this. The sound detector / localizer system can be dedicated to a single sound (in which case multiple systems need to be trained to detect, for example, whistles and kicks). Alternatively, the system can be capable of detecting and localizing a range of different sounds.

[0029] In both cases, the sound detector / locator system provides a list [PESSL PESS2, ■■■ PESSnl of time positions where the ESSs that we wish to find in the output audio signal (A 1 ) at the correct time position, i.e. at the time position corresponding to that of the specific event visible in the slow motion video signal (V r ).

[0030] To get the sounds at the right time in slow motion, we modify step b) of the method as follows: b) division of the spectrogram representing the input audio signal (A) into a set of grains (GQ, GI, G2, ...) each having a length (LQ, LI, L2, ...) and a temporal position (PQ, Pl, P2, ■■■), in which the grains (GQ, GI, G2, ...) are constituted in such a way as to include at most one ESS. This method already makes it possible to find the ESS at a temporal position which corresponds approximately to that of the specific event visible in the slow motion video signal (V r ). The difference will be little, if at all, perceptible to the viewer.

[0031] Preferably, the length of a grain containing an ESS is less than 300 ms, preferably less than or equal to 100 ms. Thus, the error in the temporal position will be extremely small and undetectable to the human ear.

[0032] Another problem that may arise is related to the probabilistic and therefore random nature of the diffusion process. Indeed, for a given inpainting configuration, between two successive grains (GQ, G ) the method can generate several different filling grain signals (Go'). As the neural network is trained on whole match sequences, containing both sequences where the crowd is calm, and sequences where the crowd is more excited and, for example, emits shouts, the generation process can randomly generate fill grains with a calmer crowd soundscape or with a more excited crowd soundscape.

[0033] To some extent, the presence of the original grains (GQ, G ) in the inpainting limits this effect and at least partly conditions the filling grain (GQ 1) to resemble the original grains (especially the sound level), but experience shows that there are often outliers, for example where an excited crowd soundscape is generated between two grains (GQ, G ) containing a quiet crowd.

[0034] To avoid this type of situation, according to an advantageous embodiment, an advanced diffusion generation method called conditional generation is used. This involves augmenting the neural network in such a way that it takes an additional vector as input called encoding, in addition to the image that undergoes the inverse diffusion process. This encoding thus contains the data with which we want to condition the generation of images.

[0035] To condition on the sound level, we use here an encoding representing the sound level of the element to be generated. In the case of a 6-second window for example with mel-spectrograms of size 256 x 1024, we can calculate such an encoding by calculating the average of the spectrogram on the scale of the mels on the y axis, which gives in this case a vector of size 256 then apply a series of filters. We can use for example, alone or in combination, a mean filter (i.e. a filter that converts the value of a point into the average of the values ​​surrounding this point), a median filter (a median filter is a non-linear digital filter, often used for noise reduction. Noise reduction is a classic preprocessing step aimed at improving the results of future processing (edge ​​detection for example).The median filter technique is widely used in digital image processing because it allows, under certain conditions, to reduce noise while preserving the contours of the image), a subsampling filter (namely a filter reducing the size of the signal by a factor F by taking a point every F in the original signal) or a filter for normalizing the values ​​so that it ends up between 0 and 1. For example, with a subsampling of 4 we have a vector of size 64 of values ​​between 0 and 1, which represents the average sound level of a spectrogram on the mel scale.

[0036] Fig. 3 shows a spectrogram on the left side of the mel scale and the corresponding encoding on the right side. Both the spectrogram and the encoding vector show that the crowd's soundscape is initially rather calm and then becomes more excited.

[0037] This vector encoding is used in two ways.

[0038] For training, we pre-compute the encodings for each mel-scale spectrogram in the dataset, and train the neural network with the combination of encodings and mel-scale spectrograms.

[0039] For inpainting with two-grain grains (Go, G ), the method to use conditional generation is as follows: - we calculate the encoding of the total audio extract with the filters described above; - we identify the parts of the encoding corresponding to the descriptions of the sound levels of the grains (Go, Gi); - we create an encoding for the filling grain (GQ 1 ) WHERE we use as values ​​those of the grains (GQ, G ) for the limits of the filling grain (GQ 1 ) (these limits also correspond respectively to the end and the beginning of the adjacent grains (GQ, G )) and where we interpolate the value of the encoding between the limits of the filling grain (GQ1 ). This can be an algorithmic interpolation, advantageously a linear interpolation.

[0040] Fig. 4 shows an example of this interpolation and the generation of the grain filling encoding vector (GQ 1 ). Once the encoding vector is generated, the inpainting is then done with the same diffusion algorithm, but with this vector as an additional input to the neural network. Thus, we directly control the level of the crowd's sound ambiance in the filling grain (GQ 1 ) generated between the grains (GQ, G ).

[0041] The present invention can be implemented on a monophonic or stereophonic audio signal.

[0042] Example of realization

[0043] An exemplary embodiment of the invention will now be described by means of figures which have no other purpose than to illustrate the present invention. These figures schematically represent:

[0044] Fig. 1a: The combination of input audio and video signals.

[0045] Fig. lb: The division of the original audio sequence into grains

[0046] Fig. 1: Inserting grains into the output audio signal (A 1 )

[0047] Fig. ld: Generation of filling grains (Go', Gi')

[0048] Fig. 2: Splitting the original audio sequence into grains with specific sound events

[0049] Fig. 3: A mel spectrogram and the corresponding encoding vector

[0050] Fig. 4, the generation of the encoding vector of a filling grain (Go').

[0051] Figs. 1a to 1d schematically show steps of the method according to the invention for generating an output audio signal A' intended to accompany a slow-motion video signal V. r. Fig. 1a shows the combination of the input audio A and video V signals. Fig. 1b shows step b) of the method according to the invention in which the input audio signal is divided into a set of grains Go, Gi, G2, each having a length LQ, LI, L2, ... and a time position PQ, Pl, P2, ■■■ The grains Go, Gi, G2, can be selected randomly from the input audio signal A, be selected by dividing the input audio signal into a plurality of intervals of the same length or selected to contain only one ESS per grain as can be seen in Fig. 2. The length of the grains GQ, GI, G2, can be the same or different for all grains. Fig. 1c illustrates the insertion of the grains GQ, GI, G2, into a draft of the output audio signal A' at a time position P'o, P'i, P'2, ■■■ Which corresponds to the original time position of the grains GQ, GI, G2, corrected by a factor depending on the speed of the slowed-down sequence. For example, if we decided to broadcast a video sequence slowed down by a factor of 2 (broadcast at a speed of 50% of the original speed), a grain positioned temporally at 2 seconds from the start of the sequence must end up at 4 seconds from the start of the sequence. The temporal position of the grain is based on the start of the grain. Fig. ld illustrates step d) of the method during which filling grains G'o, G'i, ... are generated.intended to fill the spaces between the grains GQ, GI, G2, of the output audio signal A' before inserting them into the spaces between the grains GQ, GI, G2, of the output audio signal A'. According to the invention, the filling grains G'o, G'i, ... intended to fill the spaces between the grains GQ, GI, G2, of the output audio signal A' are generated by audio inpainting and are preferably generated by diffusion-based audio inpainting.

[0052] The method of the invention may operate by performing the steps described above directly on the waveform or on an encoded version of the waveform, for example with codecs. However, advantageously, it is preferred to perform these steps on a graphical representation of the audio signal, for example in the form of a spectrogram, advantageously a mel-scale spectrogram. In this case, the method comprises a prior step (not shown) of generating a spectrogram, preferably a mel-scale spectrogram, from the input audio signal (A) as well as a final step of converting the spectrogram representing the output audio signal (A 1 ) by the reverse operation of the previous step.

[0053] Figure reference A Input audio signal A' Output audio signal V Input video signal V Slow motion video signal GQ, Gi, G2, Input audio signal grains G'o, G'i, ... Fill grains ESS Specific sound event.

Claims

Claims 1. Method for generating an output audio signal (A 1 ) intended to accompany a slow motion video signal (V r ) comprising the following steps performed in order: a) generating a first graphical representation of the input audio signal (A), b) dividing the first graphical representation of the input audio signal (A) into a set of grains (GQ, GI, G2, ...) each having a length (LQ, LI, 1-2, ...) and a time position (PQ, Pl, P2, ■■■), c) inserting the grains (GQ, GI, G2, ...) into a second graphical representation representing the output audio signal (A 1) at a time position (P'o, P'i, P'2, ■■■) which corresponds to the original time position of the grains (GQ, GI, G2, ...) corrected by a factor depending on the speed of the slowed down sequence, d) generation by audio inpainting based on the diffusion of filler grains (G'o, G'i, ...) intended to fill the spaces between the grains (GQ, GI, G2, ...) of the second graphic representation representing the output audio signal (A 1 ) and e) inserting the filler grains (G'o, G'i, ...) into the spaces between the grains (GQ, GI, G2, ...) of the second graphical representation representing the output audio signal (A 1 ), and f) converting the second graphical representation representing the output audio signal (A 1 ) by the reverse operation of step a).

2. A method according to claim 1 wherein the first graphical representation of the input audio signal (A) and the second graphical representation of the output audio signal (A 1 ) are spectrograms.

3. Method according to claim 2 wherein the first graphical representation of the input audio signal (A) and the second graphical representation of the output audio signal (A 1 ) are spectrograms at the mel scale.

4. Method according to claim 3 in which the mel-scale spectrogram is generated by applying the "stft" (Short-Time Fourier Transform) algorithm to the input audio signal, preferably by fixing the resolution of the mel scale (n_mel) and the number of frequencies chosen for the stft algorithm (n_fft) such that n_mel > n_fft / 8.

5. Method according to claim 4 in which n_mel > n_fft / 2.

6. Method according to any one of claims 4 or 5 wherein after step d) of generating the filling grains, for each trio of consecutive grains (GQ, G'O, GI, ...) of the output audio signal dl) we calculate the average amplitude (IJnit) of the grains (Go, G ) surrounding the generated filling grain (G'o); d2) we calculate the average amplitude (l_gen) of the generated filling grain (G'o) and d3) we reduce the average amplitude of the generated filling grain (G'o) to the average amplitude (IJnit) of the grains (Go, G ) by applying a factor I Jnit / I_gen to the amplitude of the generated filling grain (G'o).

7. Method according to any one of the preceding claims in which the grains (Go, G, G2, ...) are constituted so as to include at most one specific sound element, the specific sound element being defined as a specific sound linked to the course of the action.

8. Method according to claim 7 in which the length (LQ, LI, 1-2, ...) of a grain (Go, Gi, G2, ■••) containing a specific sound element is less than 300 ms and preferably less than or equal to 100 ms.

9. Method according to any one of the preceding claims in which, for each generated filling grain (G'o) intended to fill the space between two successive grains (Go, G ) of the input audio signal (A), - we generate a vector called encoding which represents the average sound level of the spectrogram at the scale of the mels of the generated filling grain (G'o), - we identify the parts of the encoding corresponding to the descriptions of the sound levels of the grains (Go, G); - we create an encoding for the filling grain (Go') where we use as values ​​those of the grains (Go, G) for the limits of the filling grain (Go') and where we interpolate the value of the encoding vector between the limits of the filling grain (Go').

10. Method according to claim 9 in which the value of the encoding vector is linearly interpolated.

Images