Method for generating an audio signal

Abstract

The invention relates to a method for generating an audio signal accompanying a slow-motion video signal, comprising: generating a first graphical representation of the input audio signal, dividing the first graphical representation into a set of grains, wherein each has a length and a temporal position, inserting the grains into a second graphical representation representing the output audio signal at a temporal position corresponding to the original temporal position of the grains, corrected by a factor dependent on the slow-motion speed, generating, by audio inpainting based on diffusion, the filling grains intended to fill the spaces between the grains of the second representation, inserting the filling grains into the spaces between the grains, converting the second representation by applying the inverse of the first step.

Description

[0001] Description

[0002] The present invention relates to a method for generating an audio signal intended to accompany a slow-motion video signal of an action.

[0003] It involves modifying the audio signal (in practice, an artificial audio signal is generated) in order to make it correspond to the duration of the slow-motion video sequence. Such a method is applicable, for example, in the live, deferred or slightly deferred broadcasting of team or individual sporting events like football, basketball or tennis matches. The present invention is particularly suitable for representing motor sport sequences. However, it should be noted that the method can be applied to any video sequence representing an action.

[0004] Description of the prior art

[0005] The first solution that comes to mind for addressing this problem involves modifying the playback speed of the audio file so as to make the duration of its playback correspond to that of the slow-motion video signal. However, the sound produced by this solution is of very poor quality as soon as the frequency of the audio signal is altered by decreasing its speed and is not realistic, or is even unpleasant for the viewer.

[0006] More realistic solutions have already been sought, approaching the soundscape of the original audio signal.

[0007] For example, document EP-A1-2509073 has proposed a method for stretching the audio signal involving dividing the original audio signal into a series of grains, inserting these grains into the generated audio signal at a distance from one another and filling the spaces between the original grains of the rendered signal with an audio texture. In terms of the audio texture, a particular proposal involves repeating grains of the audio signal that are close (disposed before or after) but not directly adjacent to the space to be filled. These grains must have been previously stored in a database, which does not allow rapid processing of the audio signal such that the use of this solution is excluded for a live or slightly deferred broadcast event.

[0008] Another solution has been proposed in document EP-A1-2960904. Like the preceding document, document EP-A1-2960904 proposes dividing the original audio signal into a series of grains and inserting these grains into the generated audio signal at a distance from one another. Silences are then inserted into the spaces between the original grains of the rendered signal. This method is not suitable for broadcasting sporting events during which a fairly large and permanent soundscape prevails such that the silences inserted into the rendered audio signal are easily identified by the viewer and the result is not satisfactory.

[0009] Therefore, it would be worthwhile providing a solution for generating an audio signal intended to accompany a video signal broadcast in slow-motion that approximates the soundscape of the original audio signal that can be applied to any type of broadcast (deferred, slightly deferred or live).

[0010] Disclosure of the invention

[0011] To this end, a method is proposed for generating an output audio signal (A') intended to accompany a video signal (Vr) read in slow-motion, comprising the following steps carried out in order: a) generating a first graphical representation of the input audio signal (A); b) dividing the first graphical representation of the input audio signal (A) into a set of grains (Gq, Gi, G2, ...) each having a length (Lq, Li, L2, ...) and a temporal position (Po, Pi, P2, •••); c) inserting the grains (Gq, Gi, G2, ...) into a second graphical representation representing the output audio signal (A') at a temporal position (P'o, P'1, P'2, •) that corresponds to the original temporal position of the grains (Gq, Gi, G2, •) corrected by a factor dependent on the speed of the slow-motion sequence; d) generating, by diffusion-based audio inpainting, filling grains (G'q, G'i, •) intended to fill the spaces between the grains (Gq, Gi, G2, •) of the second graphical representation representing the output audio signal (A'); and e) inserting the filling grains (G'q, G'i, •) into the spaces between the grains (Gq, Gi, G2, •) of the second graphical representation representing the output audio signal (A'); and f) converting the second graphical representation representing the output audio signal (A') by applying the inverse operation of step a). [qqi2] It should be noted that in steps b) and c) the temporal position (Pq, Pi, P2, •) or (P'q, P'i, P'2, •) that corresponds to the temporal position of the grains (Gq, Gi, G2, •), respectively in the original (A) or the generated (A') audio signal, is based on the temporal position of the start of the grains (Gq, Gi, G2, •). In another embodiment, the temporal position of the grains (Gq, Gi, G2, •) is based on the centers of the grains (Gq, Gi, G2, •). [qqi3] It should be noted that the method according to the invention does not involve processing the video signal. This allows the processing to be accelerated and a result to be provided that is compatible with the desired goals. [qqi4] Inpainting is the name given to the techniques for reconstructing or generating missing parts of an image. These techniques can be implemented manually or using digital tools, with or without any interactions with the user. The digital models for implementing inpainting can use artificial intelligence, for example, deep learning. The technique of audio inpainting (or sometimes also called audio interpolation) is a technique that is mainly used for the restoration or reconstruction of missing or altered portions of an audio signal, for example, as a result of faulty transmission of the signal or an error when recording. Therefore, the aim of audio inpainting is to transparently fill any gaps in the audio signal, while making the reconstructed parts indistinguishable from the original content and avoiding the introduction of audible distortions or alterations. Numerous techniques have been proposed to address the problem of audio inpainting, generally by analyzing the temporal and spectral information surrounding each missing part of the considered audio signal. Conventional methods use statistical models or digital signal processing algorithms to predict and synthesize damaged sections. By contrast, recent solutions take advantage of deep learning models, by virtue of the increasing trend to exploit methods based on data within the context of audio restoration. Diffusion-based audio inpainting is an inpainting category where the learning is based on a diffusion model. The diffusion models can be very useful for processing images or audio signals, because they offer a flexible and effective way of converting the data. The diffusion models are based on variational inference, which is a form of machine learning that allows the machines to learn not only the relationships between different sets of data, but also their hidden structure. The advantages of diffusion models for generating images are that they can be used to produce highly realistic images based on textual descriptions. The article by Eloi Moliner entitled: “Diffusion-Based Audio Inpainting”, arXiv.org, 14 September 2023 (2023-09-14), pages 1-15, DOI: 10.48550 / arxiv.2305.15266 (Retrieved from the internet: URL:https: / / arxiv.org / pdf / 2305.15266.pdf) describes a suitable example of diffusion-based audio inpainting.

[0015] In one advantageous embodiment, the first graphical representation of the input audio signal (A) and the second graphical representation of the output audio signal (A') are spectrograms, preferably mel-scale spectrograms.

[0016] In one embodiment, the filling grains (G'o, G'1,_) are generated directly on the waveform or on an encoded version of the waveform, for example, with codecs. Advantageously, the preferred audio inpainting technique involves representing the audio signal in the form of a spectrogram, advantageously, a mel-scale spectrogram. According to an advantageous variant of the invention, the inpainting operations are performed on a visual representation of the input audio signal (A).

[0017] The spectrogram is a visual representation of the frequency of the sounds with respect to time. It displays the intensity (or the power) of the various frequencies present in the sound signal over time. The mel-scale is an auditory perception scale that aims to imitate how the human ear perceives sounds. Contrary to a linear frequency scale, the mel-scale is constructed in such a way that the equal distances on the scale represent equal distances perceptible to the human ear. The techniques for generating such a spectrogram are well known. For example, such a spectrogram of an audio extract can be generated by applying the well-known “stft” (Short-Time Fourier Transform) algorithm to this extract with a finite number of frequencies. Then, the power scale obtained for these frequencies is converted on the mel-scale by a nonlinear operation involving applying specific superimposed triangular filters to these frequencies. A significant property of the mel-scale spectrogram is that the original signal can be reconstructed therefrom with differences that are practically imperceptible to the human ear.

[0018] Using the audio inpainting technique allows, in step d), a filling grain to be generated that is consistent with the grains that surround it and that correspond to sequences of the original audio signal.

[0019] In reality, the signal can experience some losses because, on the one hand, there is a notion of audio phase that is lost during the inverse operation of the stft algorithm and, on the other hand, the operation of applying superimposed triangular filters is not completely reversible. Nevertheless, these losses, even combined, are relatively small and hardly perceptible to the human ear, so that they are acceptable in light of the desired objective.

[0020] The parameters applied for obtaining the mel-scale spectrogram (resolution of the mel-scale n_mel and number of frequencies selected for the stft algorithm n_fft) can also allow this problem to be mitigated. Thus, for audio extracts of 6 seconds or less, when the parameter n_mel > n_fft / 8, the reconstruction of the audio signal from the mel-scale spectrogram exhibits practically no anomalies. Preferably, the parameters n_mel > n_fft / 2 are used. Still more preferably, n_mel and n_fft are equal to 1024.

[0021] Advantageously, the slow-motion rate is greater than 25 %, and generally ranges between 25 and 90 % of the initial speed. Indeed, a slow-motion rate of less than 25 % would involve having to generate 75 % of the audio signal and could raise perception problems.

[0022] The grains corresponding to the input signal can have identical or different lengths.

[0023] According to one embodiment, the method defined above is carried out on sequences of the input audio signal that are less than or equal to 15 seconds, preferably less than or equal to 6 seconds.

[0024] As indicated above, the inventors have observed that the operations for generating the spectrogram of the input audio signal and for converting the spectrogram corresponding to the output audio signal are not necessarily linear. Consequently, differences in amplitudes can be observed between the input and output audio signals such that the difference between the original grains and the generated grains can be perceived. In a particularly advantageous manner, the method according to the invention comprises a step in which the average amplitude of the grains corresponding to the input audio signal I_init surrounding a generated grain is computed and this average amplitude is compared with the average amplitude of the generated audio signal l_gen. The average amplitude of the generated grain is then weighted by a factor l_init / l_gen so that it corresponds to the average amplitude of the grains surrounding the generated grain. The average amplitude can be easily computed by taking the average of the absolute value of the waveform.

[0025] Preferably, the input audio signal is divided into grains with a length of more than 1 second, preferably of more than 1.5 seconds. Indeed, it has been observed that if the grains corresponding to the input audio signal are too short, there is a risk of generating filling grains whose soundscape is far enough away from the soundscape of the original audio signal.

[0026] An audio signal corresponding to a sporting event typically includes two types of sounds: the first are “ambient” sounds, which represent more or less random and chaotic noises, screams or songs not directly related to the action on the field. The second, which can be called “Specific Sound Elements” (SSEs), are sounds related to the flow of the action, such as whistle blowing by the referee, or the sounds of kicks or rackets hitting a ball. In motor sports, it can be, for example, the starting procedure buzzer. This sound signal accompanies each step of the starting sequence, notably when the red lights turn on gradually and turn off simultaneously to start the race. This buzzer is a regulatory element of the FIA, used to notify the teams and drivers of the precise starting sequence. During the race, the signal may be the whining of the turbochargers, or the squeal of tires accompanying an acceleration or sudden braking. During pit stops, it may be the sudden roar of the engines, the rapid clinking of the pneumatic guns loosening and tightening the nuts of the wheels. Contact between two cars may also cause a snapping sound of broken carbon fiber, sometimes followed by the shrill sound of a yellow flag announced over the radio. Finally, the explosion of an engine at the end of its life is accompanied by a characteristic muffled clattering sound. Several problems can arise with respect to these SSEs. A first problem is that the SSEs could be poorly temporally placed and therefore may be offset with respect to the video signal, instantaneously causing an unpleasant perception of temporal inconsistency. Furthermore, when dividing the original audio signal into grains, an SSE may be precisely located at the interface between two successive grains. Applying the inpainting technique to this type of configuration produces perceptible inconsistencies. For example, a short whistle blow corresponding to a very specific action in a game either can be duplicated (a first whistle blow at the end of the first grain and a second whistle blow at the start of the second grain) or can be stretched across the entire grain generated between the two original grains.

[0027] Therefore, the inventors have attempted to find a solution to these additional problems.

[0028] In order to overcome the first problem, a sound detector / locator system can be trained separately. Such a sound detector / locator system is fairly conventional and can be easily trained using known methods. For example, the article entitled, “Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019” by Archontis Politis, Annamaria Mesaros, Sharath Adavanne, Toni Heittola and Tuomas Virtanen in arXiv:2009.02792v2 (https: / / arxiv.org / abs / 2009.02792) describes numerous methods for achieving this. The sound detector / locator system can be dedicated to a single sound (in which case several systems must be trained to detect whistle blows and kicks, for example). Alternatively, the system can be capable of detecting and locating a series of different sounds.

[0029] In both cases, the sound detector / locator system provides a list [Pessi, PESS2, -PESSn] of temporal positions where the SSEs are located that are intended to be found in the output audio signal (A') at the correct temporal position, i.e., at the temporal position corresponding to that of the specific event visible in the slow-motion video signal (Vr).

[0030] In order to have the sounds at the correct moment in the slow-motion, step b) of the method is modified as follows: b) dividing the spectrogram representing the input audio signal (A) into a set of grains (G0, G1, G2, -) each having a length (L0, L1, L2, -) and a temporal position (P0, P1, P2, -), in which the grains (G0, G1, G2, -) are formed so as to include at most one ssE. This method already allows the ssE to be found in a temporal position that roughly corresponds to that of the specific event visible in the slow-motion video signal (Vr). The difference will be hardly, or even not, perceptible by the viewer.

[0031] Preferably, the length of a grain containing an ssE is less than 300 ms, preferably less than or equal to 100 ms. Thus, the error on the temporal position will be extremely small and undetectable to the human ear.

[0032] Another problem that can arise is related to the probabilistic and therefore random nature of the diffusion process. Indeed, for a given inpainting configuration, between two successive grains (G0, G1), the method can generate several different filling grain signals (G'0). since the neural network is trained on complete match sequences, equally containing sequences where the crowd is calm, and sequences where the crowd is more excited and, for example, screams, the generation process can randomly generate filling grains with a calmer crowd soundscape or with a more agitated crowd soundscape.

[0033] To a certain extent, the presence of the original grains (G0, G1) in the inpainting limits this effect and at least partially influences the filling grain (G'0) to resemble the original grains (in particular the sound level), but experience has shown that there are often aberrant cases, for example, where an excited crowd soundscape is generated between two grains (G0, G1) containing a calm crowd.

[0034] In order to avoid this type of situation, according to an advantageous embodiment, an advanced diffusion-generation method, called conditional generation, is used. This involves increasing the neural network in such a way that it takes an additional vector, called encoding vector, as input in addition to the image that undergoes the inverse diffusion process. This encoding thus contains the data to be used to condition the generation of images.

[0035] In order to condition the sound level, encoding representing the sound level of the element to be generated is used in this case. In the case of a 6 second window, for example, with 256 x 1024 mel-spectrograms, such encoding can be computed by computing the average of the mel-scale spectrogram on the y-axis, which in this case yields a 256-sized vector and then a series of filters can be applied. It is possible, for example, to use, alone or in combination, an average filter (namely, a filter that converts the value of a point into the average of the values surrounding this point), a median filter (a median filter is a non-linear digital filter, often used for noise reduction. Noise reduction is a conventional preprocessing step aimed at improving the results of future processing operations (detection of edges, for example). The median filter technique is widely used in digital image processing because under certain conditions it allows noise to be reduced while preserving the contours of the image), a subsampling filter (namely, a filter reducing the size of the signal by a factor F by taking one point every F in the original signal) or a filter for normalizing the values so that it is between 0 and 1. For example, subsampling of 4 yields a 64-sized vector with values between 0 and 1, which represents the average sound level of a mel-scale spectrogram.

[0036] The left-hand portion of figure 3 shows a mel-scale spectrogram and the right-hand portion shows the corresponding encoding. Equally for the spectrogram and the encoding vector, it can be seen that the soundscape of the crowd is rather calm at the beginning and then excited.

[0037] This encoding vector is used in two ways.

[0038] For training, the encoding is pre-computed for each mel-scale spectrogram of the dataset, and the neural network is trained with the combination of the encoding and the mel-scale spectrograms.

[0039] For inpainting with grains of two grains (G0, G1), the method for using the conditional generation involves the following: - computing the encoding of the total audio extract with the filters described above; - identifying the parts of the encoding corresponding to the descriptions of the sound levels of the grains (G0, G1); - creating encoding for the filling grain (G'0), wherein the values of the grains (G0, G1) are used as values for the limits of the filling grain (G'0) (these limits also respectively correspond to the end and to the beginning of the adjacent grains (G0, G1)) and wherein the value of the encoding is interpolated between the limits of the filling grain (G'0). It can involve an algorithmic interpolation, advantageously a linear interpolation.

[0040] Figure 4 shows an example of this interpolation and the generation of the encoding vector of the filling grain (G'0). Once the encoding vector has been generated, the inpainting is then completed with the same diffusion algorithm, but with this additional vector at the input of the neural network. Thus, the level of the soundscape of the crowd is directly controlled in the filling grain (G'0) generated between the grains (G0, G1).

[0041] The present invention can be implemented on a monophonic or stereophonic audio signal.

[0042] Embodiment

[0043] An embodiment of the invention will now be described by means of the figures, which are only intended to illustrate the present invention. These figures schematically show:

[0044] Figure 1a: the combination of the input audio and video signals;

[0045] Figure 1b: the division of the original audio sequence into grains;

[0046] Figure 1c: the insertion of the grains into the output audio signal (A');

[0047] Figure 1d: the generation of the filling grains (G'0, G'1);

[0048] Figure 2: the division of the original audio sequence into grains with specific sound events;

[0049] Figure 3: a mel spectrogram and the corresponding encoding vector;

[0050] Figure 4: the generation of the encoding vector of a filling grain (G'0).

[0051] Figures 1a to 1d schematically show the steps of the method according to the invention for generating an output audio signal A' intended to accompany a slow-motion video signal Vr. Figure 1a shows the combination of the input audio A and video V signals. Figure 1b shows step b) of the method according to the invention, in which step the input audio signal is divided into a set of grains Go, Gi, G2, ..., each having a length Lo, Li, L2, ... and a temporal position Po, Pi, P2, . The grains Go, Gi, G2, . can be randomly selected in the input audio signal A, can be selected by dividing the input audio signal into a plurality of intervals of the same length or can be selected so as to contain only one SSE per grain, as can be seen in figure 2. The length of the grains Go, Gi, G2, . can be identical or different for all the grains. Figure ic illustrates the insertion of the grains Go, Gi, G2, . into a blank of the output audio signal A' at a temporal position P'o, P'i, P'2, . that corresponds to the original temporal position of the grains Go, Gi, G2, . corrected by a factor dependent on the speed of the slow-motion sequence. For example, when deciding to diffuse a slow-motion video sequence by a factor of 2 (diffusion at a speed of 5o % of the original speed), a grain temporally positioned 2 seconds from the beginning of the sequence must be within 4 seconds of the beginning of the sequence. The temporal position of the grain is based on the beginning of the grain. Figure id illustrates step d) of the method, during which step filling grains G'o, G'i, . are generated that are intended to fill the spaces between the grains Go, Gi, G2, . of the output audio signal A' before inserting them into the spaces between the grains Go, Gi, G2, . of the output audio signal A'. According to the invention, the filling grains G'o, G'i, . intended to fill the spaces between the grains Go, Gi, G2, . of the output audio signal A' are generated by audio inpainting and are preferably generated by diffusion-based audio inpainting. [oo52] The method of the invention can operate by carrying out the steps described above directly on the waveform or on an encoded version of the waveform, for example, with codecs. However, advantageously, these steps are preferably carried out on a graphical representation of the audio signal, for example, in the form of a spectrogram, advantageously, a mel-scale spectrogram. In this case, the method comprises a prior step (not shown) of generating a spectrogram, preferably a mel-scale spectrogram, from the input audio signal (A), as well as a final step of converting the spectrogram representing the output audio signal (A') by applying the inverse operation of the prior step.

[0053] Reference for the figures 5   A Input audio signal A' Output audio signal V Input video signal Vr Slow-motion video signal G0, Gi, G2, ... Grains of the input audio signal 10    G'0, G'1, . Filling grains SSE Specific Sound Event

Claims

1. A method for generating an output audio signal (A') intended to accompany a slow-motion video signal (Vr), comprising the following steps carried out in order:a) generating a first graphical representation of the input audio signal (A);b) dividing the first graphical representation of the input audio signal (A) into a set of grains (Go, Gi, G2, ...) each having a length (Lo, Li, L2, ...) and a temporal position (Po, Pi, P2,...);c) inserting the grains (Go, Gi, G2, ...) into a second graphical representation representing the output audio signal (A') at a temporal position (P'o, P'i, P'2, ...) that corresponds to the original temporal position of the grains (Go, Gi, G2, ...) corrected by a factor dependent on the speed of the slow-motion sequence;d) generating, by diffusion-based audio inpainting, filling grains (G'o, G'i, ...) intended to fill the spaces between the grains (Go, Gi, G2, ...) of the second graphical representation representing the output audio signal (A'); ande) inserting the filling grains (G'o, G'i, ...) into the spaces between the grains (Go, Gi, G2, ...) of the second graphical representation representing the output audio signal (A'); andf) converting the second graphical representation representing the output audio signal (A') by applying the inverse operation of step a).

2. The method as claimed in claim i, wherein the first graphical representation of the input audio signal (A) and the second graphical representation of the output audio signal (A') are spectrograms.

3. The method as claimed in claim 2, wherein the first graphical representation of the input audio signal (A) and the second graphical representation of the output audio signal (A') are mel-scale spectrograms.

4. The method as claimed in claim 3, wherein the mel-scale spectrogram is generated by applying the Short-Time Fourier Transform (stft) algorithm to the input audio signal, preferably by setting the resolution of the mel scale (n_mel) and the number of frequencies selected for the stft algorithm (n_fft), such that n_mel > n_fft / 8.

5. The method as claimed in claim 4, wherein n_mel > n_fft / 2.

6. The method as claimed in any one of claims 4 or 5, wherein after step d) of generatingthe filling grains,for each trio of consecutive grains (Go, G'o, Gi, ...) of the output audio signal, the following steps are carried out:d1) computing the average amplitude (I_init) of the grains (G0, G1) surrounding the generated filling grain (G'0);d2) computing the average amplitude (l_gen) of the generated filling grain (G'0); andd3) returning the average amplitude of the generated filling grain (G'0) to the average amplitude (I_init) of the grains (G0, G1) by applying a factor I_init / I_gen to the amplitude of the generated filling grain (G'0).

7. The method as claimed in any one of the preceding claims, wherein the grains (G0, G1, G2, ...) are formed so as to include at most one specific sound element, the specific sound element being defined as a specific sound related to the flow of the action.

8. The method as claimed in claim 7, wherein the length (Lq, Li, L2, ...) of a grain (Go, Gi, G2, ...) containing a specific sound element is less than 300 ms and preferably less than or equal to 1qq ms.

9. The method as claimed in any one of the preceding claims, wherein, for each generated filling grain (G'0) intended to fill the space between two successive grains (G0, G1) of the input audio signal (A),- an encoding vector is generated that represents the average sound level of the mel-scale spectrogram of the generated filling grain (G'0);- the parts of the encoding corresponding to the descriptions of the sound levels of the grains (G0, G1) are identified;- an encoding for the filling grain (G'0) is created,wherein the values of the grains (G0, G1) are used as values for the limits of the filling grain (G'0); andwherein the value of the encoding vector is interpolated between the limits of the filling grain (G'0).

10. The method as claimed in claim 9, wherein the value of the encoding vector is linearly interpolated.

Images