A method and device for generating sound effects based on visual semantics
By extracting visual features in the video and using parallelized non-autoregressive acoustic models to generate sound effects, the poor sound quality and synchronization problems are solved, high-quality and rapid training of sound effects are achieved, and the robustness of sound effects is improved.
Patent Information
- Application Number
- CN202111566564.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-20
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2041-12-20
AI Technical Summary
The existing sound effect generation technology has problems such as poor sound quality, difficulty in fully synchronizing the sound effects and video images, slow model training speed, easy overfitting and unruly model.
Using a sound effect generation method based on visual semantics, the visual features of color information and action information are extracted from silent videos, and the parallelized non-autoregressive acoustic model is used to convert them into acoustic features, and the sound waveform is restored through the vocoder. Multi-category training is carried out in combination with category embedding information to avoid prediction category errors.
It improves the sound quality and synchronization of sound effects generation, improves the training speed and robustness of model, and the generated sound effects are more realistic and accurate, avoiding prediction errors.
Smart Images
Figure CN114399984B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method and device for generating sound effects based on visual semantics, and belongs to the cross-field of digital media and human-computer interaction. Background Art
[0002] Sound effects refer to the effects created by sound, which are added to the soundtrack to enhance the realism or atmosphere of a scene. For a variety of digital media applications such as movies, games, and picture books, adding sound effects corresponding to the video content on the basis of the original sound and background music will greatly enhance the audience's immersive experience. Foley is one of the ways to create sound effects. It is usually created by foley designers using props in a silent environment. This process requires their professional knowledge to produce foley that is synchronized with the content appearing in the video. It not only relies on the designer's imagination, but is also limited by environmental conditions. The creation process is extremely time-consuming and laborious.
[0003] At present, most of the sound effects of film and television dramas are still completed according to the above method, and the automatic sound effect generation technology has not been widely used in the film and television industry. This is mainly because most of the existing automatic sound effect generation solutions have the following problems: (1) The audio quality generated by the existing sound effect generation technology solutions is still poor, and the audio quality still needs to be improved; (2) The existing sound effect generation technology solutions still have missing, redundant and time mismatching phenomena in the synchronization of the generated sound effects and video images.
[0004] From a research perspective, existing automatic sound effect generation work is all based on the LSTM (Long Short-Term Memory) autoregressive model, which has problems such as slow training speed, easy overfitting, and non-robust model. Summary of the invention
[0005] The main purpose of the present invention is to propose a sound effect generation method and device based on visual semantics to solve the problems of poor sound quality, difficulty in complete synchronization of sound effects and video images, slow model training speed, easy overfitting, and non-robust model in existing sound effect generation schemes.
[0006] In order to solve the above problems, the present invention proposes the following technical solutions:
[0007] A method for generating sound effects based on visual semantics comprises the following steps: S1, extracting visual features including color information and action information from a silent video to be simulated; S2, converting the visual features into acoustic features matching the visual features using a trained parallelized non-autoregressive acoustic model; S3, restoring the sound waveform from the acoustic features using a vocoder, thereby obtaining an audio file for the simulated sound.
[0008] Furthermore, step S1 includes: for each frame of the silent video, respectively extracting the color information and the motion information, and then splicing the color information and the motion information as the visual feature of the corresponding frame.
[0009] Furthermore, in step S1, a pre-trained ResNet-50 network is used as a task extractor to extract the color information and the action information respectively; wherein, the RGB three-channel image of the current frame is read as the three-channel input of the ResNet-50 network to obtain the color information; the current frame and the previous frame and the next frame of the current frame are extracted as grayscale images respectively as the three-channel input of the ResNet-50 network to obtain the continuous actions contained in the three consecutive frames, and extended to each frame to obtain the action information.
[0010] Furthermore, the parallelized non-autoregressive acoustic model includes an encoder and a decoder based on a Transformer model; wherein, the input of the encoder based on the Transformer model is the visual feature, the input of the decoder based on the Transformer model is the superposition of the output of the encoder, category embedding and position encoding, and the output of the decoder is the acoustic feature.
[0011] Furthermore, the acoustic feature adopts a linear spectrum or a Mel spectrum; the vocoder is a vocoder based on inverse Fourier transform or a vocoder based on a neural network.
[0012] To solve the above problems, the present invention proposes the following technical solution on the other hand:
[0013] A sound effect generation device based on visual semantics, comprising: a visual feature extraction model, receiving a silent video to be simulated as input, and being used to extract visual features including color information and action information from the silent video; a parallelized non-autoregressive acoustic model, connected to the output end of the visual feature extraction model, and being used to convert the visual features into acoustic features that match them; and a vocoder, connected to the output end of the parallelized non-autoregressive acoustic model, and being used to recover a sound waveform from the acoustic features to obtain an audio file for the simulated sound.
[0014] Furthermore, the visual feature extraction model is implemented using a pre-trained ResNet-50 network, which extracts the color information and the motion information respectively; and then forms the visual features of the corresponding frame by concatenating the color information and the motion information of each frame.
[0015] Furthermore, the parallelized non-autoregressive acoustic model includes an encoder and a decoder based on a Transformer model; wherein, the input of the encoder based on the Transformer model is the visual feature, the input of the decoder based on the Transformer model is the sum of the output of the encoder, category embedding and position encoding, and the output of the decoder is the acoustic feature.
[0016] Furthermore, the encoder includes a first linear layer and a first Transformer module, wherein the first linear layer receives the visual feature as input, and its output plus the position code is used as the input of the first Transformer module, and the output of the first Transformer module is the output of the encoder; the decoder includes a second Transformer module, a second linear layer and a post-processing network, wherein the input of the second Transformer module is the input of the decoder, the output of the second Transformer module is used as the input of the second linear layer, and the output of the second linear layer is used as the input of the post-processing network, and the output of the post-processing network, the output of the second linear layer and the category average spectrum are added as the output of the decoder; wherein the post-processing network is composed of multiple layers of convolutional layers and multiple layers of batch normalization layers.
[0017] Furthermore, the acoustic feature adopts a linear spectrum or a Mel spectrum; the vocoder is a vocoder based on inverse Fourier transform or a vocoder based on a neural network.
[0018] The beneficial effects of the technical solution of the present invention are: 1) The present invention adopts a parallelized non-autoregressive acoustic model, which can be parallelized, greatly improving the training speed. 2) The parallelized non-autoregressive acoustic model used in the present invention has a complex structure and many parameters. Compared with autoregressive models such as LSTM, it has a stronger learning ability, can better fit visual features and acoustic features, and synthesize better quality and more realistic sound effects. 3) The present invention uses category embedding information to implement multi-category training, and uses the labels of each category of data as a query table, mapped to the category embedding space, and expanded to the dimensions of the hidden state of the encoder and decoder, and added to the output of the encoder to provide category information. Compared with the existing model that predicts category information by itself, the present invention avoids predicting category errors and generating sound effects of the wrong category. Ultimately, the sound effect generation scheme of the present invention has the advantages of fast model training speed, better sound effect quality, and good model robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 is a schematic diagram of a sound effect generation process based on visual semantics according to an embodiment of the present invention;
[0020] Figure 2 is a practical application flow chart of a sound effect generation device based on visual semantics according to an embodiment of the present invention;
[0021] Figure 3 is a schematic diagram of a visual feature extraction model according to an embodiment of the present invention;
[0022] Figure 4 Schematic diagram of a non-autoregressive acoustic model based on Transformer according to an embodiment of the present invention;
[0023] Figure 5 It is a schematic diagram of the structure of the Transformer model;
[0024] Figure 6 It is a schematic diagram of the structure of the post-processing network of an embodiment of the present invention. DETAILED DESCRIPTION
[0025] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.
[0026] The embodiment of the present invention provides a sound effect generation method based on visual semantics, which is used to generate sound effects for video mimicry from a silent video. The embodiment of the present invention also provides a sound effect generation device based on visual semantics that is compatible with the sound effect generation method. Figure 1 This is a schematic diagram of the sound effect generation process based on visual semantics according to an embodiment of the present invention. Figure 1 , the sound effect generation method based on visual semantics, its steps mainly include: S1, extracting visual features including color information and action information from the silent video to be simulated; S2, using the trained parallelized non-autoregressive acoustic model to convert the visual features into acoustic features that match them; S3, using a voice coder to restore the sound waveform from the acoustic features, that is, to obtain an audio file for simulation. On this basis, the user only needs to synchronize the obtained audio file to the audio track of the original video to complete the simulation of the original video. In other words, the sound effect generation method of the embodiment of the present invention can realize the generation of a simulation file from a silent video based on visual semantics. Please continue to refer to Figure 1 The network architecture of the sound effect generation device based on visual semantics is mainly composed of three parts: a visual feature extraction model 10, a parallelized non-autoregressive acoustic model 20 and a vocoder 30; wherein the visual feature extraction model receives a silent video to be simulated as input, and can extract visual features including color information and action information from the silent video; the parallelized non-autoregressive acoustic model is connected to the output end of the visual feature extraction model, and can convert the visual features into acoustic features that match the feature content contained in the visual features; the vocoder is connected to the output end of the acoustic model, and can restore the sound waveform from the acoustic features to obtain an audio file for mimicking.
[0027] Please refer to Figure 1 and Figure 3 , the visual feature extraction model of the embodiment of the present invention is implemented by using the ResNet-50 network pre-trained in a large-scale image dataset such as ImageNet. As a commonly used task extractor in visual tasks, ResNet-50 has achieved very good results in basic visual tasks such as image classification. For the construction of visual features, the embodiment of the present invention uses two parts: action information based on action frames and color information based on RGB frames to characterize the content information in continuous video frames. First, use tools such as cv2 to read the complete image sequence of each frame of the video. For action information, the embodiment of the present invention extracts the current video frame and its previous frame and next frame as grayscale images as the three-channel input of the ResNet-50 network, obtains the continuous action contained in the three consecutive frames, and extends it to each frame. For color information, the embodiment of the present invention reads the RGB image of the current video frame as the three-channel input of the ResNet-50 network to obtain the color information of each frame. Finally, the obtained action information and color information of each frame are spliced as the final visual feature of each frame. It should be understood that for each frame, the color information and action information extracted by the RGB frame and the action frame through the ResNet-50 network have the same dimensions. For example, Figure 3 As shown in , for the color information and motion information extracted from the same frame, the feature dimensions are both 860×2048. After splicing, the final visual features of the frame are obtained. The length remains unchanged at 860 frames, while the dimension becomes the superposition of the two, that is, 4096.
[0028] The existing visual feature construction extracts the color of the first frame of video, while the visual feature extraction model proposed in the embodiment of the present invention extracts color at each time step (each frame), thereby introducing more information.
[0029] Please refer to Figure 1 and Figure 4 The parallelized non-autoregressive acoustic model of the embodiment of the present invention can be established on the basis of the Transformer model, and the encoder and decoder of the acoustic model are both implemented based on the Transformer model. Specifically, Figure 4As shown, the input of the acoustic model is the visual feature. The visual feature output by the visual feature extraction model first enters the linear layer (fully connected layer) 21 of the encoder to map the visual feature to the hidden dimension of the encoder, and then adds the position code as the input of the Transformer module 22 in the encoder. After the Transformer module 22, an intermediate representation encoded from the visual feature is learned. This intermediate representation learns the visual information contained in the input visual feature. The main function of the Transformer module 22 is to learn the visual information contained in the visual feature and re-encode the learned information into a new feature space. The encoded feature space is more suitable for learning by the subsequent decoder. The position code adopts the sine and cosine method. Category embedding is a table lookup operation of the data category, which is then expanded to the dimension of the encoder output and added to the output of the encoder (i.e., the output of the Transformer module 22) and the position code as the input of the decoder. In the decoder, the input first passes through the Transformer module 23 of the decoder, and then passes through another linear layer 24 to map the features to the dimension of acoustic features (i.e., the dimension of linear spectrum or Mel spectrum). The output of the linear layer 24 enters a post-processing network 25. The post-processing network 25 provides a residual connection method, that is, the output of the linear layer 24 is connected to the input of the post-processing network 25 on the one hand, and is also connected to the output of the post-processing network 25 on the other hand and added to the output of the post-processing network 25, and then the average spectrum of the category is used as the output of the decoder.
[0030] The existing acoustic models for sound effect generation are all autoregressive models, while the embodiments of the present invention propose a parallelizable non-autoregressive acoustic model, which has stronger learning ability and can be parallelized, thereby improving the speed of the model; moreover, the proposed acoustic model is based on category embedding, which introduces category information and does not require the model to predict the category itself, so that multi-category sound effect generation can be completed. In addition to being implemented based on the Transformer model, the parallelized non-autoregressive acoustic model can also be implemented using some variant forms such as Conformer, or variant forms based on variational autoencoders (VAEs).
[0031] The network structure of the Transformer model is as follows Figure 5As shown, it includes: a multi-head attention mechanism module 01, an addition & standardization module 02 connected to the output end of the multi-head attention mechanism module 01, a one-dimensional convolution 03 connected to the output end of the addition & standardization module 02, and another addition & standardization module 04 connected to the output end of the one-dimensional convolution 03. For the Transformer model, its input is divided into multiple paths to enter the multi-head attention mechanism module 01, and on the other hand, it is also simultaneously input to the addition & standardization module 02. In addition, the output of the addition & standardization module 02 is not only sent to the one-dimensional convolution 03, but also sent to another addition & standardization module 04.
[0032] The network structure of the post-processing network 25 is as follows Figure 6 As shown, it is mainly composed of multiple convolutional layers and multiple batch normalization layers. Specifically, there are 5 repeated units in series, each of which is composed of a one-dimensional convolution 05 and a batch normalization module 06 in series; among them, the three repeated units in the middle have the same parameters; and the first and last two units play the role of dimensional transformation and have different parameters from the three middle units. For the entire post-processing network 25, the network input enters the one-dimensional convolution of the first repeated unit on the one hand, and is added to the output of the batch normalization module of the last repeated unit on the other hand, and the addition result is used as the output of the entire post-processing network.
[0033] In one embodiment of the present invention, for the acoustic features output by the acoustic model, a linear spectrum is used, that is, a spectrum with a dimension of 513, that is, the acoustic model outputs a predicted spectrum; accordingly, the vocoder uses a Fourier inverse transform vocoder, because the Fourier inverse transform vocoder runs fast and has a small amount of calculation, and can quickly restore the spectrum to a sound signal (sound waveform) to obtain an audio file. Of course, in other embodiments, the acoustic features can also use Mel spectrum, and the vocoder can also use a neural network-based vocoder to restore the sound waveform.
[0034] Continue to refer Figure 4 In the embodiment of the present invention, the average spectrum of each category is additionally introduced as a condition, added to the output of the post-processing network 25, and then the smooth L1 loss is calculated between the real spectrum. The smooth L1 loss is as follows:
[0035]
[0036]
[0037] Where x represents the spectrum generated by the acoustic model (i.e., the acoustic feature), y represents the real spectrum extracted from the original audio, n represents the number of videos contained in each mini-batch input to the model, i represents the i-th sample, and x i represents the spectrum output after the i-th sample is input into the model, yi Represents the true spectrum corresponding to the i-th sample.
[0038] By adding the category average spectrum, the acoustic model only predicts the residual loss between the average spectrum and the true spectrum. This loss is used as the loss of spectrum reconstruction. Minimizing this loss makes the spectrum generated by the model closer to the true spectrum. Finally, the generated spectrum is restored to a sound signal through an inverse Fourier transform vocoder to complete the entire sound effect generation process. Among them, the real spectrum is obtained by extracting audio features from the original video. In the model training stage, the audio features extracted from the original video are used as the real spectrum, i.e., the label, which is the goal of model learning. The goal of training is to generate a spectrum that is close enough to the real spectrum. The model training of the present invention, including a visual feature extraction model and an acoustic model, can be trained using a deep learning framework (such as TensorFlow, PyTorch, etc.).
[0039] The embodiment of the present invention is a sound effect generation device based on visual semantics, and its product-side usage process is very simple. The user only needs to upload a video that does not contain foley sound (silent video) as input. In the FastFoley system, the device of the present invention can output an audio file of foley sound. The user synchronizes the generated audio file with the input silent video to complete the foley sound.
[0040] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.
Claims
1. A method for generating sound effects based on visual semantics, characterized in that: The steps include: S1, extracting visual features including color information and motion information from the silent video to be mimicked; S2. using a trained parallelized non-autoregressive acoustic model to convert the visual features into acoustic features that match them; S3. Using a vocoder to restore the sound waveform from the acoustic features, that is, to obtain an audio file for imitation sound.
2. The method for generating sound effects based on visual semantics according to claim 1, characterized in that: Step S1 includes: for each frame of the silent video, respectively extracting the color information and the motion information, and then splicing the color information and the motion information as the visual feature of the corresponding frame.
3. The method for generating sound effects based on visual semantics as claimed in claim 2, characterized in that: In step S1, a pre-trained ResNet-50 network is used as a task extractor to extract the color information and the action information respectively; wherein, the RGB three-channel image of the current frame is read as the three-channel input of the ResNet-50 network to obtain the color information; the current frame and the previous frame and the next frame of the current frame are extracted as grayscale images respectively as the three-channel input of the ResNet-50 network to obtain the continuous actions contained in the three consecutive frames, and extended to each frame to obtain the action information.
4. The method for generating sound effects based on visual semantics as claimed in claim 1, characterized in that: The parallelized non-autoregressive acoustic model includes an encoder and a decoder based on a Transformer model; wherein the input of the encoder based on the Transformer model is the visual feature, the input of the decoder based on the Transformer model is the sum of the output of the encoder, category embedding and position encoding, and the output of the decoder is the acoustic feature.
5. The method for generating sound effects based on visual semantics as claimed in claim 1, characterized in that: The acoustic feature adopts a linear spectrum or a Mel spectrum; the vocoder is a vocoder based on inverse Fourier transform or a vocoder based on a neural network.
6. A sound effect generation device based on visual semantics, characterized in that: include: A visual feature extraction model receives a silent video to be mimicked as input and is used to extract visual features including color information and motion information from the silent video; A parallelized non-autoregressive acoustic model is connected to the output end of the visual feature extraction model and is used to convert the visual features into acoustic features that match the visual features; as well as, A vocoder is connected to the output end of the parallelized non-autoregressive acoustic model and is used to recover the sound waveform from the acoustic features to obtain an audio file for imitation sound.
7. The visual semantics-based sound effect generation device according to claim 6, characterized in that: The visual feature extraction model is implemented using a pre-trained ResNet-50 network, which extracts the color information and the motion information respectively; and then forms the visual features of the corresponding frame by splicing the color information and the motion information of each frame.
8. The visual semantics-based sound effect generation device according to claim 6, characterized in that: The parallelized non-autoregressive acoustic model includes an encoder and a decoder based on a Transformer model; wherein the input of the encoder based on the Transformer model is the visual feature, the input of the decoder based on the Transformer model is the sum of the output of the encoder, category embedding and position encoding, and the output of the decoder is the acoustic feature.
9. The visual semantics-based sound effect generation device according to claim 8, characterized in that: The encoder includes a first linear layer and a first Transformer module, wherein the first linear layer receives the visual feature as input, and its output plus the position code is used as the input of the first Transformer module, and the output of the first Transformer module is the output of the encoder; the decoder includes a second Transformer module, a second linear layer and a post-processing network, wherein the input of the second Transformer module is the input of the decoder, the output of the second Transformer module is used as the input of the second linear layer, and the output of the second linear layer is used as the input of the post-processing network, and the output of the post-processing network, the output of the second linear layer and the category average spectrum are added as the output of the decoder; wherein the post-processing network is composed of multiple layers of convolutional layers and multiple layers of batch normalization layers.
10. The visual semantics-based sound effect generation device according to claim 6, characterized in that: The acoustic feature adopts a linear spectrum or a Mel spectrum; the vocoder is a vocoder based on inverse Fourier transform or a vocoder based on a neural network.