An audio effect generation method and device, a storage medium and an electronic device

Abstract

The application discloses an audio effect generation method and device, a storage medium and an electronic device. The method comprises the following steps: inputting an image into an image encoder, extracting a global feature vector and at least one local feature vector; determining a target audio feature vector matched with the global feature vector and the local feature vector respectively in an audio feature database; generating audio based on the target audio feature vectors to obtain corresponding target audio, and obtaining output audio effects based on the target audio; wherein the audio feature vectors in the audio feature database are obtained by inputting audio materials into an audio encoder; and the audio encoder and the image encoder are aligned through joint training. According to the application, diversified audio effects can be efficiently generated based on image content.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a sound effect generation method, apparatus, storage medium and electronic device. Background Technology

[0002] In recent years, with the rapid development of computer vision and audio generation technologies, cross-modal content generation has become a research frontier in the field of artificial intelligence. In applications such as Gallery Story, the automatic story generation function lacks the ability to perceive images, resulting in a mismatch between the background audio in the generated video and the displayed image.

[0003] Current sound effect generation mainly relies on text as input or rule-based matching. However, since only text is used as the generation condition, some details or atmosphere in the image are difficult to describe accurately with text. This results in the generated audio not being able to match the image precisely, limiting the efficiency and effectiveness of the generated audio and making it difficult to guarantee quality. At the same time, the method of generating audio based on a fixed audio library results in a single audio style, making it impossible to dynamically adjust the sound effect details according to the input content. The output results lack diversity, and the generated results for each image are always exactly the same.

[0004] Furthermore, traditional diffusion models are slow, requiring hundreds of iterations to complete a single audio generation, resulting in long waiting times for users and making them unsuitable for applications with high real-time requirements. Although there is existing research applying consistency distillation to text-to-audio generation, which can accelerate diffusion models by hundreds of times, this research integrates consistency models into the text-to-audio generation process and still cannot be directly applied to image-to-sound effect generation tasks.

[0005] Therefore, how to achieve diverse sound effect generation based on image content while ensuring the quality of generated audio is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] This application provides a sound effect generation method, apparatus, storage medium, and electronic device, which can generate diverse sound effects based on image content.

[0007] To achieve the above objectives, this application adopts the following technical solution:

[0008] A sound effect generation method, comprising:

[0009] Acquire an input image, input the input image into a pre-trained image encoder, and extract the global feature vector and at least one local feature vector of the input image;

[0010] For the global feature vector and the at least one local feature vector, respectively, the target audio feature vector matching each is determined in a pre-created audio feature database;

[0011] Audio is generated based on the target audio feature vectors to obtain the corresponding target audio, and output sound effects are obtained based on the target audio.

[0012] The audio feature vectors in the audio feature database are extracted by inputting audio material into a pre-trained audio encoder; the audio encoder and the image encoder achieve feature alignment through joint training, so that the audio feature vectors and the image feature vectors are located in the same semantic space.

[0013] Preferably, the audio generation based on each of the target audio feature vectors includes:

[0014] Each of the target audio feature vectors is input into a latent space-based audio generator to obtain the audio representation in the latent space, and then decoded to obtain the corresponding target audio.

[0015] Preferably, obtaining the output sound effect based on the target audio includes:

[0016] Each decoded target audio is scored, and the target audio with a score higher than a preset threshold is mixed to obtain the output sound effect.

[0017] Preferably, the audio generator is implemented using a student diffusion model; wherein the training process of the student diffusion model includes:

[0018] The training sample audio is input into the latent space encoder and the audio encoder respectively for processing, and the latent space audio vector and the training audio feature vector are obtained accordingly.

[0019] Random noise is added to the latent space audio vector based on the initial time step. The data with added random noise, the training audio feature vector, and the initial time step are input into the student diffusion model for prediction to obtain the first prediction noise.

[0020] The denoising loss is calculated based on the comparison result between the first predicted noise and the latent spatial audio vector;

[0021] The first predicted noise is input into the ordinary differential solver based on the teacher diffusion model, and N steps are calculated in the direction of denoising to obtain the latent spatial audio vector of the corresponding time step.

[0022] The latent spatial audio vector at the corresponding time step, the training audio feature vector, and the corresponding time step are input into the student diffusion model for prediction to obtain the second prediction noise;

[0023] Calculate the consistency loss based on the first prediction noise and the second prediction noise;

[0024] The denoising loss is superimposed on the consistency loss, and the parameters of the student diffusion model are updated based on the superposition result.

[0025] Preferably, the step of extracting the global feature vector and at least one local feature vector of the input image includes:

[0026] The input image is input into the image encoder to obtain primary image features;

[0027] The primary image features are subjected to global pooling, and a global feature vector is generated by the feature extraction module.

[0028] The primary image features are subjected to average pooling, and a set of local feature vectors are generated by the feature extraction module.

[0029] Preferably, determining the respective matching target audio feature vectors in a pre-created audio feature database includes:

[0030] For each feature vector in the global feature vector and the at least one local feature vector, calculate its similarity to each audio feature vector in the audio feature database, and select a matching target audio feature vector from the audio feature database based on the similarity.

[0031] Preferably, for each feature vector, the audio feature vector with the highest similarity is selected from the audio feature database as the target audio feature vector for matching.

[0032] Preferably, the similarity is cosine similarity or L2 similarity.

[0033] Preferably, the scoring of each target audio obtained from decoding includes:

[0034] Each target audio obtained from decoding is input into the audio encoder for processing and mapped to the same semantic space to obtain a generated audio feature vector.

[0035] The similarity between the generated audio feature vector and the corresponding global feature vector or local feature vector is calculated as the score; wherein the corresponding global feature vector or local feature vector is matched with the target audio feature vector corresponding to the generated audio feature vector.

[0036] Preferably, if the score of the target audio is lower than a preset threshold, the audio representation of the latent space is regenerated using a backup audio generator, and then the corresponding target audio is decoded and scored.

[0037] Preferably, the backup audio generator has the same student model structure as the audio generator, but uses a different training dataset.

[0038] Preferably, the latent space encoder is a VAE encoder, and the ordinary differential solver is based on the DDIM teacher model.

[0039] Preferably, the audio feature database is created in the following ways:

[0040] Obtain audio materials;

[0041] The audio material is input into the audio encoder to extract the audio feature vector of the audio material;

[0042] The audio feature vector is persisted and stored in a database.

[0043] A sound effect generation device includes: an input module, an image encoder, a matching module, a target audio generation module, and a scoring and output module;

[0044] The input module is used to acquire the input image;

[0045] The image encoder is used to input the input image into a pre-trained image encoder and extract the global feature vector and at least one local feature vector of the input image;

[0046] The matching module is used to determine the target audio feature vectors for matching the global feature vector and the at least one local feature vector respectively in a pre-created audio feature database.

[0047] The target audio generation module is used to generate corresponding target audio based on each of the target audio feature vectors;

[0048] The output module is used to obtain output sound effects based on the target audio;

[0049] The audio feature vectors in the audio feature database are extracted by inputting audio material into a pre-trained audio encoder; the audio encoder and the image encoder achieve feature alignment through joint training, so that the audio feature vectors and the image feature vectors are located in the same semantic space.

[0050] A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, can implement any of the sound effect generation methods described above.

[0051] An electronic device, comprising at least a computer-readable storage medium and a processor;

[0052] The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the sound effect generation method described above.

[0053] As can be seen from the above technical solution, in this application, an input image is acquired and input into a pre-trained image encoder to extract the global feature vector and at least one local feature vector of the input image. For each of the global and at least one local feature vectors, a matching target audio feature vector is determined in a pre-created audio feature database. The audio feature vectors in the audio feature database are extracted by inputting audio material into the pre-trained audio encoder. The audio encoder and image encoder achieve feature alignment through joint training, ensuring that the audio feature vector and image feature vector reside in the same semantic space. Therefore, suitable target audio features can be matched to the input image from different global and local perspectives and regions. Next, audio is generated based on each target audio feature vector to obtain the corresponding target audio. This allows for more diverse audio content adapted to the image content. Finally, output sound effects are obtained based on the target audio. Thus, more diverse audio content adapted to the image content can yield diverse output sound effects, better matching the image content. As shown above, through the joint training of the audio encoder and image encoder, the matching of image content and audio features is effectively achieved, enabling the generation of diverse sound effects based on image content.

[0054] Furthermore, when generating audio based on the feature vectors of each target audio, each feature vector can be input into a latent space-based audio generator to obtain the audio representation in the latent space. This latent space representation is then decoded to obtain the corresponding target audio. By generating audio in the latent space, high-dimensional audio data is compressed into a low-dimensional latent representation, significantly reducing the computational complexity and memory overhead in the audio generation process, thereby significantly shortening the audio generation time and greatly improving the efficiency of audio generation. When obtaining the output sound effect based on the target audio, each decoded target audio can be scored. Target audio with scores higher than a preset threshold is mixed to obtain the output sound effect. Thus, secondary screening of the audio through scoring improves the effectiveness of the output sound effect. It is evident that by further combining latent space-based audio generation in this application, both audio diversity and generation efficiency can be effectively improved. Finally, secondary screening through scoring of the generated audio improves the effectiveness of the output sound effect, enabling the efficient generation of diverse sound effects based on image content. Attached Figure Description

[0055] Figure 1 This is a schematic diagram illustrating the basic process of the sound effect generation method in this application;

[0056] Figure 2This is a schematic diagram illustrating the specific process of the sound effect generation method in a particular embodiment of this application;

[0057] Figure 3 This is a schematic diagram of the architecture of the sound effect generation method in a specific embodiment of this application;

[0058] Figure 4 This is a schematic diagram of a single-round training process of the student diffusion model in a specific embodiment of this application;

[0059] Figure 5 This is a schematic diagram of the basic structure of the sound effect generation device in this application;

[0060] Figure 6 This is a schematic diagram of the scenario in Example 2 of this application;

[0061] Figure 7 This is a schematic diagram of the basic structure of the electronic device provided in this application. Detailed Implementation

[0062] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0063] The sound effect generation method provided in this application has the following overall idea: First, the audio encoder and the image encoder are pre-trained to achieve feature alignment, so that the feature vectors extracted by the two are located in the same semantic space; then, an audio feature database is created; global and local features of the image are extracted; the image features are matched with the features in the audio feature database to obtain the target audio feature group; audio is generated in the latent space based on the target audio features; the generated audio is scored and mixed to output the final sound effect.

[0064] Figure 1 This is a schematic diagram illustrating the basic process of the sound effect generation method in this application. For example... Figure 1 As shown, the method includes:

[0065] Step 101: Obtain the input image.

[0066] Use the image from which you want to generate sound effects as the input image.

[0067] Step 102: Input the input image into the pre-trained image encoder to extract the global feature vector and at least one local feature vector of the input image.

[0068] To generate more suitable audio for the corresponding image, this application extracts a global feature vector representing the entire image and at least one local feature vector representing the local features of the image. Different local feature vectors can be used to reflect the features of different regions of the image. The image feature vector extraction is performed using a pre-trained image encoder.

[0069] Step 103: For the global feature vector and at least one local feature vector, determine the matching target audio feature vectors in the pre-created audio feature database to form a target audio feature group.

[0070] This application pre-creates an audio feature library to store audio feature vectors extracted from audio materials. For each of the global feature vectors and at least one local feature vector extracted in step 102, a matching audio feature vector is selected from the audio feature database. The selected matching audio feature vector is called the target audio feature vector. All target audio feature vectors are combined into a target audio feature group for subsequent sound effect generation.

[0071] The audio feature vectors in the audio feature database are extracted by inputting audio material into a pre-trained audio encoder. The audio encoder and the aforementioned image encoder are jointly trained to achieve feature alignment, so that the audio feature vectors and image feature vectors are located in the same semantic space, thereby effectively selecting matching audio feature vectors for image feature vectors.

[0072] Step 104: Generate audio based on each target audio feature vector in the target audio feature group to obtain the corresponding target audio.

[0073] Through the processing in steps 102-103 above, each target audio feature vector that matches the image content has been found. In this step, corresponding audio is generated for each target audio feature vector. The generated audio is referred to as the target audio.

[0074] In audio generation based on target audio feature vectors, a more preferable approach is to input each target audio feature vector from the target audio feature group into a latent space-based audio generator to obtain the latent space audio representation, which is then decoded to obtain the corresponding target audio. This allows the latent space-based audio generator to generate the latent space audio representation, compressing high-dimensional audio data into a low-dimensional latent representation. This significantly reduces the computational complexity and memory overhead in the audio generation process, thus greatly reducing generation time while maintaining generation quality. The target audio is then obtained by decoding the latent space audio representation.

[0075] The audio generator can be implemented using a typical diffusion model, or other models such as a stream matching model. Furthermore, this application preferably uses a student diffusion model based on consistent distillation. This leverages the high-quality generation characteristics of the diffusion model while simultaneously improving generation efficiency through consistent distillation. Combined with latent space representation, this significantly enhances audio generation efficiency.

[0076] It should also be noted that in this step, each target audio feature vector in the target audio feature group will undergo the above processing to obtain the corresponding target audio. At the same time, in the aforementioned step 103, each target audio feature vector is obtained by matching with a global feature vector or local feature vector of a certain input image. Therefore, the target audio feature vector has a corresponding relationship with a certain global feature vector or local feature vector (hereinafter referred to as image feature vector). The target audio feature vector also has a corresponding relationship with the target audio. Therefore, the target audio also has a corresponding relationship with the image feature vector, that is, the target audio corresponds to a certain image feature vector.

[0077] Step 105: Obtain the output sound effect based on the target audio.

[0078] The aforementioned step 104 may yield a set of target audio files. This step processes these target audio files to obtain the final output sound effect. In the simplest way, the target audio files can be directly mixed to obtain the output sound effect. Alternatively, more preferably, the target audio files can be filtered, and the filtered results can be mixed to obtain the output sound effect.

[0079] This application provides a preferred method for filtering target audio. Specifically, each target audio can be scored, and target audio with scores higher than a preset threshold can be mixed to obtain the output sound effect.

[0080] This step is used for secondary screening of the target audio, specifically through scoring. If the score is higher than a preset threshold, the target audio is determined to meet the requirements and can be included in the sound effect mixing output. The scoring principle can be to compare the degree of matching between the target audio and its corresponding image feature vector; the higher the degree of matching, the higher the score.

[0081] The target audio that passes the secondary screening is the valid audio generated from the input image. These target audios are then mixed to obtain the output sound effect.

[0082] At this point, Figure 1 The illustrated sound effect generation method flow is now complete. As can be seen from the above flow, this application provides a framework for generating sound effects based on image content matching. This framework integrates image content-to-audio feature vector matching, latent space-based audio generation, and scoring and mixing of the generated audio, thereby efficiently generating sound effects that effectively match the image content.

[0083] More specifically, in the present application described above, a jointly trained image encoder and audio encoder are used to unify image features and audio features into the same semantic space, thereby enabling convenient and effective matching between the two. Simultaneously, global and local feature extraction is performed on the input image to comprehensively represent the image content and characteristics, and audio features are matched for each extracted image feature vector. Furthermore, latent space-based audio generation is used to effectively improve audio generation efficiency. Finally, the scoring and mixing of the generated target audio are used to achieve secondary screening of the generated audio, ensuring the effectiveness of the matching between the output sound effect and the image content.

[0084] The specific implementation of the sound effect generation method in this application is illustrated below through specific embodiments. Figure 2 This is a schematic diagram illustrating the specific process of the sound effect generation method in a particular embodiment of this application. Figure 3 This is a schematic diagram of the sound effect generation method in a specific embodiment of this application. For ease of description, the creation of the audio feature database and the joint training process of the image encoder and audio encoder are integrated into this process; this embodiment uses a Student diffusion model as the audio generator to more effectively improve audio generation efficiency. Figure 2 and Figure 3 As shown, the specific process includes:

[0085] Step 201: Jointly train the audio encoder and the image encoder.

[0086] This application trains the audio encoder and the image encoder through joint training, so that the audio feature vector extracted by the audio encoder and the image feature vector extracted by the image encoder are located in the same semantic space and are comparable.

[0087] Specifically, the audio encoder maps audio to a semantic space to obtain audio feature vectors, while the image encoder maps images to the same semantic space to obtain image feature vectors. During training, pairs of image and audio samples are input into the two encoders respectively, and a loss function is used to constrain the image feature vectors and audio feature vectors that express the same content to be as close as possible in the semantic space.

[0088] After joint training, the audio feature vector extracted by the audio encoder and the image feature vector extracted by the image encoder are located in the same semantic space, and the two are directly comparable. They can be matched by calculating similarity (such as cosine similarity).

[0089] Additionally, it should be noted that in practical applications, an image encoder can also be a video encoder, capable of both feature extraction from a single frame of input image and feature extraction from each frame of an input video stream.

[0090] Step 202: Create an audio feature database.

[0091] To improve the controllability of generated audio quality, this application creates an audio feature database to store audio feature vectors of various audio materials. If the image feature vector extracted by the image encoder matches well with the audio feature vector in the audio feature database, the matching audio feature vector is used for audio generation, thereby optimizing the audio output effect and reducing low-quality results.

[0092] Specifically, the audio feature vectors in the audio feature database can be obtained as follows:

[0093] Collect audio materials (high-quality, high-fidelity audio that meets the needs of the scene);

[0094] Input the audio material into the jointly trained audio encoder in step 201 to extract the audio feature vector;

[0095] The audio feature vectors are persisted (including index structure storage for search acceleration and quantization processing) and then stored in the audio feature database.

[0096] The audio source can include high-quality audio from the training dataset, or audio input based on specific needs during application deployment.

[0097]

[0098] Table 1

[0099] Audio selection criteria can combine environmental and object characteristics; for example, matching ocean wave sounds to ocean wave scenes, or bird calls to forest scenes. Table 1 shows an example table for selecting audio sources.

[0100] The audio feature library in this application, combined with the audio generator, can improve the quality of audio output and enhance the diversity of audio output.

[0101] Step 203: Extract image features from the input image to obtain the global feature vector and at least one local feature vector of the input image.

[0102] In this embodiment, the image encoder may include MobileNet and a feature extraction module. MobileNet is used to extract primary image features, and the feature extraction module is used to generate global and local image features based on the pooling results. The specific image feature extraction process may include:

[0103] After the original image is input, it is processed by MobileNet in the image encoder to obtain primary image features;

[0104] The primary image features are input into the feature extraction module after global pooling to generate a global feature vector.

[0105] The primary image features are processed by average pooling and input into the feature extraction module to generate a set of local feature vectors.

[0106] Global feature vectors reflect the overall semantic information of an image, while local feature vectors reflect the detailed information of different regions of the image (such as the features of different objects). By extracting both global and local feature vectors simultaneously, the image content can be captured more comprehensively, thereby generating sound effects that better match the image.

[0107] Step 204: For each of the global feature vector and at least one local feature vector, perform audio feature matching in the audio feature database to obtain the target audio feature group.

[0108] In this step, for each image feature vector, a matching audio feature vector is found in the audio feature database. Since the image encoder and audio encoder achieve feature alignment through joint training, the feature vectors extracted by the two encoders reside in the same semantic space, allowing for convenient comparison and matching. The specific matching method can be selected according to actual needs.

[0109] The exemplary matching method given in this embodiment is as follows:

[0110] For each feature vector in the global feature vector and at least one local feature vector, calculate its similarity to each audio feature vector in the audio feature database;

[0111] For each feature vector, the audio feature vector with the highest similarity is selected from the audio feature database. If the audio feature vector is higher than the preset threshold, it is used as the target audio feature vector for matching.

[0112] The target audio feature group is composed of all the target audio feature vectors obtained from the matching.

[0113] When calculating the similarity between image feature vectors and audio feature vectors, it is preferable to calculate cosine similarity or L2 similarity, etc., to mainly identify the similarity between image feature vectors and audio feature vectors in terms of angle.

[0114] Global feature vector matching yields the target global audio features, while each local feature vector is matched to obtain its corresponding target local audio features. If an image feature vector fails to match an audio feature vector with a similarity higher than a preset threshold, that image feature vector is not included in subsequent processing.

[0115] Step 205: Input each target audio feature vector in the target audio feature group into the latent space-based audio generator to obtain the audio representation in the latent space, and then obtain the corresponding target audio through latent space decoding and vocode conversion.

[0116] The audio generation process in this embodiment includes:

[0117] Each target audio feature vector in the target audio feature group is input into the latent space-based audio generator, and generation processing is performed in the latent space to obtain the audio representation in the latent space.

[0118] The audio representation in the latent space is processed by latent space decoding (such as VAE decoder) to generate a Mel spectrogram;

[0119] A vocoder is used to convert the Mel spectrogram into the target audio.

[0120] To effectively improve audio generation efficiency, the audio generator in this application is a generator trained based on consistent distillation. Consistent distillation compresses the hundreds of iterations required by traditional diffusion models into a minimum number of steps, significantly improving generation speed. Specifically, in this application, the target audio feature vector and random noise can be used as input, and the audio representation of the latent space can be obtained through sampling in a small number of iterations (e.g., 3-4 steps).

[0121] In more detail, the audio generator works as follows:

[0122] The input audio feature vector and initial time step are concatenated after embedding processing, forming a new embedding that is then processed with random noise in the diffusion model. The random noise is sequentially passed through the downsampling layer, intermediate layer, and upsampling layer of the diffusion model to obtain the predicted noise, which is embedded in the residual network of each layer and superimposed on the input noise. After obtaining the predicted noise, it is re-noised, and the time step is decreased by a predetermined value. The re-noised data and the decreased time step are then re-inputted into the diffusion model for new predictions until the time step reaches the preset value, at which point the audio representation of the latent space is output.

[0123] In order to generate more accurate audio representations stably, this application preferably further improves the training process of the student diffusion model by introducing denoising loss to guide the parameter update of the student diffusion model, so that the model distillation is more stable and it is easier to distill a model with consistency.

[0124] Specifically, the training process of the improved student diffusion model in this application is as follows: Figure 4 As shown, it includes:

[0125] S401, the training sample audio is input into the latent space encoder and the audio encoder respectively for processing, and the latent space audio vector and the training audio feature vector are obtained accordingly.

[0126] The latent space encoder can be, for example, a VAE encoder. The same training sample audio is input into both the latent space encoder and the audio encoder. After processing by the latent space encoder, the output is a latent space audio vector; after processing by the audio encoder, the output is a training audio feature vector.

[0127] S402, add random noise to the audio vector of the potential space based on the initial time step, and input the data with added random noise, the trained audio feature vector and the initial time step into the student diffusion model for prediction to obtain the first prediction noise;

[0128] This step adds random noise to the latent space audio vector obtained in S401. The noise can be added using the method used in existing diffusion model training, based on the initial time step. Next, the noise-added data, the training audio feature vector, and the initial time step are input into the target model to be trained, namely the student diffusion model. In this embodiment, the student diffusion model is specifically the student U-NET model. After model processing, the first prediction noise is obtained.

[0129] S403, calculate the denoising loss based on the comparison result between the first predicted noise and the latent space audio vector;

[0130] The first predicted noise obtained in step S402 is compared with the latent space audio vector before adding noise, and the denoising loss is calculated. The specific denoising loss function can be set according to actual needs; typically, it can be any regression loss function, for example:

[0131] L1 loss: L(pred,target) = |pred - target|

[0132] L2 loss (i.e., mean squared error loss): L(pred,target) = (pred - target)^2

[0133] Smooth L1 Loss: C is a very small constant, which can be set to 0.001.

[0134] The denoising loss function describes the current model's ability to recover noise from a noisy mixed signal, which in turn reflects the current model's ability to recover a clean audio signal from a noisy mixed signal. This ensures the accuracy of the audio generated by the model during the consistency distillation process (i.e., the process of training and generating the student diffusion model).

[0135] S404, input the first predicted noise into the ordinary differential solver based on the teacher diffusion model, calculate N steps in the direction of denoising, and obtain the latent space audio vector of the corresponding time step;

[0136] Consistent distillation is a technique that extracts the knowledge of a complex, multi-step, iterative teacher model (such as a diffusion model) into a simple and efficient student model. This allows the student model to "jump" directly to the end point or a point in the middle of the trajectory, thereby significantly reducing the steps required for generation or reasoning.

[0137] In this embodiment, a consistent distillation is performed on a mature diffusion model (hereinafter referred to as the teacher diffusion model), and the student diffusion model is gradually trained from the teacher diffusion model (such as the DDIM model) to generate the student diffusion model.

[0138] This step advances the teacher diffusion model processing N steps toward denoising by using an ordinary differential solver based on the teacher diffusion model. Specifically, the first predicted noise is input into the ordinary differential solver based on the teacher diffusion model, and N steps are calculated in the denoising direction. This yields the latent space audio vector for the corresponding time step, where the corresponding time step is XN, and X is the aforementioned initial time step.

[0139] S405, input the latent space audio vector, the training audio feature vector and the corresponding time step into the student diffusion model for prediction, and obtain the second prediction noise;

[0140] S406, calculate the consistency loss (e.g., mean square error loss) based on the first prediction noise and the second prediction noise.

[0141] The processing of S405~406 can be carried out using the existing uniform distillation method.

[0142] S407, the denoising loss and consistency loss are superimposed, and the parameters of the student diffusion model are updated based on the superposition result;

[0143] Since denoising loss is introduced in this embodiment, after obtaining the consistency loss, the denoising loss and consistency loss need to be superimposed. The model parameters are updated based on the superposition result, so as to effectively control the generation of the student diffusion model from both the accuracy and consistency of audio generation.

[0144] The superposition of denoising loss and consistency loss can be preset as needed, for example, by using a weighted sum superposition method.

[0145] The above S401~S407 constitutes one round of training. By training multiple rounds according to the above process, consistent distillation is completed, and the student diffusion model is obtained.

[0146] In the above training process, the introduction of denoising loss makes the training of the student diffusion model more stable and the generated audio quality is higher.

[0147] Based on the student diffusion model trained above, the audio generator in this embodiment is implemented. This approach can reduce the computational complexity and audio generation time by compressing the dimensionality of the audio feature representation using latent space. Furthermore, through consistent distillation, the audio generation time of the diffusion model is further compressed, achieving the generation of latent space audio representations with an extremely limited number of iterations of the student diffusion model (e.g., 3-4 sampling steps).

[0148] After generating a latent space audio representation using an audio generator, the audio's Mel spectrogram is obtained through latent space decoding (e.g., VAE decoding). Then, vocode conversion is used to convert the Mel spectrogram into an audio waveform to obtain the target audio.

[0149] Step 206: Extract features from the target audio and compare its similarity with the corresponding image feature vector to obtain a score for the target audio.

[0150] The purpose of this step is to evaluate the quality of each target audio generated in step 205, determine whether it matches the corresponding image features sufficiently, and thus select qualified target audio for subsequent mixing.

[0151] Specifically, each generated target audio is remapped back to the semantic space, and its similarity is compared with the image feature vector on which the audio was generated. The score of the target audio is obtained based on the similarity comparison result.

[0152] More specifically, the scoring process may include the following steps:

[0153] 1) Each target audio obtained by decoding in step 205 is input into the audio encoder jointly trained in step 201 for processing to obtain the corresponding audio feature vector, which is referred to as the generated audio feature vector.

[0154] For example, if a “target global audio” is generated from the “target global audio feature vector” in step 205, the target global audio is input into the audio encoder to extract its audio feature vector; similarly, each “target local audio” generated from the “target local audio feature vector” also has its own audio feature vector extracted.

[0155] 2) For each generated audio feature vector, calculate its similarity to the corresponding image feature vector.

[0156] Here, "corresponding image feature vector" refers to the image feature vector that matches the target audio feature vector corresponding to the generated audio feature vector in the feature matching in step 204.

[0157] Specifically:

[0158] The similarity between the target audio generated from the audio feature vector obtained by global feature vector matching and the global feature vector is calculated.

[0159] The similarity between the audio generated from the audio feature vector obtained by matching a certain local feature vector and that local feature vector is calculated.

[0160] The preferred methods for calculating similarity are cosine similarity and L2 similarity. The advantage of cosine similarity and L2 similarity is that they are not affected by the vector length and can accurately reflect the directional consistency of two vectors.

[0161] 3) Determine the score of the target audio based on the calculated similarity value.

[0162] The higher the similarity, the better the match between the generated audio and image features, and the better the audio quality, the higher the score should be. The simplest approach is to directly use the calculated similarity value as the score for the target audio; alternatively, other functional mapping relationships can be set between the two according to actual needs.

[0163] Step 207: Mix the target audio with an audio score higher than the set threshold to obtain the output sound effect.

[0164] The aforementioned score for the target audio is essentially a quantitative measure of the fit between the generated audio and the image content. Since the audio encoder and image encoder are jointly trained, their extracted feature vectors are comparable in semantic space. Therefore, the score directly reflects the degree of semantic matching between the generated audio and the image content. This mechanism ensures that only audio that highly matches the image content is included in the final output, thus guaranteeing the overall quality of the generated sound effects.

[0165] For each target audio, if the score is higher than or equal to the preset threshold, the target audio is deemed qualified and can participate in subsequent mixing; if the score is lower than the preset threshold, the target audio can be directly discarded and will no longer participate in subsequent mixing.

[0166] Alternatively, for target audio with a score below a preset threshold, a backup audio generator can be used to regenerate the audio representation of the latent space based on the target audio feature vector corresponding to the target audio, and then decode the corresponding target audio and score it; the following is to distinguish between the audio generator and the backup audio generator in step 205, and the audio generator used in step 205 is called the main audio generator.

[0167] The target audio generated by the backup generator is scored again. If the score is still lower than the preset threshold, the audio is discarded and will not be included in subsequent mixing.

[0168] The backup audio generator has the same model structure as the main audio generator, but uses a different training dataset, such as a training dataset focusing on foreground sound and a training dataset focusing on background sound. Specifically, in this embodiment, the backup audio generator is also based on the student diffusion model, but the training dataset for this student diffusion model is different from that used in training step 205. Because the backup audio generator and the main audio generator use different training datasets, their adaptability may differ for different audio types. Therefore, when the audio score generated by the main generator is substandard, regenerating it using the backup generator may produce better results.

[0169] The mixing of target audios that have reached a preset threshold can be configured according to actual needs. For example, the root mean square normalization of each target audio that has reached the preset threshold can be performed, the normalized audios can be linearly superimposed, and finally the superimposed audio can be normalized again to obtain the output sound effect.

[0170] At this point, Figure 2 The illustrated process flow has ended.

[0171] In the above Figure 2 In the specific embodiment of the sound effect generation method shown, the joint training of the audio encoder and the image encoder effectively achieves the matching of image content and audio features. Combined with latent space-based audio generation and scoring, it can improve the diversity of audio, generation efficiency, and effectiveness of output sound effects. Furthermore, when the audio generator is implemented based on the student diffusion model, denoising loss can be introduced during the training of the student diffusion model to further ensure the quality of audio generation. In addition, other student diffusion models can be trained using different training datasets to realize backup audio generators. When the main generator does not produce ideal results for a specific type of audio, the backup generator can provide effective alternative outputs, improve system robustness, and ensure that all audios participating in the final mixing meet the quality threshold, guaranteeing output quality. It can also adapt to the generation needs of different acoustic features, enhancing the adaptability of the scene.

[0172] The above describes a specific implementation of the sound effect generation method in this application. This application also provides a sound effect generation apparatus, which can be used to implement the above sound effect generation method. Figure 5 This is a schematic diagram of the basic structure of the sound effect generation device in this application. Figure 5 As shown, the device includes: an input module, an image encoder, a matching module, a target audio generation module, and a scoring and output module.

[0173] The input module is used to acquire the input image.

[0174] An image encoder is used to input an input image into a pre-trained image encoder to extract the global feature vector and at least one local feature vector of the input image;

[0175] The matching module is used to determine the target audio feature vectors that match the global feature vector and at least one local feature vector respectively in a pre-created audio feature database, and form a target audio feature group.

[0176] The target audio generation module is used to generate audio based on each target audio feature vector in the target audio feature group to obtain the corresponding target audio.

[0177] The output module is used to obtain the output sound effect based on the target audio.

[0178] The audio feature vectors in the audio feature database are extracted by inputting audio materials into a pre-trained audio encoder. The audio encoder and the image encoder achieve feature alignment through joint training, so that the audio feature vectors and the image feature vectors are located in the same semantic space.

[0179] Optionally, in the target audio generation module, audio generation processing based on each target audio feature vector may specifically include:

[0180] Each target audio feature vector is input into a latent space-based audio generator to obtain the audio representation in the latent space, and then decoded to obtain the corresponding target audio.

[0181] In the output module, the processing of output sound effects based on the target audio can specifically include:

[0182] Each decoded target audio is scored, and the target audio with a score higher than a preset threshold is mixed to obtain the output sound effect.

[0183] Optionally, the device may further include an audio encoder, and the audio generator may be implemented using a Student diffusion model; wherein the training process of the Student diffusion model may specifically include:

[0184] The training sample audio is input into the latent space encoder and the audio encoder respectively for processing, and the latent space audio vector and the training audio feature vector are obtained accordingly.

[0185] Random noise is added to the latent space audio vector based on the initial time step. The data with added random noise, the trained audio feature vector, and the initial time step are input into the student diffusion model for prediction to obtain the first prediction noise.

[0186] The denoising loss is calculated based on the comparison between the first predicted noise and the latent spatial audio vector;

[0187] The first predicted noise is input into the ordinary differential solver based on the teacher diffusion model, and N steps are calculated in the direction of denoising to obtain the latent spatial audio vector of the corresponding time step.

[0188] The latent spatial audio vector, the trained audio feature vector, and the corresponding time step are input into the student diffusion model for prediction to obtain the second prediction noise.

[0189] The consistency loss is calculated based on the first and second prediction noises.

[0190] The denoising loss and consistency loss are superimposed, and the parameters of the student diffusion model are updated based on the superposition result.

[0191] Optionally, the process of extracting the global feature vector and at least one local feature vector of the input image in the image encoder may specifically include:

[0192] Receive the input image and obtain the initial image features;

[0193] The primary image features are subjected to global pooling, and a global feature vector is generated by the feature extraction module.

[0194] The primary image features are subjected to average pooling, and a set of local feature vectors are generated by the feature extraction module.

[0195] Optionally, in the matching module, the processing of determining the target audio feature vectors for each match from a pre-created audio feature database may specifically include:

[0196] For each feature vector in the global feature vector and at least one local feature vector, calculate its similarity to each audio feature vector in the audio feature database, and select the matching target audio feature vector from the audio feature database based on the similarity.

[0197] Optionally, for each feature vector, the audio feature vector with the highest similarity is selected from the audio feature database as the target audio feature vector for matching.

[0198] Optionally, the similarity can be cosine similarity or L2 similarity.

[0199] Optionally, the scoring and output module may perform scoring on each decoded target audio file, which may specifically include:

[0200] Each target audio obtained from decoding is input into an audio encoder for processing to obtain a generated audio feature vector;

[0201] The similarity between the generated audio feature vector and the corresponding global or local feature vector is calculated as a score; where the corresponding global or local feature vector is matched with the target audio feature vector corresponding to the generated audio feature vector.

[0202] Optionally, if the score of the target audio is lower than a preset threshold, the audio representation of the latent space is regenerated using a backup audio generator, and then the corresponding target audio is decoded and scored.

[0203] Optionally, the backup audio generator has the same student model structure as the main audio generator, but uses a different training dataset.

[0204] Optionally, the latent space encoder can be a VAE encoder, and the ordinary differential solver is based on the DDIM teacher model.

[0205] Optionally, the specific methods for creating an audio feature database may include:

[0206] Obtain audio materials;

[0207] The audio material is input into the audio encoder to extract the audio feature vector of the audio material;

[0208] The audio feature vectors are persisted and stored in the database.

[0209] The following are some application examples of the sound effect generation method in this application in typical scenarios.

[0210] Example 1: Use Cases of Mobile Phone Photo Album

[0211] The system can use photos taken in a user's phone album as a base to generate corresponding audio materials, creating vivid life records. Specifically, when a user browses the album, the system uses the sound effect generation method of this application to generate corresponding sound effects based on the images, and outputs the sound effects along with the photos to form a more vivid memory presentation.

[0212] An exemplary process may include:

[0213] 1. The system automatically selects images, or the user uploads images that they deem meaningful, and inputs them into the sound generation model.

[0214] 2. The model extracts image features, including global features and local features.

[0215] 3. Match the image features with the audio library materials to obtain the matching audio features.

[0216] 4. Generate audio based on matched audio features.

[0217] 5. Output the generated audio, ensuring it closely matches the image content.

[0218] Then, multiple photos and their corresponding generated audio can be played in sequence to create a complete Story video. Compared with existing products, the audio generated according to the method of this application is highly consistent with the image content and is unique.

[0219] As shown above, audio vectors can be matched with global and local image features to generate customized sound effects and create more immersive and unique stories.

[0220] Example 2: Television Art Mode Scene

[0221] Once the TV enters its Art mode, it can play artworks to decorate the room. Using the sound effect generation method described in this application, appropriate audio can be automatically generated for different artworks, cultivating a refined sensibility. This allows for real-time retrieval of high-quality audio vectors matching artworks, generating a personalized soundscape and enhancing the artistic experience.

[0222] For example, in Art mode, the TV generates audio by matching the displayed image with high-quality audio vectors from an audio library. Figure 6 As shown, when displaying a picture of a towering mountain peak, the sound of howling wind and footsteps in the snow will be played automatically.

[0223] Example 3: Accessibility Assistance Scenarios

[0224] The method described in this application can provide visually impaired individuals with an image-to-audio conversion tool. When performing text-to-speech (TTS) on a webpage, the method described in this application can convert webpage images into sounds or symbolic sound effects. For visually impaired users, adding sounds that represent images enhances immersion more than simply describing the image content. Through the method described in this application, images can be converted into ambient sound effects, providing visually impaired users with auditory representations of visual content.

[0225] For example, when reading a webpage aloud, in addition to regular TTS reading, the system can recognize and inform the user that the page contains an image of "railway tracks by the sea." The user can then choose to play a sound effect of "train sounds accompanied by the sounds of waves and wind."

[0226] Example 4: AI-assisted video generation scenario

[0227] The sound effect generation method of this application can generate unique background music for videos. For example, when generating a video from a novel, the traditional method is to first generate images from the text, then connect the images with special effects transitions, and finally manually select background music based on keyframes. In this process, the final step requires manual intervention to optimize the effect. If the sound effect generation method of this application is adopted, the system can automatically generate background music. After generating the images using an industry-standard text-image model, it automatically generates matching background music for each image, forming a complete video. Thus, background music can be automatically generated for keyframes without manual post-processing.

[0228] Example 5: Interactive Immersive Learning Application Scenarios

[0229] The method described in this application can automatically generate relevant teaching audio for interactive learning scenarios. When a user clicks on an image on an exhibit panel, educational sound effects (such as animal sounds or natural sounds) related to that image can be generated instantly, enhancing learning interest and memory retention. For example, using the sound effect generation method described in this application, when illustrations of animals or landscapes in museum or zoo exhibits are clicked or scanned, relevant sound effects (such as animal sounds or simulations of scientific phenomena like sudden rainstorms or gentle breezes) can be automatically generated, increasing the immersion of the learning activity and enhancing understanding of complex knowledge through multi-sensory learning.

[0230] This application also provides a computer-readable storage medium that stores instructions, which, when executed by a processor, can perform the steps in the sound effect generation method described above. In practical applications, the computer-readable medium may be included in the devices / apparatus / systems of the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium stores instructions, which, when executed by a processor, can perform the steps in the sound effect generation method described above.

[0231] According to the embodiments disclosed in this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof, but not intended to limit the scope of protection of this application. In the embodiments disclosed in this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0232] Figure 7 An electronic device is also provided for this application. For example... Figure 7As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0233] The electronic device may include a processor 701 with one or more processing cores, a memory 702 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 702 is executed, a sound effect generation method based on registered samples can be implemented.

[0234] Specifically, in practical applications, this electronic device may also include components such as a power supply 703 and an input / output unit 704. Those skilled in the art will understand that... Figure 7 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0235] The processor 701 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 702, and calling data stored in the memory 702, it performs various functions of the server and processes data, thereby monitoring the electronic device as a whole.

[0236] The memory 702 can be used to store software programs and modules, i.e., the aforementioned computer-readable storage medium. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on server usage, etc. Furthermore, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.

[0237] The electronic device also includes a power supply 703 that supplies power to the various components. This power supply can be logically connected to the processor 701 via a power management system, enabling functions such as charging, discharging, and power consumption management. The power supply 703 may also include one or more DC or AC power supplies, a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator, or any other components.

[0238] The electronic device may also include an input / output unit 704, which can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, and optical signal inputs related to user settings and function control. The input / output unit 704 can also be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof.

[0239] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating sound effects, characterized in that, include: Acquire an input image, input the input image into a pre-trained image encoder, and extract the global feature vector and at least one local feature vector of the input image; For the global feature vector and the at least one local feature vector, respectively, the target audio feature vector matching each is determined in a pre-created audio feature database; Audio is generated based on the target audio feature vectors to obtain the corresponding target audio, and output sound effects are obtained based on the target audio. The audio feature vectors in the audio feature database are extracted by inputting audio material into a pre-trained audio encoder; the audio encoder and the image encoder achieve feature alignment through joint training, so that the audio feature vectors and the image feature vectors are located in the same semantic space.

2. The method according to claim 1, characterized in that, The audio generation based on each of the target audio feature vectors includes: Each of the target audio feature vectors is input into a latent space-based audio generator to obtain the audio representation in the latent space, and then decoded to obtain the corresponding target audio.

3. The method according to claim 1, characterized in that, The process of obtaining the output sound effect based on the target audio includes: Each decoded target audio is scored, and the target audio with a score higher than a preset threshold is mixed to obtain the output sound effect.

4. The method according to claim 2, characterized in that, The audio generator is implemented using a student diffusion model; wherein, the training process of the student diffusion model includes: The training sample audio is input into the latent space encoder and the audio encoder respectively for processing, and the latent space audio vector and the training audio feature vector are obtained accordingly. Random noise is added to the latent space audio vector based on the initial time step. The data with added random noise, the training audio feature vector, and the initial time step are input into the student diffusion model for prediction to obtain the first prediction noise. The denoising loss is calculated based on the comparison result between the first predicted noise and the latent spatial audio vector; The first predicted noise is input into the ordinary differential solver based on the teacher diffusion model, and N steps are calculated in the direction of denoising to obtain the latent spatial audio vector of the corresponding time step. The latent spatial audio vector at the corresponding time step, the training audio feature vector, and the corresponding time step are input into the student diffusion model for prediction to obtain the second prediction noise; Calculate the consistency loss based on the first prediction noise and the second prediction noise; The denoising loss is superimposed on the consistency loss, and the parameters of the student diffusion model are updated based on the superposition result.

5. The method according to claim 1, characterized in that, The extraction of the global feature vector and at least one local feature vector of the input image includes: The input image is input into the image encoder to obtain primary image features; The primary image features are subjected to global pooling, and a global feature vector is generated by the feature extraction module. The primary image features are subjected to average pooling, and a set of local feature vectors are generated by the feature extraction module.

6. The method according to claim 1, characterized in that, The step of determining the respective matching target audio feature vectors in a pre-created audio feature database includes: For each feature vector in the global feature vector and the at least one local feature vector, calculate its similarity to each audio feature vector in the audio feature database, and select a matching target audio feature vector from the audio feature database based on the similarity.

7. The method according to claim 6, characterized in that, For each feature vector, the audio feature vector with the highest similarity is selected from the audio feature database as the target audio feature vector for matching.

8. The method according to claim 6 or 7, characterized in that, The similarity is cosine similarity or L2 similarity.

9. The method according to claim 3, characterized in that, The scoring of each target audio obtained from decoding includes: Each target audio obtained from decoding is input into the audio encoder for processing and mapped to the same semantic space to obtain a generated audio feature vector. The similarity between the generated audio feature vector and the corresponding global feature vector or local feature vector is calculated as the score; wherein the corresponding global feature vector or local feature vector is matched with the target audio feature vector corresponding to the generated audio feature vector.

10. The method according to claim 1, characterized in that, If the score of the target audio is lower than a preset threshold, the backup audio generator is used to regenerate the audio representation of the latent space, and then the corresponding target audio is decoded and scored.

11. The method according to claim 10, characterized in that, The backup audio generator has the same student model structure as the audio generator, but uses a different training dataset.

12. The method according to claim 4, characterized in that, The latent space encoder is a VAE encoder, and the ordinary differential solver is based on the DDIM teacher model.

13. The method according to claim 1, characterized in that, The methods for creating the audio feature database include: Obtain audio materials; The audio material is input into the audio encoder to extract the audio feature vector of the audio material; The audio feature vector is persisted and stored in a database.

14. A sound effect generation device, characterized in that, include: Input module, image encoder, matching module, target audio generation module, scoring and output module; The input module is used to acquire the input image; The image encoder is used to input the input image into a pre-trained image encoder and extract the global feature vector and at least one local feature vector of the input image; The matching module is used to determine the target audio feature vectors for matching the global feature vector and the at least one local feature vector respectively in a pre-created audio feature database. The target audio generation module is used to generate corresponding target audio based on each of the target audio feature vectors; The output module is used to obtain output sound effects based on the target audio; The audio feature vectors in the audio feature database are extracted by inputting audio material into a pre-trained audio encoder; the audio encoder and the image encoder achieve feature alignment through joint training, so that the audio feature vectors and the image feature vectors are located in the same semantic space.

15. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instruction is executed by the processor, it can implement the sound effect generation method according to any one of claims 1 to 13.

16. An electronic device, characterized in that, The electronic device includes at least a computer-readable storage medium and a processor; The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the sound effect generation method according to any one of claims 1 to 13.

Images