An environmental sound generation method, device, storage medium and product

Abstract

The application provides an ambient sound generation method and device, a storage medium and a product. The method comprises the following steps: obtaining environment information of an environment in which a target user is located and preference information of the target user for ambient sound; wherein the environment information comprises an environment video or an environment image, and the preference information comprises positive prompt words and negative prompt words; based on the environment information and the preference information, a target ambient sound for the target user is generated through a pre-trained sound generation model; wherein the sound generation model is a diffusion model constructed based on a transformer model and a flow model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method, device, storage medium and product for generating ambient sound. Background Technology

[0002] Currently, most ambient sound generation technologies use pre-recorded, fixed audio materials that are played in specific scenarios. For example, in a park environment, ambient sound playback devices can only play preset audio such as birdsong and wind sounds. However, with the continuous development of mobile internet and electronic device technologies, and the increasing demand for immersive experiences, current ambient sound generation methods lack flexibility and personalization, making it difficult to meet the diverse needs of users. Summary of the Invention

[0003] This application provides an ambient sound generation method, device, storage medium, and product that can improve the ambient sound generation effect.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an embodiment of an ambient sound generation method, the method comprising: Obtain environmental information about the target user's environment and the target user's preference information for ambient sounds; wherein, the environmental information includes: environmental video or environmental image, and the preference information includes: positive cue words and negative cue words; Based on the environmental information and the preference information, a target ambient sound is generated for the target user using a pre-trained sound generation model; wherein the sound generation model is a diffusion model built based on a transformer model and a stream model.

[0005] In some exemplary embodiments, the sound generation model includes an encoding network, an intermediate network, and a decoding network. The intermediate network includes a conditional generation module and a diffusion processing module based on a transformer model. The decoding network adopts the streaming model. The step of generating target ambient sound for the target user based on the environmental information and the preference information using a pre-trained sound generation model includes: The environmental information, preference information, and random noise are processed through the coding network to obtain environmental features corresponding to the environmental information, preference text features corresponding to the preference information, and latent variables corresponding to the random noise. Based on the environmental features and the preference text features, a condition vector is generated through the condition generation module; Based on the environmental features, the preferred text features, the latent variables, and the conditional vector, the target audio features are obtained through the diffusion processing module. The target audio features are decoded using the streaming model to obtain the target ambient sound for the target user.

[0006] In some exemplary embodiments, the diffusion processing module includes: a plurality of multimodal transformer blocks cascaded in sequence, a plurality of single-modal transformer blocks cascaded in sequence, a normalization layer and a convolutional layer, wherein the output of the last multimodal transformer block is used as the input of the first single-modal transformer block; The process of obtaining target audio features through the diffusion processing module based on the environmental features, the preferred text features, the latent variables, and the conditional vector includes: Based on the environmental features, the preferred text features, the latent variables, and the conditional vector, the first intermediate feature is obtained through the multiple multimodal transformer blocks; Based on the first intermediate feature, a second intermediate feature is obtained through the plurality of single-mode converter blocks; Based on the second intermediate feature, the target audio feature is obtained through the normalization layer and the convolutional layer.

[0007] In some exemplary embodiments, the multimodal transformer block includes an attention module with a joint attention mechanism and three feedforward neural networks, and the single-modal transformer block includes an attention module with a self-attention mechanism and a feedforward neural network.

[0008] In some exemplary embodiments, the environmental features include: environmental synchronization features and environmental visual features; the condition vector includes: frame alignment condition vector and global condition vector; The step of generating a condition vector based on the environmental features and the preferred text features through the condition generation module includes: Based on the environmental visual features and the preferred text features, the global condition vector is obtained through the condition generation module; Based on the environmental synchronization features and the global condition vector, the frame alignment condition vector is obtained through the condition generation module.

[0009] In some exemplary embodiments, after generating the target ambient sound for the target user, the method further includes: In response to detecting one or more of the following: a change in the environment of the target user and a change in the target user's preference for ambient sound, change information is obtained; Based on the change information, the environmental sound parameters generated by the sound generation model are adjusted to obtain the adjusted environmental sound.

[0010] In some exemplary embodiments, before obtaining environmental information about the target user's environment and the target user's preference information for ambient sound, the method further includes: Obtain a multimodal sample dataset, wherein the multimodal sample dataset includes: multiple sample data, each sample data including: sample ambient sound and its corresponding sample video and sample label information; Based on the multimodal sample dataset, the model to be trained is trained to obtain the sound generation model.

[0011] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the ambient sound generation method provided in the embodiments of this application.

[0012] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, they implement the ambient sound generation method provided in this application.

[0013] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the ambient sound generation method provided in this application.

[0014] The embodiments of this application have the following beneficial effects: This application provides an ambient sound generation method, device, storage medium, and product. It obtains environmental information about the target user's environment and the target user's preference information for ambient sound. The environmental information includes environmental video or images, and the preference information includes positive and negative prompts. A sound generation model generates target ambient sound for the target user based on the environmental information and preference information. The sound generation model uses a pre-trained diffusion model built on a transformer model and a flow model. Because the sound generation model is a pre-trained diffusion model combining a transformer model and a flow model, and because it generates ambient sound based on environmental information (environmental video or images) reflecting the target user's environment, and also considers the target user's preferences during the generation process, the acoustic characteristics of the target ambient sound have a high degree of matching with the target user's environment and preferences. This facilitates a more immersive auditory experience for the user, thereby improving the flexibility, personalization, and accuracy of ambient sound generation and enhancing the overall effect. Attached Figure Description

[0015] Figure 1 A flowchart illustrating an environmental sound generation method provided in an embodiment of this application; Figure 2 This is a first structural schematic diagram of the sound generation model provided in an embodiment of this application; Figure 3 This is a second structural schematic diagram of the sound generation model provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the multimodal converter block of the sound generation model provided in the embodiments of this application; Figure 5 A schematic diagram of the structure of a single-mode converter block for a sound generation model provided in this application embodiment; Figure 6 This application provides a schematic diagram of the structure of an ambient sound generation device; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0016] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In the description of this application, references are made to "some exemplary embodiments," which describe a subset of all possible embodiments. However, it is understood that "some exemplary embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0019] In the description of this application, the terms "first," "second," etc., are used merely to distinguish similar objects and do not represent a specific order of objects, nor should they be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in a sequence other than that illustrated or described herein.

[0020] In the description of this application, "multiple" means two or more. Terms such as "one or more," "one or more items," or "one or more items" indicate any one, any two, or more than two of a plurality. For example, including one or more of A, B, and C can mean including any one, any two, or more elements selected from the set consisting of A, B, and C.

[0021] In the description of this application, the terms "module," "unit," or "component" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented, wholly or partially, using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of a larger module or unit that includes the functionality of that module or unit.

[0022] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0023] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0024] For ease of understanding, the nouns and terms used in the embodiments of this application are explained below.

[0025] (1) Artificial Intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making.

[0026] (2) Artificial Intelligence Generated Content (AIGC) system, also known as generative artificial intelligence (AI), refers to a type of artificial intelligence system that uses AI technologies such as natural language processing, natural language generation, machine learning, deep learning, computer vision, image generation, and speech processing to automatically generate various types of content, such as text, images, and audio.

[0027] (3) The diffusion model is a broad concept that describes a class of generative models based on a probabilistic diffusion process. Its core idea is to generate data through a framework of gradually adding noise (forward process) and gradually removing noise (backward process). Any model that follows this "diffusion first, then reverse diffusion" idea can be called a diffusion model.

[0028] (4) The Transformer model, also known as the Transformer structure, Transformer architecture, etc., is a multi-layer neural network based on self-attention modules. It can be used to process natural language tasks. The Transformer module mainly consists of stacked multi-head self-attention modules and feed forward neural networks (FFN).

[0029] Attention mechanisms are a method for modeling the dependencies between the hidden states of the encoder and decoder in a neural network.

[0030] A multilayer perceptron (MLP) is a typical feedforward neural network, consisting of multiple stacked perceptron layers. Its basic structure includes an input layer, hidden layers, and an output layer. Each layer consists of multiple neurons, with the hidden layers potentially having multiple neurons. Each neuron in a multilayer perceptron is connected to all neurons in the layer above it, transmitting and processing information through weights and activation functions. Activation functions can be non-linear (such as the ReLU function). A convolutional multilayer perceptron (convMLP) is a feedforward neural network that combines convolutional layers and multilayer perceptrons.

[0031] (5) Flow model, also known as normalized flow model, is a deep generative model based on reversible transformation in the field of machine learning. Its core principle is to map a simple probability distribution (such as uniform distribution or Gaussian distribution) to a complex probability distribution that fits the real data through a series of reversible nonlinear transformations, thereby completing the tasks of probability density modeling and new sample generation.

[0032] (6) The Synchformer model is a neural network model trained in a self-supervised manner based on the Transformer model, which can be used to solve the time alignment between video and audio data. The Synchformer model contains two independent encoders, one of which is a visual encoder that can extract synchronization features from video frames in the video.

[0033] (7) The Unifying Language Learning Paradigms (UL2) model is a model based on the Transformer module.

[0034] (8) The Contrastive Language-Image Pre-Training (CLIP) model is a multimodal pre-trained neural network that learns the alignment relationship between images and text through contrastive learning. The CLIP model consists of two main parts: a text encoder and a visual encoder. These two encoders convert text and images into fixed-length vector representations (embeddings), respectively, so that they can be compared in the same vector space.

[0035] (9) In a broad sense, normalization layers are a collective term for all normalization techniques, including batch normalization (BN), layer normalization (LN), root mean square normalization (RMSNorm), and other different normalization techniques. Layer normalization (LN) is the most commonly used normalization method in current architectures such as Transformer and diffusion models. Layer normalization (LN) is a normalization method mainly used to process small batches of data or single sample data. It standardizes the output of each neuron in a layer, making its mean 0 and variance 1, thereby stabilizing the output distribution of each layer, making the training process smoother, accelerating the training process, and improving the stability of the model. Adaptive layer normalization (adaLN) is an improved normalization method based on layer normalization (LN) with an adaptive mechanism. It can dynamically adjust the normalization parameters according to the input, making it more flexible and adaptable to complex scenarios than fixed layer normalization, which helps to accelerate model training and improve performance. In adaptive layer normalization, the scaling and offset parameters are dynamically generated based on the input condition vector (used to indicate condition information), while in layer normalization (LN), these two parameters are fixed.

[0036] (10) Gated fusion is an information fusion method in neural networks.

[0037] (11) Residual connection is an operation in neural networks that directly adds the input to the output.

[0038] (12) Based on Federated Learning, the distributed intelligent architecture, as a distributed machine learning framework, allows participants to jointly model without sharing data, thereby breaking down information silos and realizing distributed intelligent collaboration, while also improving model training efficiency.

[0039] (13) Model fine-tuning refers to the process of optimizing a pre-trained model (usually trained on a large-scale general dataset) for a specific task or domain. A pre-trained model is a machine learning model that has been pre-trained on a large-scale general dataset and has basic feature extraction capabilities. This type of model does not need to be trained from scratch for a specific task, but serves as a "starting point" to quickly adapt to different scenarios through fine-tuning. Model fine-tuning is divided into: full fine-tuning (fine-tuning all parameters of the model), local fine-tuning (adjusting only some or a part of the parameters), and incremental fine-tuning (fine-tuning by adding new parameters and storing the new knowledge in the new parameters).

[0040] In related technologies, some audio tools (such as Apple's Ambient Sounds mixer) adopt an "offline recording + cloud classification + local matching" model, pre-recording 50 to 200 high-definition looping audio clips and indexing them by tags (such as forest, coffee shop, rainy day, etc.). Essentially, it's "audio retrieval + fade-in / fade-out splicing," without any generation capabilities. Some open platforms (such as FMOD, Wwise, etc.) provide layered synthesis audio tools that, according to preset rules, superimpose pre-recorded background noise (such as rain sounds, wind sounds, etc.) and one-shot sounds (such as a bird call, a thunderclap, etc.) chronologically, then stylize them using equalizers (EQ) and effects (such as reverb) to finally synthesize ambient sound effects. However, the rules in these audio tools are manually written by sound engineers, with fixed parameters, and cannot support the real-time generation of personalized ambient sounds.

[0041] It is evident that most ambient sound generation technologies in related fields use pre-recorded fixed audio materials, which cannot achieve real-time, dynamic, and personalized generation of ambient sounds. They cannot generate ambient sounds that accurately reflect the user's current environment and preferences, and cannot provide users with more immersive ambient sounds.

[0042] This application provides an ambient sound generation method, device, storage medium, and product. It uses a pre-trained diffusion model based on a Transformer model and a Flow model as the sound generation model. Based on the environmental information of the target user's environment and the target user's preference information for ambient sound, it generates target ambient sound for the target user. This improves the flexibility, personalization, and accuracy of ambient sound generation, enhances the ambient sound generation effect, and improves the user experience.

[0043] In some exemplary embodiments, the ambient sound generation method provided in the various embodiments of this application can generate ambient sounds that match the user's environment and user preferences, and is applicable to various scenarios that require a more immersive, personalized and flexible auditory experience, such as leisure, games, and virtual reality (VR).

[0044] The entity executing the ambient sound generation method provided in this application embodiment can be an ambient sound generation device, which is implemented in one or more ways, such as software and hardware. For example, the ambient sound generation device can be an electronic device (e.g., a server or terminal), or a central processing unit in the electronic device, or a module for ambient sound generation in the electronic device.

[0045] In practical applications, terminals can be desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Smart in-vehicle devices can include in-vehicle navigation terminals and in-vehicle computers. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Servers can be independent physical servers, server clusters composed of multiple physical servers, or distributed systems. They can also be cloud servers or server clusters that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly through wired or wireless communication.

[0046] Figure 1 This is a flowchart illustrating an environmental sound generation method provided in an embodiment of this application. The following will be combined with... Figure 1 This will be explained in detail. For example, the entity executing this ambient sound generation method can be an ambient sound generation device, which can be integrated into an electronic device (such as a server or terminal). For example, the ambient sound generation method can be implemented by the server or terminal alone, or by the server and terminal working together. In this embodiment, the ambient sound generation method is illustrated by taking the execution of the ambient sound generation method by a terminal as an example.

[0047] like Figure 1 As shown, the ambient sound generation method may include: S101, obtain environmental information about the target user's environment and the target user's preference information for ambient sounds; wherein, the environmental information includes: environmental video or environmental image, and the preference information includes: positive cue words and negative cue words; S102, based on environmental and preference information, generates target ambient sound for the target user through a sound generation model; wherein, the sound generation model adopts a pre-trained diffusion model built based on a Transformer model and a Flow model.

[0048] Positive prompts are used to indicate to the sound generation model the sound elements that should appear in the environment, guiding the model to generate environmental sounds that fit a specific context. For example, positive prompts can be keywords such as "explosion," "fireworks," "celebration," "outdoors," "games," "stream sounds," "quieter," "lively," and "high intensity."

[0049] Positive prompts are used to indicate unwanted sound elements in the ambient sound environment to the sound generation model, guiding it to exclude unwanted sound elements during the generation process. Examples of negative prompts include keywords such as "noisy," "song," "heavy metal," and "white noise."

[0050] In some exemplary embodiments, obtaining the target user's preference information for ambient sound in S101 may include one or more of the following: Obtain the target user's pre-set preference information for ambient sound via the terminal; Obtain the target user's preference information for ambient sound as input in real time via the terminal; Obtain historical data from the target user's terminal and generate the target user's preference information for ambient sound based on the target user's historical data; It acquires real-time status data collected by sensors in the terminal used by the target user, and dynamically generates the target user's preference information for ambient sound based on the real-time status data.

[0051] For example, users can set preference parameters in the companion application (APP) of the smart glasses.

[0052] For example, the terminal can record the target user's adjustment of ambient sound in different environments to obtain historical data; when it is necessary to obtain the target user's preference information for ambient sound, the terminal can use machine learning algorithms to analyze the stored historical data to obtain the target user's preference information for ambient sound.

[0053] For example, when a user wants to reduce ambient noise, the target user can actively input preference information to indicate a reduction in ambient noise intensity through the terminal; alternatively, the target user can perform a user action (such as a click) to indicate a reduction in ambient noise intensity through the terminal, and the terminal generates preference information to indicate a reduction in ambient noise intensity based on the user action. Alternatively, the target user can input a voice command to indicate a reduction in ambient noise intensity through the terminal, and the terminal generates preference information to indicate a reduction in ambient noise intensity based on the voice command.

[0054] For example, when real-time status data (such as motion data) indicates that the target user has changed from a stationary state to a walking state, the terminal can generate preference information to indicate an increase in the ambient sound intensity.

[0055] The sound generation model is a diffusion model that combines a transformer model and a stream model for ambient sound generation tasks. It effectively captures the dependencies between environmental and preference information and the acoustic characteristics of ambient sound, generating high-quality ambient sound. The sound generation model uses a transformer model to construct the backbone network of the diffusion model and a stream model as the decoding network.

[0056] In some exemplary embodiments, the environmental information may further include auxiliary information, which includes one or more of environmental sound information and geographic location information. This facilitates the model's determination of the user's environment.

[0057] Thus, in this embodiment, environmental information of the target user's environment and the target user's preference information for ambient sound are obtained. The environmental information includes environmental videos or images, and the preference information includes positive and negative cue words. A sound generation model generates target ambient sound for the target user based on the environmental information and preference information. The sound generation model employs a pre-trained diffusion model built upon a transformer model and a flow model. Because the sound generation model is a pre-trained diffusion model combining a transformer model and a flow model, and because it generates ambient sound reflecting the target user's environment based on environmental information (environmental videos or images), and also considers the target user's preference for ambient sound during the diffusion generation process, the acoustic characteristics of the target ambient sound have a high degree of matching with the target user's environment and preferences. This facilitates a more immersive auditory experience for the user, thereby improving the flexibility, personalization, and accuracy of ambient sound generation, enhancing the overall effect and user experience.

[0058] Furthermore, compared to the rule-based splicing method used in related technologies, the sound generation model in this embodiment employs a streaming model. This allows the target ambient sound to be streamed and decoded from the latent space of the diffusion model during the diffusion generation process, effectively reducing first-frame latency and meeting the audio-visual synchronization requirements of wearable devices, thus facilitating its widespread application in wearable devices. Moreover, compared to the tag-based audio retrieval method used in related technologies, the sound generation model in this embodiment uses a transformer-based diffusion model. This ensures that the ambient sound is directly generated by AI content generation technology based entirely on environmental and preference information, without sampled segments. This results in very low similarity to audio in commercial audio libraries, avoiding copyright issues.

[0059] In some exemplary embodiments, such as Figure 2 As shown, the sound generation model 20 may include an encoding network 21, an intermediate network 22, and a decoding network 23. The intermediate network 22 may include a conditional generation module and a diffusion processing module based on a transformer model. The decoding network 23 adopts a stream model.

[0060] The encoding network encodes the input data to obtain corresponding features, compressing the high-dimensional input data into a low-dimensional latent space. The intermediate network is the backbone network of a diffusion model based on a transformer model, used to inject conditions and, guided by these conditions, model the generation of ambient sound in the latent space, gradually diffusing from random noise to approximate the target audio features. The decoding network reconstructs the target audio features in the latent space into a high-dimensional output, i.e., the final output target ambient sound.

[0061] Among them, the decoding network 23 adopts a streaming model, which essentially replaces the decoder in the diffusion model with a streaming model. It undertakes the task of reconstructing the original high-dimensional output from compressed low-dimensional features and improves the generation quality by relying on the probabilistic modeling advantages of the streaming model.

[0062] In some exemplary embodiments, the encoding network 21 may include a visual encoder, a text encoder, and an audio encoder.

[0063] In practical applications, visual encoders, text encoders, and audio encoders can be independent encoders or components from pre-trained models.

[0064] The visual encoder, also known as a video encoder or image encoder, is used to extract environmental features based on environmental information (such as environmental images or video frames from environmental videos) to inject information about the user's current environment into the backbone network of the sound generation model. The visual encoder can include a first visual encoder for extracting environmental synchronization features based on environmental information and a second visual encoder for extracting environmental visual features based on environmental information. For example, the first visual encoder can be a Syncformer visual encoder (i.e., the visual encoder in a pre-trained Syncformer model). The second visual encoder can be a CLIP visual encoder (i.e., the visual encoder in a pre-trained CLIP model). For example, taking environmental video as an example, the environmental synchronization features can be 768-dimensional data extracted at 24fps, and the environmental visual features can be 1024-dimensional data extracted at 8fps.

[0065] The text encoder is used to extract preferred text features based on preference information (such as positive and negative cue words) to inject information about the user's current preferred ambient sound into the backbone network of the sound generation model. For example, the text encoder can be a UL2 encoder (i.e., the text encoder in a pre-trained UL2 model) or a CLIP text encoder (i.e., the text encoder in a pre-trained CLIP model). For instance, the preferred text features can be 1024-dimensional data consisting of 77 tokens.

[0066] The audio encoder is used to encode the input audio data (such as ambient sound samples during training, and random noise during testing and usage) into latent variables (also called latent variables or latent representations) in the latent space, so as to inject the audio data to be diffused into the backbone network of the sound generation model. For example, the audio encoder can be a pre-trained variational autoencoder (VAE). For example, the latent variables of random noise can be 20-dimensional data extracted at 31.25 fps.

[0067] In some exemplary embodiments, such as Figure 3 As shown, the diffusion processing module based on the transformer model may include: multiple multimodal transformer blocks cascaded in sequence, multiple single-modal transformer blocks cascaded in sequence, a normalization layer, and a convolutional layer. The output of the last multimodal transformer block serves as the input of the first single-modal transformer block, and the output of the last single-modal transformer block serves as the input of the normalization layer and the convolutional layer.

[0068] Among them, multiple multimodal transformer blocks are used to realize deep interaction and fusion of cross-modal information.

[0069] Among them, multiple single-mode converter blocks are used to further refine audio features after multimodal processing, and enhance their internal structure and detail expression.

[0070] In some exemplary embodiments, such as Figure 3 As shown, the output of each multimodal transformer block serves as the input to the next multimodal transformer block. For example, the input to the (i+1)th multimodal transformer block includes the output of the ith multimodal transformer block and a condition vector (including a frame alignment condition vector and a global condition vector), where i is a positive integer greater than 1 and less than N1, i+1 is a positive integer less than or equal to N1, and N1 is the total number of multimodal transformer blocks in the diffusion processing module, where N1 is a positive integer greater than 1. The global condition vector is obtained through the condition generation module based on environmental visual features and preference text features from the environmental features. The frame alignment condition vector is obtained through the condition generation module based on environmental synchronization features and the global condition vector from the environmental features.

[0071] In some exemplary embodiments, such as Figure 3 As shown, the output of each single-mode transformer block serves as the input to the next single-mode transformer block. For example, the input to the (j+1)th multi-mode transformer block includes the output of the j-th multi-mode transformer block and the frame alignment condition vector in the condition vector, where j is a positive integer greater than 1 and less than N2, j+1 is a positive integer less than or equal to N2, and N2 is the total number of multi-mode transformer blocks in the diffusion processing module, where N2 is a positive integer greater than 1.

[0072] In some exemplary embodiments, the number of multimodal converter blocks is less than the number of single-modal converter blocks. For example, the number of multimodal converter blocks can be 4, and the number of single-modal converter blocks can be 8. Another example is that the number of multimodal converter blocks can be 2, and the number of single-modal converter blocks can be 13. Of course, other numbers are also possible, and this embodiment does not limit the specific number of such numbers.

[0073] In some exemplary embodiments, each multimodal transformer block may include a joint attention module and three feedforward neural networks. For example, the three feedforward neural networks in the multimodal transformer block may include one multilayer perceptron (MLP) and two convolutional multilayer perceptrons (convMLPs). This supports information transfer and enhancement between multimodal information. For example, the convolutional multilayer perceptron (convMLP) uses a 1D convolutional neural network with a kernel size of 3 and padding of 1.

[0074] In some exemplary embodiments, each unimodal transformer block may include a self-attention module and a feedforward neural network. For example, the feedforward neural network in the unimodal transformer block may employ a convolutional multilayer perceptron (convMLP). Thus, after multimodal processing, the audio features are refined through the unimodal transformer block, enhancing their internal structure and detailed representation.

[0075] In some exemplary embodiments, such as Figure 4 As shown, the multimodal transformer block can include three branch networks: a first branch network for environmental information (i.e., visual features), a second branch network for preference information (i.e., text features), and a third branch network for audio data (such as latent variables of random noise or latent variables of sample ambient sound). Each branch network has the general structure of a Transformer model, which includes an input layer, a normalization layer, an attention module, a normalization layer, and a feedforward neural network.

[0076] For example, the first branch network includes: an input layer, normalization layers (such as adaptive normalization layer (adaLN)), linear layers (also called linear transformation layers, fully connected layers, etc.), normalization layers (such as root mean square normalization (RMSNorm) layers, positional encoding (such as rotational positional encoding (RoPE)), attention modules, convolutional layers (such as one-dimensional convolutional (1D-Conv) layers), gated recurrent unit layers with gating mechanisms, residual connection layers, normalization layers (such as adaptive normalization layer (adaLN)), feedforward neural networks (such as convolutional multilayer perceptron (convMLP), layers with gating mechanisms, and residual connection layers. The input layer of the first branch network is used to receive visual features (such as environmental visual features and projected environmental visual features) as input.

[0077] For example, the second branch network includes: an input layer, a normalization layer (such as adaptive layer normalization (adaLN)), a linear layer, a normalization layer (such as root mean square normalization (RMSNorm) layer), an attention module, a linear layer, a gated recurrent unit layer with a gating mechanism, a residual connection layer, an adaptive layer normalization (adaLN), a feedforward neural network (such as a multilayer perceptron (MLP), a gating layer, and a residual connection layer. The input layer of the second branch network is used to receive text features (such as preferred text features and projected preferred text features) as input.

[0078] For example, the third branch network includes: an input layer, adaptive layer normalization (adaLN), linear layers, normalization layers (such as root mean square normalization (RMSNorm) layers), positional encoding (such as rotational positional encoding (RoPE)), an attention module, convolutional layers (such as one-dimensional convolutional layers), gated recurrent unit layers with gating mechanisms, residual connection layers, normalization layers (such as adaptive layer normalization (adaLN)), feedforward neural networks (such as convolutional multilayer perceptrons (convMLP), gated recurrent unit layers with gating mechanisms, and residual connection layers. Among these, rotational positional encoding (RoPE) is used to align sequences with different frame rates. The convolutional multilayer perceptron (convMLP) is used to capture local temporal structures. The joint attention module in the multimodal transformer block has a joint attention mechanism. The input layer of the third branch network receives audio features (such as latent variables and projected latent variables) as input. The linear transform layer can be used to introduce linear components. Residual connection layers and normalization layers can be used to integrate information, enhance model stability, and accelerate model convergence.

[0079] Each multimodal transformer block contains a joint attention module composed of attention modules from three branch networks. The joint attention module is used to process the query vector, key vector, and value vector obtained from the processing of the three branch networks based on the joint attention mechanism.

[0080] In some exemplary embodiments, such as Figure 5 As shown, a single-modal transformer block can include a network that only processes audio features (i.e., intermediate features output after multiple multimodal transformer blocks process the input multimodal features). The single-modal transformer block has the general structure of a Transformer model, including an input layer, a normalization layer, an attention module, a normalization layer, and a feedforward neural network. The attention module in the single-modal transformer block has a self-attention mechanism. For example, a single-modal transformer block can include: an input layer, an adaptive layer normalization (adaLN), a linear layer (also called a linear transform layer or fully connected layer), a root mean square normalization layer (RMSNorm), a rotation position encoding (RoPE), an attention module, a one-dimensional convolutional layer, a layer with a gating mechanism, a residual connection layer, an adaptive layer normalization (adaLN), a convolutional multilayer perceptron (convMLP), a layer with a gating mechanism, and a residual connection layer. The input layer of the first single-modal transformer block receives the outputs of multiple cascaded multimodal transformer blocks as input. The attention module is used to process the query vector, key vector, and value vector corresponding to audio features (i.e., intermediate features output by multiple multimodal transformer blocks after processing the input multimodal features) based on the attention mechanism.

[0081] In some exemplary embodiments, such as Figure 3As shown, the intermediate network 22 may further include a feature processing module, which may include a first projection layer, an upsampling layer, a second projection layer, a first pooling layer, a third projection layer, and a second pooling layer. The first projection layer projects the input environmental synchronization features to obtain projected environmental synchronization features; the upsampling layer upsamples the projected environmental synchronization features to obtain frame-aligned synchronization features; the second projection layer projects the input environmental visual features to obtain projected environmental visual features; the first pooling layer pools the projected environmental visual features to obtain pooled visual features; the third projection layer projects the input preference text features to obtain projected preference text features; and the second pooling layer pools the projected preference text features to obtain pooled text features.

[0082] The first, second, and third projection layers are used to uniformly project the input feature data to the target dimension. For example, the projection layers linearly project high-dimensional environmental features (including environmental synchronization features and environmental visual features) and text features to the target dimension (e.g., 512 dimensions).

[0083] In some exemplary embodiments, such as Figure 3 As shown, the feature processing module may also include: a fourth projection layer, which is used to perform projection operations on the input latent variables to obtain the projected latent variables.

[0084] In some exemplary embodiments, such as Figure 3 As shown, the condition generation module may include: a first residual connection layer, a feedforward neural network (such as an MLP), a second residual connection layer, and a third residual connection layer; wherein, the first residual connection layer is used to perform feature fusion processing on the pooled visual features and the pooled text features to obtain a first condition feature; the feedforward neural network is used to process the first condition feature to obtain a second condition feature; the second residual connection layer is used to perform feature fusion processing on the second condition feature and the embedded features of the time step to obtain a global condition vector; the third residual connection layer is used to perform feature fusion processing on the frame alignment synchronization features and the global condition vector to obtain a frame alignment condition vector.

[0085] In some exemplary embodiments, with Figure 2 Taking the model architecture shown as an example, S102, based on environmental information and preference information, generates target environmental sounds for the target user through a sound generation model, which may include: S1021, through the coding network, environmental information, preference information and random noise are processed respectively to obtain environmental features corresponding to environmental information, preference text features corresponding to preference information, and latent variables corresponding to random noise; S1022, based on environmental features and preference text features, generates condition vectors through the condition generation module; S1023, based on environmental features, preferred text features, latent variables, and conditional vectors, obtains target audio features through a diffusion processing module; S1024 uses a streaming model to decode the target audio features to obtain the target ambient sound for the target user.

[0086] Among them, latent variables refer to the latent representation of audio data in the latent space, which is a kind of feature vector.

[0087] For example, a sound generation model can randomly extract noise from a standard normal distribution or a Gaussian distribution to obtain random noise.

[0088] For example, after the sound generation model obtains random noise, it can convert the random noise into an audio waveform using a short-time Fourier transform (STFT) and extract the Mel spectrogram to obtain the spectrogram of the random noise. Then, an audio encoder encodes the spectrogram of the random noise into a latent representation. In this way, the latent variables corresponding to the random noise are obtained. For example, the audio encoder can employ a pre-trained variational autoencoder (VAE).

[0089] The condition vector is used to guide subsequent multimodal processing.

[0090] In some exemplary embodiments, the encoding network includes: a visual encoder, a text encoder, and an audio encoder; S1021, the encoding network processes environmental information, preference information, and random noise respectively to obtain environmental features corresponding to the environmental information, preference text features corresponding to the preference information, and latent variables corresponding to the random noise, which may include: The environmental information is processed by a visual encoder to obtain the environmental features corresponding to the environmental information; The preference information is processed by a text encoder to obtain the preference text features corresponding to the preference information; By processing environmental information through an audio encoder, latent variables corresponding to random noise can be obtained.

[0091] In some exemplary embodiments, the visual encoder includes: a first visual encoder and a second visual encoder; the environmental features include: environmental synchronization features and environmental visual features; processing the environmental information through the visual encoder to obtain environmental features corresponding to the environmental information may include: The environmental information is processed by the first visual encoder to obtain environmental synchronization features corresponding to the environmental information; The environmental information is processed by a second visual encoder to obtain the environmental visual features corresponding to the environmental information.

[0092] In some exemplary embodiments, the environmental features include: environmental synchronization features and environmental visual features; the condition vector includes: frame alignment condition vector and global condition vector; S1022, based on the environmental features and preferred text features, the condition vector is generated by the condition generation module, which may include: Based on environmental visual features and preference text features, a global condition vector is obtained through a condition generation module; Based on environmental synchronization features and global condition vectors, a frame alignment condition vector is obtained through a condition generation module.

[0093] In some exemplary embodiments, S1022, generating a condition vector based on environmental features and preference text features through a condition generation module may include: generating a condition vector based on environmental features, preference text features, and embedded features of time steps through a condition generation module.

[0094] The embedding features of time steps are obtained through the time step embedding operation. The time step embedding operation is a conditionally controlled process in most diffusion models (such as the DiT model). The time step (also called the time step size) determines the number of steps in the forward and backward processes. A larger time step can smooth the noise addition process and allow the model to better fit the data distribution, but it also increases computational cost. Typically, the time step value is between 1000 and 5000. For example, discrete time steps (such as discrete numbers from 0 to 1000) are encoded to obtain a fixed-dimensional initial vector; this initial vector is then transformed using a multilayer perceptron (MLP) to map to embedding features of time steps with dimensions matching the model's backbone network. This facilitates the injection of time step embedding features into the model's backbone network, allowing the model to perceive temporal information and dynamically adjust its feature processing.

[0095] In some exemplary embodiments, the environmental features include: environmental synchronization features and environmental visual features; S1022, based on the environmental features and preference text features, a condition vector is generated by the condition generation module, which may include: The environmental synchronization features are projected and upsampled to obtain frame-aligned synchronization features. The environmental visual features are projected and pooled to obtain the pooled visual features. The preferred text features are projected and pooled to obtain the pooled text features. Based on frame-aligned synchronization features, pooled visual features, pooled text features, and time-step embedded features, a conditional vector is generated through a conditional generation module.

[0096] Among them, the projection operation is used to adjust various input features (such as environmental synchronization features, environmental visual features, preference text features, latent variables, etc.) to a unified dimensional space to adapt to subsequent processing requirements.

[0097] Among them, the environmental synchronization features are aligned with the audio features after projection operation and upsampling processing to form frame-aligned synchronization features.

[0098] In some exemplary embodiments, the condition vector includes: a frame alignment condition vector and a global condition vector; the condition vector is generated by a condition generation module based on frame alignment synchronization features, pooled visual features, pooled text features, and time step embedding features, and may include: Feature fusion processing is performed on the pooled visual features, pooled text features, and embedded features at time steps to obtain a global conditional vector; The synchronization features and global condition vector for frame alignment are fused to obtain the frame alignment condition vector.

[0099] In some exemplary embodiments, the condition generation module includes: a first residual connection layer, a feedforward neural network (such as an MLP), and a second residual connection layer; performing feature fusion processing on the pooled visual features, pooled text features, and embedded features at time steps to obtain a global condition vector, which may include: The first residual connection layer in the condition generation module performs feature fusion processing on the pooled visual features and the pooled text features to obtain the first conditional features. The first conditional feature is processed by a feedforward neural network (such as MLP) in the conditional generation module to obtain the second conditional feature; The second residual connection layer in the condition generation module performs feature fusion processing on the second condition feature and the embedded features of the time step to obtain the global condition vector.

[0100] In some exemplary embodiments, the condition generation module further includes: a third residual connection layer; performing feature fusion processing on the synchronization features and global condition vector for frame alignment to obtain a frame alignment condition vector, which may include: The third residual connection layer in the condition generation module performs feature fusion processing on the synchronization features and global condition vector of frame alignment to obtain the frame alignment condition vector.

[0101] In some exemplary embodiments, the diffusion processing module includes: a plurality of multimodal transformer blocks cascaded in sequence, a plurality of single-modal transformer blocks cascaded in sequence, a normalization layer, and a convolutional layer, wherein the output of the last multimodal transformer block is used as the input of the first single-modal transformer block; S1023, based on environmental features, preference text features, latent variables, and conditional vectors, the target audio features are obtained through the diffusion processing module, which may include: Based on environmental features, preference text features, latent variables, and conditional vectors, the first intermediate feature is obtained through multiple multimodal transformer blocks; Based on the first intermediate feature, the second intermediate feature is obtained through multiple single-mode transformer blocks; Based on the second intermediate feature, the target audio features are obtained through normalization layers and convolutional layers.

[0102] Among them, the first intermediate feature, the second intermediate feature, and the target audio feature are all audio features.

[0103] For example, the normalization layer can use adaptive layer normalization (adaLN).

[0104] For example, a convolutional layer can be a one-dimensional convolutional layer (1D-Conv). For example, the kernel size of a one-dimensional convolutional layer can be 3, and the padding can be 1.

[0105] In some exemplary embodiments, the environmental features include: environmental synchronization features and environmental visual features; the condition vector includes: frame alignment condition vector and global condition vector; based on the environmental features, preference text features, latent variables, and condition vector, obtaining the first intermediate feature through multiple multimodal transformer blocks may include: Based on environmental visual features, preference text features, latent variables, frame alignment condition vectors, and global condition vectors in the environmental features, the first intermediate feature is obtained through multiple multimodal transformer blocks.

[0106] In some exemplary embodiments, the condition vector includes: a frame alignment condition vector and a global condition vector; obtaining a second intermediate feature based on the first intermediate feature through multiple single-mode transformer blocks may include: Based on the frame alignment condition vector and the first intermediate feature in the condition vector, the second intermediate feature is obtained through multiple single-mode transformer blocks.

[0107] In some exemplary embodiments, the condition vector includes: a frame alignment condition vector and a global condition vector; obtaining target audio features based on the second intermediate feature through a normalization layer and a convolutional layer may include: Based on the frame alignment condition vector and the second intermediate feature in the conditional vector, the target audio features are obtained through normalization layers and convolutional layers.

[0108] In some exemplary embodiments, after generating the target ambient sound for the target user in S102, the ambient sound generation method may further include: In response to detecting one or more changes in the target user's environment and changes in the target user's preference for ambient sound, change information is obtained; Based on the change information, the environmental sound parameters generated by the sound generation model are adjusted to obtain the adjusted environmental sound.

[0109] In some exemplary embodiments, based on change information, adjusting the environmental sound parameters generated by the sound generation model to obtain the adjusted environmental sound may include: Based on the change information, the environmental sound parameters generated by the sound generation model are adjusted; By adjusting the parameters, the matched ambient sound is adjusted to obtain the updated ambient sound.

[0110] Thus, in this embodiment, when a change in the target user's environment is detected (e.g., the user moves from a quiet corner of a park to a bustling square) or the target user's preference for ambient sound (e.g., the user requests quieter ambient sound via voice command), change information can be acquired. Then, based on this change information, the ambient sound parameters generated by the sound generation model are adjusted in a timely manner to obtain an updated ambient sound. In this way, by dynamically adjusting the generated ambient sound according to real-time changes in the environment and changes in user preferences, the ambient sound can be matched to the user's current environment and preferences.

[0111] In some exemplary embodiments, before obtaining environmental information about the target user's environment and the target user's preference information for ambient sound in S101, the ambient sound generation method may further include: Obtain a multimodal sample dataset, which includes: multiple sample data, each sample data including: sample ambient sound and its corresponding sample video and sample label information; Based on a multimodal sample dataset, the model to be trained is trained to obtain a pre-trained sound generation model.

[0112] Here, the model to be trained refers to the untrained diffusion model that combines the transformer model and the flow model.

[0113] In some exemplary embodiments, taking a server-terminal collaborative implementation as an example, training the model to be trained based on a multimodal sample dataset to obtain a sound generation model may include: The server trains the model to be trained based on a multimodal sample dataset to obtain a pre-trained sound generation model; The server sends the trained sound generation model to the terminal.

[0114] In other exemplary embodiments, taking a server-terminal collaborative implementation as an example, training the model to be trained based on a multimodal sample dataset to obtain a sound generation model may include: Federated learning is used to train the model to be trained based on a multimodal sample dataset to obtain a sound generation model.

[0115] In some exemplary embodiments, taking a server-terminal collaborative implementation as an example, federated learning is used to train the model to be trained based on a multimodal sample dataset to obtain a sound generation model, which may include: The terminal receives the initial global model parameters sent by the server; The terminal performs incremental fine-tuning on the model to be trained (such as multiple rounds of stochastic gradient descent training until the preset number of rounds) based on the locally stored multimodal sample dataset and the initial global model parameters issued by the server, to obtain the fine-tuned sound generation model and the fine-tuned model parameters. The terminal sends the fine-tuned model parameters to the server so that the server can aggregate the fine-tuned model parameters sent by multiple terminals. The terminal receives incremental model parameters sent by the server; the incremental model parameters are obtained by the server aggregating the fine-tuned model parameters sent by multiple terminals. The terminal updates the model parameters of the sound generation model based on the incremental model parameters to obtain the sound generation model.

[0116] For example, the server may aggregate fine-tuned model parameters sent by multiple terminals using a federated averaging (FedAvg) algorithm. For instance, the server may perform a weighted average of the fine-tuned model parameters sent by multiple terminals based on the proportion of data volume from each terminal.

[0117] Thus, in this embodiment, incremental fine-tuning of the model parameters of the locally stored model is achieved on the terminal side. The server uses the FedAvg algorithm to aggregate the fine-tuned model parameters, realizing unified updates of high-frequency common model parameters on the server side. Then, the terminal side updates the local model according to the incremental model parameters issued by the server, which can make the sound generation model more and more accurate with use.

[0118] To facilitate understanding, the following description, in conjunction with exemplary application embodiments, illustrates the application scenarios to which the environmental sound generation method of this application is applicable.

[0119] In some exemplary embodiments, the system architecture of the application scenario of the ambient sound generation method according to the embodiments of this application may include a terminal, a network, and a server. The ambient sound generation method may be executed by the terminal and the server individually or jointly. For example, the terminal includes, but is not limited to, at least one of the following: mobile phone, laptop computer, tablet computer, handheld computer, mobile internet device (MID), PAD, in-vehicle device, etc. For example, the network may include, but is not limited to: wired network, wireless network, wherein the wired network includes: local area network, metropolitan area network, and wide area network, and the wireless network includes: Bluetooth, Wireless Fidelity (WIFI), and other networks that enable wireless communication. For example, the server may be a single server, a server cluster composed of multiple servers, or a cloud server. Here, the embodiments of this application do not limit this.

[0120] In some exemplary embodiments, the ambient sound generation method may include the following steps: the terminal collects environmental information of the target user's environment and the target user's preference information for ambient sound, wherein the environmental information includes: environmental video or environmental image, and the preference information includes: positive prompt words and negative prompt words; the terminal sends the environmental information of the target user's environment and the target user's preference information for ambient sound to the server; the server generates a target ambient sound for the target user based on the environmental information and preference information through a sound generation model, wherein the sound generation model adopts a pre-trained diffusion model constructed based on a transformer model and a flow model; the server sends the target ambient sound for the target user to the terminal; and the terminal plays the target ambient sound.

[0121] In some exemplary embodiments, the ambient sound generation method can be executed independently by a terminal, which has a built-in sound generation model. The ambient sound generation method may include the following steps: the terminal collects environmental information about the target user's environment and the target user's preference information for ambient sound, wherein the environmental information includes environmental video or images, and the preference information includes positive and negative prompts; based on the environmental information and preference information, the terminal generates a target ambient sound for the target user using the sound generation model; wherein the sound generation model employs a pre-trained diffusion model constructed based on a transformer model and a stream model; and the terminal plays the target ambient sound.

[0122] This application provides an exemplary application embodiment, which provides a personalized ambient sound generation system and method based on AIGC. By combining the environmental information of the user's environment and the user's preference information for ambient sound, and based on the sound generation model obtained by the model architecture of diffusion model + Transformer + Flow decoding, the system generates and dynamically adjusts the ambient sound to match it in real time, which can improve the user's immersive experience and solve the problems of lack of flexibility and personalization in ambient sound generation technology.

[0123] In some exemplary embodiments, the ambient sound generation system may include: an environmental information acquisition module, a preference information acquisition module, and an ambient sound generation module. The environmental information acquisition module acquires environmental information about the environment in which the target user is located, and the preference information acquisition module acquires the target user's preference information for ambient sound. The ambient sound generation module generates a target ambient sound for the target user based on the environmental information and preference information, using a sound generation model. The sound generation model is a pre-trained AIGC model, which is constructed based on a transformer model, a flow model, and a diffusion model.

[0124] As an example, the environmental information acquisition module is used to collect environmental information about the user's current environment through sensors (such as cameras, microphones, and positioning modules) on electronic devices such as smart glasses and smartphones. For example, the camera captures environmental video or images of the user's current environment. The microphone captures environmental sound to help determine the user's current environment by analyzing sound characteristics. The positioning module, such as the Global Positioning System (GPS), obtains the user's current geographical location to help determine the user's current environment by combining it with map data.

[0125] As an example, the preference information acquisition module can obtain preference information through manual settings on the device (such as setting preference parameters in the smart glasses' accessories), voice command input, or analysis based on the user's historical usage data. For instance, the system records the user's adjustments to ambient sound in different scenarios, analyzes user preference patterns through machine learning algorithms, and thus automatically obtains user preferences.

[0126] As an example, the ambient sound generation module incorporates a pre-trained AIGC model (i.e., a sound generation model). This model is trained using a large number of sample ambient sounds, corresponding sample videos indicating the environmental scene, and pre-labeled sample information indicating preferences. It can quickly synthesize corresponding ambient sounds based on the input environmental and preference information. For instance, when the user is in a park scene and prefers to add the sound of a stream, the sound generation model can synthesize an ambient sound containing a stream with appropriate intensity and timbre, based on the park scene features indicated by the environmental information and the preference for adding the stream sound.

[0127] In some other exemplary embodiments, the ambient sound generation system may further include: an ambient sound adjustment module for adjusting the ambient sound parameters generated by the sound generation model based on change information, to obtain adjusted ambient sound. The change information may be one or more of the following: environmental change information obtained when the target user's environment changes, and preference change information obtained when the target user's preference for ambient sound changes.

[0128] As an example, the ambient sound adjustment module continuously monitors the output information of the environmental information acquisition module and the preference information acquisition module. When it detects one or more changes in the target user's environment (such as the user moving from a quiet corner of a park to a bustling square) and changes in the target user's preference for ambient sound (such as the user requesting quieter ambient sound through voice commands), it adjusts the ambient sound parameters generated by the sound generation model in the ambient sound generation module in a timely manner based on the change information, so as to dynamically adjust the generated ambient sound to adapt to the user's needs and realize the dynamic updating of ambient sound.

[0129] In some exemplary embodiments, the ambient sound generation method may include the following steps: Step A1: Obtain input data, which includes: an environmental video indicating the target user's environment and preference text indicating the target user's preference for ambient sounds. The text includes positive and negative cue words. For example, positive cue words include keywords such as "explosion," "fireworks," "celebration," and "outdoors" to guide the model in generating ambient sounds that fit the specific context; negative cue words include keywords such as "music" to guide the model in excluding unwanted sound elements.

[0130] Step A2: Input the input data into the pre-trained sound generation model.

[0131] Step A3 involves processing the environmental video, preferred text, and random noise using the encoding network in the sound generation model to obtain environmental features corresponding to the environmental video, preferred text features corresponding to the preferred text, and latent variables corresponding to the random noise. The random noise is generated by the sound generation model. For example, step A3 may include: extracting environmental synchronization features from the video frames of the environmental video using the Syncformer visual encoder in the encoding network; extracting environmental visual features from the video frames of the environmental video using the CLIP visual encoder in the encoding network; extracting preferred text features from the preferred text using the CLIP text encoder (or UL2 text encoder) in the encoding network; and converting the random noise into latent variables using an audio encoder. For example, the audio encoder is also used to encode sample environmental sounds during the training phase and to add noise to the encoded sample environmental sounds.

[0132] Step A4 involves the following processing steps performed by the feature processing module of the intermediate network (i.e., the backbone network of the diffusion model) in the sound generation model: Projecting the input environmental synchronization features to obtain projected environmental synchronization features; upsampling the projected environmental synchronization features to obtain frame-aligned synchronization features; projecting the input environmental visual features to obtain projected environmental visual features; pooling the projected environmental visual features to obtain pooled visual features; projecting the input preference text features to obtain projected preference text features; pooling the projected preference text features to obtain pooled text features; and projecting the latent variables of random noise to obtain projected latent variables.

[0133] In this way, various input features are adjusted to a uniform dimension and temporal resolution through projection operations to meet the needs of subsequent processing.

[0134] Step A5: Using the conditional generation module of the intermediate network in the sound generation model, conditional vectors are generated based on frame-aligned synchronization features, pooled visual features, pooled text features, and temporal embedding features. These conditional vectors include a frame-aligned conditional vector and a global conditional vector. The global conditional vector is obtained by fusing pooled visual features, pooled text features, and temporal embedding features. The frame-aligned conditional vector is obtained by fusing frame-aligned synchronization features and the global conditional vector. The global conditional vector guides subsequent multimodal processing.

[0135] Step A6 involves processing the projected environmental visual features, projected preference text features, projected latent variables, frame alignment conditional vector, and global conditional vector using multiple multimodal transformer blocks in the intermediate network of the sound generation model, outputting the first intermediate feature. Thus, through the multimodal Transformer blocks, deep interaction and fusion of cross-modal information are achieved. Each block contains an attention mechanism and a feedforward neural network, supporting information transfer and enhancement between modalities.

[0136] Step A7 involves processing the frame alignment condition vector and the first intermediate feature using multiple unimodal transformer blocks in the intermediate network of the sound generation model, outputting the second intermediate feature. Thus, after multimodal processing, the audio features are further refined using unimodal Transformer blocks, enhancing their internal structure and detail expression.

[0137] Step A8: The frame alignment condition vector and the second intermediate feature are processed through the normalization layer and convolutional layer of the intermediate network in the sound generation model to output the target audio features.

[0138] Thus, the intermediate network in the diffusion model combines environmental features and preferred text features to iteratively generate a compressed audio representation from random noise, gradually approximating the target audio features. Guided by visual input and natural language prompts, this process generates synchronous and realistic target audio features that are closely aligned with the prompts.

[0139] Step A9 involves decoding the target audio features using the streaming model within the sound generation model to obtain the target ambient sound for the target user. Thus, by using the streaming model as the decoding network, the compressed target audio features are restored to a complete audio waveform, completing the synthesis of personalized ambient sound and achieving the final ambient sound output.

[0140] Step A10 involves real-time monitoring of changes in the user's environment and their preferences for ambient sounds. Based on these changes, the ambient sound parameters generated by the sound generation model are adjusted to dynamically adjust and incrementally update the generated ambient sounds, adapting to the user's needs. Furthermore, this eliminates the need to re-download the entire audio file.

[0141] The inventors of this application obtained the following results through experiments: (1) Taking environmental information as an example of an environmental video used to indicate "park environment", for the same park environment, a target environmental sound with the sound of gurgling water can be generated for a user whose preference information is "likes streams", and a target environmental sound with the sound of rustling leaves concentrated in the 200 to 800 Hz can be generated for a user whose preference information is "likes white noise". In this way, compared with the solution of using pre-recorded fixed audio materials in related technologies, the embodiment of this application uses environmental information and preference information as joint conditions to generate environmental sound diffusion, which can realize personalized environmental sound generation and improve user subjective satisfaction (MOS) by 0.82.

[0142] (2) Taking the terminal's processor as Qualcomm Snapdragon 778G and the ambient sound audio specifications as 44.1kHz, 16bit, and stereo as an example, the terminal's central processing unit utilization rate is 27%, and the first frame latency of the ambient sound generated by the terminal is 168ms. The ambient sound, which loops seamlessly from 8s to 30s, can meet the rigid requirement of less than 200ms for audio-visual synchronization latency of AR glasses. Thus, compared with the rule-based splicing method used in related technologies, since the sound generation model in this application uses a streaming model, the target ambient sound is streamed and decoded from the latent space of the diffusion model during the diffusion generation process. This can effectively reduce the first frame latency, meet the audio-visual synchronization requirements of wearable devices, and is conducive to its widespread application in wearable devices.

[0143] (3) By using audio recognition tools (such as Apple's Shazam tool and YouTube's ContentID tool) for double-blind detection, the ambient sound generated by the sound generation model has a similarity of 0% (threshold 3%) to the commercial audio material resource (SoundEffects, SFX) library, which can effectively avoid copyright risks.

[0144] (4) After the terminal downloads 18MB of basic weights (i.e., the initial global model parameters) for the first time, the terminal performs incremental fine-tuning of the model based on the data stored locally. The terminal only sends 2 to 4KB of event vectors back to the server (i.e., the model parameters fine-tuned locally by the terminal). In this way, traffic can be saved by 99.2%. The server aggregates all events (i.e., aggregates the fine-tuned model parameters sent by multiple terminals) at a fixed period (e.g., every 7 days) using the FedAvg algorithm. The server sends 1.2MB of differential weights (i.e., incremental model parameters) to each terminal so that the terminal can update the model parameters of the sound generation model. In this way, the model can become more and more accurate with use.

[0145] Embodiments of this application provide an ambient sound generation device. Figure 6This application provides a schematic diagram of an ambient sound generation device, which may include: The first acquisition unit 601 is used to acquire environmental information of the target user's environment and the target user's preference information for environmental sounds; wherein, the environmental information includes: environmental video or environmental image, and the preference information includes: positive prompt words and negative prompt words; The generation unit 602 is used to generate target ambient sound for the target user based on environmental information and preference information through a pre-trained sound generation model; wherein the sound generation model is a diffusion model built based on a transformer model and a flow model.

[0146] In some exemplary embodiments, the sound generation model includes an encoding network, an intermediate network, and a decoding network. The intermediate network includes a conditional generation module and a diffusion processing module based on a transformer model. The decoding network adopts a streaming model. The generation unit processes environmental information, preference information, and random noise through an encoding network to obtain environmental features corresponding to the environmental information, preference text features corresponding to the preference information, and latent variables corresponding to the random noise. Based on the environmental features and preference text features, a conditional generation module generates a conditional vector. Based on the environmental features, preference text features, latent variables, and conditional vector, a diffusion processing module obtains the target audio features. Finally, a streaming model decodes the target audio features to obtain the target ambient sound for the target user.

[0147] In some exemplary embodiments, the diffusion processing module includes: a plurality of multimodal transformer blocks cascaded in sequence, a plurality of single-modal transformer blocks cascaded in sequence, a normalization layer and a convolutional layer, wherein the output of the last multimodal transformer block is used as the input of the first single-modal transformer block; The generation unit is used to obtain a first intermediate feature based on environmental features, preference text features, latent variables, and conditional vectors through multiple multimodal transformer blocks; based on the first intermediate feature, it obtains a second intermediate feature through multiple single-modal transformer blocks; and based on the second intermediate feature, it obtains the target audio feature through a normalization layer and a convolutional layer.

[0148] In some exemplary embodiments, the multimodal transformer block includes an attention module with a joint attention mechanism and three feedforward neural networks, and the single-modal transformer block includes an attention module with a self-attention mechanism and a feedforward neural network.

[0149] In some exemplary embodiments, the environmental features include: environmental synchronization features and environmental visual features; the condition vector includes: frame alignment condition vector and global condition vector; the generation unit is used to obtain the global condition vector based on the environmental visual features and preferred text features through the condition generation module; and to obtain the frame alignment condition vector based on the environmental synchronization features and the global condition vector through the condition generation module.

[0150] In some exemplary embodiments, the device further includes: The response unit is used to obtain change information in response to one or more of the detection of changes in the target user's environment and changes in the target user's preference for ambient sound. The second acquisition unit is used to adjust the environmental sound parameters generated by the sound generation model based on the change information, so as to obtain the adjusted environmental sound.

[0151] In some exemplary embodiments, the device further includes: The third acquisition unit is used to acquire a multimodal sample dataset, which includes: multiple sample data, each sample data including: sample ambient sound and its corresponding sample video and sample label information; The fourth acquisition unit is used to train the model to be trained based on the multimodal sample dataset to obtain the sound generation model.

[0152] Embodiments of this application provide an electronic device, which may include: Memory is used to store executable instructions or computer programs. When a processor executes computer-executable instructions or computer programs stored in memory, it implements one or more of the environmental sound generation methods provided in the embodiments of this application.

[0153] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. This electronic device can be applied to... Figure 1 In a corresponding embodiment, an ambient sound generation method is provided. For example... Figure 7 As shown, the electronic device 700 may include a processor 701, a memory 702, and a bus 703. The various components in the electronic device 700 are coupled together via the bus 703. Wherein: Bus 703 is used to realize the communication connection between processor 701 and memory 702; Memory 702 is used to store computer-executable instructions or computer programs; When the processor 701 executes computer-executable instructions or computer programs stored in the memory 702, it implements the ambient sound generation method in one or more exemplary embodiments described above.

[0154] In some exemplary embodiments, bus 703 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. For example, bus 703 may be divided into a data bus, a power bus, an address bus, a control bus, a status signal bus, etc. For ease of illustration, in... Figure 7 The bus 703 is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0155] In some exemplary embodiments, the electronic device can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or it can be implemented as a server. For example, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0156] In some exemplary embodiments, the processor may be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or any conventional processor, etc.

[0157] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the ambient sound generation method provided in this application can be implemented. For example, ... Figure 1 The method for generating ambient sound is shown.

[0158] This application provides a computer program product, which includes a computer program or computer-executable instructions. When the computer-executable instructions or the computer program are executed by a processor, the ambient sound generation method provided in this application can be implemented. For example, ... Figure 1An ambient sound generation method is illustrated. For example, the computer program or computer-executable instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform the ambient sound generation method provided in the embodiments of this application.

[0159] In some exemplary embodiments, the aforementioned computer-readable storage medium / memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; or it may be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0160] In some exemplary embodiments, a computer program or computer-executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0161] As an example, a computer program or computer-executable instructions may, but not necessarily, correspond to a file in a file system. It may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0162] As an example, a computer program or computer-executable instructions may be deployed to execute on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected by a communication network.

[0163] It should be noted that the descriptions of the above device, storage medium, or product embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device, storage medium, or product embodiments of this application, those skilled in the art should refer to the descriptions of the method embodiments of this disclosure for understanding. Further details will not be repeated here.

[0164] The features disclosed in the several methods, devices, storage media or product embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments, devices, storage media or product embodiments.

[0165] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for generating ambient sound, characterized in that, The method includes: Obtain environmental information about the target user's environment and the target user's preference information for ambient sounds; wherein, the environmental information includes: environmental video or environmental image, and the preference information includes: positive cue words and negative cue words; Based on the environmental information and the preference information, a target ambient sound is generated for the target user using a pre-trained sound generation model; wherein the sound generation model is a diffusion model built based on a transformer model and a stream model.

2. The method according to claim 1, characterized in that, The sound generation model includes an encoding network, an intermediate network, and a decoding network. The intermediate network includes a conditional generation module and a diffusion processing module based on a transformer model. The decoding network adopts the stream model. The step of generating target ambient sound for the target user based on the environmental information and the preference information using a pre-trained sound generation model includes: The environmental information, preference information, and random noise are processed through the coding network to obtain environmental features corresponding to the environmental information, preference text features corresponding to the preference information, and latent variables corresponding to the random noise. Based on the environmental features and the preference text features, a condition vector is generated through the condition generation module; Based on the environmental features, the preferred text features, the latent variables, and the conditional vector, the target audio features are obtained through the diffusion processing module. The target audio features are decoded using the streaming model to obtain the target ambient sound for the target user.

3. The method according to claim 2, characterized in that, The diffusion processing module includes: multiple multimodal transformer blocks cascaded in sequence, multiple single-modal transformer blocks cascaded in sequence, a normalization layer and a convolutional layer, with the output of the last multimodal transformer block serving as the input of the first single-modal transformer block; The process of obtaining target audio features through the diffusion processing module based on the environmental features, the preferred text features, the latent variables, and the conditional vector includes: Based on the environmental features, the preferred text features, the latent variables, and the conditional vector, the first intermediate feature is obtained through the multiple multimodal transformer blocks; Based on the first intermediate feature, a second intermediate feature is obtained through the plurality of single-mode converter blocks; Based on the second intermediate feature, the target audio feature is obtained through the normalization layer and the convolutional layer.

4. The method according to claim 3, characterized in that, The multimodal converter block includes an attention module with a joint attention mechanism and three feedforward neural networks, and the single-modal converter block includes an attention module with a self-attention mechanism and a feedforward neural network.

5. The method according to claim 2, characterized in that, The environmental features include: environmental synchronization features and environmental visual features; the condition vector includes: frame alignment condition vector and global condition vector. The step of generating a condition vector based on the environmental features and the preferred text features through the condition generation module includes: Based on the environmental visual features and the preferred text features, the global condition vector is obtained through the condition generation module; Based on the environmental synchronization features and the global condition vector, the frame alignment condition vector is obtained through the condition generation module.

6. The method according to claim 1, characterized in that, After generating the target ambient sound for the target user, the method further includes: In response to detecting one or more of the following: a change in the environment of the target user and a change in the target user's preference for ambient sound, change information is obtained; Based on the change information, the environmental sound parameters generated by the sound generation model are adjusted to obtain the adjusted environmental sound.

7. The method according to claim 1, characterized in that, Before obtaining the environmental information of the target user's environment and the target user's preference information for ambient sound, the method further includes: Obtain a multimodal sample dataset, wherein the multimodal sample dataset includes: multiple sample data, each sample data including: sample ambient sound and its corresponding sample video and sample label information; Based on the multimodal sample dataset, the model to be trained is trained to obtain the sound generation model.

8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, the method as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, the method as described in any one of claims 1 to 7 is implemented.

Images