Audio generation method, training of spatial audio generation model, pre-training method

Abstract

Embodiments of the present specification provide an audio generation method, training of a spatial audio generation model, and a pre-training method, wherein the audio generation method comprises: obtaining a target image; performing visual coding on the target image based on spatial layout information of the target image to obtain visual spatial features; fusing the visual spatial features and audio features of random noise to obtain visual spatial audio features; and performing transformation mapping processing on the visual spatial audio features to generate spatial audio data corresponding to the target image. By obtaining the target image and performing visual coding based on the spatial layout information in the target image, the spatial information in the image can be accurately captured, providing a visual basis that fits the spatial content for subsequent audio generation. By performing transformation mapping processing on the fused visual spatial audio features to generate spatial audio data, spatial audio that is highly adapted to the spatial video can be generated based on the visual information of the target image, so that the audio content accurately fits the video content.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to an audio generation method, a training method for a spatial audio generation model, and a pre-training method. Background Technology

[0002] With the continuous development and advancement of technology, the demand for immersive and realistic audiovisual experiences continues to grow. A 360-degree camera is a device capable of capturing panoramic images or videos, capturing scene information from all 360 degrees around the shooting point. In recent years, the application of 360-degree cameras has become increasingly widespread, and consequently, the demand for spatial audio generation technology that can be adapted to panoramic videos is also increasing daily.

[0003] However, audio generated by traditional audio generation techniques is mostly non-spatial audio, and the audio content is difficult to match with the panoramic video content, making it difficult to construct a complete auditory scene that conforms to human perception. Therefore, there is an urgent need for a spatial audio generation method for panoramic videos. Summary of the Invention

[0004] In view of this, embodiments of this specification provide an audio generation method. One or more embodiments of this specification also relate to a training method for a spatial audio generation model, a pre-training method for a spatial audio generation model, a task platform, a model training platform, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an audio generation method is provided, comprising:

[0006] Acquire the target image;

[0007] Based on the spatial layout information of the target image, the target image is visually encoded to obtain visual spatial features;

[0008] The audio features of visual spatial features and random noise are fused to obtain visual spatial audio features;

[0009] The visual spatial audio features are transformed and mapped to generate spatial audio data corresponding to the target image.

[0010] According to a second aspect of the embodiments of this specification, a method for training a spatial audio generation model is provided, comprising:

[0011] Obtain the sample image and the corresponding label space audio data of the sample image;

[0012] Based on the sample spatial layout information of the sample images, the sample images are visually encoded to obtain the visual spatial features of the samples.

[0013] The visual spatial features of the sample and the audio features of random noise are fused to obtain the visual spatial audio features of the sample.

[0014] The sample visual spatial audio features are input into the initial spatial audio generation model. The sample visual spatial audio features are then transformed and mapped to generate the predicted spatial audio data corresponding to the sample image. The initial spatial audio generation model is obtained by pre-training the flow model. The transformation and mapping process is used to transform and map the sample visual spatial audio features into a sample spatial audio distribution used to generate the predicted spatial audio data.

[0015] Based on the predicted spatial audio data and the labeled spatial audio data, an initial spatial audio generation model is trained to obtain a trained spatial audio generation model.

[0016] According to a third aspect of the embodiments of this specification, a pre-training method for a spatial audio generation model is provided, comprising:

[0017] Acquire pre-trained spatial audio data;

[0018] Audio encoding is performed on the pre-trained spatial audio data to obtain pre-trained spatial audio features;

[0019] Add a mask to the pre-trained spatial audio features to obtain the pre-trained spatial audio mask features;

[0020] The pre-trained spatial audio mask features and the audio features of random noise are fused to obtain the pre-trained spatial audio mask features with added noise.

[0021] The pre-trained spatial audio mask features with added noise are input into the stream model, and the pre-trained spatial audio mask features with added noise are transformed and mapped to generate predicted spatial audio features.

[0022] Based on predicted spatial audio features and pre-trained spatial audio features, a pre-trained flow model is used to obtain a pre-trained initial spatial audio generation model.

[0023] According to a fourth aspect of the embodiments of this specification, a task platform is provided, including a request interface and an audio generation response unit;

[0024] The request interface is used to receive task generation requests sent by terminal devices, wherein the task generation request includes request information;

[0025] The audio generation response unit is used to acquire a target image based on request information; visually encode the target image based on its spatial layout information to obtain visual spatial features; fuse the visual spatial features and the audio features of random noise to obtain visual spatial audio features; perform transformation mapping processing on the visual spatial audio features to generate spatial audio data corresponding to the target image; and feed the spatial audio data back to the terminal device.

[0026] According to a fifth aspect of the embodiments of this specification, a model training platform is provided, including a training interface and a training response unit;

[0027] The training interface is used to receive model training requests sent by terminal devices. The model training request includes request information.

[0028] The training response unit is used to acquire sample images and corresponding label spatial audio data based on request information; visually encode the sample images based on the sample spatial layout information to obtain sample visual spatial features; fuse the sample visual spatial features and the audio features of random noise to obtain sample visual spatial audio features; input the sample visual spatial audio features into an initial spatial audio generation model, perform transformation mapping processing on the sample visual spatial audio features, and generate predicted spatial audio data corresponding to the sample images. The initial spatial audio generation model is obtained by pre-training a convection model, and the transformation mapping processing is used to transform and map the sample visual spatial audio features into a sample spatial audio distribution used to generate the predicted spatial audio data; train the initial spatial audio generation model based on the predicted spatial audio data and label spatial audio data to obtain a trained spatial audio generation model; and feed back the model parameters of the trained spatial audio generation model to the terminal device.

[0029] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0030] Memory and processor;

[0031] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0032] According to a seventh aspect of an embodiment of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the method described above.

[0033] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0034] One embodiment of this specification implements the following: acquiring a target image; visually encoding the target image based on its spatial layout information to obtain visual spatial features; fusing the visual spatial features with the audio features of random noise to obtain visual spatial audio features; and performing transformation mapping processing on the visual spatial audio features to generate spatial audio data corresponding to the target image. By acquiring the target image and visually encoding it based on its spatial layout information to obtain visual spatial features, the spatial information in the image can be accurately captured, providing a visual basis for subsequent audio generation that fits the spatial content. Fusing the visual spatial features with the audio features of random noise increases the diversity and complexity of the audio features, making the generated audio richer than in a real scene. Performing transformation mapping processing on the fused visual spatial audio features to generate spatial audio data can generate spatial audio that is highly adapted to the spatial video based on the visual information of the target image, thereby making the audio content accurately fit the video content, constructing a complete auditory scene that conforms to human perception, improving the accuracy of spatial audio generation, and enhancing the immersive audiovisual experience. Attached Figure Description

[0035] Figure 1 This is an application architecture diagram for audio generation provided in one embodiment of this specification;

[0036] Figure 2 This is a flowchart of an audio generation method provided in one embodiment of this specification;

[0037] Figure 3 This is a flowchart illustrating the processing steps of an audio generation method provided in one embodiment of this specification.

[0038] Figure 4 This is a flowchart illustrating a training method for a spatial audio generation model provided in one embodiment of this specification;

[0039] Figure 5 This is a flowchart illustrating a pre-training method for a spatial audio generation model provided in one embodiment of this specification;

[0040] Figure 6 This is a schematic diagram of the pre-training stage of a spatial audio generation model provided in one embodiment of this specification;

[0041] Figure 7 This is a schematic diagram of the fine-tuning stage of a spatial audio generation model provided in one embodiment of this specification;

[0042] Figure 8 This is a schematic diagram illustrating the construction process of a 360-degree sphere dataset provided in one embodiment of this specification;

[0043] Figure 9This is a schematic diagram of the structure of a task platform provided in one embodiment of this specification;

[0044] Figure 10 This is a schematic diagram of the structure of a model training platform provided in one embodiment of this specification;

[0045] Figure 11 This is a schematic diagram of the structure of an audio generation device provided in one embodiment of this specification;

[0046] Figure 12 This is a schematic diagram of the structure of a training device for a spatial audio generation model provided in one embodiment of this specification;

[0047] Figure 13 This is a schematic diagram of the structure of a pre-training device for a spatial audio generation model provided in one embodiment of this specification;

[0048] Figure 14 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0049] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0050] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0051] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0052] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0053] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0054] 360V2SA (360-degree video to spatial audio generation) is a task that generates real-time, immersive spatial audio from 360-degree video.

[0055] FoV (Field-of-View): This refers to the limited field of view that a camera can capture in traditional video. It can be understood as a local field of view or a fixed-direction field of view.

[0056] Non-spatial audio: is an audio format that does not contain directional information.

[0057] FOA (First-order omnidirectional audio): is a 3D audio standard format that captures the directionality of sound and is suitable for real 3D audio playback.

[0058] Omn iAud io (Omn iAud io): is a spatial audio generation framework for generating first-order omnidirectional audio from 360-degree video.

[0059] Sphere360 is a new dataset designed for the 360V2SA mission, containing real-world video-audio pairing data.

[0060] Dual-branch Framework: A system architecture that uses two independent paths to process information from different dimensions in order to improve system performance.

[0061] This specification provides an audio generation method. It also relates to a training method for a spatial audio generation model, a pre-training method for a spatial audio generation model, a task platform, a model training platform, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0062] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the audio generation method provided in this application can be applied to, for example, Figure 1 The application scenarios shown are not limited to these. In, for example... Figure 1 In the application scenario shown, the large model is deployed on server 104. Server 104 can connect to one or more clients 102 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Clients 102 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Clients 102 can interact with users through a graphical user interface to invoke the large model, thereby implementing the methods provided in the embodiments of this specification.

[0063] In the embodiments of this specification, the system consisting of a client and a server can perform the following steps: The client sends the video to be processed to the server. The server obtains the target image corresponding to the video to be processed; based on the spatial layout information of the target image, it performs visual encoding on the target image to obtain visual spatial features; it fuses the visual spatial features and the audio features of random noise to obtain visual spatial audio features; it performs transformation mapping processing on the visual spatial audio features to generate spatial audio data corresponding to the target image. It should be noted that, provided that the client's runtime resources can meet the deployment and running conditions of a large model, the embodiments of this application can be performed on the client.

[0064] See Figure 2 , Figure 2 A flowchart of an audio generation method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0065] Step 202: Obtain the target image.

[0066] The embodiments described in this specification can be applied to applications, platforms, or systems with task processing capabilities. Specific tasks may include audio generation tasks, video-to-audio generation tasks, etc. For example, it could be a 360V2SA task, i.e., a 360-degree video-to-spatial audio generation task.

[0067] In practical applications, the target image can be obtained during the process of generating corresponding audio for a video.

[0068] Specifically, the target image can be understood as the image data used as input during the audio generation process. Its content typically includes information such as the scene and objects related to the audio to be generated. The target image can be a still picture or a frame from a video.

[0069] According to one alternative implementation, a pre-stored video file can be read from a local storage device (such as a hard drive, SD card, etc.) to obtain the target image; or the video can be downloaded from a remote server via a network interface, such as from a video library or cloud storage; for real-time application scenarios, images can also be captured in real time via an image acquisition device (such as a camera).

[0070] By acquiring the target image, a data foundation is provided for the subsequent audio generation process.

[0071] Step 204: Based on the spatial layout information of the target image, perform visual encoding on the target image to obtain visual spatial features.

[0072] In practical applications, based on the acquisition of the target image, visual encoding can be performed on the target image based on its spatial layout information to obtain visual spatial features.

[0073] Specifically, the spatial layout information of a target image can be understood as the distribution of different elements in the target image in terms of spatial location, including the position, size, shape of objects, and their relative positions and layout relationships. When the target image is a panoramic image, the spatial layout information can include the position, orientation, size, and interrelationships of each element in three-dimensional space, such as scene structure information, object position and orientation information, depth information, and the spatial proportions between different objects in the image. Visual encoding can be understood as the process of converting image information into feature representations that computers can understand and process, extracting key information from the image through specific algorithms. Visual spatial features can be understood as feature vectors or matrices obtained after visual encoding that reflect the spatial distribution characteristics of the target image, including the spatial structure and semantic information of the target image.

[0074] According to one alternative implementation, a pre-trained image feature extraction model can be used to extract visual spatial features from the target image; alternatively, a specific neural network, such as a combination of multi-layer convolutional neural networks and pooling layers, can be used to extract visual spatial features from the target image.

[0075] By visually encoding the target image and extracting visual spatial features, the complex information in the image can be transformed into a feature representation that can be processed by a computer. These features preserve the spatial layout information of the image, providing an accurate visual basis for subsequent fusion with audio features, making the generated audio more in line with human auditory perception.

[0076] Step 206: Fuse the visual spatial features and the audio features of random noise to obtain visual spatial audio features.

[0077] In practical applications, based on visual encoding of the target image to obtain visual spatial features, the visual spatial features and the audio features of random noise can be fused to obtain visual spatial audio features.

[0078] Specifically, the audio characteristics of random noise can be understood as a random variable that follows a normal distribution. For example, it can be a simple random variable that follows a standard normal distribution N(x0, x...). t Visuospatial audio features can characterize the spatial distribution of randomly generated audio, which is usually a simple distribution. Visuospatial audio features can be understood as visual and audio fusion features that integrate visual and audio information. They include spatial layout information and semantic information of the target image, as well as random audio features, and can be used to guide the subsequent audio generation process.

[0079] By fusing visual spatial features with audio features derived from random noise, visual information provides scene and content constraints for audio generation. Under these constraints, complex audio features can be derived step-by-step from a simple random audio distribution, thereby guiding the generation of richer and more realistic spatial audio.

[0080] Step 208: Perform transformation mapping on the visual spatial audio features to generate spatial audio data corresponding to the target image.

[0081] In practical applications, based on the fusion of visual spatial features and audio features of random noise to obtain visual spatial audio features, the visual spatial audio features can be transformed and mapped to generate spatial audio data corresponding to the target image.

[0082] Specifically, transform mapping can be understood as a flow-based generative method that, under the constraints of visual spatial features, can generate complex data distributions from a simple distribution (such as a Gaussian distribution). In audio generation, it can generate high-quality spatial audio data from visual spatial audio features. Spatial audio data can be understood as audio data with spatial information; specifically, it can be FOA format audio data, which is the FOA audio corresponding to the target image, capable of simulating the propagation and localization effects of sound in three-dimensional space.

[0083] According to an optional implementation, visual spatial audio features can be input into a trained streaming model, and the streaming model can perform transformation mapping on the visual spatial audio features to generate spatial audio data corresponding to the target image.

[0084] Furthermore, the flow model, based on long differential equations, simulates the continuously changing trajectories of each data point in the latent space over time, thereby progressively inferring a complex target distribution from a random distribution. Here, the random distribution can be understood as visual spatial audio features, and the target distribution can be understood as the distribution corresponding to the spatial audio data.

[0085] This embodiment involves acquiring a target image; visually encoding the target image based on its spatial layout information to obtain visual spatial features; fusing the visual spatial features with the audio features of random noise to obtain visual spatial audio features; and performing transformation mapping on the visual spatial audio features to generate spatial audio data corresponding to the target image. By acquiring the target image and visually encoding it based on its spatial layout information to obtain visual spatial features, the spatial information in the image can be accurately captured, providing a visual basis for subsequent audio generation that fits the spatial content. Fusing the visual spatial features with the audio features of random noise increases the diversity and complexity of the audio features, making the generated audio richer than in a real scene. Performing transformation mapping on the fused visual spatial audio features to generate spatial audio data can generate spatial audio that is highly adapted to the spatial video based on the visual information of the target image, thereby ensuring that the audio content accurately matches the video content, constructing a complete auditory scene that conforms to human perception, improving the accuracy of spatial audio generation, and enhancing the immersive audiovisual experience.

[0086] In one optional embodiment, the target image is a panoramic image; before visually encoding the target image based on the spatial layout information of the target image to obtain visual spatial features, it may further include: performing field-view segmentation processing on the target image based on the target field-view angle to obtain a target view image, wherein the target view image is an image of the target image under the target field-view angle.

[0087] Accordingly, based on the spatial layout information of the target image, the target image is visually encoded to obtain visual spatial features, including: based on the spatial layout information of the target view image under the target field of view angle, the target view image is visually encoded to obtain visual spatial features.

[0088] Specifically, the target image can be understood as a video frame in the panoramic video to be processed. The panoramic video to be processed is a 360-degree video, obtained based on a 360-degree camera. The target image is a panoramic image containing spatial information. View segmentation can be understood as dividing the panoramic image according to different view angle ranges, extracting a portion of the image from a specific angle range from the complete information of the panorama, so as to process it for a specific viewpoint later. The target view angle can be understood as a specific view angle range of interest preset by the user or system, which is used to determine the viewpoint boundary of the portion of the image segmented from the panoramic image. For example, the target image can contain multiple view angles such as front and back (front and back), left and right, and up and down, and the target view angle can be the default frontal view.

[0089] According to an optional implementation, the target image is segmented to obtain a target image with a target field of view angle, which can be achieved through coordinate transformation.

[0090] In practical applications, panoramic images are typically stored in a specific format, such as a rectangular projection format. Using mathematical coordinate transformation formulas, the two-dimensional planar coordinates of the panoramic image can be mapped to three-dimensional spherical coordinates. Based on the target field of view angle, a corresponding range is determined in spherical coordinates. Then, the coordinates within this range are inversely transformed back to a two-dimensional plane, thereby extracting the target image from the target field of view angle. For example, when the target field of view angle is a frontal view, the extracted image is the range where both the horizontal and vertical angles are zero.

[0091] According to another alternative implementation, the panoramic image can also be preprocessed using an image segmentation algorithm to convert the target field of view angle information into input parameters that the algorithm can recognize. The algorithm identifies the set of pixels within the target field of view angle by classifying each pixel in the image, and then extracts the target image.

[0092] Specifically, visual spatial features can be understood as local visual features from the perspective of the target's field of view, including spatial structural information and image semantic information from the perspective of the target's field of view.

[0093] According to an alternative implementation, a pre-trained image encoder can be used to visually encode the target image based on the spatial layout information of the target image at the target field of view, thereby obtaining visual spatial features.

[0094] Specifically, the pre-trained image encoder can be the MetaCLIP-Huge image encoder. The MetaCLIP-Huge image encoder is pre-trained on large-scale image datasets, accumulating rich experience in image feature extraction and semantic understanding. Using it for visual encoding of target images can leverage this prior knowledge to more efficiently and accurately extract the spatial distribution features of the target image from the target's visual perspective, significantly reducing training time and data requirements compared to training a model from scratch. It should be noted that other image encoders with similar functionalities can also be selected based on the specific needs of the application.

[0095] By performing field-view segmentation on the target image and visually encoding it based on the spatial layout information of the target image at the target field-view angle, visual spatial features are obtained. This enables the subsequently generated visual spatial features and the final audio to better match the scene that the user is actually interested in, improving the accuracy of the experience. The pre-trained image encoder can deeply mine the spatial structure information of the target image at a specific field-view angle, and the generated visual spatial features can more accurately reflect the spatial positional relationships of objects in the scene. This allows the generated audio to better match the image scene in terms of spatial dimensions such as direction and distance, enhancing the realism of audiovisual fusion.

[0096] In one optional embodiment, fusing visual spatial features and audio features of random noise to obtain visual spatial audio features may include: aligning the visual spatial features with the audio features in terms of feature dimensions to obtain aligned visual spatial features; and concatenating the audio features with the aligned visual spatial features element by element to obtain visual spatial audio features.

[0097] Specifically, aligned visual spatial features can be understood as visual spatial features aligned with audio features in terms of feature dimensions. Feature dimension alignment means ensuring consistency between visual spatial features and audio features in terms of feature dimensions, enabling effective feature fusion operations later. Element concatenation can be understood as a feature fusion method that achieves feature fusion by adding elements of the two features at the same positions. Visual spatial audio features can be understood as the features resulting from the fusion of visual spatial features and audio features, including spatial structural information of the target image, visual semantic information, and random audio distribution features. These can serve as constraints to guide the subsequent spatial audio generation process.

[0098] According to one optional implementation, aligning the visual spatial features with the audio features in terms of feature dimensions to obtain aligned visual spatial features may include: sampling the visual spatial features to the same length as the audio features. Further, sampling the visual spatial features to the same length as the audio features may include: aligning the length of the visual spatial features with the length of the audio features through interpolation.

[0099] According to one optional implementation, element-wise concatenation of audio features and aligned visual spatial features may include fusing the audio features and aligned visual spatial features through element-wise addition. Further, fusing the audio features and aligned visual spatial features through element-wise addition may include adding the element or variable values ​​of the two features at the same position.

[0100] By aligning visual spatial features with audio features in terms of feature dimensions, this embodiment enables temporal alignment between visual spatial features and audio features, ensuring temporal consistency between them. By fusing visual spatial features and audio features, spatial information in the image can be better combined with the characteristics of the audio when generating spatial audio, making the generated spatial audio more in line with human auditory perception and the needs of actual scenarios.

[0101] In an optional embodiment, before performing transformation mapping processing on the visual spatial audio features to generate spatial audio data corresponding to the target image, the method may further include: visually encoding the target image based on the visual content information of the target image to obtain visual semantic features.

[0102] Accordingly, performing transformation mapping processing on visual spatial audio features to generate spatial audio data corresponding to the target image can include: performing transformation mapping processing on visual spatial audio features under the guidance of visual semantic features to generate spatial audio data corresponding to the target image.

[0103] Specifically, visual content information can be understood as semantic information such as specific objects, scenes, and actions contained in the target image. For example, an image might depict a forest or ocean scene, someone running, or the location and color of vehicles. Visual semantic features can be understood as features obtained by encoding the visual content information of the target image. They reflect the semantic meaning of the target image and help understand the theme and key information expressed by the image. Visual semantic features can also be called global features of the target image. In the case of a panoramic image, visual semantic features are global features obtained by encoding the content from various viewpoints within the panoramic image. Visuospatial audio features can be understood as feature representations that integrate visual spatial information and audio features (such as the audio features of random noise), including the spatial distribution of the image and certain audio characteristics. Transformation mapping processing can be understood as a generation method based on a flow model structure, capable of inferring complex data distributions from a simple distribution (such as a Gaussian distribution). Here, it is used to generate spatial audio data from visual spatial audio features. A flow model can be understood as a generative model, consisting of a series of reversible transformations that can transform a simple distribution (such as the standard normal distribution) into a complex data distribution.

[0104] According to an optional implementation, visual encoding of the target image based on the visual content information of the target image to obtain visual semantic features may include: visual encoding of the target image based on the visual content information of the target image through a pre-trained image encoder to obtain visual semantic features.

[0105] Specifically, the pre-trained image encoder can be the same image encoder used for extracting local visual features as described above. For example, a pre-trained MetaCLIP-Huge image encoder can be used.

[0106] According to an optional implementation, under the guidance of visual semantic features, a transformation mapping process is performed on visual spatial audio features to generate spatial audio data corresponding to the target image. This may include: inputting visual semantic features and visual spatial audio features into a trained spatial audio generation model; performing a transformation mapping process on visual spatial audio features under the constraints of visual semantic features to obtain the target spatial audio distribution; and restoring the target spatial audio distribution to spatial audio data.

[0107] Furthermore, reconstructing the target spatial audio distribution into spatial audio data can include:

[0108] The target spatial audio distribution is restored to spatial audio data through a variational autoencoder.

[0109] Specifically, a spatial audio generation model can be understood as a generative model within a streaming model architecture. The target spatial audio distribution can be understood as a complex audio distribution derived from a simple audio distribution under constraints by the streaming model. A variational autoencoder can be understood as a pre-trained spatial audio encoder. A variational autoencoder can compress FOA-formatted audio data into a latent audio representation, and it can also restore the spatial audio distribution to spatial audio data.

[0110] According to one alternative implementation, the variational autoencoder can be trained using non-spatial format audio data; after initializing the trained variational autoencoder, it is then fine-tuned using spatial audio data from the Sphere 360 ​​dataset.

[0111] In this embodiment, through a dual-branch framework and a pre-trained image encoder, one branch performs field-view segmentation on the target image and extracts local field-view features from the target field-view angle; the other branch extracts global features from the target image. Thus, through the dual-branch video representation architecture, the fusion of local and global information is achieved, making the generated spatial audio and video content closely integrated.

[0112] In another optional embodiment, guided by visual semantic features, the visual spatial audio features are transformed and mapped to generate spatial audio data corresponding to the target image. This may further include: guided by visual semantic features, performing at least one time-step transformation and mapping on the visual spatial audio features to obtain the target spatial audio distribution, wherein the time-step transformation and mapping is performed by transforming and mapping the visual spatial audio features or the transformation and mapping result of the previous time step under the guidance of visual semantic features; and decoding the target spatial audio distribution to generate spatial audio data corresponding to the target image.

[0113] In practical implementation, the flow model structure can include one or more transformation mapping layers, each corresponding to a transformation mapping process at a given time step. When the time step is the first time step, the first transformation mapping layer, guided by visual semantic features, performs transformation mapping on the visual-spatial audio features to obtain the transformation mapping result. When the time steps are other than the first time step, each transformation mapping layer, guided by visual semantic features, performs transformation mapping on the transformation mapping result of the previous time step, ultimately obtaining the target spatial audio distribution via the last transformation mapping layer. It can be understood that the flow model simulates the flow trajectory of data in the latent space through a series of transformation mapping processes, ultimately transforming the original distribution (i.e., visual-spatial audio features) into the target spatial audio distribution.

[0114] Optionally, the target spatial audio distribution is decoded to generate spatial audio data corresponding to the target image, which is equivalent to restoring the distribution in the latent space to spatial audio data. This can be achieved through the decoding layer of a variational autoencoder.

[0115] In one optional embodiment, the target image is at least one video frame; after performing transformation mapping processing on visual spatial audio features based on visual semantic features to generate spatial audio data corresponding to the target image, the method may further include: generating a target video based on at least one video frame and the spatial audio data corresponding to at least one video frame.

[0116] Specifically, a video frame can be understood as any one of a series of consecutive images in the video to be processed; each image can be called a video frame, which is the basic unit of video. The target video can be understood as a complete audio-visual experience generated by combining at least one video frame with its corresponding spatial audio data, possessing a synchronized visual and auditory effect. Setting a time step can be understood as a constraint in the streaming model's processing; by setting the time step t, the model can know which stage of the diffusion process it is currently in, and thus perform corresponding processing based on the characteristics of that stage. For example, the time step t can serve as conditional information for the streaming model, influencing the model's parameters or transformation methods.

[0117] According to an optional implementation, generating a target video based on at least one video frame and spatial audio data corresponding to at least one video frame may include: combining at least one video frame and spatial audio data corresponding to at least one video frame using a video generation model, video editing software, or by using a programming language in conjunction with a video processing library to generate the target video.

[0118] By introducing a time step as a constraint in the transformation mapping process, this embodiment enables specific transformation operations on spatial audio features based on each time step. This allows for the gradual mapping of spatial audio features to the target distribution of spatial audio, resulting in more accurate spatial audio data and a high degree of matching between visual and auditory information in the target video. This provides users with a more realistic and coherent audiovisual experience. Guided by the set time steps and visual semantic information, the generated spatial audio data maintains a high degree of consistency with the video content, thus better conveying the semantics and emotions of the video and enriching its expressive layers.

[0119] For example, in a video featuring a person running, spatial audio can accurately synchronize the sounds of footsteps and breathing, enhancing the video's immersive experience. In a story-telling video, audio can be used to enhance the story's impact through different sound effects and music, tailored to the plot and chronological order.

[0120] In one optional embodiment, transforming and mapping visual spatial audio features to generate spatial audio data corresponding to a target image may include: inputting visual spatial audio features into a spatial audio generation model, performing transform and mapping on the visual spatial audio features, and generating spatial audio data corresponding to the target image. The spatial audio generation model is obtained by training an initial spatial audio generation model, and the transform and mapping process is used to transform and map the visual spatial audio features into a target spatial audio distribution for generating spatial audio data.

[0121] Specifically, a spatial audio generation model can be understood as a model built on a stream model architecture, obtained by training (or fine-tuning) an initial spatial audio generation model. The spatial audio generation model can generate spatial audio data corresponding to the target image through transformation mapping based on the input visual spatial audio features. The initial spatial audio generation model can be understood as the starting point for training the spatial audio generation model; it is a pre-trained stream model that has not yet undergone fine-tuning. The spatial audio data corresponding to the target image can be understood as FOA audio that matches the target image in terms of content and spatial information, simulating the propagation and localization effects of sound in three-dimensional space, providing users with an auditory experience that matches the target image scene.

[0122] Furthermore, the transformation mapping process may include, but is not limited to, affine coupling transformation, invertible convolution transformation, permutation transformation, and invertible nonlinear transformation, etc. The specific method can be determined according to the needs of the actual application, and this specification does not impose any limitations on it.

[0123] According to one alternative implementation, the spatial audio generation model can be obtained by training an initial spatial audio generation model; the initial spatial audio generation model can be obtained by pre-training a convection model.

[0124] Applying this embodiment, based on a flow model architecture, the flow model possesses powerful modeling capabilities and can learn complex data distributions. Through training on a large amount of image-audio pairing data, the model can accurately capture the mapping relationship between visual spatial audio features and spatial audio data, thereby generating high-quality spatial audio data that highly matches the target image, realizing the generation of spatial audio from panoramic video.

[0125] The following is in conjunction with the appendix Figure 3 Taking the application of the audio generation method provided in this specification in generating the first-order omnidirectional audio corresponding to panoramic video as an example, the audio generation method will be further explained. Among them, Figure 3 The present specification shows a flowchart of an audio generation method according to an embodiment, which includes the following steps.

[0126] Step 302: Obtain the panoramic video to be processed, which includes multiple panoramic images.

[0127] Step 304: Based on the spatial layout information of the target panoramic image in multiple panoramic images, perform visual encoding on the target panoramic image to obtain local field-of-view features.

[0128] Specifically, visual encoding of the target panoramic image to obtain local field-of-view features can include:

[0129] Field of view segmentation is performed on the target panoramic image to obtain the target panoramic image from the target field of view angle; based on the spatial layout information from the target field of view angle, the target panoramic image is visually encoded to obtain local field of view features.

[0130] Step 304': Based on the visual content information of the target panoramic image, perform visual encoding on the target panoramic image to obtain global features.

[0131] Step 306: Align the local visual features with the audio latent representation of random noise, and fuse the aligned local visual features with the audio latent representation through element-wise addition to obtain the visual spatial audio features.

[0132] Step 308: Based on global features, perform transformation mapping on the audio features in the visual space to obtain the audio spatial distribution.

[0133] Step 310: Use a variational autoencoder to restore the audio spatial distribution to the first-order omnidirectional audio data corresponding to the target panoramic image.

[0134] Step 312: Generate target panoramic audio and video based on multiple panoramic images and the first-order omnidirectional audio data corresponding to the multiple panoramic images.

[0135] In this embodiment, a dual-branch video representation architecture is used. One branch performs field-of-view segmentation on the target panoramic image and extracts local field-of-view features from the target field-of-view angle; the other branch extracts global features from the target panoramic image. Thus, the dual-branch video representation architecture achieves the fusion of local and global information, making the generated first-order omnidirectional audio data more closely match the video content of the panoramic video, realizing the generation of spatial audio from panoramic video and improving the accuracy of spatial audio generation.

[0136] See Figure 4 , Figure 4 A flowchart is shown of a training method for a spatial audio generation model according to an embodiment of this specification, which specifically includes the following steps.

[0137] Step 402: Obtain the sample image and the corresponding label space audio data of the sample image.

[0138] The embodiments in this specification can be applied to a model training platform, specifically for training an initial spatial audio generation model, which is part of the fine-tuning stage.

[0139] Specifically, sample images can be understood as image data used to train the spatial audio generation model. They can cover various scenes and types and serve as the initial visual input data for the spatial audio generation model. Labeled spatial audio data can be understood as real spatial audio data corresponding to the sample images. It possesses spatial information and can simulate the propagation and localization of sound in three-dimensional space; it is the target output for model training. Furthermore, labeled spatial audio data can be obtained from the Sphere 360 ​​dataset. The Sphere 360 ​​dataset includes panoramic videos and spatial audio matched to them. Spatial audio obtained based on the Sphere 360 ​​dataset constitutes labeled spatial audio data. Panoramic images contained in panoramic videos obtained based on the Sphere 360 ​​dataset constitute sample images.

[0140] According to one optional implementation, acquiring sample images and corresponding label spatial audio data may include: acquiring the Sphere 360 ​​dataset; and acquiring sample images and corresponding label spatial audio data based on multiple panoramic video-spatial audio pairing data included in the Sphere 360 ​​dataset.

[0141] Step 404: Based on the sample spatial layout information of the sample image, perform visual encoding on the sample image to obtain the sample visual spatial features.

[0142] Specifically, sample spatial layout information can be understood as the spatial distribution of different elements in a sample image, including object position, size, shape, and interrelationships. Visual encoding can be understood as the process of converting image information into feature representations that computers can understand and process, extracting key image information using specific algorithms. Sample visual spatial features can be understood as feature vectors or matrices obtained after visual encoding of a sample image, reflecting the spatial distribution characteristics of the sample image, and containing image spatial structure and semantic information.

[0143] In one optional embodiment, the sample image is a panoramic image; before encoding the sample image based on the sample spatial layout information of the sample image to obtain the sample visual spatial features, it may further include: performing field-of-view segmentation on the sample image to obtain a sample image of the target field-of-view angle.

[0144] Accordingly, visual encoding of the sample image based on the sample spatial layout information to obtain sample visual spatial features may include: visual encoding of the sample image based on the spatial layout information of the sample image under the target field of view to obtain sample visual spatial features.

[0145] Step 406: Fuse the visual spatial features of the sample with the audio features of random noise to obtain the visual spatial audio features of the sample.

[0146] In one optional embodiment, fusing sample visual space features and audio features of random noise to obtain sample visual space audio features may include: aligning the feature dimensions of the sample visual space features with the audio features of random noise to obtain aligned sample visual space features; and concatenating the audio features with the aligned sample visual space features element by element to obtain sample visual space audio features.

[0147] Specifically, the audio features of the sample visual space can be understood as features formed by fusing the audio features of the sample visual space and random noise. It contains both spatial information of the sample image and a certain degree of randomness.

[0148] Step 408: Input the sample visual spatial audio features into the initial spatial audio generation model, perform transformation mapping processing on the sample visual spatial audio features, and generate the predicted spatial audio data corresponding to the sample image. The initial spatial audio generation model is obtained by pre-training the flow model, and the transformation mapping processing is used to transform and map the sample visual spatial audio features into the sample spatial audio distribution used to generate the predicted spatial audio data.

[0149] Specifically, predicting spatial audio data can be understood as spatial audio data generated by the initial spatial audio generation model after transformation and mapping processing based on the visual spatial audio features of the samples.

[0150] In an optional embodiment, before inputting the sample visual spatial audio features into the initial spatial audio generation model, performing transformation mapping processing on the sample visual spatial audio features, and generating the predicted spatial audio data corresponding to the sample image, the method may further include: semantically encoding the sample image based on the visual content information of the sample image to obtain sample visual semantic features.

[0151] Accordingly, the sample visual spatial audio features are input into the initial spatial audio generation model, and the sample visual spatial audio features are transformed and mapped to generate the predicted spatial audio data corresponding to the sample image. This can include: inputting the sample visual semantic features and sample visual spatial audio features into the initial spatial audio generation model, and transforming and mapping the sample visual spatial audio features based on the sample visual semantic features to generate the predicted spatial audio data corresponding to the sample image.

[0152] It should be noted that the specific implementation methods of steps 402-408 are the same as those of steps 202-208 described above. The specific implementation methods of steps 402-408 can be found in steps 202-208, and will not be repeated here.

[0153] Step 410: Based on the predicted spatial audio data and the labeled spatial audio data, train the initial spatial audio generation model to obtain the trained spatial audio generation model.

[0154] Specifically, a trained spatial audio generation model can be understood as a model obtained after training the initial spatial audio generation model, which can accurately generate corresponding spatial audio data based on the input image.

[0155] By applying this embodiment, acquiring sample images and corresponding labeled spatial audio data provides a rich data foundation for training the spatial audio generation model. Visual encoding based on the spatial layout information of the sample images to obtain sample visual spatial features enables the model to better learn the spatial structure and semantics of the images. Fusing the sample visual spatial features with the audio features of random noise allows the model to learn the reasoning process from arbitrary distributions to the target audio spatial distribution. Transforming and mapping the fused features using the initial spatial audio generation model enables the flow model structure to guide the data from Gaussian noise along a trajectory to the real data distribution, thereby generating high-quality predicted spatial audio data and realizing the generation of spatial audio from video.

[0156] See Figure 5 , Figure 5 A flowchart is shown of a pre-training method for a spatial audio generation model according to an embodiment of this specification, which specifically includes the following steps.

[0157] Step 502: Obtain pre-trained spatial audio data.

[0158] The embodiments in this specification can be applied to a model training platform, specifically to pre-train a flow model to obtain an initial spatial audio generation model, which belongs to the pre-training stage.

[0159] Specifically, pre-trained spatial audio data can be understood as spatial audio data used for model pre-training. It can contain rich audio scenes and spatial information, providing a foundation for the model to learn the intrinsic features and distribution patterns of audio.

[0160] According to one optional implementation, the pre-trained spatial audio data can be obtained based on the Sphere 360 ​​dataset. Further, spatial audio from multiple panoramic video-spatial audio pairings included in the Sphere 360 ​​dataset can be used as the pre-trained spatial audio data.

[0161] According to another alternative implementation, the pre-trained spatial audio data can also be obtained based on non-spatial audio data.

[0162] In one optional embodiment, obtaining pre-trained spatial audio data may include: obtaining pre-trained audio data; performing spatial audio encoding conversion processing on the pre-trained audio data to obtain pre-trained spatial audio data, wherein the spatial audio encoding conversion processing is used to convert the pre-trained audio data from a non-spatial audio format to a spatial audio format.

[0163] Specifically, pre-training audio data can be understood as non-spatial audio data, or as a large amount of unlabeled non-spatial audio data. Spatial audio encoding and conversion processing can be understood as the process of converting non-spatial pre-training audio data into audio data with spatial information. Spatial audio data can specifically be understood as audio data in FOA format.

[0164] According to one alternative implementation, spatial dimension features can be added to non-spatial audio data using specific algorithms and techniques, enabling it to simulate the propagation and localization effects of sound in three-dimensional space. For example, if spatial audio data includes four channels and non-spatial audio data includes two channels, the information from the two channels of the non-spatial audio data can be mapped to the four channels, or information from the other two channels not included in the non-spatial audio data can be filled in, thereby converting the non-spatial audio data into spatial audio data.

[0165] By applying this embodiment, spatial audio encoding conversion processing is performed on the pre-trained audio data to obtain pre-trained spatial audio data. This can overcome the problem of scarce audio data and solve the defect of lack of directionality in non-spatial audio in traditional methods, thereby improving the model's ability to generate spatial audio.

[0166] In practical applications, the streaming model can be pre-trained using pre-trained spatial audio data obtained by spatial audio encoding and transformation of pre-trained audio data; then, it can be pre-trained using the same process with pre-trained spatial audio data obtained from the Sphere 360 ​​dataset. Alternatively, the streaming model can be pre-trained using both pre-trained spatial audio data obtained by spatial audio encoding and transformation of pre-trained audio data and pre-trained spatial audio data obtained from the Sphere 360 ​​dataset.

[0167] Step 504: Perform audio encoding on the pre-trained spatial audio data to obtain pre-trained spatial audio features.

[0168] Specifically, audio encoding can be understood as the process of converting raw audio data into a feature representation in a specific format, enabling models to better process and analyze audio information. By extracting key audio features, such as frequency and amplitude, audio information is transformed into a form that computers can understand. Pre-trained spatial audio features can be understood as the latent audio representation obtained after audio encoding of pre-trained spatial audio data. These features reflect the essential characteristics of the audio data and contain spatial information, such as timbre, pitch, and the spatial location of the audio, serving as crucial input for subsequent model processing.

[0169] In one optional embodiment, audio encoding of pre-trained spatial audio data to obtain pre-trained spatial audio features may include: using a spatial audio encoder to compress and encode the pre-trained spatial audio data to obtain pre-trained spatial audio features.

[0170] Specifically, the spatial audio encoder can be a pre-trained variational autoencoder, capable of directly compressing pre-trained spatial audio data and encoding it into a latent representation carrying spatial information. It should be noted that other encoders with similar functionality to the variational autoencoder can also be selected based on the specific needs of the application.

[0171] By using this embodiment, the pre-trained spatial audio data is compressed and encoded using a spatial audio encoder to obtain pre-trained spatial audio features. This enables the latent representation to include not only audio information but also spatial information, thereby allowing the streaming model to learn the spatial distribution of audio.

[0172] Step 506: Add a mask to the pre-trained spatial audio features to obtain the pre-trained spatial audio mask features.

[0173] Specifically, a mask can be understood as a technique used to obscure or hide parts of data. In audio feature processing, a mask can randomly set some audio feature values ​​to specific values ​​(such as 0) or replace them with other features to simulate missing data or enhance the model's ability to learn partial information. Pre-trained spatial audio mask features can be understood as feature representations obtained by adding a mask to pre-trained spatial audio features. In this case, some audio features are masked, and the model needs to learn how to recover or infer the complete audio features from this masked data.

[0174] According to an optional implementation, adding a mask to the pre-trained spatial audio features to obtain pre-trained spatial audio mask features may include: using a random masking strategy to add a random mask to the pre-trained spatial audio features to obtain pre-trained spatial audio mask features.

[0175] Step 508: Fuse the pre-trained spatial audio mask features and the audio features of random noise to obtain the pre-trained spatial audio mask features with added noise.

[0176] Step 510: Input the noisy pre-trained spatial audio mask features into the stream model, perform transformation mapping on the noisy pre-trained spatial audio mask features, and generate predicted spatial audio features.

[0177] Specifically, predicting spatial audio features can be understood as generating audio features by transforming and mapping the pre-trained spatial audio mask features after adding noise using a streaming model. These features are the model's predictions of the original audio features based on learned patterns. The pre-trained initial spatial audio generation model can be understood as having learned some basic features and distribution patterns of the pre-trained spatial audio data during the pre-training phase, possessing a preliminary ability to generate spatial audio. However, further fine-tuning or training is needed to better apply it to real-world tasks.

[0178] In one optional embodiment, inputting the noisy pre-trained spatial audio mask features into the stream model and performing a transform mapping process on the noisy pre-trained spatial audio mask features to generate predicted spatial audio features may include: inputting the noisy pre-trained spatial audio mask features into the stream model and performing a transform mapping process on the noisy pre-trained spatial audio mask features at least one time step to generate predicted spatial audio features, wherein the transform mapping process at each time step is to perform a transform mapping process on the noisy pre-trained spatial audio mask features or the transform mapping result of the previous time step.

[0179] By introducing time step constraints, this embodiment enables the model to perform transformation operations corresponding to each time step during the transformation mapping process. This allows for a gradual mapping from the pre-trained spatial audio mask features with added noise to the target distribution, improving the accuracy of predicted spatial audio features, ensuring a natural and smooth transition in audio, and enabling the audio to accurately correspond to the visual image, greatly enhancing the audiovisual immersion and interactive experience.

[0180] Step 512: Based on the predicted spatial audio features and the pre-trained spatial audio features, pre-train the flow model to obtain the pre-trained initial spatial audio generation model.

[0181] By applying this embodiment, the problem of scarce audio data can be overcome by converting a large amount of unlabeled non-spatial audio data into spatial audio data, providing the model with sufficient learning material. By compressing and encoding the audio data, a potential audio representation containing both audio and spatial information can be obtained, thereby accurately extracting key audio features and enabling the model to understand audio information in essence. By adding a random mask, the situation of missing data can be simulated, prompting the model to learn to recover complete audio features from partial information, enhancing the model's robustness and adaptability to data incompleteness. By fusing mask features with random noise audio features and using a streaming model for transformation mapping, the advantages of streaming models in learning complex data distributions based on reversible transformations can be leveraged, enabling the model to master the ability to recover audio features from noise.

[0182] The following is in conjunction with the appendix Figures 6-7 This document further explains the pre-training and training methods for the spatial audio generation model provided in this manual. Figure 6 This diagram illustrates the pre-training phase of a spatial audio generation model according to an embodiment of this specification. Figure 7 This diagram illustrates the fine-tuning process of a spatial audio generation model provided in one embodiment of this specification.

[0183] In practical applications, a framework for generating spatial audio from panoramic video can be constructed. In one optional implementation, this framework could specifically be the Omn iAudio framework. This framework can generate spatial audio that closely matches the 360-degree video, and its construction includes two stages: pre-training and fine-tuning. Figure 6 This demonstrates the pre-training phase of the Omn iAudio framework. Figure 7 This illustrates the fine-tuning phase of the Omn iAudio framework. The Omn iAudio framework primarily consists of a VAE Encoder and DiT Blocks. The VAE Encoder is the encoder portion of the VAE, used to compress spatial audio data into a latent audio representation. The VAE also includes a decoder portion, used to restore the spatial audio distribution predicted by the model back to spatial audio data. DiT Blocks, also known as diffusion transformers, can be understood as the spatial audio generation model within the Omn iAudio framework. Both the pre-training and fine-tuning phases involve parameter adjustments to the DiTBlocks.

[0184] like Figure 6As shown, in the pre-training phase, the pre-training data can include two parts: Non-Spatial data and Spatial data. Non-Spatial data consists of a large amount of unlabeled non-spatial audio data, while Spatial data is spatial audio data obtained based on the Sphere 360 ​​dataset. First, Non-Spatial data can be converted to FOA format spatial audio data through spatial audio encoding conversion. Then, the FOA format spatial audio data is compressed into an audio latent representation using a VAE Encoder. Finally, a random mask (masked token) is added to the audio latent representation using a random masking strategy to obtain the masked audio latent representation. Random noise is sampled from a Gaussian distribution to obtain the audio features of the random noise (Noised Latents). t The process involves channel concatenation of the audio features of random noise with the latent representation of the masked audio. The fused noisy masked audio features are then linearly projected before being input into the diffusion transformer. The temporally encoded features obtained at time step t are also input into the diffusion transformer. Based on time step t, the diffusion transformer performs a transform mapping on the fused noisy masked audio features to generate the predicted spatial audio features (v). t Based on predicted spatial audio features (v t ) and spatial audio data in FOA format (u t The Omn iAud io model is pre-trained by adjusting the parameters of the diffusion transformer to obtain an intermediate pre-trained model.

[0185] Furthermore, using spatial audio data obtained from the 360 ​​Sphere dataset, the intermediate pre-trained model is pre-trained using the same procedure as described above to obtain the initial model after pre-training.

[0186] like Figure 7As shown, in the fine-tuning stage, 360-degree videos are acquired from the 360-degree sphere dataset, and the panoramic images from these videos are input into the pre-trained MetaCLIP-Huge image encoder. One branch of the MetaCLIP-Huge image encoder adjusts the resolution of the panoramic images and performs visual encoding based on the visual semantic information of the panoramic images to extract global features; the other branch performs field-of-view segmentation on the panoramic images to obtain panoramic images of the target field-of-view angle. Based on the spatial layout information under the target field-of-view angle, it performs visual encoding on the panoramic images of the target field-of-view angle to extract local field-of-view features. After aligning the local field-of-view features with the audio features of random noise in the feature dimension, the aligned local field-of-view features and audio features are fused by element-wise addition to obtain the visual-spatial audio features. The global features, visual-spatial audio features, and temporal encoded features at time step t are input into the diffusion transformer. Under the condition of panoramic features, the diffusion transformer performs transformation mapping processing on the visual-spatial audio features to generate predicted spatial audio data.

[0187] Furthermore, based on the predicted spatial audio data and the panoramic audio corresponding to the panoramic video in the Sphere 360 ​​dataset, the pre-trained initial model is fine-tuned by adjusting the parameters of the diffusion transformer to obtain the trained model.

[0188] See Figure 8 , Figure 8 This diagram illustrates the construction process of a Ball360 dataset according to an embodiment of this specification. Specifically, the construction process of the Ball360 dataset may include three stages: basic search term construction, initial pairing data construction, and target pairing data construction.

[0189] like Figure 8 As shown, the construction of basic search terms can include the construction of search tags. Based on event tags and limiting conditions, a rich variety of basic search terms can be constructed. The limiting conditions can be spatial panoramic audio and video. The construction of initial pairing data can include acquiring multiple sets of audio and video data through a type-scale filter via a video acquisition interface, and filtering out 360-degree video data and FOA audio data from these sets; the initial pairing data is composed of the 360-degree video data and the corresponding FOA audio data. The construction of target pairing data can include: performing channel-based data analysis and manual screening on the 360-degree video data, and performing channel-based item crawling to obtain target 360-degree video data; performing video-based channel filtering and manual screening on the FOA audio data, and performing video-based item crawling to obtain target FOA audio data. Thus, target pairing data is composed of the crawled target 360-degree video data and target FOA audio data. The target pairing data constitutes the 360-degree sphere dataset.

[0190] Corresponding to the above method embodiments, this specification also provides task platform embodiments. Figure 9 A schematic diagram of the structure of a task platform provided in one embodiment of this specification is shown. Figure 9 As shown, the task platform 900 includes a request interface 902 and an audio generation response unit 904.

[0191] Request interface 902: Used to receive task generation requests sent by terminal devices, wherein the task generation request includes request information.

[0192] Audio generation response unit 904: is used to acquire a target image based on request information; to perform visual encoding on the target image based on the spatial layout information of the target image to obtain visual spatial features; to fuse the visual spatial features and the audio features of random noise to obtain visual spatial audio features; to perform transformation mapping processing on the visual spatial audio features to generate spatial audio data corresponding to the target image; and to feed the spatial audio data back to the terminal device.

[0193] Optionally, the task platform also includes a model library, which stores multiple models; the request information includes the task scenario identifier of the audio generation task, or the task model identifier.

[0194] The audio generation response unit 904 is specifically used to determine the target scene template from multiple preset scene templates based on the task scene identifier, and to search for a spatial audio generation model from the model library based on the target scene template, wherein the model library stores multiple models; or, to search for a spatial audio generation model from the model library based on the task model identifier.

[0195] In the embodiments described in this specification, the task platform adapts to user needs to obtain spatial audio generation models, realizes personalized model services, provides users with an efficient, flexible and easy-to-use model service platform, and improves user experience.

[0196] The above is an illustrative scheme of a task platform according to this embodiment. It should be noted that the technical solution of this task platform and the technical solution of the audio generation method described above belong to the same concept. For details not described in detail in the technical solution of the task platform, please refer to the description of the technical solution of the audio generation method described above.

[0197] Corresponding to the above method embodiments, this specification also provides embodiments of a model training platform. Figure 10 A schematic diagram of the structure of a model training platform provided in one embodiment of this specification is shown. Figure 10 As shown, the model training platform 1000 includes a training interface 1002 and a training response unit 1004.

[0198] Training interface 1002: Used to receive model training requests sent by terminal devices. The model training request includes request information.

[0199] Training response unit 1004: Based on request information, it acquires sample images and corresponding label spatial audio data; based on the sample spatial layout information of the sample images, it performs visual encoding on the sample images to obtain sample visual spatial features; it fuses the sample visual spatial features with the audio features of random noise to obtain sample visual spatial audio features; it inputs the sample visual spatial audio features into an initial spatial audio generation model, performs transformation mapping on the sample visual spatial audio features, and generates predicted spatial audio data corresponding to the sample images, wherein the initial spatial audio generation model is obtained by pre-training a convection model; based on the predicted spatial audio data and label spatial audio data, it trains the initial spatial audio generation model to obtain a trained spatial audio generation model; and it feeds back the model parameters of the trained spatial audio generation model to the terminal device.

[0200] Optionally, the model training platform 1000 also includes a pre-trained response unit 1006;

[0201] Pre-trained response unit 1006: Used to acquire pre-trained spatial audio data; perform audio encoding on the pre-trained spatial audio data to obtain pre-trained spatial audio features; add a mask to the pre-trained spatial audio features to obtain pre-trained spatial audio mask features; fuse the pre-trained spatial audio mask features and the audio features of random noise to obtain noisy pre-trained spatial audio mask features; input the noisy pre-trained spatial audio mask features into the stream model, perform transformation mapping processing on the noisy pre-trained spatial audio mask features to generate predicted spatial audio features; based on the predicted spatial audio features and the pre-trained spatial audio features, pre-train the stream model to obtain the pre-trained initial spatial audio generation model.

[0202] The above is an illustrative scheme of a model training platform according to this embodiment. It should be noted that the technical solution of this model training platform belongs to the same concept as the technical solutions of the model training method and the pre-training method described above. For details not described in detail in the technical solution of the model training platform, please refer to the description of the technical solutions of the model training method and the pre-training method described above.

[0203] Corresponding to the above method embodiments, this specification also provides embodiments of an audio generation apparatus. Figure 11 A schematic diagram of an audio generation apparatus according to one embodiment of this specification is shown. Figure 11 As shown, the device includes:

[0204] First acquisition module 1102: is configured to acquire the target image.

[0205] The first encoding module 1104 is configured to perform visual encoding on the target image based on the spatial layout information of the target image to obtain visual spatial features.

[0206] The first fusion module 1106 is configured to fuse visual spatial features and audio features of random noise to obtain visual spatial audio features.

[0207] The first generation module 1108 is configured to perform transformation mapping processing on visual spatial audio features to generate spatial audio data corresponding to the target image.

[0208] Optionally, the target image is a panoramic image; the first encoding module 1104 is further configured to:

[0209] The target image is segmented to obtain the target image at the target field of view angle; based on the spatial layout information of the target image at the target field of view angle, the target image is visually encoded to obtain visual spatial features.

[0210] Optionally, the first fusion module 1106 is further configured as follows:

[0211] Align the visual spatial features with the audio features according to their feature dimensions to obtain the aligned visual spatial features; then concatenate the audio features with the aligned visual spatial features to obtain the visual spatial audio features.

[0212] Optionally, the first generation module 1108 is further configured as follows:

[0213] Based on the visual content information of the target image, the target image is visually encoded to obtain visual semantic features; based on the visual semantic features, the visual spatial audio features are transformed and mapped to generate spatial audio data corresponding to the target image.

[0214] Optionally, the target image is at least one video frame; the first encoding module 1104 is further configured to:

[0215] Based on visual semantic features, a transformation mapping process is performed on visual spatial audio features under a set time step constraint to generate spatial audio data corresponding to the target image; and a target video is generated based on at least one video frame and the spatial audio data corresponding to at least one video frame.

[0216] Optionally, the first generation module 1108 is further configured as follows:

[0217] The visual spatial audio features are input into the spatial audio generation model, and the visual spatial audio features are transformed and mapped to generate spatial audio data corresponding to the target image. The spatial audio generation model is obtained by training the initial spatial audio generation model.

[0218] In this embodiment, the first acquisition module 1102 acquires the target image, and the first encoding module 1104 performs visual encoding based on the spatial layout information in the target image to obtain visual spatial features. This accurately captures the spatial information in the image, providing a visual basis that fits the spatial content for subsequent audio generation. The first fusion module 1106 fuses the visual spatial features with the audio features of random noise, increasing the diversity and complexity of the audio features and making the generated audio more rich in real-world scenarios. The first generation module 1108 performs transformation mapping processing on the fused visual spatial audio features to generate spatial audio data. This enables the generation of spatial audio that is highly adapted to the spatial video based on the visual information of the target image, thereby ensuring that the audio content accurately matches the video content, constructing a complete auditory scene that conforms to human perception, improving the accuracy of spatial audio generation, and enhancing the immersive audiovisual experience.

[0219] The above is an illustrative scheme of an audio generation device according to this embodiment. It should be noted that the technical solution of this audio generation device and the technical solution of the audio generation method described above belong to the same concept. For details not described in detail in the technical solution of the audio generation device, please refer to the description of the technical solution of the audio generation method described above.

[0220] Corresponding to the above method embodiments, this specification also provides embodiments of a training device for a spatial audio generation model. Figure 12 A schematic diagram of a training apparatus for a spatial audio generation model according to one embodiment of this specification is shown. Figure 12 As shown, the device includes:

[0221] The second acquisition module 1202 is configured to acquire sample images and the corresponding label space audio data of the sample images.

[0222] The second encoding module 1204 is configured to perform visual encoding on the sample image based on the sample spatial layout information of the sample image to obtain the sample visual spatial features.

[0223] The second fusion module 1206 is configured to fuse the visual spatial features of the sample and the audio features of random noise to obtain the visual spatial audio features of the sample.

[0224] The second generation module 1208 is configured to input the visual spatial audio features of the sample into the initial spatial audio generation model, perform transformation mapping processing on the visual spatial audio features of the sample, and generate the predicted spatial audio data corresponding to the sample image. The initial spatial audio generation model is obtained by pre-training the flow model.

[0225] Training module 1210: is configured to train an initial spatial audio generation model based on predicted spatial audio data and labeled spatial audio data, and obtain a trained spatial audio generation model.

[0226] Optionally, the sample image is a panoramic image; the second encoding module 1204 is further configured to:

[0227] The sample image is segmented to obtain a sample image of the target field of view angle; based on the spatial layout information of the sample image under the target field of view angle, the sample image is visually encoded to obtain the visual spatial features of the sample.

[0228] Optionally, the second fusion module 1206 is further configured as follows:

[0229] Align the visual space features of the sample with the audio features of random noise by feature dimension to obtain the aligned visual space features of the sample; then concatenate the audio features with the aligned visual space features of the sample to obtain the audio features of the visual space of the sample.

[0230] Optionally, the second encoding module 1204 is further configured as follows:

[0231] Based on the visual content information of the sample images, semantic encoding is performed on the sample images to obtain the visual semantic features of the samples.

[0232] Accordingly, the second generation module 1208 is further configured as follows:

[0233] The sample visual semantic features and sample visual spatial audio features are input into the initial spatial audio generation model. Based on the sample visual semantic features, the sample visual spatial audio features are transformed and mapped to generate the predicted spatial audio data corresponding to the sample image.

[0234] In this embodiment, the second acquisition module 1202 acquires sample images and corresponding label spatial audio data, providing a rich data foundation for training the spatial audio generation model. The second encoding module 1204 performs visual encoding based on the spatial layout information of the sample images to acquire sample visual spatial features, enabling the model to better learn the spatial structure and semantics of the images. The second fusion module 1206 fuses the sample visual spatial features with the audio features of random noise, enabling the model to learn the reasoning process from arbitrary distribution to the target audio spatial distribution. The second generation module 1208 performs transformation mapping processing on the fused features by the initial spatial audio generation model, enabling the flow model structure to guide the flow of Gaussian noise along a trajectory to the real data distribution, thereby generating high-quality predicted spatial audio data and realizing the generation of video into spatial audio.

[0235] The above is a schematic scheme of a training device for a spatial audio generation model according to this embodiment. It should be noted that the technical solution of this training device for a spatial audio generation model and the technical solution of the training method for a spatial audio generation model described above belong to the same concept. For details not described in detail in the technical solution of the training device for a spatial audio generation model, please refer to the description of the technical solution of the training method for a spatial audio generation model described above.

[0236] Corresponding to the above method embodiments, this specification also provides embodiments of a pre-training device for a spatial audio generation model. Figure 13 This specification illustrates a schematic diagram of the structure of a pre-training device for a spatial audio generation model according to one embodiment. Figure 13 As shown, the device includes:

[0237] The third acquisition module 1302 is configured to acquire pre-trained spatial audio data.

[0238] The third encoding module 1304 is configured to perform audio encoding on the pre-trained spatial audio data to obtain pre-trained spatial audio features.

[0239] Masking module 1306: is configured to add a mask to the pre-trained spatial audio features to obtain pre-trained spatial audio mask features.

[0240] The third fusion module 1308 is configured to fuse the pre-trained spatial audio mask features and the audio features of random noise to obtain the pre-trained spatial audio mask features with added noise.

[0241] The third generation module 1310 is configured to input the noisy pre-trained spatial audio mask features into the stream model, perform transformation mapping processing on the noisy pre-trained spatial audio mask features, and generate predicted spatial audio features.

[0242] Pre-training module 1312: is configured to pre-train a stream model based on predicted spatial audio features and pre-trained spatial audio features, and obtain a pre-trained initial spatial audio generation model.

[0243] Optionally, the third acquisition module 1302 is further configured to: acquire pre-trained audio data; and perform spatial audio encoding conversion processing on the pre-trained audio data to obtain pre-trained spatial audio data.

[0244] Optionally, the third encoding module 1304 is further configured to: use a spatial audio encoder to compress and encode the pre-trained spatial audio data to obtain pre-trained spatial audio features.

[0245] Optionally, the third generation module 1310 is further configured to: input the noisy pre-trained spatial audio mask features into the stream model, perform transformation mapping processing on the noisy pre-trained spatial audio mask features under a set time step constraint, and generate predicted spatial audio features.

[0246] In this embodiment, the third acquisition module 1302 converts a large amount of unlabeled non-spatial audio data into spatial audio data, overcoming the problem of scarce audio data and providing sufficient learning material for the model. The third encoding module 1304 compresses and encodes the audio data, obtaining a potential audio representation containing both audio and spatial information, thereby accurately extracting key audio features and enabling the model to understand audio information in essence. The masking module 1306 adds a random mask, simulating data loss and prompting the model to learn to recover complete audio features from partial information, enhancing the model's robustness and adaptability to data incompleteness. The third fusion module 1308 fuses the mask features with random noise audio features, and the third generation module 1310 uses a streaming model for transformation mapping, leveraging the advantages of the streaming model in learning complex data distributions based on reversible transformations, enabling the model to master the ability to recover audio features from noise.

[0247] The above is an illustrative scheme of a pre-training device for a spatial audio generation model according to this embodiment. It should be noted that the technical solution of this pre-training device for a spatial audio generation model and the technical solution of the pre-training method for a spatial audio generation model described above belong to the same concept. For details not described in detail in the technical solution of the pre-training device for a spatial audio generation model, please refer to the description of the technical solution of the pre-training method for a spatial audio generation model described above.

[0248] Figure 14 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 1400 include, but are not limited to, a memory 1410 and a processor 1420. The processor 1420 is connected to the memory 1410 via a bus 1430, and a database 1450 is used to store data.

[0249] The computing device 1400 also includes an access device 1440, which enables the computing device 1400 to communicate via one or more networks 1460. Examples of such networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. Access device 1440 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as IEEE 802.11 Wireless Local Area Network (WLAN) interface, Wi-MAX (Worldwide Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, and Near Field Communication (NFC).

[0250] In one embodiment of this specification, the above-described components of the computing device 1400 and Figure 14 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 14 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0251] The computing device 1400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1400 can also be a mobile or stationary server.

[0252] The processor 1420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0253] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the above method.

[0254] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0255] The above is an illustrative embodiment of a computer-readable storage medium. It should be noted that the technical solution of this storage medium and the technical solution of the method described above belong to the same concept; details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the method described above.

[0256] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0257] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computer program product can be referred to in the description of the technical solution of the above method.

[0258] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0259] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0260] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0261] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0262] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An audio generation method, comprising: Acquire the target image; Based on the spatial layout information of the target image, the target image is visually encoded to obtain visual spatial features; The visual spatial features and the audio features of random noise are fused to obtain visual spatial audio features; The visual spatial audio features are transformed and mapped to generate spatial audio data corresponding to the target image.

2. The method according to claim 1, wherein the target image is a panoramic image; Before visually encoding the target image based on its spatial layout information to obtain visual spatial features, the method further includes: The target image is segmented based on the target field of view angle to obtain a target view image, wherein the target view image is the image of the target image at the target field of view angle; The step of visually encoding the target image based on its spatial layout information to obtain visual spatial features includes: Based on the spatial layout information of the target view image under the target field of view angle, the target view image is visually encoded to obtain the visual spatial features.

3. The method according to claim 1, wherein fusing the visual spatial features and the audio features of random noise to obtain visual spatial audio features includes: Align the visual spatial features with the audio features by feature dimensions to obtain the aligned visual spatial features; The audio features and the aligned visual spatial features are concatenated element by element to obtain the visual spatial audio features.

4. The method according to any one of claims 1-3, further comprising, before performing transformation mapping processing on the visual spatial audio features to generate spatial audio data corresponding to the target image: Based on the visual content information of the target image, the target image is visually encoded to obtain visual semantic features; The step of performing a transformation mapping process on the visual spatial audio features to generate spatial audio data corresponding to the target image includes: Guided by the visual semantic features, the visual spatial audio features are transformed and mapped to generate spatial audio data corresponding to the target image.

5. The method according to claim 4, wherein the step of performing a transformation mapping process on the visual spatial audio features under the guidance of the visual semantic features to generate spatial audio data corresponding to the target image includes: Guided by the visual semantic features, at least one time step transformation mapping process is performed on the visual spatial audio features to obtain the target spatial audio distribution. The time step transformation mapping process is performed on the visual spatial audio features or the transformation mapping result of the previous time step under the guidance of the visual semantic features. The target spatial audio distribution is decoded to generate spatial audio data corresponding to the target image.

6. The method according to claim 1, wherein the target image is at least one video frame; After performing transformation mapping processing on the visual spatial audio features to generate spatial audio data corresponding to the target image, the method further includes: A target video is generated based on the at least one video frame and the spatial audio data corresponding to the at least one video frame.

7. The method according to claim 1, wherein performing transformation mapping processing on the visual spatial audio features to generate spatial audio data corresponding to the target image includes: The visual spatial audio features are input into the spatial audio generation model, and the visual spatial audio features are transformed and mapped to generate spatial audio data corresponding to the target image. The spatial audio generation model is obtained by training an initial spatial audio generation model, and the transformation and mapping process is used to transform and map the visual spatial audio features into a target spatial audio distribution for generating the spatial audio data.

8. A training method for a spatial audio generation model, comprising: Acquire sample images and the corresponding label space audio data of the sample images; Based on the sample spatial layout information of the sample image, the sample image is visually encoded to obtain sample visual spatial features; The visual spatial features of the sample and the audio features of random noise are fused to obtain the visual spatial audio features of the sample. The sample visual spatial audio features are input into an initial spatial audio generation model, and the sample visual spatial audio features are transformed and mapped to generate predicted spatial audio data corresponding to the sample image. The initial spatial audio generation model is obtained by pre-training a convection model, and the transformation and mapping process is used to transform and map the sample visual spatial audio features into a sample spatial audio distribution used to generate the predicted spatial audio data. Based on the predicted spatial audio data and the labeled spatial audio data, the initial spatial audio generation model is trained to obtain the trained spatial audio generation model.

9. The method according to claim 8, wherein the sample image is a panoramic image; Before encoding the sample image based on the sample spatial layout information to obtain the sample visual spatial features, the method further includes: Based on the target field of view angle, the sample image is segmented to obtain a sample target view image, wherein the sample target view image is the image of the sample image under the target field of view angle; The step of visually encoding the sample image based on the sample spatial layout information to obtain sample visual spatial features includes: Based on the spatial layout information of the sample target view image under the target field of view angle, the sample target view image is visually encoded to obtain the sample visual spatial features.

10. The method according to claim 8, wherein fusing the visual spatial features of the sample and the audio features of random noise to obtain the visual spatial audio features of the sample comprises: The visual space features of the sample are aligned with the audio features of random noise by feature dimension to obtain the aligned visual space features of the sample. The audio features and the aligned sample visual space features are concatenated element by element to obtain the sample visual space audio features.

11. The method according to any one of claims 8-10, further comprising, before inputting the sample visual-spatial audio features into an initial spatial audio generation model, performing a transformation mapping process on the sample visual-spatial audio features, and generating the predicted spatial audio data corresponding to the sample image: Based on the visual content information of the sample images, semantic encoding is performed on the sample images to obtain sample visual semantic features; The step of inputting the sample visual-spatial audio features into an initial spatial audio generation model, performing transformation mapping processing on the sample visual-spatial audio features, and generating predicted spatial audio data corresponding to the sample image includes: The sample visual semantic features and the sample visual spatial audio features are input into the initial spatial audio generation model. Based on the sample visual semantic features, the sample visual spatial audio features are transformed and mapped to generate the predicted spatial audio data corresponding to the sample image.

12. A pre-training method for a spatial audio generation model, comprising: Acquire pre-trained spatial audio data; The pre-trained spatial audio data is audio encoded to obtain pre-trained spatial audio features; Add a mask to the pre-trained spatial audio features to obtain pre-trained spatial audio mask features; The pre-trained spatial audio mask features and the audio features of random noise are fused to obtain the pre-trained spatial audio mask features with added noise. The noise-added pre-trained spatial audio mask features are input into the stream model, and the noise-added pre-trained spatial audio mask features are transformed and mapped to generate predicted spatial audio features. Based on the predicted spatial audio features and the pre-trained spatial audio features, the flow model is pre-trained to obtain the pre-trained initial spatial audio generation model.

13. The method according to claim 12, wherein acquiring the pre-trained spatial audio data comprises: Acquire pre-trained audio data; The pre-trained audio data is subjected to spatial audio encoding conversion processing to obtain pre-trained spatial audio data, wherein the spatial audio encoding conversion processing is used to convert the pre-trained audio data from a non-spatial audio format to a spatial audio format.

14. The method according to claim 13, wherein the step of audio encoding the pre-trained spatial audio data to obtain pre-trained spatial audio features includes: The pre-trained spatial audio data is compressed and encoded using a spatial audio encoder to obtain the pre-trained spatial audio features.

15. The method according to any one of claims 12-14, wherein inputting the noisy pre-trained spatial audio mask features into a stream model and performing a transform mapping process on the noisy pre-trained spatial audio mask features to generate predicted spatial audio features includes: The noisy pre-trained spatial audio mask features are input into the stream model, and a transformation mapping process is performed on the noisy pre-trained spatial audio mask features at at least one time step to generate predicted spatial audio features. The transformation mapping process at each time step is to perform a transformation mapping process on the noisy pre-trained spatial audio mask features or the transformation mapping result of the previous time step.

16. A task platform, comprising a request interface and an audio generation response unit; The request interface is used to receive a task generation request sent by a terminal device, wherein the task generation request includes request information; The audio generation response unit is configured to: acquire a target image based on the request information; perform visual encoding on the target image based on the spatial layout information of the target image to obtain visual spatial features; fuse the visual spatial features and the audio features of random noise to obtain visual spatial audio features; perform transformation mapping processing on the visual spatial audio features to generate spatial audio data corresponding to the target image; and feed the spatial audio data back to the terminal device.

17. A model training platform, comprising a training interface and a training response unit; The training interface is used to receive model training requests sent by terminal devices, and the model training requests include request information. The training response unit is used to acquire a sample image and the corresponding label space audio data based on the request information; and to perform visual encoding on the sample image based on the sample space layout information of the sample image to obtain sample visual space features. The visual spatial features of the sample and the audio features of random noise are fused to obtain the visual spatial audio features of the sample. The sample visual-spatial audio features are input into an initial spatial audio generation model. A transformation mapping process is then performed on the sample visual-spatial audio features to generate predicted spatial audio data corresponding to the sample image. The initial spatial audio generation model is obtained by pre-training a convection model. The transformation mapping process transforms and maps the sample visual-spatial audio features into a sample spatial audio distribution used to generate the predicted spatial audio data. Based on the predicted spatial audio data and the labeled spatial audio data, the initial spatial audio generation model is trained to obtain a trained spatial audio generation model. The model parameters of the trained spatial audio generation model are then fed back to the terminal device.

18. The model training platform according to claim 17 further includes a pre-trained response unit; The pre-trained response unit is used to acquire pre-trained spatial audio data; encode the pre-trained spatial audio data to obtain pre-trained spatial audio features; add a mask to the pre-trained spatial audio features to obtain pre-trained spatial audio mask features; fuse the pre-trained spatial audio mask features and the audio features of random noise to obtain noisy pre-trained spatial audio mask features; input the noisy pre-trained spatial audio mask features into a streaming model, perform transformation mapping processing on the noisy pre-trained spatial audio mask features to generate predicted spatial audio features; and pre-train the streaming model based on the predicted spatial audio features and the pre-trained spatial audio features to obtain a pre-trained initial spatial audio generation model.

19. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1-15.

20. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-15.

21. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-15.

Images