Audio generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an audio generation method and device, electronic equipment and a storage medium, and the method comprises the steps: extracting first text features of an object text used for describing a target object, and carrying out the weighted summation of the first text features in a same video frame image through a mask score, according to the method, the second text feature representing the target object in the video frame image more accurately can be obtained, and on the basis, the target text feature is obtained by fusing the second text features of the plurality of video frame images, so that the target text feature can provide frame-level and fine-grained semantic information; and subsequently, obtaining a randomly generated audio signal with noise, denoising the audio signal with noise by taking the target video feature and the target text feature as denoising conditions to obtain a target audio signal, and generating a target audio corresponding to the target video based on the target audio signal to realize frame-level audio generation. The alignment effect of the pictures in the target audio and the target video is optimized, so that the audio generation quality is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an audio generation method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence generated content (AIGC) technology, video generation has attracted widespread attention in various fields. In order to make the generated video closer to the real scene, it is necessary to generate speech synchronized with the picture for the video. However, since the video contains complex semantic information, it is often difficult to achieve a good video-audio alignment effect when generating audio, and the quality of audio generation needs to be improved. Summary of the invention

[0003] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present disclosure provide an audio generation method, device, electronic device, and storage medium, which can improve the quality of audio generation.

[0005] In one aspect, an embodiment of the present disclosure provides an audio generation method, comprising:

[0006] Acquire a target video, and extract target video features of the target video;

[0007] Performing image segmentation on the video frame image in the target video to obtain a mask score of an object mask in the video frame image, wherein the object mask is used to mark the target object in the video frame image;

[0008] Extracting a first text feature of an object text used to describe the target object, performing weighted summation on the first text features in the same video frame image using the mask score to obtain a second text feature of the video frame image, and fusing the second text features of a plurality of the video frame images to obtain a target text feature;

[0009] A randomly generated noisy audio signal is obtained, and the noisy audio signal is denoised by using the target video feature and the target text feature as denoising conditions to obtain a target audio signal, and a target audio corresponding to the target video is generated based on the target audio signal.

[0010] On the other hand, an embodiment of the present disclosure provides an audio generating device, including:

[0011] A first feature extraction module is used to obtain a target video and extract target video features of the target video;

[0012] a segmentation module, configured to perform image segmentation on a video frame image in the target video to obtain a mask score of an object mask in the video frame image, wherein the object mask is used to mark a target object in the video frame image;

[0013] A second feature extraction module is used to extract a first text feature of an object text used to describe the target object, perform weighted summation of the first text features in the same video frame image using the mask score to obtain a second text feature of the video frame image, and fuse the second text features of multiple video frame images to obtain a target text feature;

[0014] A generation module is used to obtain a randomly generated noisy audio signal, denoise the noisy audio signal using the target video feature and the target text feature as denoising conditions to obtain a target audio signal, and generate a target audio corresponding to the target video based on the target audio signal.

[0015] Furthermore, the generation module is specifically used for:

[0016] Inputting the noisy audio signal, the target video feature and the target text feature into a diffusion model, wherein the diffusion model includes a UNet network, and the UNet network is provided with a plurality of processing modules cascaded in sequence;

[0017] For each of the processing modules, the cross-attention result obtained based on the target video feature is summed with the target text feature and then a feature transformation is performed to obtain the output of the current processing module, and the target audio signal is obtained based on the output of the last processing module.

[0018] Furthermore, the generation module is specifically used for:

[0019] Performing self-attention processing on the noisy frequency signal or the output of the previous processing module to obtain a first self-attention feature;

[0020] constructing a first query matrix based on the first self-attention feature, constructing a first value matrix and a first key matrix based on the target video feature, and performing cross-attention processing based on the first query matrix, the first key matrix and the first value matrix to obtain a first cross-attention feature;

[0021] The first cross-attention feature is summed with the target text feature and then subjected to feature transformation to obtain the output of the current processing module.

[0022] Furthermore, the audio generation device further includes a training module, and the training module is used to:

[0023] Acquire a sample video and a sample audio signal of the sample video, and extract sample video features and sample text features of the sample video, wherein the method of extracting the sample text features is the same as the method of extracting the target text features;

[0024] The diffusion model is called to use the sample video features and the sample text features as denoising conditions to denoise the sample audio signal after adding the label noise signal to obtain a sample noise signal;

[0025] A model loss is determined according to a difference between the sample noise signal and the label noise signal, and the diffusion model is trained based on the model loss.

[0026] Furthermore, the generation module is specifically used for:

[0027] De-noising the noisy audio signal without configuring a de-noising condition to obtain a reference audio signal;

[0028] Performing weighted summation on the target audio signal and the reference audio signal to obtain a weighted audio signal;

[0029] A target audio corresponding to the target video is generated based on the weighted audio signal.

[0030] Furthermore, the generation module is specifically used for:

[0031] Inputting the target audio signal and the reference audio signal into a weight prediction model for regression to obtain target weights corresponding to the target audio signal and the reference audio signal respectively;

[0032] The target audio signal and the reference audio signal are weighted and summed based on the target weight to obtain a weighted audio signal.

[0033] Furthermore, the segmentation module is specifically used for:

[0034] Obtaining object prompt information of a video frame image in the target video, and extracting information features of the object prompt information, wherein the object prompt information is used to prompt the target object to be marked;

[0035] Extracting image features of the video frame image, and fusing the image features with the information features to obtain a first fused feature;

[0036] Perform image segmentation on the video frame image based on the first fusion feature to obtain a mask score of the object mask.

[0037] Furthermore, before extracting the first text feature of the object text for describing the target object, the segmentation module is further used to:

[0038] When the object prompt information includes prompt information of text type, using the object prompt information as object text for describing the target object;

[0039] Alternatively, when the object prompt information includes other types of prompt information besides text type, the target object is highlighted in the video frame image based on the object mask to obtain a target image, and the target image is input into the visual large language model to perform text prediction on the target object to obtain object text for describing the target object.

[0040] Furthermore, the segmentation module is specifically used for:

[0041] Performing self-attention processing on the information feature to obtain a second self-attention feature;

[0042] constructing a second query matrix based on the second self-attention feature, constructing a second value matrix and a second key matrix based on the image feature, performing cross-attention processing based on the second query matrix, the second key matrix and the second value matrix to obtain a second cross-attention feature;

[0043] Performing feature transformation on the second cross-attention feature to obtain a transformed feature;

[0044] constructing a third query matrix based on the image features, constructing a third value matrix and a third key matrix based on the transformation features, performing cross-attention processing based on the third query matrix, the third key matrix and the third value matrix to obtain a third cross-attention feature;

[0045] A fourth query matrix is ​​constructed based on the transformed features, a fourth value matrix and a fourth key matrix are constructed based on the third cross-attention features, and cross-attention processing is performed based on the fourth query matrix, the fourth key matrix and the fourth value matrix to obtain fused features.

[0046] Furthermore, the generation module is specifically used for:

[0047] Fusing the target video feature with the target text feature to obtain a second fused feature;

[0048] The noisy audio signal is denoised using the second fusion feature as a denoising condition to obtain a target audio signal.

[0049] Furthermore, the generation module is specifically used for:

[0050] Inputting the target audio signal into a Mel spectrum encoder for encoding to obtain an encoding result;

[0051] Input the encoding result into a Mel spectrum decoder for decoding to obtain an audio Mel spectrum;

[0052] The audio Mel spectrum is input into a vocoder for audio generation to generate a target audio corresponding to the video frame image.

[0053] On the other hand, an embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned audio generation method when executing the computer program.

[0054] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned audio generation method.

[0055] On the other hand, the embodiment of the present disclosure further provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned audio generation method.

[0056] The disclosed embodiments include at least the following beneficial effects: by performing image segmentation on the video frame images in the target video, a mask score of the object mask is obtained; since the mask score can indicate the prediction confidence of the object mask, a first text feature of the object text used to describe the target object is extracted; and the first text feature in the same video frame image is weighted and summed using the mask score to obtain a second text feature that more accurately represents the target object in the video frame image; on this basis, a target text feature is obtained by fusing the second text features of multiple video frame images, so that the target text feature can provide frame-level and fine-grained semantic information; subsequently, a randomly generated noisy audio signal is obtained, and the noisy audio signal is denoised using the target video feature and the target text feature as denoising conditions to obtain a target audio signal; and a target audio corresponding to the target video is generated based on the target audio signal, thereby achieving frame-level audio generation and optimizing the alignment effect between the target audio and the picture in the target video, thereby improving the quality of audio generation.

[0057] Other features and advantages of the present disclosure will be set forth in the following description, and in part will be apparent from the description, or may be learned by practicing the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0059] Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure;

[0060] Figure 2 An optional flowchart of the audio generation method provided in the embodiment of the present disclosure;

[0061] Figure 3 An optional schematic diagram of object prompt information provided by an embodiment of the present disclosure;

[0062] Figure 4 An optional schematic diagram of a segmentation model provided in an embodiment of the present disclosure;

[0063] Figure 5 An optional schematic diagram of a mask generation module for a refined segmentation model provided in an embodiment of the present disclosure;

[0064] Figure 6 An optional schematic diagram of a decoding module provided in an embodiment of the present disclosure;

[0065] Figure 7 An optional overall flow chart of the audio generation method provided in the embodiment of the present disclosure;

[0066] Figure 8 A schematic diagram of an optional process for determining a mask score provided in an embodiment of the present application;

[0067] Fig. 9 A schematic diagram of the structure of an audio generating device provided in an embodiment of the present disclosure;

[0068] Fig.10 A partial structural block diagram of a terminal provided in an embodiment of the present disclosure;

[0069] Fig.11 A partial structural block diagram of a server provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0071] It should be noted that in various specific embodiments of the present disclosure, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present disclosure needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present disclosure will be obtained.

[0072] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0073] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, some key terms used in the embodiments of the present disclosure are explained here:

[0074] Segmentation model: Segment the input image or video frame into several pixel-level regions. These regions can be similar regions based on features such as color, texture, shape, or objects with specific semantic information.

[0075] Diffusion model: It is a generative model based on probability theory, including forward diffusion process and reverse diffusion process. The forward diffusion process is a process of gradually transforming data distribution into Gaussian noise distribution, and the reverse diffusion process is a gradual denoising process. High-quality data can be reconstructed from noise by simulating the two processes of forward diffusion and reverse diffusion.

[0076] Mask: As a special binary image, a specific area in the image can be clearly marked by a value of 0 or 1 for each pixel. This technology is widely used in the field of image segmentation. In the mask, pixels with a value of "1" indicate the segmented object area that is clearly selected or edited, while pixels with a value of "0" indicate other objects or background areas that are not of concern to the current segmentation task. The accuracy of the segmentation model can be evaluated by the quality of mask generation.

[0077] Mel spectrum: It is an important technology in audio signal processing. Based on the characteristics of the human auditory system, it converts the sound signal into an easy-to-process frequency domain representation through a series of mathematical transformations.

[0078] With the development of artificial intelligence generated content (AIGC) technology, video generation has attracted widespread attention in various fields. In order to make the generated video closer to the real scene, it is necessary to generate speech synchronized with the picture for the video. Inspired by the "text-to-audio" task (TTA), the current research on the "video-to-audio" task (VTA) mainly focuses on the latent diffusion model, which aims to improve the accuracy of semantic alignment through the latent diffusion model framework, combined with the strategy of comparing pre-trained audio, video, and text embedding, and data augmentation technology. However, since the video contains rich visual information and complex semantic information, the existing models are difficult to process these complex video features. Therefore, when effectively converting the complex features in the video into matching audio signals, they often face the problem of time synchronization between video and audio, which restricts the quality of audio generation.

[0079] Based on this, the embodiments of the present disclosure provide an audio generation method, device, electronic device and storage medium, which can generate target audio corresponding to the target video based on the target audio signal in a refined manner, realize frame-level audio generation, and thus improve the quality of audio generation.

[0080] Reference Figure 1 , Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure, wherein the implementation environment includes a terminal 101.

[0081] Exemplarily, a target video can be obtained in terminal 101, target video features of the target video can be extracted, image segmentation can be performed on the video frame images in the target video to obtain the mask score of the object mask in the video frame images, the first text feature of the object text used to describe the target object can be extracted, the mask score can be used to perform weighted summation on the first text feature in the same video frame image to obtain the second text feature of the video frame image, the second text features of multiple video frame images can be fused to obtain the target text feature, a randomly generated noisy audio signal can be obtained, the noisy audio signal can be denoised using the target video feature and the target text feature as denoising conditions to obtain a target audio signal, and a target audio corresponding to the target video can be generated based on the target audio signal.

[0082] It is understandable that the process of generating the target audio can also be performed in the server. Specifically, the terminal 101 can upload the target video to the server, extract the target video features of the target video in the server, perform image segmentation on the video frame image in the target video, obtain the mask score of the object mask in the video frame image, extract the first text feature of the object text used to describe the target object, use the mask score to weighted sum the first text feature in the same video frame image, obtain the second text feature of the video frame image, fuse the second text features of multiple video frame images to obtain the target text feature, obtain a randomly generated noisy audio signal, denoise the noisy audio signal using the target video feature and the target text feature as denoising conditions, obtain the target audio signal, generate the target audio corresponding to the target video based on the target audio signal, and finally, the server sends the generated target audio to the terminal 101.

[0083] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In addition, the server can also be a node server in the blockchain network.

[0084] The terminal 101 may be a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, an intelligent wearable device, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal 101 and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present disclosure.

[0085] Reference Figure 2 , Figure 2 An optional flowchart of an audio generation method provided for an embodiment of the present disclosure. The audio generation method can be executed by a terminal, or can be executed by a server, or can be executed by a terminal and a server in cooperation. The audio generation method includes but is not limited to the following steps S201 to S204.

[0086] Step S201: Acquire a target video and extract target video features of the target video;

[0087] Among them, the target video is the video for audio generation. When extracting the target video features of the target video, the video frame image can be input into the video coding model to encode the target video, and the target video features of the target video are output based on the video coding result. The video coding model is used to extract the target video features of the target video, and can be a multimodal pre-training model (Contrastive Language-Image Pre-training, CLIP). The present disclosure does not specifically limit the type of video coding model used to extract target video features.

[0088] Step S202: performing image segmentation on the video frame image in the target video to obtain a mask score of the object mask in the video frame image.

[0089] Among them, the object mask can be a matrix or image with the same size as the video frame image, which is used to mark the target object or area in the video frame image. The target object is the object that appears in the video frame image, such as a car, a cat, a tree, etc. in the video frame image; the mask score is a key indicator that can measure the prediction confidence of the object mask, which is used to indicate the probability of occurrence of the target object marked by the object mask.

[0090] In one possible implementation, the video frame image can be segmented by a segmentation model. In the process of obtaining the mask score of the object mask, the object prompt information of the video frame image in the target video can be obtained, the information features of the object prompt information can be extracted, the image features of the video frame image can be extracted, the image features and the information features can be fused to obtain a first fusion feature, and the video frame image can be segmented based on the first fusion feature to obtain the mask score of the object mask. Among them, the object prompt information is used to prompt the target object to be marked, and can be prompt information of the type of text, border, pixel point, mask, etc.; the information features of the object prompt information are used to describe and represent the features of the pixel points or areas marked by the object prompt information in the video frame image, including semantic features, position information, hierarchical features, etc.; the position information is used to indicate the relative position of the prompted object, which can be represented by two-dimensional coordinates, or by spatial position information after position encoding based on the two-dimensional coordinates; the hierarchical features are used to indicate that the prompted object belongs to the foreground of the video frame image, or belongs to the background of the video frame image. Reference Figure 3 , Figure 3An optional schematic diagram of object prompt information provided in an embodiment of the present disclosure, when the object prompt information is a text prompt, the two wheels of a vehicle in a video frame image can be marked by the prompt word "wheel"; when the object prompt information is a pixel point prompt, a five-pointed star is displayed on the front wheel of the vehicle, and the five-pointed star is used to represent the pixel point, and the front wheel of the vehicle in the video frame image can be marked by the pixel point prompt; when the object prompt information is a border prompt, the target object within the border is the vehicle, and the entire vehicle in the video frame image can be marked by the border prompt.

[0091] The above process can be expressed as:

[0092]

[0093] Among them, S represents the segmentation model, i represents the sequence number of the video frame image, and v i represents the video frame image, t represents the object prompt information, represents the object mask of the first target object in the first video frame image, The mask score representing the object mask of the first target object in the first video frame image.

[0094] Specifically, the segmentation model includes a prompt information processing module, an image processing module, a mask processing module, a mask generation module and other modules. The prompt information processing module is used to process the prompt information of the object as a text type, a pixel type and a frame type. The image processing module is used to process the input video frame image. The mask processing module is used to process the prompt information of the object as a mask type. The mask generation module generates a corresponding object mask based on the features output by the prompt information processing module, the image processing module or the mask processing module. The acquired object prompt information is input into the prompt information processing module or the mask processing module so that the prompt information processing module or the mask processing module extracts the corresponding information features. The video frame image is input into the image processing module so that the image processing module extracts the corresponding image features. The image features are cross-fused with the information features to obtain the first fused features. The first fused features are input into the mask generation module to perform image segmentation on the video frame image. The object mask of the object corresponding to the information feature is generated based on the segmentation result. The object mask and the mask score of the object mask are output. By fusing image features with information features, a first fused feature containing more information can be obtained. The segmentation model can capture the details and overall structure of the image through the first fused feature, thereby improving the accuracy of segmentation, generating a more accurate object mask, and further improving the mask score.

[0095] Reference Figure 4 , Figure 4An optional schematic diagram of a segmentation model provided in an embodiment of the present disclosure is provided. A video frame image showing a car is input into an image processing module to obtain an image feature f1, wherein the image processing module may be an image coding model. When the object prompt information is mask prompt information, the mask prompt information is input into a mask processing module to obtain a mask information feature f2. The image feature f1 and the mask information feature f2 are feature fused to obtain a first fused feature F1. The first fused feature F1 is input into a mask generation module to perform image segmentation, generate a mask A corresponding to the mask prompt information, and output mask A and a mask score score_1 of mask A. When the object prompt information is text prompt information, the text prompt information is input into a prompt information processing module to obtain a text feature f3. The text feature f3 and the image feature f1 are input into a mask generation module, cross-fused in the mask generation module, and image segmentation is performed based on the first fused feature to generate a mask A corresponding to the text prompt information, and output mask A and a mask score score_1 of mask A. When the object prompt information is pixel prompt information, the pixel prompt information is input into the prompt information processing module to obtain the pixel feature f4, the pixel feature f4 and the image feature f1 are input into the mask generation module, cross-fused in the mask generation module, image segmentation is performed based on the first fusion feature, mask B corresponding to the pixel prompt information is generated, and mask B and mask score score_2 of mask B are output. When the object prompt information is border prompt information, the border prompt information is input into the prompt information processing module to obtain the border feature f5, the border feature f5 and the image feature f1 are input into the mask generation module, cross-fused in the mask generation module, image segmentation is performed based on the first fusion feature, mask C corresponding to the border prompt information is generated, and mask C and mask score score_3 of mask C are output. It should be noted that when the input object prompt information is text prompt information, pixel prompt information or border prompt information, the mask processing module will output an "unmasked" mask feature (the feature map is an all-0 matrix). After the "unmasked" mask feature is fused with the image feature f1, the first fused feature obtained is essentially the image feature f1. At this time, the input feature F1 of the input mask generation module is the image feature f1.

[0096] In a possible implementation, in the process of obtaining the object text for describing the target object, specifically, when the object prompt information includes prompt information of text type, the object prompt information is used as the object text for describing the target object. Specifically, when the object prompt information is prompt information of text type, the natural language processing toolkit can be used to extract the object prompt information from the input text sentence as the object text for describing the target object. For example, if the input text sentence is "a person is driving a car", the natural language processing toolkit can be used to extract the object prompt information "person" and "car" from the input text sentence. "Person" and "car" can also be used as the object text of the target object. In the above manner, the object text required for audio generation can be quickly extracted from complex object prompt information.

[0097] Alternatively, when the object prompt information includes prompt information of other types other than text type, the target object is highlighted in the video frame image based on the object mask to obtain the target image, and the target image is input into the visual large language model to perform text prediction on the target object to obtain the object text for describing the target object. Specifically, when the object prompt information is prompt information of other types other than text type, an object mask is generated based on the prompt information of other types, the target object is highlighted in the corresponding video frame image based on the object mask, the target object is used as the target image, the target image is input into the visual large language model to perform text prediction on the target object, and the object text capable of describing the target object is output. For example, when the object prompt information is prompt information of the mask type, the input mask prompt information is processed by the mask processing module of the segmentation model, a mask feature having the same size as the feature map of the image feature is output, the mask feature is feature-fused with the image feature to obtain a first fused feature, the first fused feature is input into the mask generation module of the segmentation model to obtain the object mask, the target object is highlighted in the video frame image based on the object mask, and the corresponding target image is obtained. Assuming that the target image is an image showing "rabbit", the target image is input into the visual language model for text prediction, and the object text obtained is "rabbit".

[0098] Alternatively, when the object prompt information is a pixel point type prompt information, the prompt information is used to predict the object text of the object at the location of the pixel point. The pixel point type prompt information includes the two-dimensional coordinates of the pixel point in the video frame image, and the position encoding is performed based on the two-dimensional coordinates of the pixel point to obtain the spatial position information that can uniquely identify the pixel point, and the hierarchical features corresponding to the pixel point are extracted based on the pixel point. The spatial position information and the hierarchical features are input into the prompt information processing module of the segmentation model for combination, and the pixel point features are output. The pixel point features and the image features are input into the mask generation module of the segmentation model, so that the mask generation module can find other pixels with high similarity to the pixel point features based on the pixel point features, regard all pixels with high similarity and continuous as part of the same object and perform edge segmentation, generate an object mask based on the contour of the edge segmentation, highlight the target object in the corresponding video frame image based on the object mask, take the target object as the target image, input the target image into the visual large language model to perform text prediction on the target object, and output the object text that can describe the target object.

[0099] Alternatively, when the object hint information is a hint information of a border type, the hint information is used to predict the object text of at least one object in the area surrounded by the border. The hint information of the border type includes a set of diagonal vertices (upper left corner and lower right corner, or lower left corner and upper right corner) of the border in the two-dimensional coordinates of the video frame image, and position encoding is performed based on the two-dimensional coordinates of a set of diagonal vertices of the border to obtain spatial position information that can uniquely identify the two-dimensional coordinates of the set of diagonal vertices. The spatial position information can be a digital sequence or a vector representation. An orientation description vector is generated based on the orientation description of the diagonal vertex. The orientation description vector is used to indicate the diagonal orientation of the vertex. For example, there is a set of diagonal vertices, vertex A and vertex B. The orientation description vector of vertex A indicates that vertex A is located in the upper left corner, and then the orientation description vector of vertex B indicates that vertex B is located in the lower right corner. Then, the spatial position information and the orientation description vector are input into the hint information processing module of the segmentation model for feature fusion, and the border feature is output. The border features and image features are input into the mask generation module of the segmentation model, so that the mask generation module generates an object mask for at least one object in the corresponding area based on the border features, and the target object is highlighted in the corresponding video frame based on the object mask. The target object is used as a target image, and the target image is input into the visual large language model to perform text prediction on the target object, and the object text that can describe the target object is output.

[0100] When the object hint information is other types of hint information besides text type, the method of obtaining the target image based on the object mask and performing text prediction using the visual large language model is not limited to the specific type of hint information and can cope with various types of object hint information, thereby improving the adaptability of the segmentation model to various types of object hint information and the flexibility of processing object hint information.

[0101] In a possible implementation, in the process of fusing the image feature with the information feature to obtain the first fused feature, the information feature may be subjected to self-attention processing to obtain the second self-attention feature;

[0102] A second query matrix is ​​constructed based on the second self-attention feature, a second value matrix and a second key matrix are constructed based on the image feature, and cross-attention processing is performed based on the second query matrix, the second key matrix and the second value matrix to obtain a second cross-attention feature; feature transformation is performed on the second cross-attention feature to obtain a transformed feature; a third query matrix is ​​constructed based on the image feature, a third value matrix and a third key matrix are constructed based on the transformed feature, and cross-attention processing is performed based on the third query matrix, the third key matrix and the third value matrix to obtain a third cross-attention feature; a fourth query matrix is ​​constructed based on the transformed feature, a fourth value matrix and a fourth key matrix are constructed based on the third cross-attention feature, and cross-attention processing is performed based on the fourth query matrix, the fourth key matrix and the fourth value matrix to obtain a first fusion feature.

[0103] Specifically, the image features and the information features can be input into the mask generation module of the segmentation model, and the image features and the information features are fused in the mask generation module to obtain the first fused features. Figure 5 , Figure 5An optional schematic diagram of a mask generation module for a refined segmentation model provided in an embodiment of the present disclosure, inputting information feature F1 into a self-attention mechanism layer for feature extraction, obtaining a second self-attention feature f1, constructing a second query matrix Q1 based on the second self-attention feature f1, constructing a second value matrix V1 and a second key matrix K1 based on the image feature F2, inputting the second query matrix Q1, the second value matrix V1 and the second key matrix K1 into a cross-attention layer 1 for cross-attention processing, and obtaining a second cross-attention feature f2. Then, inputting the second cross-attention feature f2 into a multi-layer perceptron layer for feature transformation of the second cross-attention feature f2, and obtaining a transformation feature f3. Constructing a third query matrix Q2 based on the image feature F2, constructing a third value matrix V2 and a third key matrix K2 based on the transformation feature f3, inputting the third query matrix Q2, the third value matrix V2 and the third key matrix K2 into a cross-attention layer 2 for cross-attention processing, and obtaining a third cross-attention feature f4. Then, the fourth query matrix Q3 is constructed based on the transformed feature f3, and the fourth value matrix V3 and the fourth key matrix K3 are constructed based on the third cross-attention feature f4. The fourth query matrix Q3, the fourth value matrix V3 and the fourth key matrix K3 are input into the cross-attention layer 3 for feature extraction to obtain the first fusion feature f5.

[0104] When cross-attention is applied to information features and image features, the cross-attention mechanism can capture the correlation and dependency between the two features. When the two are cross-attentioned multiple times, the interaction between information features and image features can be further deepened, so that the segmentation model can more comprehensively understand the complex relationship between the two features, thereby improving the robustness of the segmentation model and enabling the segmentation model to generate more accurate object masks.

[0105] Step S203: extract the first text feature of the object text used to describe the target object, use the mask score to perform weighted summation on the first text features in the same video frame image to obtain the second text feature of the video frame image, and fuse the second text features of multiple video frame images to obtain the target text feature.

[0106] In a possible implementation, the first text feature may be obtained by encoding the object text through a text encoding model, and the text encoding model may also be a multimodal pre-trained model. It is understandable that when the object prompt information is used as the object text for describing the target object, the information feature is also a text feature, and the information feature and the first text feature may be obtained through the same text encoding model, or may be obtained through different text encoding models.

[0107] Specifically, since the mask score can measure the prediction confidence of the object mask, the confidence reflects the segmentation model's estimation of the accuracy of the object mask. The higher the confidence, the higher the accuracy of the generated mask. Based on this feature, for the target object corresponding to the object mask with a higher mask score, the first text feature of the object text of the target object has a higher weight when weighted summing, and thus a second text feature that can accurately represent the target object in the video frame image can be obtained. Among them, in order to make the weighted summation more reasonable, when the mask score is not in the interval [0,1], the mask score can be normalized to the interval [0,1]. When the normalized mask score is high, the first text feature weighted with the mask score will obtain a higher weight. Similarly, when the normalized mask score is low, the first text feature weighted with the mask score will obtain a lower weight. The first text feature is weighted summed based on the normalized mask score, and the weighted first text feature is used as the second text feature. The above weighted summation process can be expressed as:

[0108]

[0109] Among them, s i represents the second text feature corresponding to the i-th video frame image, represents the mask score of the jth object mask in the i-th video frame image, ε t represents the text encoding model, Represents the object text of the jth target object in the i-th video frame image.

[0110] The fusion method of the second text features of multiple video frame images can be to splice them into a feature set, that is, the target text feature, which can be specifically expressed as:

[0111]

[0112] Wherein, T represents the total number of video frame images.

[0113] It should also be noted that in order to avoid the situation where the text features corresponding to low weight values ​​are mistakenly regarded as unimportant features due to too low weight values, when the mask score of the object mask is too low, it is necessary to confirm whether the score is too low due to incorrect object prompt information, image blur, image occlusion, etc. At the same time, it is also necessary to verify whether the model can correctly identify the target object. When the model cannot correctly identify the target object, it means that the segmentation model is insufficient in capturing the image features of the target object corresponding to the object mask, or fails to fully understand the text features associated with the object mask. At this time, targeted training is required for samples of the category to which the target object belongs.

[0114] By weighting the first text feature based on the mask score, the confidence of the mask score can be effectively used to optimize the representation of the first text feature, thereby obtaining a second text feature that can accurately reflect the characteristics and contextual information of the target object. The second text feature not only contains the semantic information of the original object prompt information, but also incorporates the segmentation model's understanding and confidence of the video frame image content, providing a richer and more accurate feature representation for subsequent audio generation.

[0115] Step S204: obtain a randomly generated noisy audio signal, denoise the noisy audio signal using target video features and target text features as denoising conditions to obtain a target audio signal, and generate a target audio corresponding to the target video based on the target audio signal.

[0116] Among them, the noisy audio signal can be a noise signal randomly generated from the distribution N(0,1); the denoising condition is a conditional information that combines the characteristics of the video and text and is used to guide the denoising process of the diffusion model; the target audio signal refers to the result obtained by restoring the original input after gradually denoising the noisy audio signal.

[0117] By performing image segmentation on the video frame images in the target video, the mask score of the object mask is obtained. Since the mask score can indicate the prediction confidence of the object mask, the first text feature of the object text used to describe the target object is extracted, and the first text feature in the same video frame image is weighted and summed using the mask score to obtain the second text feature that more accurately represents the target object in the video frame image. On this basis, the target text feature is obtained by fusing the second text features of multiple video frame images, so that the target text feature can provide frame-level and fine-grained semantic information. Subsequently, a randomly generated noisy audio signal is obtained, and the noisy audio signal is denoised using the target video feature and the target text feature as denoising conditions to obtain a target audio signal. The target audio corresponding to the target video is generated based on the target audio signal, and frame-level audio generation is achieved. The alignment effect of the target audio and the picture in the target video is optimized, thereby improving the quality of audio generation.

[0118] In a possible implementation, in the process of denoising a noisy audio signal by using target video features and target text features as denoising conditions to obtain a target audio signal, the noisy audio signal, the target video features and the target text features can be specifically input into a diffusion model; for each processing module, the cross-attention result obtained based on the target video features is summed with the target text features and then a feature transformation is performed to obtain the output of the current processing module, and the target audio signal is obtained based on the output of the last processing module.

[0119] The diffusion model includes an encoding module and a decoding module. The encoding module is used to randomly generate a noisy audio signal from the distribution N(0,1). Then, the noisy audio signal, the target video features, and the target text features are input into the decoding module of the diffusion model. The noisy audio signal is denoised using the target video features and the target text features as denoising conditions, and the target noise signal is predicted. Then, the target noise signal is removed from the noisy audio signal to restore the target audio signal. For each processing module, the processing process can be expressed as:

[0120] Z′ t =Attention(Z t ,c v ,c v )+c o

[0121] Among them, Z t ′ represents the sum of the cross attention result based on the target video feature and the target text feature, Z t represents a noisy frequency signal, c v represents the target video feature, c o Represents the target text features.

[0122] By summing the cross-attention results obtained based on the target video features with the target text features, the target video features can be introduced in the denoising process to perform retrieval operations based on the cross-attention results, while the target text features can be introduced to increase semantics. Since the target text features have fine-grained temporal information, they can effectively improve the accuracy of the target audio signal, thereby improving the generation quality of the target audio.

[0123] In one possible implementation, when the processing module sums the cross-attention result obtained based on the target video feature with the target text feature and then performs feature transformation to obtain the output of the current processing module, it can specifically perform self-attention processing on the noisy frequency signal or the output of the previous processing module to obtain a first self-attention feature; construct a first query matrix based on the first self-attention feature, construct a first value matrix and a first key matrix based on the target video feature, perform cross-attention processing based on the first query matrix, the first key matrix and the first value matrix to obtain a first cross-attention feature; sum the first cross-attention feature with the target text feature and then perform feature transformation to obtain the output of the current processing module.

[0124] Specifically, the decoding module of the diffusion model is a UNet network, and the processing module of the UNet network may include a self-attention layer, a cross-attention layer, and a feedforward network layer. In order to enhance the extraction capability of the UNet network and enable the UNet to pay attention to the correlation between different features, a cross-attention mechanism is introduced in the processing module of the UNet network. For the first processing module, the input of the self-attention layer is a noisy frequency signal, and self-attention processing is performed based on the noisy frequency signal. For the remaining processing modules, the input of the self-attention layer is the output of the previous processing module, and self-attention processing is performed based on the output of the previous processing module. For each processing module, the input of the cross-attention layer is the output of the self-attention layer, and a first query matrix is ​​constructed based on the first self-attention feature output by the self-attention layer, a first value matrix and a first key matrix are constructed based on the target video feature, and cross-attention processing is performed based on the first query matrix, the first key matrix, and the first value matrix, and the first cross-attention feature is summed with the target text feature; the input of the feedforward network layer is the output of the cross-attention layer, and the summed result of the cross-attention layer output is subjected to feature transformation to obtain the output of the current processing module.

[0125] After passing through multiple processing modules, the target audio signal is obtained based on the output of the last processing module. By introducing the cross-attention mechanism in the UNet network, the UNet network's ability to capture and process global information can be enhanced, which helps to improve the generalization ability of the UNet network, so that the UNet network can process features of multiple modalities, thereby being able to predict accurate noise and improve the reliability of denoising. At the same time, when the self-attention layer is set at the first layer of the processing module, the global or cross-regional feature integration capability can be introduced before feature extraction, so that the UNet network can learn the key features of noisy audio signals earlier and focus on important information during the learning process, thereby reducing redundant information and enhancing the robustness of the UNet network, which helps to improve the UNet network's ability to capture the key features of noisy audio signals. When the cross-attention layer is set at the middle processing layer of the processing module, the extracted preliminary noise features can be refined and enhanced, or feature fusion at different levels can be performed, which helps to improve the accuracy of noise prediction by the UNet network.

[0126] For example, refer to Figure 6 , Figure 6 Schematic diagram of an optional decoding module provided in an embodiment of the present disclosure. The decoding module is provided with N processing modules, and processing module 1 is the first processing module of the decoding module. t Input to the processing module 1 of the decoding module, the noisy frequency signal Z t After the self-attention layer performs self-attention processing, the output feature Z t 1 , based on feature Zt 1 Construct the query matrix Q, construct the key matrix K and the value matrix V based on the target video features, input the query matrix Q, key matrix K and value matrix V into the cross attention layer for cross attention processing, and then sum them with the target text features to output the features Then the features Input to the feedforward network layer and output features As the output of processing module 1, the feature It will be used as the input of processing module 2 for further processing. After passing through multiple processing modules, the noise added at time step t is predicted based on the output of processing module N, and the noisy audio signal is denoised based on the predicted noise to obtain the noise Z at time step t-1. t-1 , the noise Z t-1 Re-input into processing module 1 for processing, and after the above steps, the noise at time step t-2 is obtained. After multiple iterations, processing module N will output the predicted target noise signal, and then restore the target audio signal based on the predicted target noise signal. In addition, by connecting processing module 1 and processing module N, processing module 2 and processing module N-1 and other modules through jump connections, shallow features (such as the output features of processing module 1 and processing module 2) can be combined with deep features (such as the output features of processing module N-1 and processing module N) to improve the accuracy of predicted noise. It should be noted that since the denoising process is based on step-by-step denoising, the number of iterations is the same as the number of time steps, that is, when the time step is t, the number of iterations is also t.

[0127] In a possible implementation, before the noisy audio signal, the target video features and the target text features are input into the diffusion model, the diffusion model needs to be trained, specifically by obtaining a sample video and a sample audio signal of the sample video, extracting the sample video features and the sample text features of the sample video, wherein the sample text features are extracted in the same manner as the target text features; calling the diffusion model to use the sample video features and the sample text features as denoising conditions, denoising the sample audio signal after adding the label noise signal, and obtaining the sample noise signal; determining the model loss according to the difference between the sample noise signal and the label noise signal, and training the diffusion model based on the model loss.

[0128] Specifically, during the training process, the encoding module of the diffusion model is mainly used to add noise signals to the pure audio signal, and the decoding module is used to gradually restore the original audio signal from the noisy audio signal; the audio signal is used to describe the acoustic characteristics of the target audio, which is captured based on the audio representation by mapping the target audio to an audio representation, and the target audio is the audio that matches the video composed of all video frame images, and the audio representation is used to describe the audio signal of the target audio, which can be a Mel spectrum. Specifically, the audio signal in the original input space is encoded into a latent audio signal in the latent space through the encoding module, and a random noise signal is gradually added to the latent audio signal based on the time step. In the process of adding the noise signal, the latent audio signal Z1 for the next time step t1 is generated based on the latent audio signal Z0 at the current time step t0, and the latent audio signal Z1 adds more noise to the latent audio signal Z0. Continue to add noise signals to the latent audio signal until the latent audio signal completely obeys the Gaussian distribution. At this time, the time step is recorded as t, and the latent audio signal that completely obeys the Gaussian distribution is recorded as Z t , that is, the sample audio signal after adding the labeled noise signal.

[0129] The sample video is a video containing sample audio aligned with the picture. The sample audio signal is obtained by encoding the sample audio. The method for extracting the sample video feature is the same as the method for extracting the target video feature, which will not be repeated here. Similarly, the video frame image in the sample video can be segmented to obtain the mask score of the object mask in the video frame image, extract the text feature of the object text used to describe the object in the video frame image, and use the mask score to weight the text features in the same video frame image to obtain the text feature of the video frame image. The text features of multiple video frame images are fused to obtain the sample text feature. The sample audio, sample video features and sample text features are input into the diffusion model for training, and the encoding module of the diffusion model converts the sample audio into a sample audio signal. A known label noise signal is gradually added to the sample audio signal based on the time step. The diffusion model then uses the sample video features and the sample text features as denoising conditions to denoise the sample audio signal after adding the label noise signal to obtain a sample noise signal. For the sample noise signal at each time step, the model loss is determined according to the difference between the sample noise signal and the label noise signal, and the training of the diffusion model is guided based on the model loss.

[0130] In the process of determining the model loss based on the difference between the sample noise signal and the label noise signal, the difference between the label noise signal and the sample noise signal can be calculated, a norm operation is performed on the difference, the degree of difference is quantified by the norm result obtained by the norm operation, and the obtained norm result is used as the model loss. According to the above description, the model loss Loss noiseIt can be expressed by the following formula, where ∈ represents the label noise signal and the decoding module ∈ θ (Z t ,t,c v ,c o ) is the predicted sample noise signal, Z t represents a noisy frequency signal, t represents the time step, c v represents the sample video features, c o Represents sample text features.

[0131]

[0132] Furthermore, the diffusion model is trained for multiple rounds to obtain the model loss Loss corresponding to each round of training. noise , configure a unique index for each round of model loss, and accurately locate the diffusion model of the corresponding round based on the index. Find the round with the smallest model loss from multiple model losses, locate the corresponding diffusion model based on the index of the round model loss, and use the parameter settings of the diffusion model corresponding to the round as the parameter settings of the diffusion model used for reasoning. Based on the above description, the diffusion model θ used for reasoning can be obtained by the following formula, where, represents the expected function, t~U(1,T) indicates that the time step t obeys the interval U(1,T), and the function arg min is used to find the round with the smallest model loss from the model losses of all rounds, and use the parameter settings of the diffusion model corresponding to the round as the parameter settings of the diffusion model used for inference.

[0133]

[0134] Guiding the training of the diffusion model based on the difference between the sample noise signal and the label noise signal can provide a clear quantitative indicator for the model. Based on this quantitative indicator, the model can more accurately predict the sample noise signal. At the same time, the arg min function is used to automatically select the diffusion model with the smallest model loss, which not only saves the time of manual selection, but also avoids the human errors that may occur when manually selecting the model, thereby improving the accuracy of the diffusion model in predicting the sample noise signal.

[0135] In a possible implementation, when generating a target audio corresponding to a target video based on a target audio signal, the noisy audio signal can be denoised without configuring a denoising condition to obtain a reference audio signal; the target audio signal and the reference audio signal are weighted and summed to obtain a weighted audio signal; and the target audio corresponding to the target video is generated based on the weighted audio signal. θ It can be expressed by the following formula.

[0136] ∈′ θ=α×∈ θ (z t ,t,c v ,c o )+(1-α)×∈ θ (Z t ,t)

[0137] Among them, Z t is a noisy frequency signal, t is the time step, which is used to describe the different stages of the diffusion process, c v represents the target video feature, c o represents the target text feature, α is the noise weight, ∈ θ (·) is the decoding module. t , time step t, target video feature c and second text feature s are input to the decoding module ∈ θ (·), we get ∈ θ (Z t ,t,c v ,c o ) is the target audio signal. Similarly, we get ∈ θ (Z t ,t) is the reference audio signal.

[0138] Since the target audio signal and the reference audio signal each have their own emphasis, by configuring weights for the target audio signal and the reference audio signal and performing weighted summation, the weighted audio signal obtained combines the characteristics of the two audio signals, effectively improving the quality of the weighted audio signal, thereby ensuring the quality of the subsequent generated target audio.

[0139] In one possible implementation, in the process of weighted summing the target audio signal and the reference audio signal to obtain the weighted audio signal, it is necessary to first determine the noise weight, which can be specifically inputting the target audio signal and the reference audio signal into a weight prediction model for regression to obtain the target weights corresponding to the target audio signal and the reference audio signal respectively; based on the target weights, the target audio signal and the reference audio signal are weighted summed to obtain the weighted audio signal. Among them, the weight prediction model is used to output the noise weight, which is obtained by joint training with the diffusion model, thereby forming a joint architecture with the diffusion model. The two models complement each other, so that the quality of the target audio finally generated is better. The weight prediction model can be a regression model, such as linear regression, logistic regression, random forest, etc.

[0140] Specifically, the weight prediction model is first trained, and the training data samples include sample audio, target audio signals under denoising conditions, reference audio signals under non-denoising conditions, and their corresponding sample weights. The sample weights can be preset based on specific configuration standards or automatically determined by an optimization algorithm, wherein the specific configuration standards can be signal-to-noise ratio, mean square error, etc. After obtaining the target audio signal and the reference audio signal, the target audio signal and the reference audio signal are input into the weight prediction model to extract the corresponding denoising result features, and the extracted denoising result features are batch normalized to eliminate the differences between the features of the target audio signal and the reference audio signal. Based on the denoising result features after batch normalization, the corresponding target weights are predicted, and the predicted target weights are normalized (i.e., the target weights are automatically normalized to a certain range, for example, the numerical values ​​of the target weights are normalized to the interval of [0,1]), and the target audio signal and the reference audio signal are weighted and summed based on the normalized target weights to obtain a weighted audio signal. Since the calculation process of the target audio signal involves denoising conditions, more computing resources will be consumed in the process of calculating the target audio signal. Therefore, by predicting the weights through the weight prediction model, the weights of the target audio signal and the reference audio signal can be dynamically adjusted to obtain the best weighted audio signal, so that the weighted audio signal can retain important audio signals, thereby improving the denoising effect of the diffusion model. In addition, the weights predicted by the weight prediction model can also dynamically adjust the computational complexity of the target audio signal and the reference audio signal, while ensuring the denoising effect, reducing the consumption of computing resources, thereby speeding up the processing speed of the diffusion model, and achieving a balance between improving the denoising effect, reducing computing resources, and speeding up the processing speed.

[0141] In a possible implementation, after obtaining the target audio signal, the target audio corresponding to the target video is generated based on the target audio signal. In the process of generating the target audio, the target audio signal may be input into a Mel spectrum encoder for encoding to obtain the encoding result, the encoding result may be input into a Mel spectrum decoder for decoding to obtain the audio Mel spectrum, the audio Mel spectrum may be input into a vocoder for audio synthesis, and the target audio corresponding to the target video may be generated. The vocoder generates high-quality audio by learning and simulating the physical characteristics of human voice, and may be a Vocoder, WaveNet, etc.

[0142] Specifically, the target audio signal is input into a Mel spectrum encoder for encoding. The audio signal is divided into multiple audio frames according to a preset frame length and folding rate, and Fourier transform is applied to each audio frame. The frequency distribution of the audio signal within the audio frame length is obtained based on the Fourier transform result, and an audio spectrum is generated based on the frequency distribution. Then, the audio spectrum is mapped to the Mel scale, and the audio spectrum is encoded to obtain an encoding result. The encoding result is then input into a Mel spectrum decoder for decoding to obtain an audio Mel spectrum. The audio Mel spectrum is input into a vocoder for audio synthesis to generate the target audio corresponding to the target video.

[0143] In a possible implementation, when the target video feature and the target text feature are used as denoising conditions to denoise the noisy video signal to obtain the target audio signal, the target video feature and the target text feature may be fused to obtain a second fused feature, and the second fused feature is used as the denoising condition to denoise the noisy video signal to obtain the target audio signal. At this time, for each processing module, its processing process can be expressed as:

[0144] Z′ t =Attention(Z t ,c v ,c v ,c o ,c o )

[0145] The above method fuses the target video feature and the target text feature into a second fused feature, that is, the target video feature and the target text feature participate in the cross-attention processing process at the same time, so that the cross-attention processing process can introduce fine-grained semantic features, making the cross-attention processing more accurate, thereby making the subsequently generated target audio quality higher.

[0146] The target video feature and the target text feature may be merged by directly concatenating them, or by summing the target video feature and the target text feature. Since the target video feature and the second text feature are output from different models, the data formats of the target video feature and the second text feature are different. Before summing the target video feature and the second text feature, the data formats of the target video feature and the second text feature may be adjusted so that the data formats of the target video feature and the second text feature are the same.

[0147] Reference Figure 7 , Figure 7 An optional overall flow chart of the audio generation method provided in the embodiment of the present disclosure is provided below. The principle of the audio generation method in the embodiment of the present disclosure is generally described as follows:

[0148] When generating the target audio corresponding to the target video, the target video is first divided into multiple video frame images according to a certain video frame rate. The object texts "car" and "person" are extracted from the sentence "a person is driving a car" describing the target video as the object prompt information of the video frame image. Figure 8 , Figure 8 An optional process schematic diagram for determining a mask score provided in an embodiment of the present application, after the video frame image and object prompt information are input into the segmentation model, the video frame image is encoded based on the image encoding module to obtain image features, and the object text is encoded based on the text encoding module to obtain information features, which are retrieved based on the information features and cross-fused with the image features to obtain a first fused feature, and the mask score of the object mask is output based on the first fused feature.

[0149] In addition, the object texts "car" and "person" are respectively encoded using the text encoding model to obtain the first text features corresponding to the object texts "car" and "person", and the first text features corresponding to "car" and "person" are weighted based on the corresponding mask scores to obtain the second text features. The second text features corresponding to multiple video frame images are fused into target text features, and the target video is encoded using the video encoding model to obtain target video features. The target video features and target text features are input into the diffusion model as diffusion conditions.

[0150] Next, a noisy audio signal is randomly generated through the diffusion model, and the noisy audio signal, target video features and target text features are input into the UNet network. The noisy audio signal is denoised using the target video features and target text features as denoising conditions to obtain the target audio signal. At the same time, the noisy audio signal is denoised without configuring the denoising conditions to obtain a reference audio signal. A weighted sum is performed based on the target audio signal and the reference audio signal to obtain the target audio signal.

[0151] Next, the target audio signal is input into the Mel spectrum encoder for encoding. The audio signal is divided into multiple audio frames according to the preset frame length and folding rate, and Fourier transform is applied to each audio frame. The frequency distribution of the audio signal within the audio frame length is obtained based on the Fourier transform result, and the audio spectrum is generated based on the frequency distribution. Then, the audio spectrum is mapped to the Mel scale, and the audio spectrum is encoded to obtain the encoding result. The encoding result is then input into the Mel spectrum decoder for decoding to obtain the audio Mel spectrum. The audio Mel spectrum is input into the vocoder for audio synthesis to generate the target audio corresponding to the target video.

[0152] By performing image segmentation on the video frame images in the target video, the mask score of the object mask is obtained. Since the mask score can indicate the prediction confidence of the object mask, the first text feature of the object text used to describe the target object is extracted, and the first text feature in the same video frame image is weighted and summed using the mask score to obtain the second text feature that more accurately represents the target object in the video frame image. On this basis, the target text feature is obtained by fusing the second text features of multiple video frame images, so that the target text feature can provide frame-level and fine-grained semantic information. Subsequently, a randomly generated noisy audio signal is obtained, and the noisy audio signal is denoised using the target video feature and the target text feature as denoising conditions to obtain a target audio signal. The target audio corresponding to the target video is generated based on the target audio signal, and frame-level audio generation is achieved. The alignment effect of the target audio and the picture in the target video is optimized, thereby improving the quality of audio generation.

[0153] Referring to Table 1, the embodiments of the present disclosure use FD, FAD and KL to measure the semantic similarity between the generated target audio and the real audio, and AV-Align to measure the degree of time alignment between the generated target audio and the target video. First, compared with the Tango model, the Tango model is a model that generates audio through text. Compared with this, the audio generation method provided by the embodiments of the present disclosure shows better performance in terms of semantics and time alignment. As for the Diff-Foley model, the VTA-LDM model, and the FoleyCrafter model, these models are all used to generate audio through video. The audio generation method provided by the embodiments of the present disclosure also shows better performance in terms of semantics and time alignment.

[0154] Table 1

[0155]

[0156] It is to be understood that, although the steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0157] Reference Fig. 9 , Fig. 9This is a schematic diagram of the structure of an audio generating device provided in an embodiment of the present disclosure. The audio generating device 900 includes:

[0158] The first feature extraction module 901 is used to obtain a target video and extract target video features of the target video;

[0159] A segmentation module 902 is used to perform image segmentation on a video frame image in a target video to obtain a mask score of an object mask in the video frame image, wherein the object mask is used to mark a target object in the video frame image;

[0160] The second feature extraction module 903 is used to extract the first text feature of the object text used to describe the target object, perform weighted summation of the first text features in the same video frame image using the mask score to obtain the second text feature of the video frame image, and fuse the second text features of multiple video frame images to obtain the target text feature;

[0161] The generating module 904 is used to obtain a randomly generated noisy audio signal, denoise the noisy audio signal using target video features and target text features as denoising conditions to obtain a target audio signal, and generate a target audio corresponding to the target video based on the target audio signal.

[0162] Further, the generating module 904 is specifically used for:

[0163] The noisy video signal, the target video feature and the target text feature are input into a diffusion model, wherein the diffusion model includes a UNet network, and the UNet network is provided with a plurality of processing modules cascaded in sequence;

[0164] For each processing module, the cross-attention result based on the target video feature is summed with the target text feature and then feature transformation is performed to obtain the output of the current processing module, and the target audio signal is obtained based on the output of the last processing module.

[0165] Further, the generating module 904 is specifically used for:

[0166] Perform self-attention processing on the noisy frequency signal or the output of the previous processing module to obtain a first self-attention feature;

[0167] constructing a first query matrix based on the first self-attention feature, constructing a first value matrix and a first key matrix based on the target video feature, performing cross-attention processing based on the first query matrix, the first key matrix and the first value matrix to obtain a first cross-attention feature;

[0168] The first cross-attention feature is summed with the target text feature and then feature transformation is performed to obtain the output of the current processing module.

[0169] Furthermore, the audio generation device further includes a training module (not shown in the drawings), which is used to:

[0170] Obtaining a sample video and a sample audio signal of the sample video, and extracting sample video features and sample text features of the sample video, wherein the method for extracting the sample text features is the same as the method for extracting the target text features;

[0171] The diffusion model is called to use the sample video features and the sample text features as denoising conditions to denoise the sample audio signal after adding the label noise signal to obtain the sample noise signal;

[0172] The model loss is determined according to the difference between the sample noise signal and the label noise signal, and the diffusion model is trained based on the model loss.

[0173] Further, the generating module 904 is specifically used for:

[0174] De-noising the noisy audio signal without configuring a de-noising condition to obtain a reference audio signal;

[0175] Performing weighted summation on the target audio signal and the reference audio signal to obtain a weighted audio signal;

[0176] A target audio corresponding to the target video is generated based on the weighted audio signal.

[0177] Further, the generating module 904 is specifically used for:

[0178] Inputting the target audio signal and the reference audio signal into the weight prediction model for regression to obtain the target weights corresponding to the target audio signal and the reference audio signal respectively;

[0179] The target audio signal and the reference audio signal are weighted and summed based on the target weight to obtain a weighted audio signal.

[0180] Further, the segmentation module 902 is specifically used for:

[0181] Obtaining object prompt information of a video frame image in a target video, and extracting information features of the object prompt information, wherein the object prompt information is used to prompt a target object to be marked;

[0182] Extracting image features of the video frame image, fusing the image features with the information features to obtain a first fused feature;

[0183] The video frame image is segmented based on the first fusion feature to obtain a mask score of the object mask.

[0184] Furthermore, before extracting the first text feature of the object text for describing the target object, the segmentation module 902 is further used to:

[0185] When the object prompt information includes prompt information of text type, the object prompt information is used as object text for describing the target object;

[0186] Alternatively, when the object prompt information includes other types of prompt information in addition to the text type, the target object is highlighted in the video frame image based on the object mask to obtain a target image, and the target image is input into the visual large language model to perform text prediction on the target object to obtain the object text used to describe the target object.

[0187] Further, the segmentation module 902 is specifically used for:

[0188] Perform self-attention processing on the information feature to obtain the second self-attention feature;

[0189] constructing a second query matrix based on the second self-attention feature, constructing a second value matrix and a second key matrix based on the image feature, performing cross-attention processing based on the second query matrix, the second key matrix and the second value matrix to obtain a second cross-attention feature;

[0190] Performing feature transformation on the second cross-attention feature to obtain a transformed feature;

[0191] constructing a third query matrix based on the image features, constructing a third value matrix and a third key matrix based on the transformation features, performing cross-attention processing based on the third query matrix, the third key matrix and the third value matrix to obtain a third cross-attention feature;

[0192] A fourth query matrix is ​​constructed based on the transformed features, a fourth value matrix and a fourth key matrix are constructed based on the third cross-attention features, and cross-attention processing is performed based on the fourth query matrix, the fourth key matrix and the fourth value matrix to obtain fused features.

[0193] Further, the generating module 904 is specifically used for:

[0194] Fusing the target video feature with the target text feature to obtain a second fused feature;

[0195] The noisy audio signal is denoised using the second fusion feature as a denoising condition to obtain a target audio signal.

[0196] Further, the generating module 904 is specifically used for:

[0197] Input the target audio signal into the Mel spectrum encoder for encoding to obtain an encoding result;

[0198] The encoding result is input into the Mel spectrum decoder for decoding to obtain the audio Mel spectrum;

[0199] The audio Mel spectrum is input into the vocoder for audio generation to generate the target audio corresponding to the video frame image.

[0200] The audio generation device 900 and the audio generation method provided in the embodiment of the present disclosure are based on the same inventive concept. By performing image segmentation on the video frame image in the target video, a mask score of the object mask is obtained. Since the mask score can indicate the prediction confidence of the object mask, the first text feature of the object text used to describe the target object is extracted, and the first text feature in the same video frame image is weighted and summed using the mask score to obtain a second text feature that more accurately represents the target object in the video frame image. On this basis, the target text feature is obtained by fusing the second text features of multiple video frame images, so that the target text feature can provide frame-level and fine-grained semantic information. Subsequently, a randomly generated noisy audio signal is obtained, and the noisy audio signal is denoised using the target video feature and the target text feature as denoising conditions to obtain a target audio signal. The target audio corresponding to the target video is generated based on the target audio signal, frame-level audio generation is achieved, and the alignment effect of the target audio and the picture in the target video is optimized, thereby improving the quality of audio generation.

[0201] The electronic device for executing the above-mentioned audio generation method provided in the embodiment of the present disclosure may be a terminal. Fig.10 , Fig.10 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure, and the terminal includes: a camera assembly 1010, a first memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a first processor 1080, and a first power supply 1090. Those skilled in the art can understand that Fig.10 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0202] The camera assembly 1010 can be used to capture images or videos. Optionally, the camera assembly 1010 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions.

[0203] The first memory 1020 may be used to store software programs and modules. The first processor 1080 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the first memory 1020 .

[0204] The input unit 1030 may be used to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032 .

[0205] The display unit 1040 may be used to display input information or provided information and various menus of the terminal. The display unit 1040 may include a display panel 1041.

[0206] The audio circuit 1060 , the speaker 1061 , and the microphone 1062 may provide an audio interface.

[0207] The first power source 1090 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0208] The number of sensors 1050 may be one or more, and the one or more sensors 1050 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:

[0209] The acceleration sensor can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor can be used to detect the components of gravity acceleration on the three coordinate axes. The first processor 1080 can control the display unit 1040 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for collecting motion data of games or users.

[0210] The gyroscope sensor can detect the body direction and rotation angle of the terminal, and the gyroscope sensor can cooperate with the acceleration sensor to collect the user's 3D actions on the terminal. The first processor 1080 can implement the following functions based on the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0211] The pressure sensor can be set in the side frame of the terminal and / or the lower layer of the display unit 1040. When the pressure sensor is set in the side frame of the terminal, the user's holding signal of the terminal can be detected, and the first processor 1080 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor. When the pressure sensor is set in the lower layer of the display unit 1040, the first processor 1080 controls the operability controls on the UI interface according to the user's pressure operation on the display unit 1040. The operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0212] The optical sensor is used to collect the ambient light intensity. In one embodiment, the first processor 1080 can control the display brightness of the display unit 1040 according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1040 is increased; when the ambient light intensity is low, the display brightness of the display unit 1040 is decreased. In another embodiment, the first processor 1080 can also dynamically adjust the shooting parameters of the camera assembly 1010 according to the ambient light intensity collected by the optical sensor.

[0213] In this embodiment, the first processor 1080 included in the terminal can execute the audio generation method of the previous embodiment.

[0214] The electronic device for executing the above-mentioned audio generation method provided in the embodiment of the present disclosure may also be a server. Fig.11 , Fig.11 This is a partial structural block diagram of a server provided in an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more second processors 1110 and a second memory 1130, and one or more storage media 1140 (e.g., one or more mass storage devices) storing application programs 1143 or data 1142. Among them, the second memory 1130 and the storage medium 1140 may be short-term storage or persistent storage. The program stored in the storage medium 1140 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the second processor 1110 may be configured to communicate with the storage medium 1140 to execute a series of instruction operations in the storage medium 1140 on the server.

[0215] The server may also include one or more second power supplies 1120, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1160, and / or one or more operating systems 1141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0216] The second processor 1110 in the server may be configured to execute the audio generation method.

[0217] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the audio generation method of each of the aforementioned embodiments.

[0218] The present disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned audio generation method.

[0219] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate to describe the embodiments of the present disclosure, such as being able to be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0220] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0221] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0222] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0223] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0225] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0226] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0227] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. An audio generation method, characterized in that: include: Acquire a target video, and extract target video features of the target video; Performing image segmentation on the video frame image in the target video to obtain a mask score of an object mask in the video frame image, wherein the object mask is used to mark the target object in the video frame image; Extracting a first text feature of an object text used to describe the target object, performing weighted summation on the first text features in the same video frame image using the mask score to obtain a second text feature of the video frame image, and fusing the second text features of a plurality of the video frame images to obtain a target text feature; A randomly generated noisy audio signal is obtained, and the noisy audio signal is denoised by using the target video feature and the target text feature as denoising conditions to obtain a target audio signal, and a target audio corresponding to the target video is generated based on the target audio signal.

2. The audio generation method according to claim 1, characterized in that: The step of denoising the noisy audio signal by using the target video feature and the target text feature as denoising conditions to obtain a target audio signal includes: Inputting the noisy audio signal, the target video feature and the target text feature into a diffusion model, wherein the diffusion model includes a UNet network, and the UNet network is provided with a plurality of processing modules cascaded in sequence; For each of the processing modules, the cross-attention result obtained based on the target video feature is summed with the target text feature and then a feature transformation is performed to obtain the output of the current processing module, and the target audio signal is obtained based on the output of the last processing module.

3. The audio generation method according to claim 2, characterized in that: The step of summing the cross attention result obtained based on the target video feature and the target text feature and performing feature transformation to obtain the output of the current processing module includes: Performing self-attention processing on the noisy frequency signal or the output of the previous processing module to obtain a first self-attention feature; constructing a first query matrix based on the first self-attention feature, constructing a first value matrix and a first key matrix based on the target video feature, and performing cross-attention processing based on the first query matrix, the first key matrix and the first value matrix to obtain a first cross-attention feature; The first cross-attention feature is summed with the target text feature and then subjected to feature transformation to obtain the output of the current processing module.

4. The audio generation method according to claim 2, characterized in that: Before inputting the noisy audio signal, the target video feature and the target text feature into the diffusion model, the audio generation method further includes: Acquire a sample video and a sample audio signal of the sample video, and extract sample video features and sample text features of the sample video, wherein the method of extracting the sample text features is the same as the method of extracting the target text features; The diffusion model is called to use the sample video features and the sample text features as denoising conditions to denoise the sample audio signal after adding the label noise signal to obtain a sample noise signal; A model loss is determined according to a difference between the sample noise signal and the label noise signal, and the diffusion model is trained based on the model loss.

5. The audio generation method according to claim 1, characterized in that: The generating the target audio corresponding to the target video based on the target audio signal includes: De-noising the noisy audio signal without configuring a de-noising condition to obtain a reference audio signal; Performing weighted summation on the target audio signal and the reference audio signal to obtain a weighted audio signal; A target audio corresponding to the target video is generated based on the weighted audio signal.

6. The audio generation method according to claim 5, characterized in that: The step of performing weighted summation on the target audio signal and the reference audio signal to obtain a weighted audio signal includes: Inputting the target audio signal and the reference audio signal into a weight prediction model for regression to obtain target weights corresponding to the target audio signal and the reference audio signal respectively; The target audio signal and the reference audio signal are weighted and summed based on the target weight to obtain a weighted audio signal.

7. The audio generation method according to claim 1, characterized in that: The performing image segmentation on the video frame image in the target video to obtain a mask score of the object mask in the video frame image includes: Obtaining object prompt information of a video frame image in the target video, and extracting information features of the object prompt information, wherein the object prompt information is used to prompt the target object to be marked; Extracting image features of the video frame image, and fusing the image features with the information features to obtain a first fused feature; Perform image segmentation on the video frame image based on the first fusion feature to obtain a mask score of the object mask.

8. The audio generation method according to claim 7, characterized in that: Before extracting the first text feature of the object text for describing the target object, the audio generation method further includes: When the object prompt information includes prompt information of text type, using the object prompt information as object text for describing the target object; Alternatively, when the object prompt information includes other types of prompt information besides text type, the target object is highlighted in the video frame image based on the object mask to obtain a target image, and the target image is input into the visual large language model to perform text prediction on the target object to obtain object text for describing the target object.

9. The audio generation method according to claim 7, characterized in that: The fusing the image feature with the information feature to obtain a first fused feature includes: Performing self-attention processing on the information feature to obtain a second self-attention feature; constructing a second query matrix based on the second self-attention feature, constructing a second value matrix and a second key matrix based on the image feature, performing cross-attention processing based on the second query matrix, the second key matrix and the second value matrix to obtain a second cross-attention feature; Performing feature transformation on the second cross-attention feature to obtain a transformed feature; constructing a third query matrix based on the image features, constructing a third value matrix and a third key matrix based on the transformation features, performing cross-attention processing based on the third query matrix, the third key matrix and the third value matrix to obtain a third cross-attention feature; A fourth query matrix is ​​constructed based on the transformed features, a fourth value matrix and a fourth key matrix are constructed based on the third cross-attention features, and cross-attention processing is performed based on the fourth query matrix, the fourth key matrix and the fourth value matrix to obtain a first fusion feature.

10. The audio generation method according to claim 1, characterized in that: The step of denoising the noisy audio signal by using the target video feature and the target text feature as denoising conditions to obtain a target audio signal includes: Fusing the target video feature with the target text feature to obtain a second fused feature; The noisy audio signal is denoised using the second fusion feature as a denoising condition to obtain a target audio signal.

11. The audio generation method according to claim 1, characterized in that: The generating the target audio corresponding to the target video based on the target audio signal includes: Inputting the target audio signal into a Mel spectrum encoder for encoding to obtain an encoding result; Input the encoding result into a Mel spectrum decoder for decoding to obtain an audio Mel spectrum; The audio Mel spectrum is input into a vocoder for audio generation to generate a target audio corresponding to the target video.

12. An audio generating device, characterized in that: include: A first feature extraction module, used to obtain a target video and extract target video features of the target video; a segmentation module, configured to perform image segmentation on a video frame image in the target video to obtain a mask score of an object mask in the video frame image, wherein the object mask is used to mark a target object in the video frame image; A second feature extraction module is used to extract a first text feature of an object text used to describe the target object, perform weighted summation of the first text features in the same video frame image using the mask score to obtain a second text feature of the video frame image, and fuse the second text features of multiple video frame images to obtain a target text feature; A generation module is used to obtain a randomly generated noisy audio signal, denoise the noisy audio signal using the target video feature and the target text feature as denoising conditions to obtain a target audio signal, and generate a target audio corresponding to the target video based on the target audio signal.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the audio generation method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the audio generation method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the audio generation method according to any one of claims 1 to 11 is implemented.

Images