A method and device for generating three-dimensional model based on audio and image fusion

Abstract

The present invention relates to the field of computer vision technology, and discloses a method and device for generating a three-dimensional model based on audio and image fusion, the method comprising acquiring audio data; extracting features from the audio data to obtain an audio feature vector; mapping the audio feature vector to a plurality of first two-dimensional images through a trained adversarial diffusion generation model; initializing a three-dimensional model, and using the audio feature vector to render the three-dimensional model from a plurality of random angles to generate a plurality of second two-dimensional images of viewing angles; iteratively optimizing the three-dimensional model according to the audio feature vector, the first two-dimensional image and the second two-dimensional image to generate a final three-dimensional model. The present invention can learn the complex mapping relationship between audio and image, ensures that the three-dimensional model learns the characteristics of the three-dimensional scene from multiple viewing angles, does not require a large-scale three-dimensional annotation data set, and realizes the generation of three-dimensional models with both high quality and diversity.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to a method and device for generating a three-dimensional model based on audio and image fusion. Background Art

[0002] In the field of artificial intelligence, the technology of converting audio and pictures into three-dimensional models is gradually maturing, which is of great significance to industries such as augmented reality, virtual reality and game development. This technology can provide a richer and more interactive user experience.

[0003] Existing methods usually rely on a large amount of 3D training data and combine deep learning and computer vision technology. These methods extract features from audio and images and map them into 3D space to generate 3D models. However, these technologies face a large amount of computation in the data processing process, and the generated models are often out of sync with the input data, the naturalness of expressions and movements is difficult to ensure, and the generated 3D models often lack details and realism, and have defects of low quality and low diversity. Summary of the invention

[0004] In order to overcome the defects of low quality and low diversity of the generated three-dimensional models in the prior art, the present invention proposes the following technical solutions:

[0005] In a first aspect, the present invention proposes a method for generating a three-dimensional model based on audio and image fusion, comprising:

[0006] Get audio data.

[0007] Feature extraction is performed on the audio data to obtain an audio feature vector.

[0008] The audio feature vector is mapped into a plurality of first two-dimensional images through a trained adversarial diffusion generation model.

[0009] A three-dimensional model is initialized, and the three-dimensional model is rendered from a plurality of random angles using the audio feature vector to generate a second two-dimensional image at a plurality of viewing angles.

[0010] The three-dimensional model is iteratively optimized according to the audio feature vector, the first two-dimensional image and the second two-dimensional image to generate a final three-dimensional model.

[0011] As a preferred technical solution, before mapping the audio feature vector to a plurality of first two-dimensional images through a trained adversarial diffusion generative model, the method further includes training an adversarial diffusion generative model. The adversarial diffusion generative model includes an image encoder, a UNet decoder, and a discriminator, and training the adversarial diffusion generative model includes:

[0012] Initialize a noisy image.

[0013] The image encoder is used to encode the noisy image into a noise image feature vector.

[0014] The audio feature vector and the noise image feature vector are fused to obtain a fused feature vector.

[0015] The fused feature vector is decoded using the UNet decoder to generate a two-dimensional image.

[0016] The discriminator performs classification processing according to the generated two-dimensional image and the input real two-dimensional image, and outputs a probability value that the generated two-dimensional image is a real two-dimensional image.

[0017] The discriminator and the UNet decoder are iteratively optimized according to the generated two-dimensional image, the input real two-dimensional image and the probability value to obtain a trained adversarial diffusion generation model.

[0018] As a preferred technical solution, the discriminator performs classification processing according to the generated two-dimensional image and the input real two-dimensional image, and outputs a probability value that the generated two-dimensional image is a real two-dimensional image, including:

[0019] Feature extraction is performed on the generated two-dimensional image and the real two-dimensional image to generate a pair of image feature vectors of the same length, and then the pair of image feature vectors of the same length are concatenated and input into the self-attention layer for feature interaction, and the feature vector after feature interaction is processed through a fully connected layer, and a probability value of the generated two-dimensional image being a real two-dimensional image is output.

[0020] As a preferred technical solution, the discriminator and the UNet decoder are iteratively optimized according to the generated two-dimensional image, the input real two-dimensional image and the probability value, including:

[0021] The KL divergence and the mean square error of the generated two-dimensional image and the real two-dimensional image are calculated, and the loss function of the UNet decoder is constructed using the KL divergence and the mean square error.

[0022] A binary cross entropy loss function is constructed according to the probability value as the loss function of the discriminator.

[0023] The UNet decoder and the discriminator are iteratively trained according to the loss function of the UNet decoder and the loss function of the discriminator, respectively, until a predetermined number of training rounds is reached to obtain a trained adversarial diffusion generation model.

[0024] As a preferred technical solution, feature extraction is performed on the audio data to obtain an audio feature vector, including:

[0025] The audio data is converted into a spectrogram.

[0026] The spectrogram is input into an audio encoder for feature extraction to obtain an audio feature vector.

[0027] As a preferred technical solution, the spectrogram is a Mel-spectrogram or a linear spectrogram. The audio encoder is a convolutional neural network or a recurrent neural network.

[0028] As a preferred technical solution, after initializing a three-dimensional model, the method further includes:

[0029] The cosine similarity between every two audio feature vectors is calculated, and a group of audio feature vectors whose cosine similarity is greater than a preset threshold is screened out from the audio feature vectors.

[0030] The audio feature vector groups are mapped into a plurality of first two-dimensional image groups through a trained adversarial diffusion generation model, and the three-dimensional model is rendered from a plurality of random angles to generate a plurality of second two-dimensional image groups of viewing angles.

[0031] As a preferred technical solution, the three-dimensional model is iteratively optimized according to the audio feature vector, the first two-dimensional image and the second two-dimensional image to generate a final three-dimensional model, including:

[0032] According to the audio feature vector, the first two-dimensional image and the second two-dimensional image, a loss function of the three-dimensional model is constructed, and its expression is as follows:

[0033] Loss=Loss_audio_cos+Loss_audio_KL+Loss_img_cos+Loss_img_KL

[0034] +Loss_SSIM_1+Loss_SSIM_2

[0035] Among them, Loss_audio_cos and Loss_audio_KL are the cosine similarity loss and KL divergence loss between the audio feature vector and the second two-dimensional image, Loss_img_cos and Loss_img_KL are the cosine similarity loss and KL divergence loss between the first two-dimensional image and the second two-dimensional image, Loss_SSIM_1 and Loss_SSIM_2 are the two groups of structural similarity index losses between the first two-dimensional image group and the second two-dimensional image group.

[0036] The three-dimensional model is iteratively trained by a gradient descent method according to the loss function of the three-dimensional model until a predetermined number of training rounds is reached to generate a final three-dimensional model.

[0037] As a preferred technical solution, the initializing a three-dimensional model includes:

[0038] Initialize a 3D model using the Neural Radiance Field architecture.

[0039] In a second aspect, the present invention further proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the operations performed by a three-dimensional model generation method based on audio and image fusion as described in any one of the schemes in the first aspect are implemented.

[0040] The beneficial effects of the present invention include at least:

[0041] The present invention first obtains audio data and performs feature extraction to generate an audio feature vector, and then maps the audio feature vector to a plurality of first two-dimensional images through a trained adversarial diffusion generation model, thereby solving the problem of generating a matching two-dimensional image from audio data, and being able to learn the complex mapping relationship between audio and image. The generated image is not only of high quality, but also can faithfully reflect the rhythm, intonation and emotional features in the audio, and provides key visual information for subsequent three-dimensional model generation. In addition, by combining the pre-trained audio-to-image adversarial diffusion generation model with the three-dimensional model generation technology, the first two-dimensional image is used to render the three-dimensional model from a random angle, and a plurality of second two-dimensional images of different viewing angles are generated. The three-dimensional model is iteratively optimized based on the audio feature vector, the first two-dimensional image and the second two-dimensional image, thereby ensuring that the three-dimensional model learns the characteristics of the three-dimensional scene from multiple viewing angles, and does not require a large-scale three-dimensional annotation data set, thereby achieving the generation of a three-dimensional model with both high quality and diversity. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a three-dimensional model generation method based on audio and image fusion provided in an embodiment of the present application.

[0043] Figure 2 This is a general principle block diagram of the three-dimensional model generation method based on audio and image fusion provided in an embodiment of the present application.

[0044] Figure 3 This is a principle block diagram of the training adversarial diffusion generation model proposed in an embodiment of the present application.

[0045] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0047] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0048] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0049] Specifically, Figure 1 A schematic diagram of a flow chart of a method for generating a three-dimensional model based on audio and image fusion provided in an embodiment of the present application. Figure 2 The overall principle block diagram of the 3D model generation method based on audio and image fusion provided in the embodiment of the present application includes the following steps:

[0050] S1: Get audio data.

[0051] In the specific implementation process, a data set containing both audio and corresponding three-dimensional models is collected. The three-dimensional model in the data set is a collection of two-dimensional images taken from multiple angles, ensuring that each audio has a corresponding three-dimensional model.

[0052] S2: Extract features from the audio data to obtain an audio feature vector.

[0053] Optionally, in one embodiment of the present application, feature extraction is performed on the audio data to obtain an audio feature vector, including:

[0054] The audio data is converted into a spectrogram.

[0055] The spectrogram is input into an audio encoder for feature extraction to obtain an audio feature vector.

[0056] Optionally, in one embodiment of the present application, the spectrogram is a Mel-spectrogram or a linear spectrogram. The audio encoder is a convolutional neural network or a recurrent neural network.

[0057] In the specific implementation process, the input audio data is preprocessed such as noise reduction and normalization to ensure data quality. Then, the audio data is converted into a spectrogram (which can be converted into a Mel Spectrogram or a linear spectrogram) and then input into an audio encoder (such as a convolutional neural network (CNN) or a recurrent neural network (RNN) model). Through multi-layer network structure learning, audio features are extracted and time series information is captured, and finally an audio feature vector of fixed length is output.

[0058] S3: Mapping the audio feature vector into a plurality of first two-dimensional images through a trained adversarial diffusion generation model.

[0059] Optionally, in one embodiment of the present application, before mapping the audio feature vector to a plurality of first two-dimensional images through the trained adversarial diffusion generative model, the method further includes training an adversarial diffusion generative model. The adversarial diffusion generative model includes an image encoder, a UNet decoder and a discriminator, and the adversarial diffusion generative model is trained, such as Figure 3 As shown, including:

[0060] Initialize a noisy image.

[0061] The image encoder is used to encode the noisy image into a noise image feature vector.

[0062] The audio feature vector and the noise image feature vector are fused to obtain a fused feature vector.

[0063] The fused feature vector is decoded using the UNet decoder to generate a two-dimensional image.

[0064] The discriminator performs classification processing according to the generated two-dimensional image and the input real two-dimensional image, and outputs a probability value that the generated two-dimensional image is a real two-dimensional image.

[0065] The discriminator and the UNet decoder are iteratively optimized according to the generated two-dimensional image, the input real two-dimensional image and the probability value to obtain a trained adversarial diffusion generation model.

[0066] In the specific implementation process, a noisy image is randomly initialized, encoded into an image feature vector through the image encoder VAE, and then feature fused with the audio feature vector. This synthetic feature vector is fed to the self-attention layer of the UNet encoder for feature interaction. The purpose of this process is to use the audio features as conditional information to guide the image generation process and ensure that the generated image can fully reflect the content and emotion of the audio. In each iteration, the UNet decoder predicts the noise components in the current noisy image based on the features extracted by the encoder. Subsequently, the model subtracts these predicted noises from the noisy image and gradually restores a clearer image. In this training step, the KL divergence and MSE mean square error between the two images are calculated, and then the loss is calculated as a weighted sum: Loss = 0.5*Loss_KL+0.5*Loss_MSE. The overall goal is to minimize the difference between the generated content and the real image content in terms of richness and detail.

[0067] It can be understood that by training an adversarial diffusion generative model for generating matching image content from audio data, it aims to capture features such as rhythm, intonation, emotion, etc. in the audio and convert them into various visual elements in the image, such as scenery, objects, human expressions, body movements, etc.

[0068] Optionally, in one embodiment of the present application, the discriminator performs classification processing according to the generated two-dimensional image and the input real two-dimensional image, and outputs a probability value that the generated two-dimensional image is a real two-dimensional image, including:

[0069] Feature extraction is performed on the generated two-dimensional image and the real two-dimensional image to generate a pair of image feature vectors of the same length, and then the pair of image feature vectors of the same length are concatenated and input into the self-attention layer for feature interaction, and the feature vector after feature interaction is processed through a fully connected layer, and a probability value of the generated two-dimensional image being a real two-dimensional image is output.

[0070] In the specific implementation process, the discriminator receives two inputs: one is the image generated by the decoder of UNet, and the other is the corresponding real image. The discriminator first uses VAE to perform the same feature extraction on the pair of input images to generate a fixed-length image feature vector, which contains the key features of the image, such as edges, textures, shapes and other information. These feature vectors are concatenated and input to the self-attention layer for feature interaction, and finally the fused features are output. The extracted features then pass through a series of fully connected layers or other layers, and finally the discriminator outputs a probability value, indicating the possibility that the image generated by the decoder of UNet is a real image. The training of the discriminator usually uses the binary cross entropy loss function Loss_cls, which aims to minimize the difference between the predicted probability and the actual label between the generated image and the real image.

[0071] Optionally, in one embodiment of the present application, the discriminator and the UNet decoder are iteratively optimized according to the generated two-dimensional image, the input real two-dimensional image and the probability value, including:

[0072] The KL divergence and the mean square error of the generated two-dimensional image and the real two-dimensional image are calculated, and the loss function of the UNet decoder is constructed using the KL divergence and the mean square error.

[0073] A binary cross entropy loss function is constructed according to the probability value as the loss function of the discriminator.

[0074] The UNet decoder and the discriminator are iteratively trained according to the loss function of the UNet decoder and the loss function of the discriminator, respectively, until a predetermined number of training rounds is reached to obtain a trained adversarial diffusion generation model.

[0075] In the specific implementation process, the training of the discriminator is performed alternately with the adversarial training of the generator. The discriminator identifies the difference between the generated image and the real image through feature extraction and classification, and continuously improves the identification ability. The generator iteratively optimizes the model parameters to better achieve the conversion from noise to clear images and ensure that the generated images are associated with the audio. After the predetermined number of training rounds, the training process ends.

[0076] In addition, this embodiment uses an independent validation set to evaluate the performance of the model to ensure that the model can accurately generate corresponding images based on the audio. Based on the evaluation results, the model may need to be fine-tuned to improve the relevance and quality of the generated images. Through this training method, the adversarial diffusion model of audio and two-dimensional images can learn how to generate corresponding images based on audio signals, and continuously improve the model to better reflect the content and emotion of the audio.

[0077] S4: Initialize a three-dimensional model, and use the audio feature vector to render the three-dimensional model from a plurality of random angles to generate a second two-dimensional image of a plurality of viewing angles.

[0078] Optionally, in one embodiment of the present application, a three-dimensional model is initialized using a Neural Radiance Fields (NeRF) architecture.

[0079] In the specific implementation process, a series of viewpoints are randomly selected in 3D space, including different camera positions, directions, and possible focal lengths. Random sampling ensures that a variety of views can be captured from the 3D model, which helps to fully understand and represent the 3D scene. The rendering process means that for each randomly selected viewpoint, the system uses the neural radiance field model to render a 2D image. The neural radiance field is a neural network that learns the volume density and color distribution of the 3D scene and can generate a corresponding 2D image based on the spatial position and viewing direction.

[0080] S5: Iteratively optimize the three-dimensional model according to the audio feature vector, the first two-dimensional image, and the second two-dimensional image to generate a final three-dimensional model.

[0081] Optionally, in one embodiment of the present application, after initializing a three-dimensional model, the method further includes:

[0082] The cosine similarity between every two audio feature vectors is calculated, and a group of audio feature vectors whose cosine similarity is greater than a preset threshold is screened out from the audio feature vectors.

[0083] The audio feature vector groups are mapped into a plurality of first two-dimensional image groups through a trained adversarial diffusion generation model, and the three-dimensional model is rendered from a plurality of random angles to generate a plurality of second two-dimensional image groups of viewing angles.

[0084] Optionally, in one embodiment of the present application, iteratively optimizing the three-dimensional model according to the audio feature vector, the first two-dimensional image, and the second two-dimensional image to generate a final three-dimensional model includes:

[0085] According to the audio feature vector, the first two-dimensional image and the second two-dimensional image, a loss function of the three-dimensional model is constructed, and its expression is as follows:

[0086] Loss=Loss_audio_cos+Loss_audio_KL+Loss_img_cos+Loss_img_KL

[0087] +Loss_SSIM_1+Loss_SSIM_2

[0088] Among them, Loss_audio_cos and Loss_audio_KL are the cosine similarity loss and KL divergence loss between the audio feature vector and the second two-dimensional image, Loss_img_cos and Loss_img_KL are the cosine similarity loss and KL divergence loss between the first two-dimensional image and the second two-dimensional image, Loss_SSIM_1 and Loss_SSIM_2 are the two groups of structural similarity index losses between the first two-dimensional image group and the second two-dimensional image group.

[0089] The three-dimensional model is iteratively trained by a gradient descent method according to the loss function of the three-dimensional model until a predetermined number of training rounds is reached to generate a final three-dimensional model.

[0090] In the specific implementation process, the two-dimensional image img_1 rendered by the three-dimensional model, the audio feature audio_1 and the image img_2 output by the audio input adversarial diffusion model are output using the same image encoder InceptionNet to calculate the KL divergence and cosine similarity of img_1 and img_2, and the KL divergence and cosine similarity of img_1 and audio_1. These indicators are added as the loss function: Loss_audio_cos+Loss_audio_KL+Loss_img_cos+Loss_img_KL.

[0091] Then, some audio pairs are randomly selected from the group with high similarity Group_most_sim, and the similarity Sim_score calculated previously, and the audio pairs are input into the 3D model to obtain the rendered 2D image pairs img_3D_1 and img_3D_2. The audio pairs are input into the adversarial diffusion model to output the image pairs img_diffusion_1 and img_diffusion_2. Similarly, the same image encoder InceptionNet is used to output image features, and the SSIM (structural similarity index) of img_diffusion_1 and img_3D_2, namely Loss_SSIM_1, and the SSIM index of img_diffusion_2 and img_3D_1, namely Loss_SSIM_2, are calculated respectively.

[0092] Combining the above two steps, the overall loss function is:

[0093] Loss=Loss_audio_cos+Loss_audio_KL+Loss_img_cos+Loss_img_KL

[0094] +Loss_SSIM_1+Loss_SSIM_2

[0095] As an exemplary description, this embodiment uses the loss function Loss to adjust the parameters of the three-dimensional model through a gradient descent algorithm and dynamic quantization training so that the rendered two-dimensional image better reflects the audio content and emotion.

[0096] By adopting a dynamic selection method for important weights, it is determined which weights should be kept in the original high-precision format (such as FP16) and which can be quantized to lower precision (such as INT4) based on the importance of the weights. The quantization operation can significantly reduce the storage space requirements of the model, making it easier to deploy the model on devices with limited storage resources and improve the reasoning speed, perform forward propagation, generate activation data, and collect activation outputs for each layer. These activation outputs are intermediate representations generated by the model when processing input data.

[0097] In addition, this embodiment performs statistical analysis on the activation output of each layer and calculates the distribution of activation amplitudes. Activations with larger activation amplitudes are identified, usually corresponding to more important features in the model. According to the distribution of activation amplitudes, the importance of each weight channel is evaluated. The weight importance is determined by calculating the average (or maximum) activation amplitude of each channel. The initialization threshold parameter T is 0.05, indicating that the top 5% of the weights with the largest activation amplitudes are retained and do not participate in quantization, and T will be updated during training. Except for the weight parameters that do not participate in quantization, the remaining weights are quantized by dynamically adjusted quantization steps. The quantization step size is a key scaling factor that determines the accuracy of mapping floating point numbers (such as FP32 or FP16) to low-precision integer representations (such as INT8 or INT4). A strategy for dynamically adjusting the quantization step size is adopted, and a larger value is set in the initial stage of training, corresponding to a wider quantization range, to reduce errors in the quantization process. As the training proceeds, the quantization step size is gradually reduced according to a predetermined attenuation parameter to quantize the model weights more finely and approach the original floating point values. This dynamic quantization step method helps to avoid quantization errors affecting model performance, reduce the quantization step in the later stages of training, improve the accuracy of quantization weights, reduce model size and accelerate inference.

[0098] After multiple iterations of optimization, the resulting 3D model can accurately reflect the characteristics and emotions of the input audio. This 3D model can render a 2D image that matches the audio content from any angle. The invention provides new possibilities for audio-driven 3D content creation and has potential applications in game development, virtual reality, music video production and other fields.

[0099] It is understandable that by dynamically adjusting the quantization step size, the model is successfully compressed and accelerated, while maintaining the performance of the model. This innovation not only solves the problem of model deployment and reasoning speed, but also makes it possible to use high-quality 3D model generation on resource-constrained devices.

[0100] It should be noted that this embodiment uses CNN or RNN to extract audio features, and other types of neural networks, such as LSTM or GRU, can also be used. This embodiment uses the UNet structure, and other generation models, such as conditional GAN ​​or CVAE, can also be tried. The discriminator of this embodiment uses VAE for feature extraction, and the Inception network or other pre-trained image recognition models can also be used. In view of the lack of large-scale three-dimensional annotation data sets, this embodiment can also use data enhancement and transfer learning methods to replace part of the data requirements. For example, pseudo data generated by existing two-dimensional image data sets and some three-dimensional model generation algorithms are used to assist training.

[0101] It is understandable that the audio data is obtained and feature extracted to generate an audio feature vector, and then the audio feature vector is mapped to a number of first two-dimensional images through a trained adversarial diffusion generative model, which solves the problem of generating a matching two-dimensional image from the audio data, and can learn the complex mapping relationship between audio and image. The generated image is not only of high quality, but also can faithfully reflect the rhythm, intonation and emotional characteristics in the audio, providing key visual information for the subsequent three-dimensional model generation. In addition, by combining the pre-trained audio-to-image adversarial diffusion generative model with the three-dimensional model generation technology, the first two-dimensional image is used to render the three-dimensional model from a random angle, and the second two-dimensional images of several perspectives are generated. The three-dimensional model is iteratively optimized based on the audio feature vector, the first two-dimensional image and the second two-dimensional image, which ensures that the three-dimensional model learns the characteristics of the three-dimensional scene from multiple perspectives, does not require a large-scale three-dimensional annotation data set, and achieves the generation of three-dimensional models with both high quality and diversity.

[0102] Figure 4 The electronic device 100 provided in the embodiment of the present application is a schematic diagram of the structure of the electronic device 100. The electronic device 100 includes: a memory 101, a processor 102, and a computer program stored in the memory 101 and executable on the processor 102.

[0103] When the processor 102 executes the program, a three-dimensional model generation method based on audio and image fusion provided in the above embodiment is implemented.

[0104] Furthermore, the electronic device 100 also includes: a communication interface 103 for communication between the memory 101 and the processor 102 .

[0105] The memory 101 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0106] If the memory 101, the processor 102 and the communication interface 103 are implemented independently, the communication interface 103, the memory 101 and the processor 102 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0107] Optionally, in a specific implementation, if the memory 101, the processor 102 and the communication interface 103 are integrated on a chip, the memory 101, the processor 102 and the communication interface 103 can communicate with each other through an internal interface.

[0108] The processor 102 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0109] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a three-dimensional model generation method based on audio and image fusion as described in the above embodiment is implemented.

[0110] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0111] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0112] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0113] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0114] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0115] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A three-dimensional model generation method based on audio and image fusion, characterized in that: include: Get audio data; Extracting features from the audio data to obtain an audio feature vector; Calculate the cosine similarity between every two audio feature vectors, and select the audio feature vector group whose cosine similarity is greater than a preset threshold from the audio feature vectors; Initializing a three-dimensional model, using the audio feature vector group to map into a plurality of first two-dimensional image groups through a trained adversarial diffusion generation model, and rendering the three-dimensional model from a plurality of random angles to generate a plurality of second two-dimensional image groups of viewing angles; According to the audio feature vector, the first two-dimensional image and the second two-dimensional image, a loss function of the three-dimensional model is constructed, and its expression is as follows: Loss=Loss_audio_cos+Loss_audio_KL+Loss_img_cos+Loss_img_KL +Loss_SSIM_1+Loss_SSIM_2 Among them, Loss_audio_cos and Loss_audio_KL are the cosine similarity loss and KL divergence loss between the audio feature vector and the second two-dimensional image, Loss_img_cos and Loss_img_KL are the cosine similarity loss and KL divergence loss between the first two-dimensional image and the second two-dimensional image, Loss_SSIM_1 and Loss_SSIM_2 are the two groups of structural similarity index losses between the first two-dimensional image group and the second two-dimensional image group; The three-dimensional model is iteratively trained by a gradient descent method according to the loss function of the three-dimensional model until a predetermined number of training rounds is reached to generate a final three-dimensional model.

2. The method for generating a three-dimensional model based on audio and image fusion according to claim 1, characterized in that: The method also includes training an adversarial diffusion generative model; The adversarial diffusion generation model includes an image encoder, a UNet decoder and a discriminator, and training the adversarial diffusion generation model includes: Initialize a noise image; Encoding the noisy image into a noise image feature vector using the image encoder; Fusing the audio feature vector and the noise image feature vector to obtain a fused feature vector; Decoding the fused feature vector using the UNet decoder to generate a two-dimensional image; The discriminator performs classification processing according to the generated two-dimensional image and the input real two-dimensional image, and outputs a probability value that the generated two-dimensional image is a real two-dimensional image; The discriminator and the UNet decoder are iteratively optimized according to the generated two-dimensional image, the input real two-dimensional image and the probability value to obtain a trained adversarial diffusion generation model.

3. The method for generating a three-dimensional model based on audio and image fusion according to claim 2, characterized in that: The discriminator performs classification processing according to the generated two-dimensional image and the input real two-dimensional image, and outputs a probability value that the generated two-dimensional image is a real two-dimensional image, including: Feature extraction is performed on the generated two-dimensional image and the real two-dimensional image to generate a pair of image feature vectors of the same length, and then the pair of image feature vectors of the same length are concatenated and input into the self-attention layer for feature interaction, and the feature vector after feature interaction is processed through a fully connected layer, and a probability value of the generated two-dimensional image being a real two-dimensional image is output.

4. The method for generating a three-dimensional model based on audio and image fusion according to claim 2, characterized in that: Iteratively optimizing the discriminator and the UNet decoder according to the generated two-dimensional image, the input real two-dimensional image, and the probability value, including: Calculating the KL divergence and mean square error of the generated two-dimensional image and the real two-dimensional image, and constructing the loss function of the UNet decoder using the KL divergence and the mean square error; Constructing a binary cross entropy loss function according to the probability value as the loss function of the discriminator; The UNet decoder and the discriminator are iteratively trained according to the loss function of the UNet decoder and the loss function of the discriminator, respectively, until a predetermined number of training rounds is reached to obtain a trained adversarial diffusion generation model.

5. The method for generating a three-dimensional model based on audio and image fusion according to claim 1, characterized in that: Performing feature extraction on the audio data to obtain an audio feature vector includes: Converting the audio data into a spectrogram; The spectrogram is input into an audio encoder for feature extraction to obtain an audio feature vector.

6. The method for generating a three-dimensional model based on audio and image fusion according to claim 5, characterized in that: The spectrogram is a Mel-spectrogram or a linear spectrogram; the audio encoder is a convolutional neural network or a recurrent neural network.

7. The method for generating a three-dimensional model based on audio and image fusion according to claim 1, characterized in that: The initialization of a three-dimensional model comprises: Initialize a 3D model using the Neural Radiance Field architecture.

8. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the operations performed by the three-dimensional model generation method based on audio and image fusion as described in any one of claims 1 to 7 are implemented.

Images