Audio visualization method and device, equipment and storage medium

Abstract

The invention discloses an audio visualization method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring an input audio; according to the input audio, obtaining an animation track corresponding to the input audio, the animation track being a motion track of an object described by the input audio in a to-be-generated animation; according to the input audio, an animation sequence corresponding to the input audio and scene data corresponding to the input audio are obtained, the animation sequence comprises at least one animation image, and each animation image is used for indicating a part of audio meanings in the input audio; the scene data is used for indicating scene content in the at least one animation image and a display view angle of the at least one animation image; and generating an output animation corresponding to the input audio according to the animation track, the animation sequence and the scene data. According to the invention, the relevance and accuracy of the output animation and the input audio are improved, the technical scheme of the invention is realized by the client, and the resources of the server are saved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an audio visualization method, device, equipment and storage medium. Background Art

[0002] Music visualization refers to a popular communication method that takes vision as the core, music as the carrier, the general public as the target of appeal, and uses a variety of new media technologies and other communication media to interpret the content of music through pictures and images, combining audio and video.

[0003] In related technologies, the animation trajectory is first calculated offline based on the frequency and loudness of the audio. For example, the animation trajectory refers to the height of the ball bouncing in each frame. The louder the audio, the higher the ball bounces. The designer designs the animation image in advance, and the client pulls the animation trajectory and animation image, and uses the rendering tool on the client to perform audio visualization rendering to obtain the animation corresponding to the audio.

[0004] However, the animation images obtained by the above method are fixed and lack variation, which easily leads to a small correlation between the rendering results and the audio. Summary of the invention

[0005] The embodiment of the present application provides an audio visualization method, device, equipment and storage medium. The technical solution provided by the embodiment of the present application is as follows:

[0006] According to one aspect of an embodiment of the present application, there is provided an audio visualization method, the method comprising:

[0007] Get input audio;

[0008] According to the input audio, an animation trajectory corresponding to the input audio is obtained, wherein the animation trajectory is a motion trajectory of an object described by the input audio in an animation to be generated;

[0009] According to the input audio, an animation sequence corresponding to the input audio and scene data corresponding to the input audio are obtained, wherein the animation sequence includes at least one animation image, each animation image is used to indicate a meaning of a portion of the audio in the input audio, and the scene data is used to indicate scene content in the at least one animation image and a display viewing angle of the at least one animation image;

[0010] An output animation corresponding to the input audio is generated according to the animation trajectory, the animation sequence and the scene data.

[0011] According to one aspect of an embodiment of the present application, there is provided an audio visualization device, the device comprising:

[0012] An audio acquisition module, used to acquire input audio;

[0013] A trajectory determination module, used to obtain an animation trajectory corresponding to the input audio according to the input audio, wherein the animation trajectory is a motion trajectory of the object described by the input audio in the animation to be generated;

[0014] a data determination module, configured to obtain, according to the input audio, an animation sequence corresponding to the input audio and scene data corresponding to the input audio, wherein the animation sequence includes at least one animation image, each animation image is used to indicate a meaning of a portion of the audio in the input audio, and the scene data is used to indicate a scene content in the at least one animation image and a display viewing angle of the at least one animation image;

[0015] The animation generation module is used to generate an output animation corresponding to the input audio according to the animation trajectory, the animation sequence and the scene data.

[0016] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned audio visualization method.

[0017] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned audio visualization method.

[0018] According to one aspect of an embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. The computer program is loaded and executed by a processor to implement the above-mentioned audio visualization method.

[0019] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0020] Based on the input audio, the animation sequence corresponding to the input audio and the scene data corresponding to the input audio are obtained, so that the obtained animation sequence and scene data are closely related to the audio data of the input audio, and fully represent the audio meaning of the input audio. Compared with the use of pre-designed animation images in related technologies, this application improves the flexibility of obtaining animation sequences and scene data, and improves the correlation between animation sequences and scene data and input audio, so that output animations are generated according to animation trajectories, animation sequences and scene data, which can improve the correlation and accuracy of output animations and input audio, and enhance the animation effect of output animations. In addition, the technical solution of this application is implemented by the client of the terminal device, which only occupies the processing resources of the client, thereby consuming the processing resources of the terminal device and saving the resources of the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of audio visualization provided by an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a complete process of audio visualization provided by an embodiment of the present application;

[0024] Figure 4 is a block diagram of an audio visualization device provided by an embodiment of the present application;

[0025] Figure 5 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0027] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment can be implemented as an audio visualization system. The solution implementation environment may include: a terminal device 10 and a server 20.

[0028] The number of terminal devices 10 can be one or more. The terminal device 10 can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a game console, an e-book reader, a multimedia player, a wearable device, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc.

[0029] The terminal device 10 may be installed with a client of a target application, and the target application has an audio visualization function. When a user inputs audio into the target application, the target application generates an output animation corresponding to the input audio. The present application does not limit the type of the target application, including but not limited to question-and-answer applications, music applications, video applications, game applications, web applications, etc. Optionally, the target application may be an application that needs to be downloaded and installed, or may be a click-to-use application, which is not limited in the present application.

[0030] The server 20 is used to provide background services for the client of the target application installed and running in the terminal device 10. For example, the server 20 can be the background server of the above-mentioned target application. The server 20 can be an independent physical server, or a server cluster composed of multiple servers, or a cloud computing service center. Optionally, the server 20 provides background services for the target applications in multiple terminal devices 10 at the same time. The terminal device 10 and the server 20 can communicate with each other through the network.

[0031] In the embodiments of the present application, the audio visualization method is implemented on a terminal device. The user inputs audio in a target application. The client of the target application generates an animation trajectory corresponding to the input audio based on the input audio, and generates an animation sequence and scene data corresponding to the input audio based on the input audio. Then, based on the animation trajectory, animation sequence and scene data, an output animation corresponding to the input audio is generated, and the output animation is displayed on the display interface of the target application.

[0032] Please refer to Figure 2 , which shows a flow chart of an audio visualization method provided by an embodiment of the present application. The execution subject of each step of the method may be a terminal device. The method may include at least one of the following steps 210 to 240:

[0033] Step 210: Obtain input audio.

[0034] Optionally, the input audio may be a single audio data, or may be audio data obtained by synthesizing multiple audio data, which is not limited in this application. The audio data includes at least one of the following: network audio data, local audio data, and audio data collected in real time. Network audio data refers to audio data downloaded from the network in real time, local audio data refers to audio data pre-stored in a local file, and audio data collected in real time refers to audio data collected in real time through a microphone. Optionally, the input audio may be any one of network audio data, local audio data, and audio data collected in real time, or may be audio data obtained by synthesizing at least two of network audio data, local audio data, and audio data collected in real time, which is not limited in this application.

[0035] Step 220, obtaining an animation trajectory corresponding to the input audio according to the input audio, where the animation trajectory is a motion trajectory of the object described by the input audio in the animation to be generated.

[0036] The object described by the input audio may be at least one arbitrary object. Optionally, the object described by the input audio may be one arbitrary object or multiple arbitrary objects. Exemplarily, when the input audio includes one audio track, the object described by the input audio is one object, and when the input audio includes multiple audio tracks, the object described by the input audio is multiple objects.

[0037] Optionally, one audio track may correspond to one object or multiple objects. Exemplarily, when one audio track included in the input audio is a vocal track, the object described by the input audio is determined based on the vocals in the vocal track. For example, if the vocals in the vocal track only include vocals of one voice line, the object described by the input audio is one object; if the vocals in the vocal track include vocals of multiple different voice lines, the object described by the input audio is multiple objects; or, if the vocals in the vocal track only involve one object, the object described by the input audio is one object; if the vocals in the vocal track involve multiple different objects, the object described by the input audio is multiple objects.

[0038] For the objects mentioned above, when the input audio contains a human voice track and the human voice track involves at least one object, the object described by the input audio may be the object involved in the human voice track, for example, it may be an object mentioned in the human voice such as a bird, a fish, a pedestrian, a kite, etc. When the input audio contains a human voice track and the human voice track does not involve at least one object, the object described by the input audio may be a default defined object, or, when the input audio does not contain a human voice track, the object described by the input audio may be a default defined object. The default defined objects include but are not limited to any objects such as small balls, silk threads, water flows, etc., and this application does not limit them.

[0039] If the object described by the input audio is multiple objects, the multiple objects can be objects of different types. For example, if the input audio contains two tracks, one of the two tracks can correspond to a ball, and the other track can correspond to a thread. The multiple objects can also be objects of the same type, presented in different forms. For example, if the input audio contains two tracks, one of the two tracks can correspond to a red ball, and the other track can correspond to a blue ball. This application does not limit the specific setting of the object described by the input audio.

[0040] The animation trajectory is the motion trajectory of the object described by the input audio in the animation to be generated. If the object described by the input audio is one object, the animation trajectory refers to the motion trajectory of the one object in the animation to be generated. If the object described by the input audio is multiple objects, the animation trajectory includes the motion trajectories of the multiple objects in the animation to be generated. According to the input audio, the object described by the input audio is obtained, and according to the input audio, the position of the object described by the input audio in each frame of the animation to be generated is determined. According to the position of the object described by the input audio in each frame of the animation to be generated, the animation trajectory corresponding to the input audio is obtained. Exemplarily, if the object described by the input audio is a small ball, the animation trajectory refers to the motion trajectory of the small ball in the animation to be generated. According to the input audio, the position of the small ball in each frame of the animation to be generated is determined, and then the position of the small ball in each frame of the animation to be generated is obtained to obtain the animation trajectory corresponding to the small ball.

[0041] Step 230, based on the input audio, obtain an animation sequence corresponding to the input audio and scene data corresponding to the input audio, the animation sequence includes at least one animation image, each animation image is used to indicate a part of the audio meaning in the input audio, and the scene data is used to indicate the scene content in at least one animation image and the display viewing angle of at least one animation image.

[0042] The animation sequence corresponding to the input audio includes at least one animation image, and the at least one animation image is used to indicate a part of the audio meaning in the input audio in turn, and the animation sequence is used to indicate the entire audio meaning of the input audio. The scene data corresponds to the animation sequence, and the scene data includes at least one sub-scene data. Each animation image corresponds to a sub-scene data, and each sub-scene data is used to indicate the scene content in the animation image corresponding to the sub-scene data and the display viewing angle of the animation image corresponding to the sub-scene data. In addition, the animation trajectory is essentially the motion trajectory of the object described by the input audio in the animation sequence, which includes the position of the object described by the input audio in at least one animation image. And the position of the object described by the input audio in adjacent animation images can be the same or different, which is specifically related to the audio data of the input audio and is not limited in this application.

[0043] For the animation sequence and scene data, please refer to the following embodiments for details, which will not be introduced here.

[0044] Step 240: Generate an output animation corresponding to the input audio according to the animation trajectory, animation sequence and scene data.

[0045] The animation trajectory is the motion trajectory of the object described by the input audio in the animation to be generated. The animation sequence is used to indicate the audio meaning in the input audio. The scene data is used to indicate the scene content in the animation sequence and the display perspective of the animation sequence.

[0046] The output animation corresponding to the input audio is generated by an image rendering tool according to the animation trajectory, animation sequence and scene data. The specific generation process can be referred to the following embodiment and will not be introduced here.

[0047] The technical solution provided by the embodiment of the present application obtains the animation sequence corresponding to the input audio and the scene data corresponding to the input audio based on the input audio, so that the obtained animation sequence and scene data are closely related to the audio data of the input audio, and fully represent the audio meaning of the input audio. Compared with the use of pre-designed animation images in the related art, the present application improves the flexibility of obtaining animation sequences and scene data, and improves the correlation between animation sequences and scene data and input audio, so as to generate output animation according to animation trajectory, animation sequence and scene data, which can improve the correlation and accuracy of output animation and input audio, and enhance the animation effect of output animation. In addition, the technical solution of the present application is implemented by the client of the terminal device, which only occupies the processing resources of the client, thereby consuming the processing resources of the terminal device and saving the resources of the server.

[0048] In some embodiments, step 210 includes at least one of sub-steps 211 - 212 .

[0049] Sub-step 211, obtaining at least two audio data, where the at least two audio data include at least one of the following: network audio data, local audio data, and real-time collected audio data.

[0050] Network audio data refers to audio data downloaded from the Internet in real time, local audio data refers to audio data pre-stored in local files, and real-time audio data refers to audio data collected in real time through a microphone. Audio data can be an audio stream containing human voice, such as songs, voices, calls, etc., or it can be an audio stream that does not contain human voice, such as pure music, instrument sounds, accompaniment, sound effects, and audio streams collected from nature that do not contain human voice.

[0051] The at least two audio data are at least two of network audio data, local audio data, and audio data collected in real time. Optionally, the at least two audio data may include network audio data and local audio data, or may include network audio data and audio data collected in real time, or may include local audio data and audio data collected in real time, or may include network audio data, local audio data, and audio data collected in real time.

[0052] Sub-step 212, performing mixing processing on at least two audio data to obtain input audio.

[0053] Mixing refers to the process of combining at least two audio data into one output audio stream. This process requires processing and merging each audio data to generate the final mixed audio, which is the input audio. When the client performs mixing, it is usually necessary to control the volume, channel, time, etc. of each audio data to ensure that the final mixing effect is natural and meets the requirements.

[0054] Input audio is generated by performing mixing processing on at least two audio data, thereby enriching the acquisition sources of the input audio and improving the diversity and extensiveness of audio visualization.

[0055] In some embodiments, sub-step 212 includes at least one of sub-steps 2121 - 2125 .

[0056] Sub-step 2121, preprocessing is performed on at least two audio data to obtain at least two preprocessed audio data, where the preprocessing includes at least one of the following: sampling rate conversion processing, channel adjustment processing, and audio format unification processing.

[0057] Sampling rate conversion processing is used to ensure that the sampling rates of multiple audio data are consistent to avoid incompatibility issues during mixing. Channel adjustment processing is used to ensure that the number of channels of multiple audio data is consistent, such as stereo, mono and other processing methods. Audio format unification processing is used to convert multiple audio data into a unified format for processing, such as WAV (Waveform Audio File Format), PCM (Pulse Code Modulation) and other formats.

[0058] Sub-step 2122: Perform gain adjustment processing on the at least two pre-processed audio data respectively to obtain at least two audio data after volume adjustment.

[0059] Perform gain or attenuation processing on each pre-processed audio data to adjust the volume of each audio data to avoid excessive volume or distortion after audio superposition.

[0060] Sub-step 2123, respectively setting channel positions for the at least two audio data after the volume is adjusted, to obtain the at least two audio data after the channel is set.

[0061] The channel position, such as the left channel or the right channel, is set for each audio stream after the volume is adjusted, so that when mixing multiple audio data, some audio data can be biased to one side to provide a sense of space for the left and right channels, such as audio panning, stereo effects, etc.

[0062] Sub-step 2124, performing effect processing on the at least two audio data after the channel setting, to obtain at least two audio data after effect processing, the effect processing including at least one of the following: sound effect adding processing, filtering processing, compression processing.

[0063] After the channel is set, the corresponding effect processing is performed on the different audio data, such as reverberation, delay, equalization, etc., to ensure the sound coordination of each audio data and enhance the expressiveness and coordination of the input audio. If the audio data comes from different sources, each audio data may also need to be filtered, compressed, etc. to ensure that there is no frequency conflict between different audio data.

[0064] Sub-step 2125, performing track superposition on the at least two audio data after the effect processing according to the playing time of the at least two audio data after the effect processing to obtain the input audio.

[0065] According to the play time of at least two audio data after effect processing, the play time of at least two audio data is aligned to avoid asynchronism between different audio data. For pre-recorded audio data and real-time generated audio data, synchronization processing may be required.

[0066] Perform track superposition on at least two audio data after effect processing, that is, add the waveforms of at least two audio data to generate final mixed audio data, and dynamic range compression may be required during the superposition process to avoid distortion caused by excessive volume. Merge all audio tracks into a final output audio stream and export it to a file (such as WAV, MP3, etc.) to obtain the input audio.

[0067] By performing mixing processing on at least two audio data through the above method, the synchronization and coordination between the various audio data are effectively guaranteed, and the audio quality of the generated input audio is guaranteed.

[0068] In some embodiments, step 220 includes at least one of sub-steps 221 - 224 .

[0069] Sub-step 221, separating the input audio tracks through a multi-track separation model to obtain instrument tracks and vocal tracks in the input audio.

[0070] The Music Source Separation (MSS) task usually uses a deep learning model to separate audio data into different tracks, such as instrument tracks (Instrument Stems) and vocal tracks (Vocal Stems). The multi-track separation model here can be any publicly available multi-track separation model, such as a model based on a deep learning structure trained with a large amount of data, such as a sample level of more than 100 million, which is not limited in this application.

[0071] Exemplarily, the multi-track separation model can be a Wave-U-Net model, which gradually restores the audio signal by extracting and upsampling the multi-level features of the audio waveform, while retaining the independent characteristics between the human voice and the musical instrument, so as to achieve the separation of the human voice from the background music, such as the separation of the instrument track and the human voice track. Exemplarily, the multi-track separation model can be an Open-Unmix model, which converts the audio to the frequency domain through the short-time Fourier transform, uses a neural network to operate on the spectrum, predicts the amplitude spectrogram of each source (such as human voice or musical instrument), and finally reconstructs the separated audio signal through the inverse short-time Fourier transform to obtain the instrument track and the human voice track.

[0072] Through the multi-track separation model, different audio sources can be extracted from the input audio, such as instrument tracks, vocal tracks and other tracks. Other tracks can be background noise, environmental sounds and other tracks other than instrument tracks and vocal tracks.

[0073] The instrument track in the input audio refers to the track generated based on the performance of the instrument. The instrument mentioned here can be an actual instrument, such as a piano, violin, erhu, etc., and the instrument track is obtained based on the real-time performance of the instrument; it can also be an electronic instrument, and the instrument track is obtained based on data operation; it can also include actual instruments and electronic instruments, and the above instrument tracks are obtained based on the tracks obtained by real-time performance and the tracks obtained by data operation.

[0074] The human voice track in the input audio refers to a track generated based on human voice, which is a track obtained by collecting human voice. The human voice track can be a track sung by a person, or a track recited by a person, etc., which is not limited in this application.

[0075] Sub-step 222, performing track analysis on the instrument track in the input audio through the track analysis model to obtain instrument data corresponding to the instrument track, where the instrument data is used to indicate the audio style and audio attributes in the instrument track.

[0076] The audio track analysis model is a model obtained by adjusting and training a large language model based on sample audio data. It is used to perform audio track analysis on the audio track to obtain the audio track data corresponding to the audio track.

[0077] The instrument track in the input audio is input to the track analysis model, and the instrument data corresponding to the instrument track is output. The instrument data is feature data for indicating the audio style and audio attributes in the instrument track. The instrument data includes at least one of the following: rhythm data, frequency data, loudness data, and style data. The rhythm data is used to indicate the rhythm characteristics of the audio stream in the instrument track, the frequency data is used to indicate the frequency characteristics of the audio stream in the instrument track, the loudness data is used to indicate the loudness characteristics of the audio stream in the instrument track, and the style data is used to indicate the emotion and style contained in the audio stream in the instrument track.

[0078] Sub-step 223, performing character analysis on the vocal track in the input audio through the audio track analysis model to obtain character data corresponding to the vocal track, where the character data is used to indicate the character attributes in the vocal track.

[0079] The vocal track in the input audio is input to the audio track analysis model, and the character data corresponding to the vocal track is output. The character data is feature data used to indicate the attributes of the character in the vocal track. The character data includes at least one of the following: character age data, character gender data, character personality data, and character voice data. The character personality data is used to indicate the character characteristics of the character in the vocal track, and the character voice data is used to indicate the voice characteristics of the character in the vocal track.

[0080] Sub-step 224, obtaining the animation trajectory corresponding to the input audio according to the instrument data and the character data through the trajectory extraction model.

[0081] The trajectory extraction model is a model obtained by adjusting and training the large language model based on sample instrument data and sample character data, and is used to generate animation trajectories corresponding to input data.

[0082] By separating the audio tracks of the input audio and performing audio track analysis on the instrument track and the vocal track respectively, the instrument data corresponding to the instrument track and the character data corresponding to the vocal track are obtained, and the instrument data and the character data are used to represent the audio features of the input audio, so as to generate an animation trajectory based on the instrument data and the character data, which can improve the accuracy of the animation trajectory and improve the correlation between the animation trajectory and the input audio.

[0083] In some embodiments, the training process of the trajectory extraction model is as follows: obtain at least one training sample of the large language model, each training sample includes sample data and sample trajectory, and the sample data includes sample instrument data and sample character data; generate a predicted trajectory based on the sample instrument data and the sample character data through the large language model; adjust the parameters of the large language model according to the difference between the predicted trajectory and the sample trajectory to obtain the trajectory extraction model.

[0084] The large language model here can be any publicly available large language model, such as a natural language model based on a transformer structure trained with a large amount of data. The large amount of data can reach a sample level of more than 100 million, and this application does not limit this.

[0085] Exemplarily, according to the difference between the predicted trajectory and the sample trajectory, a cross entropy loss function is calculated, and the parameters of the large language model are adjusted with the goal of minimizing the loss function value to obtain a trajectory extraction model.

[0086] By using sample instrument data and sample character data to train a large language model, the trajectory extraction model can be better applied to the trajectory extraction process of the present application, thereby improving the accuracy of the generated animation trajectory.

[0087] In some embodiments, step 230 includes at least one of sub-steps 231 - 234 .

[0088] Sub-step 231, obtaining an audio description text corresponding to the input audio through the audio understanding model based on the input audio.

[0089] The audio description text is a description text used to describe the audio meaning of the input audio. The audio understanding model here can be any publicly available multimodal understanding model, and this application does not limit this.

[0090] Sub-step 232, performing semantic analysis on the audio description text through a semantic analysis model to obtain at least one description text corresponding to the audio description text.

[0091] The semantic analysis model is a model obtained by adjusting and training a large language model based on sample text. It is used to divide the input text into lines according to the semantics of the input text and generate at least one sub-text.

[0092] The audio description text is input into the semantic analysis model, and at least one description text corresponding to the audio description text is output. Each description text is used to describe a part of the audio meaning of the input audio, and at least one description text is used in turn to describe a part of the audio meaning of the input audio. Each description text includes at least one character.

[0093] Sub-step 233, generating animation images corresponding to at least one segment of description text according to at least one segment of description text by using a diffusion model, and obtaining an animation sequence.

[0094] The Stable Diffusion (SD) model is a deep learning-based text-to-image generation model that can generate high-quality animated images based on a given description text. Its generation process combines technologies such as Convolutional Neural Networks (CNN) and Denoising Diffusion Probabilistic Models (DDPM) to gradually generate images from noise.

[0095] The basic process of the diffusion model to generate animated images is as follows: the description text is input into the diffusion model, and the description text is used to guide the diffusion model to generate animated images. First, the description text is converted into a feature representation through a text encoder (usually using a Transformer model, such as the CLIP (Contrastive Language-Image Pre-Training) model). This text feature captures the semantic information of the description text and serves as a condition for the subsequent generation of animated images. Secondly, the diffusion model generates a random noise image, which is usually completely random and the pixel values ​​are randomly initialized. The random noise image will be added with noise through a series of steps and gradually become a pure noise image. The core of the diffusion model is a U-Net structure, which is a convolutional neural network architecture suitable for image generation and denoising tasks. U-Net combines the input text conditions in each layer to achieve the association between text and image. In the process of generating animated images, each layer of U-Net will denoise the noise image based on the text features to ensure that the generated animated image is consistent with the description text. Each denoising step will gradually restore the structure and details of the image, and each denoising step will generate a result that is closer and closer to the final image. Each step will perform denoising operations based on the results of the previous step and the text features. Usually, the diffusion model uses multiple denoising steps (for example, 25 to 50 iterations) to gradually transform the noisy image into a clear image. Finally, after multiple denoising processes, a clear animation image is obtained, which should conform to the input description text. In addition, after the animation image is generated, there may be some additional post-processing steps, such as adjusting the brightness, contrast or detail optimization of the image, to improve the quality of the animation image.

[0096] Sub-step 234, performing scene analysis on at least one section of description text respectively through the scene analysis model to obtain scene data corresponding to the input audio.

[0097] The scenario analysis model is a model obtained by adjusting and training a large language model based on sample text. It is used to perform scenario analysis on the input text according to the semantics of the input text and obtain the scenario data corresponding to the input text.

[0098] At least one segment of description text is input into the scene analysis model respectively, and sub-scene data corresponding to the at least one segment of description text is output. According to the sub-scene data corresponding to the at least one segment of description text, scene data corresponding to the input audio is obtained.

[0099] By dividing the audio description text into lines according to the text semantics, the at least one obtained animation image can respectively represent a part of the audio semantics in the input audio, thereby improving the rationality and accuracy of the animation sequence, thereby enhancing the logic of the output animation, and performing scene analysis on at least one section of the description text, so that the scene data can fully represent the scene content indicated by the description text, further improving the accuracy of the output animation.

[0100] In some embodiments, the scene data corresponding to the input audio includes coordinate data corresponding to at least one segment of description text and camera data corresponding to at least one segment of description text. For each segment of description text, the scene analysis model is used to obtain the coordinate data corresponding to the description text and the camera data corresponding to the description text according to the description text; wherein the coordinate data is used to indicate the scene content in the animation image corresponding to the description text, and the camera data is used to indicate the display angle of the animation image corresponding to the description text.

[0101] The scene data corresponding to the input audio includes at least one sub-scene data, each sub-scene data corresponds to an animated image, and each sub-scene data corresponds to a description text. The scene data corresponding to the input audio includes coordinate data corresponding to at least one description text and camera data corresponding to at least one description text, which essentially means that each sub-scene data includes coordinate data corresponding to the description text and camera data corresponding to the description text.

[0102] The scene data is used to indicate the scene content in at least one animated image and the display viewing angle of at least one animated image. The coordinate data included in each sub-scene data is used to indicate the scene content in the animated image corresponding to the description text. The camera data included in each sub-scene data is used to indicate the display viewing angle of the animated image corresponding to the description text.

[0103] At least one segment of description text is input into the scene analysis model respectively, and coordinate data corresponding to the at least one segment of description text and camera data corresponding to the at least one segment of description text are output respectively.

[0104] By dividing scene data into coordinate data and camera data, using coordinate data to represent the scene content in the animated image, and using camera data to represent the display perspective of the animated image, the display content and display features of the animated image can be more comprehensively represented, enhancing the visual understanding of the descriptive text, thereby improving the accuracy of the generated output animation.

[0105] In some embodiments, the coordinate data includes at least one of the following: attribute data of at least one object in the animated image, position data of at least one object in the animated image, and background data in the animated image. The camera data includes at least one of the following: camera position data, camera orientation data, and camera field of view; the camera is used to shoot the animated image.

[0106] In the case where the input audio includes a human voice track, and the human voice track involves at least one object, the coordinate data is data for the at least one object. The attribute data of at least one object in the animated image includes at least one of the following: the type of at least one object, the size of at least one object, the color of at least one object, and the shape of at least one object. Exemplarily, if at least one object includes a sphere and a cube, the attribute data of at least one object in the animated image includes type data (sphere and cube), the size of the sphere and the cube, the color of the sphere and the cube, and other data. The position data of at least one object in the animated image is used to indicate the position of at least one object in the animated image. For example, if at least one object includes a sphere, the position data of at least one object in the animated image is: the sphere is on the left side in the animated image. Optionally, if the animated image includes multiple objects, the position data of at least one object in the animated image is used to indicate the positions of the multiple objects in the animated image and the relative positions and relative angles of the multiple objects in the animated image. For example, if at least one object includes a sphere and a cube, the position data of at least one object in the animated image is: the sphere is on the left side of the animated image, the cube is on the right side of the animated image, and the sphere and the cube are placed in parallel. The background data in the animated image is used to indicate the background features in the animated image. For example, if the background of the animated image is the seaside, the background data in the animated image includes feature data such as the location of the seaside, the weather at the seaside, and the environment at the seaside.

[0107] The camera is used to shoot the animation image, and the camera position data is used to indicate the position of the camera in the three-dimensional space to shoot the animation image. The camera orientation data is used to indicate the shooting angle of the camera relative to the animation image, and can also be interpreted as the relative direction data of the camera relative to at least one object in the animation image. The camera field of view is used to indicate the shooting range of the camera, and can also be interpreted as the visible range of the animation image.

[0108] According to the coordinate data and camera data corresponding to each description text, the animation image corresponding to the description text can be constructed, so that according to the coordinate data and camera data, a three-dimensional animation scene can be rendered to generate an output animation, thereby ensuring the accuracy of the output animation.

[0109] The specific process of scene analysis model generating scene data is as follows: First, the input description text is parsed and understood. The description text is usually used to describe at least one object, position, relative relationship, etc. in the scene. For example, the description text is "a red sphere is in the center of the table, there is a green cube on the left, and the background is a blue sky". Then the description text is parsed to understand the entities, position relationships, object attributes, etc., mainly including: extracting the attribute data of at least one object in the scene (such as sphere, cube, color, size, etc.), and analyzing the position data of at least one object, such as the relative position ("left", "center") and relative angle between at least one object, and parsing the background data (such as sky, ground, etc.). Secondly, the three-dimensional coordinates of at least one object in the scene are calculated, and the position and spatial relationship in the description text are converted into three-dimensional coordinates. The Cartesian coordinate system is usually used, where (x, y, z) represents the horizontal direction, vertical direction and depth direction respectively. Based on the described spatial relationship, the relative coordinates between different objects are calculated. For example, "the sphere is in the center of the table, and there is a cube on the left" may be converted into the relative position of the sphere and the cube in three-dimensional space. For example, "the center of the table" may be mapped to a fixed coordinate point (e.g., (0,0,0)), and the "left side" can be calculated relative to the fixed coordinate point. The sphere position can be (0,0,0), and the cube position can be (-2,0,0). Then, the camera data is calculated to calculate the camera's position, orientation, and field of view according to the viewing angle information in the description text (if any). The camera position is used to define the position of the viewing camera in three-dimensional space. For example, the description text "from a distance" may correspond to a large z value (far away from the object). The orientation data is used to determine the camera's orientation (the direction of the target object), which can be obtained by calculating the relative direction between the center position of the target object and the camera. The field of view is used to calculate the field of view (FOV) to determine the camera's visible range based on the description in the text (e.g., "distant view" or "close view"). For example, if the description text mentions "the camera is viewed from the front", the camera position may be (0,5,10) and the orientation is pointing to an object in the scene. Finally, the scene analysis model outputs coordinate data and camera data. The coordinate data converts the position, size, direction, etc. of at least one object into three-dimensional coordinates and outputs them, for example: sphere position (0,0,0), cube position (-2,0,0), ground position (0,-1,0). Camera data may include: camera position (0,5,10), camera orientation (0,0,0) (i.e. pointing to the center of the scene), and field of view angle of 60 degrees (based on descriptive text, such as "distant view").

[0110] In some embodiments, the animation trajectory includes at least one trajectory parameter, and each trajectory parameter is used to indicate the position of the object described by the input audio in the animation image corresponding to the trajectory parameter. Step 240 includes at least one of sub-steps 241-242.

[0111] Sub-step 241 : sequentially matching at least one trajectory parameter in the animation trajectory and at least one animation image in the animation sequence through an animation mapping model to obtain at least one set of animation mapping parameters.

[0112] Animation mapping is a matching process, in which at least one trajectory parameter in the animation trajectory and at least one animation image in the animation sequence are matched in sequence to form a one-to-one mapping relationship. Each set of animation mapping parameters includes a trajectory parameter and an animation image. Animation mapping parameters are renderable parameters, and their parameter format is a renderable parameter format.

[0113] If the number of trajectory parameters is less than the number of animation images, or the number of animation images is less than the number of trajectory parameters, the last data of the smaller party is recycled for matching. Exemplarily, at least one trajectory parameter includes trajectory 1, trajectory 2, and trajectory 3, and at least one animation image includes image 1 and image 2, then at least one set of animation mapping parameters generated is: trajectory 1 and image 1, trajectory 2 and image 2, and trajectory 3 and image 2.

[0114] Sub-step 242, generating an output animation according to the scene data and at least one set of animation mapping parameters by an image rendering tool.

[0115] The image rendering tool here can be any publicly available image rendering tool, and this application does not limit this.

[0116] At least one sub-scene data in the scene data corresponds to at least one set of animation mapping parameters. The scene data and the at least one set of animation mapping parameters are input into an image rendering tool to generate an output animation.

[0117] By performing animation mapping on the trajectory parameters and the animation image, renderable parameters are generated, and then the output animation is generated according to the scene data and the animation mapping parameters, so that the generated output animation can achieve a perfect match between the scene content and the animation effect, thereby improving the rendering effect of the output animation.

[0118] Figure 3The schematic diagram of the complete process of audio visualization is shown. First, the input audio is obtained, and the input audio includes multiple audios of network audio, local audio and real-time collected audio. Mixing processing is performed on the multiple audios to obtain mixed audio. For the mixed audio, on the one hand, the audio is separated by a multi-track separation model to obtain instrument tracks, vocal tracks and other tracks, and then the instrument tracks and vocal tracks are analyzed by the track analysis model to obtain instrument data corresponding to the instrument tracks and character data corresponding to the vocal tracks. Then, the track extraction model is used to obtain the animation track corresponding to the audio according to the instrument data and character data. On the other hand, the audio understanding model is used to understand the mixed audio to obtain the corresponding audio description text, and the semantic analysis model is used to perform semantic analysis on the audio description text to obtain at least one description text corresponding to the audio description text. Then, the diffusion model is used to generate animation images corresponding to at least one description text according to at least one description text to obtain an animation sequence, and the scene analysis model is used to perform scene analysis on at least one description text to obtain scene data. After obtaining the animation trajectory and the animation sequence, at least one trajectory parameter in the animation trajectory and at least one animation image in the animation sequence are matched in sequence through the animation mapping model to obtain at least one set of animation mapping parameters. Then, an image rendering tool is used to generate an output animation based on the scene data and at least one set of animation mapping parameters to complete the animation rendering.

[0119] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0120] Please refer to Figure 4 , which shows a block diagram of an audio visualization device provided by an embodiment of the present application. The device has the function of implementing the above-mentioned audio visualization method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the terminal device introduced above, or it can be set in the terminal device. Figure 4 As shown, the device 400 may include: an audio acquisition module 410 , a trajectory determination module 420 , a data determination module 430 and an animation generation module 440 .

[0121] The audio acquisition module 410 is used to acquire input audio.

[0122] The trajectory determination module 420 is used to obtain an animation trajectory corresponding to the input audio according to the input audio, where the animation trajectory is a motion trajectory of the object described by the input audio in the animation to be generated.

[0123] The data determination module 430 is used to obtain an animation sequence corresponding to the input audio and scene data corresponding to the input audio based on the input audio, wherein the animation sequence includes at least one animation image, each animation image is used to indicate a part of the audio meaning in the input audio, and the scene data is used to indicate the scene content in the at least one animation image and the display viewing angle of the at least one animation image.

[0124] The animation generation module 440 is used to generate an output animation corresponding to the input audio according to the animation trajectory, the animation sequence and the scene data.

[0125] In some embodiments, the trajectory determination module 420 is used to:

[0126] Separating the input audio tracks by a multi-track separation model to obtain instrument tracks and vocal tracks in the input audio;

[0127] Performing track analysis on the instrument track in the input audio through a track analysis model to obtain instrument data corresponding to the instrument track, wherein the instrument data is used to indicate the audio style and audio attributes in the instrument track;

[0128] Performing character analysis on the vocal track in the input audio by using the audio track analysis model to obtain character data corresponding to the vocal track, wherein the character data is used to indicate the character attributes in the vocal track;

[0129] The animation trajectory corresponding to the input audio is obtained according to the instrument data and the character data through a trajectory extraction model.

[0130] In some embodiments, the training process of the trajectory extraction model is as follows:

[0131] Acquire at least one training sample of a large language model, each training sample comprising sample data and sample trajectory, wherein the sample data comprises sample instrument data and sample character data;

[0132] Generate a prediction trajectory according to the sample instrument data and the sample person data by using the large language model;

[0133] According to the difference between the predicted trajectory and the sample trajectory, the parameters of the large language model are adjusted to obtain the trajectory extraction model.

[0134] In some embodiments, the data determination module 430 is used to:

[0135] Obtaining an audio description text corresponding to the input audio according to the input audio through an audio understanding model;

[0136] Performing semantic analysis on the audio description text by using a semantic analysis model to obtain at least one description text corresponding to the audio description text;

[0137] Generate the animation images corresponding to the at least one description text respectively according to the at least one description text by using a diffusion model to obtain the animation sequence;

[0138] The scene analysis model is used to perform scene analysis on the at least one description text to obtain scene data corresponding to the input audio.

[0139] In some embodiments, the scene data corresponding to the input audio includes coordinate data corresponding to the at least one description text and camera data corresponding to the at least one description text; the data determination module 430 is used to:

[0140] For each description text, obtaining coordinate data corresponding to the description text and camera data corresponding to the description text according to the description text through the scene analysis model;

[0141] The coordinate data is used to indicate the scene content in the animation image corresponding to the description text, and the camera data is used to indicate the display viewing angle of the animation image corresponding to the description text.

[0142] In some embodiments, the coordinate data includes at least one of the following: attribute data of at least one object in the animated image, position data of at least one object in the animated image, background data in the animated image;

[0143] The camera data includes at least one of the following: camera position data, camera orientation data, and camera field of view angle; the camera is used to shoot the animation image.

[0144] In some embodiments, the animation trajectory includes at least one trajectory parameter, each trajectory parameter is used to indicate the position of the object described by the input audio in the animation image corresponding to the trajectory parameter; the animation generation module 440 is used to:

[0145] Matching at least one trajectory parameter in the animation trajectory and at least one animation image in the animation sequence in sequence through an animation mapping model to obtain at least one set of animation mapping parameters;

[0146] The output animation is generated by an image rendering tool according to the scene data and the at least one set of animation mapping parameters.

[0147] In some embodiments, the audio acquisition module 410 is used to:

[0148] Acquire at least two audio data, wherein the at least two audio data include at least one of the following: network audio data, local audio data, and audio data collected in real time;

[0149] A mixing process is performed on the at least two audio data to obtain the input audio.

[0150] In some embodiments, the audio acquisition module 410 is used to:

[0151] Performing preprocessing on the at least two audio data to obtain at least two preprocessed audio data, wherein the preprocessing includes at least one of the following: sampling rate conversion processing, channel adjustment processing, and audio format unification processing;

[0152] Performing gain adjustment processing on the at least two pre-processed audio data respectively to obtain at least two audio data after volume adjustment;

[0153] respectively setting channel positions for the at least two audio data after the volume adjustment to obtain at least two audio data after the channel setting;

[0154] Performing effect processing on the at least two audio data after the channel setting respectively to obtain at least two audio data after effect processing, wherein the effect processing includes at least one of the following: sound effect adding processing, filtering processing, and compression processing;

[0155] According to the playing time of the at least two audio data after the effect processing, the at least two audio data after the effect processing are subjected to track superposition to obtain the input audio.

[0156] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0157] Please refer to Figure 5 , which shows a block diagram of a computer device 500 provided in one embodiment of the present application. The computer device 500 may be any electronic device with data calculation, processing and storage functions. The computer device 500 may be used to implement the audio visualization method provided in the above embodiment.

[0158] Typically, the computer device 500 includes a processor 501 and a memory 502 .

[0159] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0160] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned audio visualization method.

[0161] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the computer device 500, and the computer device 500 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0162] In an illustrative embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and the computer program implements the above-mentioned audio visualization method when executed by a processor of a computer device. Optionally, the above-mentioned computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0163] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above-mentioned audio visualization method.

[0164] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0165] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An audio visualization method, characterized in that: The method comprises: Get input audio; According to the input audio, an animation trajectory corresponding to the input audio is obtained, wherein the animation trajectory is a motion trajectory of an object described by the input audio in an animation to be generated; According to the input audio, an animation sequence corresponding to the input audio and scene data corresponding to the input audio are obtained, wherein the animation sequence includes at least one animation image, each animation image is used to indicate a meaning of a portion of the audio in the input audio, and the scene data is used to indicate scene content in the at least one animation image and a display viewing angle of the at least one animation image; An output animation corresponding to the input audio is generated according to the animation trajectory, the animation sequence and the scene data.

2. The method according to claim 1, characterized in that The step of obtaining an animation track corresponding to the input audio according to the input audio includes: Separating the input audio tracks by a multi-track separation model to obtain instrument tracks and vocal tracks in the input audio; Performing track analysis on the instrument track in the input audio through a track analysis model to obtain instrument data corresponding to the instrument track, wherein the instrument data is used to indicate the audio style and audio attributes in the instrument track; Performing character analysis on the vocal track in the input audio by using the audio track analysis model to obtain character data corresponding to the vocal track, wherein the character data is used to indicate the character attributes in the vocal track; The animation trajectory corresponding to the input audio is obtained according to the instrument data and the character data through a trajectory extraction model.

3. The method according to claim 2, characterized in that The training process of the trajectory extraction model is as follows: Acquire at least one training sample of a large language model, each training sample comprising sample data and sample trajectory, wherein the sample data comprises sample instrument data and sample character data; Generate a prediction trajectory according to the sample instrument data and the sample person data by using the large language model; According to the difference between the predicted trajectory and the sample trajectory, the parameters of the large language model are adjusted to obtain the trajectory extraction model.

4. The method according to claim 1, characterized in that The step of obtaining, according to the input audio, an animation sequence corresponding to the input audio and scene data corresponding to the input audio comprises: Obtaining an audio description text corresponding to the input audio according to the input audio through an audio understanding model; Performing semantic analysis on the audio description text by using a semantic analysis model to obtain at least one description text corresponding to the audio description text; Generate the animation images corresponding to the at least one description text respectively according to the at least one description text by using a diffusion model to obtain the animation sequence; The scene analysis model is used to perform scene analysis on the at least one description text to obtain scene data corresponding to the input audio.

5. The method according to claim 4, characterized in that The scene data corresponding to the input audio includes coordinate data corresponding to the at least one description text and camera data corresponding to the at least one description text; The performing scene analysis on the at least one description text respectively by using a scene analysis model to obtain scene data corresponding to the input audio includes: For each description text, obtaining coordinate data corresponding to the description text and camera data corresponding to the description text according to the description text through the scene analysis model; The coordinate data is used to indicate the scene content in the animation image corresponding to the description text, and the camera data is used to indicate the display viewing angle of the animation image corresponding to the description text.

6. The method according to claim 5, characterized in that The coordinate data includes at least one of the following: attribute data of at least one object in the animation image, position data of at least one object in the animation image, and background data in the animation image; The camera data includes at least one of the following: camera position data, camera orientation data, and camera field of view angle; the camera is used to shoot the animation image.

7. The method according to claim 1, characterized in that The animation trajectory includes at least one trajectory parameter, each trajectory parameter is used to indicate the position of the object described by the input audio in the animation image corresponding to the trajectory parameter; The step of generating an output animation corresponding to the input audio according to the animation trajectory, the animation sequence and the scene data comprises: sequentially matching at least one trajectory parameter in the animation trajectory and at least one animation image in the animation sequence through an animation mapping model to obtain at least one set of animation mapping parameters; The output animation is generated by an image rendering tool according to the scene data and the at least one set of animation mapping parameters.

8. The method according to claim 1, characterized in that The step of obtaining the input audio comprises: Acquire at least two audio data, wherein the at least two audio data include at least one of the following: network audio data, local audio data, and audio data collected in real time; A mixing process is performed on the at least two audio data to obtain the input audio.

9. The method according to claim 8, characterized in that The performing mixing processing on the at least two audio data to obtain the input audio includes: Performing preprocessing on the at least two audio data to obtain at least two preprocessed audio data, wherein the preprocessing includes at least one of the following: sampling rate conversion processing, channel adjustment processing, and audio format unification processing; Performing gain adjustment processing on the at least two pre-processed audio data respectively to obtain at least two audio data after volume adjustment; respectively setting channel positions for the at least two audio data after the volume adjustment, to obtain at least two audio data after the channel setting; Performing effect processing on the at least two audio data after the channel setting respectively to obtain at least two audio data after effect processing, wherein the effect processing includes at least one of the following: sound effect adding processing, filtering processing, and compression processing; According to the playing time of the at least two audio data after the effect processing, the at least two audio data after the effect processing are subjected to track superposition to obtain the input audio.

10. An audio visualization device, characterized in that: The device comprises: An audio acquisition module, used to acquire input audio; A trajectory determination module, used to obtain an animation trajectory corresponding to the input audio according to the input audio, wherein the animation trajectory is a motion trajectory of the object described by the input audio in the animation to be generated; a data determination module, configured to obtain, according to the input audio, an animation sequence corresponding to the input audio and scene data corresponding to the input audio, wherein the animation sequence includes at least one animation image, each animation image is used to indicate a meaning of a portion of the audio in the input audio, and the scene data is used to indicate a scene content in the at least one animation image and a display viewing angle of the at least one animation image; The animation generation module is used to generate an output animation corresponding to the input audio according to the animation trajectory, the animation sequence and the scene data.

11. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the audio visualization method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the audio visualization method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program, which is loaded and executed by a processor to implement the audio visualization method according to any one of claims 1 to 9.

Images