Audio special effect rendering method and device, electronic equipment and storage medium

Abstract

This application provides an audio effects rendering method, apparatus, electronic device, and storage medium. The method includes: acquiring a time-frequency map of a target audio file, the time-frequency map indicating the energy distribution of each audio frame in the frequency domain; sampling the time-frequency map to obtain the energy of multiple frequencies in each audio frame; determining the amplitude values ​​of multiple frequencies in each audio frame based on their energy; and performing effects rendering on a preset rendering model corresponding to the target audio file based on the amplitude values ​​of the multiple frequencies in each audio frame. This solution performs effects rendering on a preset rendering model corresponding to the target audio file using a time-frequency map, eliminating the need for real-time audio parsing, reducing processing overhead, improving the performance of audio visualization effects, and avoiding interference from other sound effects by only acquiring the time-frequency map of the target audio file, thus improving the accuracy of effects rendering.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an audio effects rendering method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of game technology, audio-driven animation effects are often created in games to complement the audio playback effects in the game. Audio-driven effects can match the auditory and visual stimuli of game users to achieve a synesthetic sensory experience.

[0003] Currently, game engines typically use audio analysis modules to analyze audio files to obtain the amplitude of each frequency in the audio file, and then apply the amplitude of each frequency to special effects to generate dynamic effects.

[0004] However, the aforementioned audio analysis module performs real-time analysis, and dynamic effects are rendered in real-time, resulting in high computational costs. Furthermore, when reading audio files, it may also read other sound effects that occur during game runtime, making it impossible to read only audio files. Summary of the Invention

[0005] In view of this, embodiments of this application provide an audio effects rendering method, apparatus, electronic device, and storage medium to reduce the computational cost of real-time audio file parsing, and to obtain only the time-frequency diagram of the target audio file, avoiding interference from other sound effects and improving the accuracy of effects rendering.

[0006] In a first aspect, embodiments of this application provide an audio effects rendering method, including:

[0007] Obtain a time-frequency map of the target audio file, wherein the time-frequency map is used to indicate the energy distribution of each audio frame of the target audio file in the frequency domain;

[0008] The time-frequency map is sampled to obtain the energy of multiple frequencies in each audio frame;

[0009] The amplitude values ​​of multiple frequencies in each audio frame are determined based on the energy of multiple frequencies in each audio frame.

[0010] Based on the amplitude values ​​of multiple frequencies in each audio frame, special effects rendering is performed on the preset rendering model corresponding to the target audio file.

[0011] Secondly, embodiments of this application also provide an audio effects rendering apparatus, including:

[0012] The acquisition module is used to acquire a time-frequency map of a target audio file, wherein the time-frequency map is used to indicate the energy distribution of each audio frame of the target audio file in the frequency domain.

[0013] The sampling module is used to sample the time-frequency map to obtain the energy of multiple frequencies in each audio frame;

[0014] The determining module is used to determine the amplitude values ​​of multiple frequencies in each audio frame based on the energy of multiple frequencies in each audio frame;

[0015] The rendering module is used to perform special effects rendering on the preset rendering model corresponding to the target audio file based on the amplitude values ​​of multiple frequencies in each audio frame.

[0016] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the method described in any of the first aspects.

[0017] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method described in any of the first aspects.

[0018] This application provides an audio effects rendering method, apparatus, electronic device, and storage medium. The method includes: acquiring a time-frequency map of a target audio file, the time-frequency map indicating the energy distribution of each audio frame in the frequency domain; sampling the time-frequency map to obtain the energy of multiple frequencies in each audio frame; determining the amplitude values ​​of multiple frequencies in each audio frame based on their energy; and performing effects rendering on a preset rendering model corresponding to the target audio file based on the amplitude values ​​of the multiple frequencies in each audio frame. This solution performs effects rendering on a preset rendering model corresponding to the target audio file using a time-frequency map, eliminating the need for real-time audio parsing, reducing processing overhead, improving the performance of audio visualization effects, and avoiding interference from other sound effects by only acquiring the time-frequency map of the target audio file, thus improving the accuracy of effects rendering. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the audio effects rendering method provided in this application embodiment;

[0021] Figure 2 This is a schematic diagram of a time-frequency texture provided in an embodiment of this application;

[0022] Figure 3 Another flowchart illustrating the audio effects rendering method provided in this application embodiment;

[0023] Figure 4 A schematic diagram of the special effects rendering effect of an audio frame provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the special effects rendering effect of another audio frame provided in an embodiment of this application;

[0025] Figure 6 A flowchart illustrating the audio effects rendering method provided in this application embodiment;

[0026] Figure 7 This is a schematic diagram of the special effects rendering effect of another audio frame provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the special effects rendering effect of another audio frame provided in an embodiment of this application;

[0028] Figure 9 A flowchart illustrating another audio effects rendering method provided in this application embodiment;

[0029] Figure 10 A flowchart illustrating another audio effects rendering method provided in this application embodiment;

[0030] Figure 11 A flowchart illustrating another audio effects rendering method provided in this application embodiment;

[0031] Figure 12 This is a schematic diagram of the audio effects rendering device provided in the embodiments of this application;

[0032] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0034] To address the current approach to creating dynamic audio effects, which involves real-time analysis and rendering, resulting in high computational costs and the potential for reading other game sound effects while reading audio files, thus preventing the selection of only the audio files to be analyzed, this application proposes a method for offline generation of time-frequency maps and their application in game engines to create audio effects. This method satisfies the variability requirements of real-time rendering, eliminates the need for real-time audio parsing, reduces processing overhead, and improves the performance of audio visualization effects. Furthermore, this method is a general audio processing approach that allows modification of audio files to create different audio effects.

[0035] In one embodiment of this application, the audio effects rendering method can run on a local terminal device or a server. When the audio effects rendering method runs on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0036] In an optional implementation, various cloud applications, such as cloud gaming, can run under the cloud interaction system. Taking cloud gaming as an example, cloud gaming refers to a gaming method based on cloud computing. In the cloud gaming operating mode, the game program and the game screen presentation are separated. The storage and execution of audio effects rendering methods are completed on the cloud gaming server. The client device is used for data reception, transmission, and game screen presentation. For example, the client device can be a display device with data transmission capabilities located close to the user, such as a mobile terminal, television, computer, or PDA; however, the information processing is performed by the cloud gaming server in the cloud. When playing the game, the player operates the client device to send operation commands to the cloud gaming server. The cloud gaming server runs the game according to the operation commands, encodes and compresses game screen data, returns it to the client device via the network, and finally, the client device decodes and outputs the game screen.

[0037] In an optional implementation, taking a game as an example, the local terminal device stores the game program and is used to display the game screen. The local terminal device is used to interact with the player through a graphical user interface (GUI), i.e., conventionally by downloading, installing, and running the game program via an electronic device. The local terminal device can provide the GUI to the player in various ways, such as rendering it on the terminal's display screen or providing it to the player via holographic projection. For example, the local terminal device can include a display screen for displaying the GUI, which includes game screens, and a processor for running the game, generating the GUI, and controlling the display of the GUI on the display screen.

[0038] In one possible implementation, this invention provides an audio effects rendering method that provides a graphical user interface through a terminal device. The terminal device can be either the aforementioned local terminal device or a client device in the aforementioned cloud interaction system.

[0039] Figure 1 This is a flowchart illustrating the audio effects rendering method provided in this application embodiment. The execution subject of this embodiment can be an electronic device, such as a terminal device or a server.

[0040] like Figure 1 As shown, the method may include:

[0041] S101. Obtain the time-frequency texture of the target audio file.

[0042] The time-frequency map is used to indicate the energy distribution of each audio frame in the frequency domain of the target audio file. The target audio file can be any audio file, and the target audio file has multiple audio frames. Each audio frame is a specific time segment of the target audio file. The energy of each audio frame in the frequency domain corresponds to the amplitude value of each audio frame in the frequency domain. The larger the amplitude value, the greater the energy, and the smaller the amplitude value, the smaller the energy.

[0043] Figure 2 This is a schematic diagram of the time-frequency mapping provided in the embodiments of this application, as shown below. Figure 2 As shown, when viewing the time-frequency map from the UV coordinate system, the value of U represents frequency, the value of V represents time, the brightness represents the energy value of frequency U at time V, and the brightness of a row represents the energy distribution of the audio in the frequency domain recorded in the time-frequency map at time V.

[0044] S102. Sample the time-frequency map to obtain the energy of multiple frequencies in each audio frame.

[0045] S103. Determine the amplitude values ​​of multiple frequencies in each audio frame based on the energy of multiple frequencies in each audio frame.

[0046] Each audio frame contains multiple frequencies. The time-frequency map is sampled to obtain the energy of multiple frequencies in each audio frame, and the amplitude value of multiple frequencies is determined based on the energy of multiple frequencies.

[0047] In some embodiments, the game engine creates material functions for an audio effects system (such as the Niagara system) based on the total number of audio frames (FrameCounts), the frame rate exported from the default texture (Frames Per Second, FPS), and multiple frequencies in the audio frames. These material functions are then used to sample the time-frequency map to obtain the amplitude values ​​of multiple frequencies in each audio frame. These multiple frequencies in the audio frame can be normalized frequencies (SampleNormalizedIndex).

[0048] S104. Based on the amplitude values ​​of multiple frequencies in each audio frame, perform special effects rendering on the preset rendering model corresponding to the target audio file.

[0049] Based on the amplitude values ​​of multiple frequencies in each audio frame, special effects rendering is performed on the preset rendering model corresponding to the target audio file to obtain the rendering model of each audio frame. In other words, for each audio frame, the amplitude values ​​of multiple frequencies in each audio frame are different, and the special effects of the preset rendering model are different. The special effects of the preset rendering model can include static special effects or floating special effects. Static special effects, for example, can control the preset rendering model to present brightness and color that match the amplitude values ​​of multiple frequencies in each audio frame. Floating special effects, for example, can control the preset animation model to float in a way that matches the amplitude values ​​of multiple frequencies in each audio frame. Thus, while playing the target audio file, the preset rendering model can be controlled to change according to the amplitude values ​​of multiple frequencies in each audio frame of the target audio file, so as to match the auditory and visual stimuli of the game user.

[0050] It is worth noting that steps S101-S104 are the application process of audio effects rendering in the game engine.

[0051] In this embodiment, the method of generating time-frequency maps offline and applying them to the game engine to create audio effects meets the variability requirements of real-time rendering, and does not require real-time audio parsing, thus reducing processing overhead and improving the performance of audio visualization effects. Furthermore, this method is a general audio processing method that can modify audio files to create different audio effects.

[0052] In an optional implementation, step S104, based on the amplitude values ​​of multiple frequencies in each audio frame, performs special effects rendering on the preset rendering model corresponding to the target audio file, including:

[0053] Based on the amplitude values ​​of multiple frequencies in each audio frame, the preset animation model corresponding to the target audio file is rendered to obtain a floating effect that matches the preset animation model with the rhythm of the amplitude values.

[0054] For each audio frame, based on the amplitude values ​​of multiple frequencies in each audio frame, the preset animation model corresponding to the target audio file is rendered with special effects so that the preset animation model presents a floating effect that matches the rhythm of the amplitude values. In other words, based on the amplitude values ​​of multiple frequencies, the preset animation model is controlled to present dynamic changes, and the dynamic change method matches the amplitude values.

[0055] Figure 3 Another flowchart illustrating the audio effects rendering method provided in this application embodiment is as follows: Figure 3 As shown, in an optional implementation, based on the amplitude values ​​of multiple frequencies in each audio frame, a preset animation model effect corresponding to the target audio file is rendered to obtain a floating effect where the preset animation model and the amplitude values ​​rhythmically match. This may include:

[0056] S201. Map multiple frequencies in each audio frame to the edge vertices of the preset rendering model.

[0057] Based on the frequency unfolding order, multiple frequencies in each audio frame are mapped to the edge vertices of the preset rendering model to obtain the mapping relationship between frequencies and edge vertices. The frequency unfolding order can be, for example, from largest to smallest or from smallest to largest. The preset rendering model has multiple edge vertices, and multiple frequencies in each audio frame are displayed on the edge vertices respectively.

[0058] In some embodiments, the preset animation model can be a toroidal model, which is unfolded along the direction of increasing U according to the UV unfolding method, where U represents frequency. This allows the mapping relationship between frequency and edge vertices to be obtained, thereby enabling the display of all frequency information through the toroidal model.

[0059] S202. Based on the amplitude values ​​of multiple frequencies in each audio frame, perform floating effect rendering on the edge vertices of the preset rendering model.

[0060] Based on the amplitude values ​​of multiple frequencies in each audio frame, floating effects are applied to the edge vertices of the preset rendering model corresponding to the frequencies, so that the edge vertices of the preset rendering model present dynamic effects corresponding to the amplitude values ​​of the corresponding frequencies. For example, the dynamic effect can be a blinking effect of the edge vertices. For example, the larger the amplitude value, the higher the blinking frequency, and the smaller the amplitude value, the lower the blinking frequency.

[0061] In this embodiment, floating effects are rendered on the edge vertices of the preset rendering model according to the amplitude value of the frequency, so that the edge vertices of the preset rendering model present rich dynamic effects and improve the visual effect for game users.

[0062] In an optional implementation, step S202, rendering edge vertex effects of a preset rendering model based on the amplitude values ​​of multiple frequencies in each audio frame, may include:

[0063] Based on the amplitude values ​​of multiple frequencies in each audio frame, the edge vertices of the preset rendering model are controlled and rendered to scale according to the amplitude values.

[0064] Based on the amplitude values ​​of multiple frequencies in each audio frame, the edge vertices of the preset rendering model are controlled and rendered so that the edge vertices expand and contract according to the amplitude values. The expansion and contraction of the edge vertices can be understood as the extension length of the preset animation model to the inside and outside based on the amplitude value of the edge vertices. For example, the larger the amplitude value, the longer the extension length to the inside and outside, and the smaller the amplitude value, the shorter the extension length to the inside and outside.

[0065] Figure 4 This is a schematic diagram of the special effects rendering effect of an audio frame provided in an embodiment of this application. Figure 5 This is an illustration of the special effects rendering effect of another audio frame provided in an embodiment of this application, such as... Figure 4 , Figure 5 As shown, the preset animation model is a circular model. The edge of the circular model expands in the direction of increasing U. Based on the amplitude values ​​of multiple frequencies in the audio frame, the edge vertices of the preset animation model are controlled and rendered so that the edge vertices expand and contract in different ways. It can be seen that because the amplitude values ​​of multiple frequencies in two audio frames are different, the expansion and contraction of the edge vertices are different. This creates an audio visualization effect that changes with the frequencies in each audio frame when the target audio file is played continuously.

[0066] In this embodiment, the edge vertices of the preset rendering model are stretched and resized according to the amplitude value of the frequency, so that the edge vertices of the preset rendering model present rich dynamic effects and improve the visual effect for game users.

[0067] Figure 6 The flowchart of the audio effects rendering method provided in the embodiments of this application is as follows: Figure 6 As shown, in an optional implementation, based on the amplitude values ​​of multiple frequencies in each audio frame, a preset animation model effect corresponding to the target audio file is rendered to obtain a floating effect where the preset animation model and the amplitude values ​​rhythmically match. This may include:

[0068] S301. Based on the amplitude values ​​of multiple frequencies in each audio frame, obtain the motion velocity of particles in multiple preset particle models.

[0069] The preset animation model includes: a preset particle model. Based on the amplitude values ​​of multiple frequencies in each audio frame, the motion speed of multiple preset particle models is obtained. There is a mapping relationship between the preset particle model and the frequency. For example, one preset particle model corresponds to one frequency. Similarly, there is a mapping relationship between the amplitude value of the frequency and the motion speed of the particle model. For example, the larger the amplitude value, the greater the motion speed of the particle model, and the smaller the amplitude value, the smaller the motion speed of the particle model.

[0070] In some embodiments, amplitude values ​​can be pre-stored as variables in the audio effects system, for example, in amplitude parameters. When rendering model effects for each audio frame, the amplitude values ​​of the frequencies in each audio frame are read from the amplitude parameters.

[0071] S302. Render the multiple preset particle models according to their movement speeds to control their movement.

[0072] Based on the movement speed of multiple preset particle models, each preset particle model is rendered to control the movement of multiple preset particles so that the movement speed of each preset particle model matches the amplitude value of the corresponding frequency.

[0073] Figure 7 This is a schematic diagram of the special effects rendering effect of another audio frame provided in an embodiment of this application. Figure 8 This is a schematic diagram of the special effects rendering effect of another audio frame provided in an embodiment of this application. Figure 7 , Figure 8 As shown, a dot represents an example model. Based on the amplitude values ​​of multiple frequencies in the audio frame, the preset particle model is controlled and rendered to move. It can be seen that because the amplitude values ​​of multiple frequencies in the two audio frames are different, the particle movement is different.

[0074] In this embodiment, based on the amplitude value of the frequency, a preset particle model is controlled and rendered to move, resulting in particle motion changes that are linked to the frequency of each audio frame, thereby improving the visual effect for game users.

[0075] Figure 9 This is a flowchart illustrating another audio effects rendering method provided in an embodiment of this application, as follows: Figure 9 As shown, in an optional implementation, step S102, sampling the time-frequency map to obtain the energy of multiple frequencies in each audio frame, may include:

[0076] S401. Determine the starting audio frame from each audio frame according to the preset audio frame playback identifier.

[0077] The preset audio frame playback identifier is used as the identifier of the starting audio frame. Based on the preset audio frame playback identifier, the starting audio frame is determined from each audio frame.

[0078] S402. Sample the time-frequency map to obtain the energy of multiple frequencies in the starting audio frame and subsequent audio frames, so as to obtain the amplitude values ​​of multiple frequencies.

[0079] By sampling the time-frequency map, the energy of multiple frequencies in the starting audio frame and subsequent audio frames is obtained. In other words, by sampling the time-frequency map using a specified preset audio frame playback identifier, the amplitude values ​​of multiple frequencies in the starting audio frame and the energy of multiple frequencies in subsequent audio frames can be obtained. Then, based on the energy of multiple frequencies in the starting audio frame and subsequent audio frames, the amplitude values ​​of multiple frequencies in the starting audio frame and subsequent audio frames can be determined. This allows for the rendering of special effects on the preset rendering model corresponding to the target audio file, resulting in the rendering model for each audio frame.

[0080] In some embodiments, the value of a parameter (such as Dynamic MaterialParameter) in the audio effects system can be set to a normalized age value (TimeFromZero). The normalized age value is used to determine the playback identifier of a preset audio frame. The normalized age value is a normalized representation of the playback identifier of the preset audio frame. In this way, when rendering the model, it is possible to control the reading to start from the amplitude value of the frequency in the specified starting audio frame. As the effect playback ends, the amplitude values ​​of the frequency in the audio frames after the starting audio frame are read.

[0081] When setting parameters in the audio effects system, an interface creation function (such as DynamicParameter) can be used to create an interface that connects to the audio effects system. By inputting a normalized age value into the audio effects system, the starting audio frame can be specified, and the rendering can be performed according to the particle age growth during subsequent model rendering.

[0082] In this embodiment, by specifying a starting audio frame, the model can be rendered based on the amplitude values ​​of multiple frequencies in the specified audio frame according to actual needs, which improves the flexibility of model rendering.

[0083] Figure 10 This is a flowchart illustrating another audio effects rendering method provided in an embodiment of this application, as follows: Figure 10 As shown, in an optional implementation, step S101, obtaining the time-frequency map of the target audio file, may include:

[0084] S501. Obtain the audio data of each audio frame based on the target audio file.

[0085] S502. Perform spectral analysis on the audio data of each audio frame to generate a spectrum diagram of each audio frame.

[0086] The target audio file is parsed to obtain the audio data of each audio frame of the target audio file. An audio frame is a specific time segment of the target audio file, and the audio data of the audio frame is the audio data of the specific time segment in the target audio file.

[0087] In some embodiments, open texture creation software (such as Houdini), and through the visual interface, right-click under the texture creation node (such as obj Network) to create a new Geometry node. Double-click the node to enter its interior, and create a new child node (such as CHOP Network). The child node is used for audio data processing. Double-click to enter the child node, create a new File node, select the target audio file to be processed in the Channel File option, and create a new Spectrum node. Connect the input of the Spectrum node to the File node. In the Spectrum node, enter 0 in the Start value and 1 in the End value. The Spectrum node is used to analyze the audio spectrum, which includes changes in the frequency domain and time domain. 0 and 1 indicate that the analysis is performed from the lowest frequency to the highest frequency of the target audio file, that is, the entire frequency information of the target audio file is analyzed.

[0088] Within the Spectrum node, you can open the Motion FX View window on the left to view the spectral waveform of the target audio file being analyzed. When the target audio file is played, the spectral waveform in the Spectrum View window continuously shifts to the right, and the horizontal axis value of the waveform's starting point continuously increases. This means that as time progresses during playback, the spectral waveform of the previous audio frame shifts to the right. Based on this, a new Shift node is created to shift the waveform. This node is connected below the Spectrum node, with the Unit Value set to Absolute and the Reference parameter set to the default Start Position, keeping the Start parameter at its default value of 0. This shift node will then shift the waveform to the left over time, ensuring that the horizontal axis of the waveform's starting point remains at 1. The amount of leftward shift cancels out the amount of rightward shift before using the Shift node, thus setting the horizontal axis of the spectral waveform to frequency, eliminating time-domain variations, and generating the spectrogram for each audio frame.

[0089] S503. Create a spectrum segment based on multiple preset line segment sampling points.

[0090] Among them, the preset line segment sampling points can be pre-created sampling points, and the number of multiple preset line segment sampling points can be in the range of 200-400, such as 256. A spectrum line segment is created based on the multiple preset line segment sampling points, and the multiple preset line segment sampling points are sampling points on the spectrum line segment.

[0091] In some embodiments, return to the geometry node, right-click under the texture creation node (such as obj Network) to create a new line node, and set the number of sampling points to 200-400, for example, 256.

[0092] S504. Map the amplitude values ​​of multiple frequencies in the spectrum diagram to multiple preset sampling points in the spectrum line segments to generate the target spectrum diagram of each audio frame.

[0093] The spectrogram is used to characterize the relationship between frequency and amplitude. The amplitude values ​​of multiple frequencies in the spectrogram of each audio frame are mapped to multiple preset line segment sampling points in the spectrum. In this way, the multiple preset line segment sampling points have height attributes. It can be understood that the multiple preset line segment sampling points are pulled up according to the corresponding amplitude values ​​to generate the target spectrogram of each audio frame. In this case, the amplitude value of one frequency is mapped to one preset line segment sampling point.

[0094] In some embodiments, a new attribute create node is created, the group type is set to Points, a new height attribute is created, the height attribute is used to receive the amplitude value of the frequency in the spectrum, and a new channel node is created. The function of this node is to map the amplitude value of the frequency in the spectrum to the height attribute. The channel node can obtain the target spectrogram output by the child node in the geometry node. The node path to be mapped to the geometry node is filled in the child node parameter, that is, the last node that generates the target spectrogram, such as / obj / geo1 / chopnet1 / Trim. Here, the channel scope can be filled in with any channel, such as the left channel (chan0m). For the target spectrogram of each audio frame, there are left channel target spectrogram and right channel target spectrogram. The attribute scope is filled in with height. In this way, by reading the set parameters, the amplitude value of a selected audio channel can be mapped to the height attribute of the spectrum segment.

[0095] S505. Generate a time-frequency map based on the target spectrogram of each audio frame.

[0096] Export the target spectrogram of each audio frame to generate a time-frequency texture.

[0097] In some embodiments, the texture creation software can be opened, and the texture export node (out network) can be switched. In this node, the texture export method (Mode / Target Engine), texture format (such as PNG format), export location, and identifier of each audio frame can be set. Then, according to the target spectrogram of each audio frame, the texture export method is used to export the texture, generate a time-frequency texture in texture format, and export the time-frequency texture to the set export location.

[0098] In an optional implementation, before step S504, which maps the amplitude values ​​of multiple frequencies in the spectrogram to multiple preset sampling points in the spectral line segments to generate the target spectrogram for each audio frame, the method may further include:

[0099] The spectrum is trimmed based on the amplitude distribution in the spectrum.

[0100] Based on the amplitude distribution in the spectrogram of each audio frame, the spectrogram is trimmed to remove areas where the amplitude change is less than a preset range. Since the audio effects in areas where the amplitude change is less than the preset range are not obvious and are relatively flat with weak contrast, in order to ensure a certain audio effect contrast, areas where the amplitude change is less than the preset range in the spectrogram of each audio frame can be trimmed. Then, the amplitude values ​​of multiple frequencies in the trimmed spectrogram are mapped to multiple preset line segment sampling points in the spectrum line segment to generate the target spectrogram of each audio frame.

[0101] In some embodiments, a new Trim node is created. The purpose of this node is to determine the frequency range. The Trim node is visualized, and clicking on the spectrum display window shows the frequency range of the spectrum of each audio frame. If the amplitude changes of the highest and lowest frequencies are not obvious, the values ​​of the start and end points in the Trim node can be adjusted. For example, the value of the start point can be adjusted to 0.0011 and the value of the end point can be adjusted to -0.9951 to trim the latter half of the horizontal axis in the spectrum.

[0102] In an optional implementation, before step S504, which maps the amplitude values ​​of multiple frequencies in the spectrogram to multiple preset sampling points in the spectral line segments to generate the target spectrogram for each audio frame, the method may further include:

[0103] Multiple frequency sets are obtained by classifying the frequencies in the spectrum.

[0104] According to the preset adjustment rules of each frequency set, the amplitude value of the frequency in each frequency set is adjusted respectively.

[0105] Multiple frequencies in the spectrum are classified according to different frequency intensities to obtain multiple frequency sets. Each frequency set includes multiple frequencies of a certain frequency intensity, such as low frequency, mid frequency, and high frequency.

[0106] Each frequency set corresponds to a preset adjustment rule. Based on the preset adjustment rule of each frequency set, the amplitude value of the frequencies in each frequency set is adjusted. The amplitude value of low frequency is increased, and the amplitude value of mid-high frequency is decreased, so that the amplitude value is mapped from one range to another. Thus, the low frequency is in a state of rapid rise, and the mid-high frequency is in a state of stability, so that the overall height of the spectrum is more even, avoiding the difference between high and low points too large, and the details are rich. The spectrum before adjustment maintains a uniform rise, with details concentrated in the low frequency and dense changes, but the high frequency is very sharp and the changes are too violent, which is not suitable for special effects.

[0107] In some embodiments, a new attribute remap node is created. By adjusting the remap slider in the attribute remap node, the amplitude values ​​of frequencies in each frequency set are adaptively adjusted to adjust the spectrum to a shape where the first part (i.e., low frequency) rises rapidly and the second part (mid-high frequency) is relatively stable. The original name is filled with the height, and the new name is filled with the mapped height (height_remap). The mapped height is the new attribute name after mapping.

[0108] In this embodiment, the amplitude values ​​are remapped according to the characteristics of different audio frequencies to create a greater difference in amplitude variations. By adjusting the amplitude of frequencies with different intensities accordingly, the generated audio effects are improved.

[0109] In an optional implementation, before step S504, which maps the amplitude values ​​of multiple frequencies in the spectrogram to multiple preset sampling points in the spectral line segments to generate the target spectrogram for each audio frame, the method may further include:

[0110] The amplitude values ​​of multiple frequencies in the spectrum are normalized.

[0111] Since the maximum brightness of the texture corresponds to an amplitude value of 1, in order to avoid information loss when generating the texture due to excessively large amplitude values, the amplitude values ​​of multiple frequencies in the spectrum are normalized to keep the amplitude values ​​within the range of 0-1. This way, the maximum amplitude value will not exceed 1 when exporting the texture, and therefore there will be no information loss.

[0112] In some embodiments, in the attribute remapping node, the maximum input value (Input Max) is filled with the maximum amplitude value, such as 2, in order to reduce the height of the line segment from 2 to 1 in the maximum output value (Output Max).

[0113] It is worth noting that before generating the target spectrogram, at least one of the following operations can be performed on the spectrogram: trimming, adjusting the amplitude values ​​of frequencies in the spectrogram, and normalization. For specific implementation details, please refer to the relevant descriptions above.

[0114] In this embodiment, the time-frequency map is generated by baking, which avoids the overhead of real-time audio parsing, improves the performance of audio visualization effects, and allows for the selection of a target audio file for visualization and spectrum information preprocessing. This eliminates the need to process spectrum data during the real-time rendering stage, achieving the goal of audio visualization and avoiding real-time adjustment operations. Furthermore, the audio spectrum of the effects can be previewed in advance during the creation of the time-frequency map, allowing for corresponding preprocessing operations, enriching the use of map information, and reducing the overhead of real-time adjustments.

[0115] Figure 11 This is a flowchart illustrating another audio effects rendering method provided in an embodiment of this application, as follows: Figure 11 As shown, in an optional implementation, step S504, mapping the amplitude values ​​of multiple frequencies in the spectrum to multiple preset sampling points in the spectrum segments to generate the target spectrum for each audio frame, may include:

[0116] S601. Normalize multiple frequencies in the spectrum to obtain multiple normalized frequencies.

[0117] Normalize multiple frequencies in the spectrum to obtain multiple normalized frequencies. Each frequency corresponds to one normalized frequency. The frequency starts at 0 and ends at 1, and all frequencies are mapped to 0-1 to obtain the normalized frequency.

[0118] S602. Normalize the preset point numbers of multiple preset line segment sampling points to obtain multiple normalized point numbers.

[0119] Normalize the preset point numbers of multiple preset line segment sampling points to obtain multiple normalized point numbers corresponding to multiple preset line segment sampling points. One preset line segment sampling point corresponds to one normalized point number. The preset point number of the preset line segment sampling point can be divided by the number of multiple preset line segment sampling points to map the preset point number to 0-1, thus obtaining the normalized point number of the preset line segment sampling point.

[0120] It is worth noting that the preset point numbers of multiple preset line segment sampling points are distributed in a regular manner. The normalized point number of the point closest to the origin is 0, the normalized point number of the point farthest from the origin is 1, and the normalized point numbers of the points in the middle are arranged in the order of 0-1. Therefore, the normalized point numbers of the spectrum line segment increase from 0 to 1 from beginning to end.

[0121] In some embodiments, a new attribute expression node is created, and the position in the attribute parameter remains unchanged. This indicates that the coordinates of the preset segment sampling point in the spectrum segment will be manipulated. The expression parameter can be filled with self+@height_remap*set(0,1,0), which is to add the remapping height attribute carrying the amplitude value to the height of the preset segment sampling point.

[0122] Here, `self` represents the coordinates of the preset line segment sampling point, `@height_remap` is the remapped height attribute, and `set(0,1,0)` refers to a 3D vector of size (0,1,0), pointing upwards, with a length of 1. `@height_remap*set(0,1,0)` represents a 3D vector pointing upwards with a length of `@height_remap`. `self+@height_remap*set(0,1,0)` means adding the remapped height attribute to the height of the preset line segment sampling point.

[0123] S603. Based on multiple normalized frequencies and multiple normalized point numbers, map the amplitude values ​​of multiple frequencies in the spectrum diagram to the height attribute of the corresponding preset line segment sampling points to generate the target spectrum diagram of each audio frame.

[0124] Matching is performed based on multiple normalized frequencies and multiple normalized point numbers. The amplitude values ​​of multiple frequencies in the spectrum are mapped to the height attribute of the preset line segment sampling points corresponding to each frequency, generating the target spectrum of each audio frame. The target spectrum is a waveform diagram composed of the amplitude lines connecting each frequency in each audio frame.

[0125] The preset line segment sampling points corresponding to each frequency are the preset line segment sampling points corresponding to the normalized point numbers that are equal to the normalized frequency corresponding to each frequency. In other words, the value of a sampling point in the height attribute is equal to the amplitude value corresponding to the normalized frequency that is equal to the normalized point number of this sampling point.

[0126] In this embodiment, the frequency amplitude value is mapped to a preset line segment sampling point through normalization processing, which achieves the purpose of visualizing the audio, avoids real-time adjustment operations, and allows for previewing of the audio spectrum of the special effect in advance during the creation of the time-frequency texture map, and corresponding preprocessing operations are performed to enrich the use of texture map information and reduce the consumption of real-time adjustments.

[0127] Based on the same inventive concept, this application also provides an audio effects rendering device corresponding to the audio effects rendering method. Since the principle of the device in this application is similar to the audio effects rendering method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0128] Figure 12 This is a schematic diagram of the audio effects rendering device provided in an embodiment of this application. This device can be integrated into an electronic device. Figure 12 As shown, the device may include:

[0129] The acquisition module 701 is used to acquire the time-frequency map of the target audio file. The time-frequency map is used to indicate the energy distribution of each audio frame of the target audio file in the frequency domain.

[0130] The sampling module 702 is used to sample the time-frequency map to obtain the energy of multiple frequencies in each audio frame;

[0131] The determining module 703 is used to determine the amplitude value of multiple frequencies in each audio frame based on the energy of multiple frequencies in each audio frame;

[0132] The rendering module 704 is used to perform special effects rendering on the preset rendering model corresponding to the target audio file based on the amplitude values ​​of multiple frequencies in each audio frame.

[0133] In an optional implementation, the rendering module 704 is specifically used for:

[0134] Based on the amplitude values ​​of multiple frequencies in each audio frame, the preset animation model corresponding to the target audio file is rendered to obtain a floating effect that matches the preset animation model with the rhythm of the amplitude values.

[0135] In an optional implementation, the rendering module 704 is specifically used for:

[0136] Map multiple frequencies in each audio frame to the edge vertices of a preset rendering model;

[0137] Based on the amplitude values ​​of multiple frequencies in each audio frame, floating effects are rendered on the edge vertices of the preset rendering model.

[0138] In an optional implementation, the rendering module 704 is specifically used for:

[0139] Based on the amplitude values ​​of multiple frequencies in each audio frame, the edge vertices of the preset rendering model are controlled and rendered to scale according to the amplitude values.

[0140] In one optional implementation, the preset animation model includes: a preset particle model;

[0141] The rendering module is specifically used for:

[0142] Based on the amplitude values ​​of multiple frequencies in each audio frame, the motion velocities of multiple preset particle models are obtained respectively;

[0143] Based on the movement speed of multiple preset particle models, the multiple preset particle models are rendered respectively to control the movement of the multiple preset particle models.

[0144] In an optional implementation, the sampling module 702 is specifically used for:

[0145] The starting audio frame is determined from each audio frame based on the preset audio frame playback identifier.

[0146] The time-frequency map is sampled to obtain the energy of multiple frequencies in the starting audio frame and subsequent audio frames, so as to obtain the amplitude values ​​of multiple frequencies.

[0147] In an optional implementation, the acquisition module 701 is specifically used for:

[0148] Based on the target audio file, obtain the audio data of each audio frame;

[0149] Perform spectral analysis on the audio data of each audio frame to generate a spectrogram of each audio frame;

[0150] Create a spectrum segment based on multiple preset line segment sampling points;

[0151] The amplitude values ​​of multiple frequencies in the spectrum are mapped to multiple preset sampling points in the spectrum line segment to generate the target spectrum of each audio frame.

[0152] A time-frequency map is generated based on the target spectrogram of each audio frame.

[0153] In an optional implementation, the acquisition module 701 is specifically used for:

[0154] Normalize multiple frequencies in the spectrum to obtain multiple normalized frequencies;

[0155] The preset point numbers of multiple preset line segment sampling points are normalized to obtain multiple normalized point numbers;

[0156] Based on multiple normalized frequencies and multiple normalized point numbers, the amplitude values ​​of multiple frequencies in the spectrum are mapped to the height attribute of the corresponding preset line segment sampling points.

[0157] In an optional embodiment, the device further includes:

[0158] Processing module 705 is used to trim the spectrum diagram according to the amplitude distribution in the spectrum diagram.

[0159] In an optional implementation, the processing module 705 is further configured to:

[0160] Multiple frequency sets are obtained by classifying the frequencies in the spectrum graph.

[0161] According to the preset adjustment rules of each frequency set, the amplitude value of the frequency in each frequency set is adjusted respectively.

[0162] In this embodiment, the acquisition module acquires a time-frequency map of the target audio file, which indicates the energy distribution of each audio frame in the frequency domain. The sampling module samples the time-frequency map to obtain the energy of multiple frequencies in each audio frame. The determination module determines the amplitude values ​​of multiple frequencies in each audio frame based on their energy. The rendering module performs special effects rendering on the preset rendering model corresponding to the target audio file based on the amplitude values ​​of multiple frequencies in each audio frame. This solution performs special effects rendering on the preset rendering model corresponding to the target audio file using a time-frequency map, eliminating the need for real-time audio parsing, reducing processing overhead, improving the performance of audio visualization effects, and avoiding interference from other sound effects by only acquiring the time-frequency map of the target audio file, thus improving the accuracy of special effects rendering.

[0163] Figure 13 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 13 As shown, the device may include a processor 801, a memory 802, and a bus 803. The memory 802 stores machine-readable instructions executable by the processor 801. When the electronic device is running, the processor 801 communicates with the memory 802 via the bus 803. The processor 801 executes the machine-readable instructions to perform the following steps:

[0164] Obtain the time-frequency map of the target audio file. The time-frequency map is used to indicate the energy distribution of each audio frame in the frequency domain of the target audio file.

[0165] The time-frequency map is sampled to obtain the energy of multiple frequencies in each audio frame;

[0166] The amplitude values ​​of multiple frequencies in each audio frame are determined based on the energy of multiple frequencies in each audio frame.

[0167] Based on the amplitude values ​​of multiple frequencies in each audio frame, special effects are rendered on the preset rendering model corresponding to the target audio file.

[0168] In an optional implementation, special effects rendering is performed on a preset rendering model corresponding to the target audio file based on the amplitude values ​​of multiple frequencies in each audio frame, including:

[0169] Based on the amplitude values ​​of multiple frequencies in each audio frame, the preset animation model corresponding to the target audio file is rendered to obtain a floating effect that matches the preset animation model with the rhythm of the amplitude values.

[0170] In one optional implementation, based on the amplitude values ​​of multiple frequencies in each audio frame, a preset animation model corresponding to the target audio file is rendered with special effects to obtain a floating effect where the preset animation model and the amplitude values ​​rhythmically match, including:

[0171] Map multiple frequencies in each audio frame to the edge vertices of a preset rendering model;

[0172] Based on the amplitude values ​​of multiple frequencies in each audio frame, floating effects are rendered on the edge vertices of the preset rendering model.

[0173] In an optional implementation, edge vertex effects of a preset rendering model are rendered based on the amplitude values ​​of multiple frequencies in each audio frame, including:

[0174] Based on the amplitude values ​​of multiple frequencies in each audio frame, the edge vertices of the preset rendering model are controlled and rendered to scale according to the amplitude values.

[0175] In one optional implementation, the preset animation model includes: a preset particle model;

[0176] Based on the amplitude values ​​of multiple frequencies in each audio frame, special effects rendering is performed on the preset animation model corresponding to the target audio file to obtain floating effects that match the rhythm of the preset animation model and the amplitude values, including:

[0177] Based on the amplitude values ​​of multiple frequencies in each audio frame, the motion velocities of multiple preset particle models are obtained respectively;

[0178] Based on the movement speed of multiple preset particle models, render the multiple preset particle models separately to control their movement.

[0179] In an optional implementation, the time-frequency map is sampled to obtain the energy of multiple frequencies in each audio frame, including:

[0180] The starting audio frame is determined from each audio frame based on the preset audio frame playback identifier.

[0181] The time-frequency map is sampled to obtain the energy of multiple frequencies in the starting audio frame and subsequent audio frames, so as to obtain the amplitude values ​​of multiple frequencies.

[0182] In an optional implementation, obtaining the time-frequency map of the target audio file includes:

[0183] Based on the target audio file, obtain the audio data of each audio frame;

[0184] Perform spectral analysis on the audio data of each audio frame to generate a spectrogram of each audio frame;

[0185] Create a spectrum segment based on multiple preset line segment sampling points;

[0186] The amplitude values ​​of multiple frequencies in the spectrum are mapped to multiple preset sampling points in the spectrum line segment to generate the target spectrum of each audio frame.

[0187] A time-frequency map is generated based on the target spectrogram of each audio frame.

[0188] In an optional implementation, the amplitude values ​​of multiple frequencies in the spectrogram are mapped to multiple preset sampling points in the spectral line segments to generate target spectrograms for each audio frame, including:

[0189] Normalize multiple frequencies in the spectrum to obtain multiple normalized frequencies;

[0190] The preset point numbers of multiple preset line segment sampling points are normalized to obtain multiple normalized point numbers;

[0191] Based on multiple normalized frequencies and multiple normalized point numbers, the amplitude values ​​of multiple frequencies in the spectrum are mapped to the height attribute of the corresponding preset line segment sampling points to generate the target spectrum of each audio frame.

[0192] In an optional implementation, before mapping the amplitude values ​​of multiple frequencies in the spectrogram to multiple preset sampling points in the spectral segments to generate the target spectrogram for each audio frame, the method further includes:

[0193] The spectrum is trimmed based on the amplitude distribution in the spectrum.

[0194] In an optional implementation, before mapping the amplitude values ​​of multiple frequencies in the spectrogram to multiple preset sampling points in the spectral segments to generate the target spectrogram for each audio frame, the method further includes:

[0195] Multiple frequency sets are obtained by classifying the frequencies in the spectrum graph.

[0196] According to the preset adjustment rules of each frequency set, the amplitude value of the frequency in each frequency set is adjusted respectively.

[0197] In this embodiment, the processor executes machine-readable instructions to acquire a time-frequency map of the target audio file. The time-frequency map indicates the energy distribution of each audio frame in the frequency domain. The time-frequency map is sampled to obtain the energy of multiple frequencies in each audio frame. Based on the energy of these frequencies, the amplitude values ​​of the frequencies in each audio frame are determined. Based on the amplitude values ​​of these frequencies, special effects rendering is performed on the preset rendering model corresponding to the target audio file. This solution performs special effects rendering on the preset rendering model corresponding to the target audio file using a time-frequency map, eliminating the need for real-time audio parsing, reducing processing overhead, improving the performance of audio visualization effects, and avoiding interference from other sound effects by only acquiring the time-frequency map of the target audio file, thus improving the accuracy of special effects rendering.

[0198] This application embodiment also provides a computer-readable storage medium storing a computer program, which is executed by a processor, and the processor performs the following steps:

[0199] Obtain the time-frequency map of the target audio file. The time-frequency map is used to indicate the energy distribution of each audio frame in the frequency domain of the target audio file.

[0200] The time-frequency map is sampled to obtain the energy of multiple frequencies in each audio frame;

[0201] The amplitude values ​​of multiple frequencies in each audio frame are determined based on the energy of multiple frequencies in each audio frame.

[0202] Based on the amplitude values ​​of multiple frequencies in each audio frame, special effects are rendered on the preset rendering model corresponding to the target audio file.

[0203] In an optional implementation, special effects rendering is performed on a preset rendering model corresponding to the target audio file based on the amplitude values ​​of multiple frequencies in each audio frame, including:

[0204] Based on the amplitude values ​​of multiple frequencies in each audio frame, the preset animation model corresponding to the target audio file is rendered to obtain a floating effect that matches the preset animation model with the rhythm of the amplitude values.

[0205] In one optional implementation, based on the amplitude values ​​of multiple frequencies in each audio frame, a preset animation model corresponding to the target audio file is rendered with special effects to obtain a floating effect where the preset animation model and the amplitude values ​​rhythmically match, including:

[0206] Map multiple frequencies in each audio frame to the edge vertices of a preset rendering model;

[0207] Based on the amplitude values ​​of multiple frequencies in each audio frame, floating effects are rendered on the edge vertices of the preset rendering model.

[0208] In an optional implementation, edge vertex effects of a preset rendering model are rendered based on the amplitude values ​​of multiple frequencies in each audio frame, including:

[0209] Based on the amplitude values ​​of multiple frequencies in each audio frame, the edge vertices of the preset rendering model are controlled and rendered to scale according to the amplitude values.

[0210] In one optional implementation, the preset animation model includes: a preset particle model;

[0211] Based on the amplitude values ​​of multiple frequencies in each audio frame, special effects rendering is performed on the preset animation model corresponding to the target audio file to obtain floating effects that match the rhythm of the preset animation model and the amplitude values, including:

[0212] Based on the amplitude values ​​of multiple frequencies in each audio frame, the motion velocities of multiple preset particle models are obtained respectively;

[0213] Based on the movement speed of multiple preset particle models, the multiple preset particle models are rendered respectively to control the movement of the multiple preset particle models.

[0214] In an optional implementation, the time-frequency map is sampled to obtain the energy of multiple frequencies in each audio frame, including:

[0215] The starting audio frame is determined from each audio frame based on the preset audio frame playback identifier.

[0216] The time-frequency map is sampled to obtain the energy of multiple frequencies in the starting audio frame and subsequent audio frames, so as to obtain the amplitude values ​​of multiple frequencies.

[0217] In an optional implementation, obtaining the time-frequency map of the target audio file includes:

[0218] Based on the target audio file, obtain the audio data of each audio frame;

[0219] Perform spectral analysis on the audio data of each audio frame to generate a spectrogram of each audio frame;

[0220] Create a spectrum segment based on multiple preset line segment sampling points;

[0221] The amplitude values ​​of multiple frequencies in the spectrum are mapped to multiple preset sampling points in the spectrum line segment to generate the target spectrum of each audio frame.

[0222] A time-frequency map is generated based on the target spectrogram of each audio frame.

[0223] In an optional implementation, the amplitude values ​​of multiple frequencies in the spectrogram are mapped to multiple preset sampling points in the spectral line segments to generate target spectrograms for each audio frame, including:

[0224] Normalize multiple frequencies in the spectrum to obtain multiple normalized frequencies;

[0225] The preset point numbers of multiple preset line segment sampling points are normalized to obtain multiple normalized point numbers;

[0226] Based on multiple normalized frequencies and multiple normalized point numbers, the amplitude values ​​of multiple frequencies in the spectrum are mapped to the height attribute of the corresponding preset line segment sampling points to generate the target spectrum of each audio frame.

[0227] In an optional implementation, before mapping the amplitude values ​​of multiple frequencies in the spectrogram to multiple preset sampling points in the spectral segments to generate the target spectrogram for each audio frame, the method further includes:

[0228] The spectrum is trimmed based on the amplitude distribution in the spectrum.

[0229] In an optional implementation, before mapping the amplitude values ​​of multiple frequencies in the spectrogram to multiple preset sampling points in the spectral segments to generate the target spectrogram for each audio frame, the method further includes:

[0230] Multiple frequency sets are obtained by classifying the frequencies in the spectrum graph.

[0231] According to the preset adjustment rules of each frequency set, the amplitude value of the frequency in each frequency set is adjusted respectively.

[0232] In this embodiment, the computer program is executed by the processor. The processor acquires a time-frequency map of the target audio file. The time-frequency map indicates the energy distribution of each audio frame in the frequency domain. The time-frequency map is sampled to obtain the amplitude values ​​of multiple frequencies in each audio frame. Based on the energy of multiple frequencies in each audio frame, the amplitude values ​​of multiple frequencies in each audio frame are determined, and special effects rendering is performed on the preset rendering model corresponding to the target audio file. This solution performs special effects rendering on the preset rendering model corresponding to the target audio file using a time-frequency map, eliminating the need for real-time audio parsing, reducing processing overhead, improving the performance of audio visualization effects, and avoiding interference from other sound effects by only acquiring the time-frequency map of the target audio file, thus improving the accuracy of special effects rendering.

[0233] In this embodiment, the computer program, when run by the processor, can also execute other machine-readable instructions to perform other methods as described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0234] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0235] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0236] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0237] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0238] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0239] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. An audio special effect rendering method, characterized by, include: Obtain a time-frequency map of the target audio file, wherein the time-frequency map is used to indicate the energy distribution of each audio frame of the target audio file in the frequency domain; The time-frequency map is sampled to obtain the energy of multiple frequencies in each audio frame; The amplitude values ​​of multiple frequencies in each audio frame are determined based on the energy of multiple frequencies in each audio frame. Based on the amplitude values ​​of multiple frequencies in each audio frame, special effects rendering is performed on the preset rendering model corresponding to the target audio file; The step of rendering special effects on the preset rendering model corresponding to the target audio file based on the amplitude values ​​of multiple frequencies in each audio frame includes: Based on the amplitude values ​​of multiple frequencies in each audio frame, special effects rendering is performed on the preset animation model corresponding to the target audio file to obtain a floating effect that rhythmically matches the preset animation model with the amplitude values. The step of rendering special effects on a preset animation model corresponding to the target audio file based on the amplitude values ​​of multiple frequencies in each audio frame to obtain a floating effect in which the preset animation model rhythmically matches the amplitude values ​​includes: Map multiple frequencies in each audio frame to the edge vertices of the preset animation model; Based on the amplitude values ​​of multiple frequencies in each audio frame, floating effects are rendered on the edge vertices of the preset animation model.

2. The audio special effect rendering method of claim 1, wherein, The step of rendering floating effects on the edge vertices of the preset animation model based on the amplitude values ​​of multiple frequencies in each audio frame includes: Based on the amplitude values ​​of multiple frequencies in each audio frame, the edge vertices of the preset animation model are controlled and rendered to scale according to the amplitude values.

3. The audio special effect rendering method of claim 1, wherein, The preset animation model includes: a preset particle model; The step of rendering special effects on a preset animation model corresponding to the target audio file based on the amplitude values ​​of multiple frequencies in each audio frame to obtain a floating effect in which the preset animation model rhythmically matches the amplitude values ​​includes: Based on the amplitude values ​​of multiple frequencies in each audio frame, the motion speeds of multiple preset particle models are obtained respectively; Based on the movement speed of the multiple preset particle models, the multiple preset particle models are rendered respectively to control the movement of the multiple preset particle models.

4. The audio effects rendering method of any one of claims 1-3, wherein, The step of sampling the time-frequency map to obtain the energy of multiple frequencies in each audio frame includes: The starting audio frame is determined from each audio frame according to the preset audio frame playback identifier; The time-frequency map is sampled to obtain the energy of the starting audio frame and the audio frames after the starting audio frame, so as to obtain the amplitude values ​​of multiple frequencies.

5. The audio special effect rendering method of claim 1, wherein, The process of obtaining the time-frequency texture of the target audio file includes: Based on the target audio file, obtain the audio data of each audio frame; Perform spectral analysis on the audio data of each audio frame to generate a spectrogram of each audio frame; Create a spectrum segment based on multiple preset line segment sampling points; The amplitude values ​​of multiple frequencies in the spectrum are mapped to the multiple preset line segment sampling points in the spectrum line segment to generate the target spectrum of each audio frame; The time-frequency map is generated based on the target spectrogram of each audio frame.

6. The audio special effect rendering method of claim 5, wherein, The step of mapping the amplitude values ​​of multiple frequencies in the spectrum to the multiple preset sampling points in the spectrum segment includes: The frequencies in the spectrum are normalized to obtain multiple normalized frequencies; The preset point numbers of the multiple preset line segment sampling points are normalized respectively to obtain multiple normalized point numbers; Based on the multiple normalized frequencies and the multiple normalized point numbers, the amplitude values ​​of multiple frequencies in the spectrum are mapped to the height attribute of the corresponding preset line segment sampling points.

7. The audio special effect rendering method of claim 5, wherein, Before mapping the amplitude values ​​of multiple frequencies in the spectrum to the multiple preset sampling points in the spectrum segments to generate the target spectrum for each audio frame, the method further includes: The spectrum is trimmed based on the amplitude distribution in the spectrum.

8. The audio special effect rendering method of claim 5, wherein, Before mapping the amplitude values ​​of multiple frequencies in the spectrum to the multiple preset sampling points in the spectrum segments to generate the target spectrum for each audio frame, the method further includes: Multiple frequency sets are obtained by classifying the frequencies in the spectrum. According to the preset adjustment rules of each frequency set, the amplitude value of the frequency in each frequency set is adjusted respectively.

9. An audio special effect rendering apparatus, characterized by comprising: include: The acquisition module is used to acquire a time-frequency map of a target audio file, wherein the time-frequency map is used to indicate the energy distribution of each audio frame of the target audio file in the frequency domain. The sampling module is used to sample the time-frequency map to obtain the energy of multiple frequencies in each audio frame; The determining module is used to determine the amplitude values ​​of multiple frequencies in each audio frame based on the energy of multiple frequencies in each audio frame; The rendering module is used to perform special effects rendering on the preset rendering model corresponding to the target audio file based on the amplitude values ​​of multiple frequencies in each audio frame. The rendering module is specifically used for: Based on the amplitude values ​​of multiple frequencies in each audio frame, special effects rendering is performed on the preset animation model corresponding to the target audio file to obtain a floating effect that rhythmically matches the preset animation model with the amplitude values. The rendering module is specifically used for: Map multiple frequencies in each audio frame to the edge vertices of the preset animation model; Based on the amplitude values ​​of multiple frequencies in each audio frame, floating effects are rendered on the edge vertices of the preset animation model.

10. An electronic device, comprising: include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the audio effects rendering method according to any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the audio effects rendering method according to any one of claims 1 to 8.

Images