Image rendering method and device based on audio data and computer device

Abstract

The application provides an image rendering method and device based on audio data, a computer device and a storage medium. The method converts the frequency spectrum data of the audio data into a time-frequency spectrum diagram. When rendering an audio visualization effect diagram, a continuous audio column drawing area is defined on an arbitrary virtual model. Texture coordinates are generated for the audio column drawing area. The frequency spectrum data of each audio column in the audio column drawing area is obtained through the texture coordinates. The drawing area corresponding to the audio column is rendered to generate an audio column effect diagram. The visualization animation effect of the audio data is realized on the surface of an arbitrary model.

Description

Technical Field

[0001] This application relates to the field of image technology, and specifically to an image rendering method, apparatus, computer device, and computer-readable storage medium based on audio data. Background Technology

[0002] In fields involving computer graphics, such as gaming and film, audio data visualization is often used to represent the rhythm of music, providing users with a more immersive auditory experience. Audio data visualization typically involves converting music data into a corresponding histogram, using the rise and fall of different musical bars to represent the rhythm of the music. However, current technologies for histogram-based visualization... Figure 1 Generally, it can only be displayed through a fixed display window, and it is difficult to render it onto the surface of any virtual model. Summary of the Invention

[0003] Therefore, it is necessary to provide an image rendering method, apparatus, and computer device based on audio data to address the aforementioned technical problems, so as to render the visualization effect of audio data onto the surface of any virtual model and improve the universality of audio data visualization effect rendering.

[0004] In a first aspect, this application provides an image rendering method based on audio data, the method comprising:

[0005] Obtain the time-spectrum graph of the audio data;

[0006] Determine the audio column drawing area on the virtual model, and obtain the texture coordinates of the audio column drawing area based on the vertex coordinates of each target vertex in the audio column drawing area;

[0007] The audio bar drawing area is divided into multiple drawing areas corresponding to the audio bars, and the spectral data corresponding to the audio bars under different time frames is obtained in the time-spectrum graph according to the texture coordinates of the audio bar drawing area.

[0008] Based on the spectral data corresponding to the audio bars in different time frames, the drawing area corresponding to the audio bars is rendered to obtain the audio bar effect diagram corresponding to the audio data.

[0009] Secondly, this application provides an image rendering apparatus based on audio data, the apparatus comprising:

[0010] The time-spectrum acquisition module is used to acquire the time-spectrum of audio data;

[0011] The drawing area acquisition module is used to determine the audio column drawing area on the virtual model, and to obtain the texture coordinates of the audio column drawing area based on the vertex coordinates of each target vertex in the audio column drawing area.

[0012] The spectrum data sampling module is used to divide the audio bar drawing area into multiple drawing areas corresponding to the audio bars, and to obtain the spectrum data corresponding to the audio bars in different time frames in the time spectrum diagram according to the texture coordinates of the audio bar drawing area.

[0013] The rendering module is used to render the drawing area corresponding to the audio column based on the spectrum data corresponding to the audio column under different time frames, so as to obtain the audio column rendering corresponding to the audio data.

[0014] Thirdly, this application also provides a computer device, which includes:

[0015] One or more processors;

[0016] Memory; and

[0017] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement an image rendering method based on audio data.

[0018] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps in the image rendering method based on audio data.

[0019] The aforementioned image rendering method, apparatus, computer device, and storage medium based on audio data acquire a temporal spectrogram of the audio data; determine the audio bar drawing area on the virtual model, and obtain the texture coordinates of the audio bar drawing area based on the vertex coordinates of each target vertex in the audio bar drawing area; divide the audio bar drawing area to obtain multiple drawing areas corresponding to the audio bars, and obtain the spectral data corresponding to the audio bars in different time frames from the temporal spectrogram based on the texture coordinates of the audio bar drawing area; render the drawing areas corresponding to the audio bars based on the spectral data corresponding to the audio bars in different time frames to obtain the audio bar effect image corresponding to the audio data. By converting the spectral data of the audio data into a temporal spectrogram, when rendering the audio visualization effect image, after defining a continuous audio bar drawing area in any virtual model, texture coordinates are generated for the audio bar drawing area, and the spectral data of each audio bar in the audio bar drawing area is obtained through the texture coordinates to render the drawing areas corresponding to the audio bars to generate the audio bar effect image, thereby realizing the visualization animation effect of rendering audio data on the surface of any model. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the image rendering method based on audio data in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the time-frequency spectrum in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the drawing area corresponding to each audio column in the embodiments of this application;

[0024] Figure 4A This is a schematic diagram of the audio column effect in the embodiments of this application;

[0025] Figure 4B This is another schematic diagram of the audio column effect in the embodiments of this application;

[0026] Figure 5A This is a schematic diagram of the bar chart corresponding to the spectrum data in the embodiments of this application;

[0027] Figure 5B This is another schematic diagram of the bar chart corresponding to the spectrum data in the embodiments of this application;

[0028] Figure 5C This is yet another schematic diagram of the bar chart corresponding to the spectrum data in the embodiments of this application;

[0029] Figure 5D This is another schematic diagram of the bar chart corresponding to the spectrum data in the embodiments of this application;

[0030] Figure 6A This is a schematic diagram of the projection bounding box in an embodiment of this application;

[0031] Figure 6B This is a schematic diagram of the audio bar drawing area in an embodiment of this application;

[0032] Figure 7 This is a schematic diagram of the structure of the image rendering device based on audio data in the embodiments of this application;

[0033] Figure 8 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0036] In the description of this application, the word "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0037] In existing technologies, there are two main methods for generating visualizations of audio data. One method involves converting the audio data into corresponding spectral information, then creating a dynamic histogram based on this information. The histogram is then saved as a sequence frame texture in a frame-by-frame animation manner. When rendering to a virtual model, each frame texture (i.e., the histogram) is extracted along the timeline and rendered onto the virtual model. The other method involves acquiring the spectral information of the audio data in real time, dividing the continuous spectral data into several audio partitions, and performing signal processing on the spectral information of each partition to obtain the corresponding spectral intensity. During audio data visualization, the same number of histograms (hereinafter referred to as audio bars) are generated, with the height of each audio bar corresponding to the spectral intensity of the corresponding audio partition. This allows the audio bars to move dynamically as the audio data is played, achieving the desired audio visualization effect.

[0038] However, for the first method of generating audio data visualizations, the pre-generated histogram has a fixed resolution. When rendering the histogram onto a large virtual model, the rendering effect is blurry and lacks adaptability. Furthermore, the histogram needs to be displayed on a separate, flat surface. When the virtual model's surface is too complex, such as a curved surface, the histogram struggles to fit the surface. Even worse, when the texture coordinates (i.e., UV coordinates) of the virtual model's surface are discontinuous, the histogram cannot be directly rendered onto the virtual model's surface. For the second method of generating audio data visualizations, which uses histogram geometry to represent the audio data visualization, this method struggles to render the effect onto the surface of the virtual model.

[0039] To address the aforementioned issues, this application provides an image rendering method based on audio data, which can run on a terminal device or a server. The terminal device can be a local terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Taking the application of this audio data-based image rendering method in a video game scenario as an example, the video game server can render an audio histogram corresponding to the background music on any virtual object model in the game scene based on this audio data-based image rendering method, realizing the visualization of the background music and enriching the user's audiovisual experience during gameplay.

[0040] See Figure 1 This application provides an image rendering method based on audio data. The method is illustrated by its application to a server. The method includes steps S110 to S140, as detailed below:

[0041] Step S110: Obtain the time-spectrum diagram of the audio data.

[0042] Among them, audio data refers to music files to be visualized; taking a game scene as an example, audio data refers to the audio data of background music, special effects music, and other music in the game scene.

[0043] Among them, the time-spectrum map is used to record the texture map of the spectral changes of audio data. Specifically, the continuous audio data can be converted into discrete spectral data through the Fourier transform algorithm, and then the discrete spectral data is normalized to the data between 0 and 1. This value is then saved as a grayscale value in the texture map, thereby compressing the spectral data corresponding to the audio data into a grayscale texture map.

[0044] Specifically, in one embodiment, the step of obtaining the time-spectrum map of audio data includes: sampling the audio data based on a preset sampling window to obtain audio data segments corresponding to multiple time frames; performing Fourier transform on the audio data segments corresponding to each time frame to obtain spectrum data corresponding to multiple time frames; wherein the spectrum data includes amplitude values ​​at different frequencies; normalizing the amplitude values ​​at different frequencies in each time frame; and saving the spectrum data of the audio data in multiple time frames using the frame number of the time frame as a first variable, the frequency values ​​of each frequency in the same time frame as a second variable, and the amplitude values ​​at different frequencies in different time frames as pixel data to obtain the time-spectrum map.

[0045] Specifically, the audio data is first windowed using a preset sampling window to obtain audio data segments corresponding to multiple sampling windows. Each sampling window is considered a time frame, and audio data segments for multiple time frames are obtained according to the sampling order. Then, for any audio data segment in a time frame, a Fast Fourier Transform is performed on the audio data segment to obtain the amplitude values ​​at different frequencies in that time frame. The preset sampling window can be a Hamming window, a Blackman window, etc., and is not limited here.

[0046] After obtaining the amplitude values ​​at different frequencies in different time frames, the amplitude values ​​at different frequencies in each time frame can be normalized to obtain amplitude values ​​between 0 and 1. Then, the spectral data of the audio data in multiple time frames can be converted into a time-spectrum graph and saved, with the change of time frame as the first coordinate axis, the change of frequency as the second coordinate axis, and the amplitude values ​​of different frequencies in different time frames as the pixel data at the corresponding coordinate points.

[0047] For example, see Figure 2 A grayscale map is drawn by using time (instantaneous frame) as the horizontal axis, frequency as the vertical axis, and the amplitude values ​​of different frequencies in different time frames as the grayscale values ​​at the corresponding coordinate points. This grayscale map is an instantaneous spectrum map.

[0048] Furthermore, when visualizing audio data, the spectral data of any time-series frame is typically rendered into a spectral histogram sequentially according to the time sequence of the time-series frames. That is, one frequency is rendered as one audio bar, and the amplitude value at the corresponding frequency determines the height of the audio bar. In order to represent the dynamic effect of the audio data, in one embodiment, the spectral data of the audio data in multiple time-series frames is saved using the frame number of the time-series frame as the first variable, the frequency values ​​of each frequency within the same time-series frame as the second variable, and the amplitude values ​​at different frequencies in different time-series frames as pixel data. Before the step of obtaining the time-spectrum graph, the method further includes: for any target time-series frame in different time-series frames, adjusting the amplitude values ​​of the target time-series frame at different frequencies according to the amplitude values ​​of the corresponding reference time-series frame at the same frequency; and / or smoothing the amplitude values ​​of the target time-series frame at different frequencies to obtain the adjusted amplitude values ​​of the target time-series frame at different frequencies.

[0049] The target time frame and its corresponding reference time frame are time-continuous frames. For example, the target time frame is the frame at time t, and the reference time frame is the frame at time (t-1).

[0050] Specifically, based on the amplitude values ​​of the target and reference timing frames at the same frequency, an amplitude decrease value at that frequency can be calculated, and then the amplitude value of the target timing frame at that frequency can be adjusted accordingly. Further, the amplitude values ​​of the target and reference timing frames at the same frequency can be compared. If the amplitude value of the target timing frame at a certain frequency is greater than that of the reference timing frame at the same frequency, the amplitude adjustment value can be set to 0, keeping the amplitude value of the target timing frame at that frequency unchanged, so that the height of the audio column reflects the increase in the spectrum. If the amplitude value of the target timing frame at a certain frequency is less than that of the reference timing frame at that frequency, the amplitude decrease value can be set to a positive value, and the amplitude value of the target timing frame at that frequency can be adjusted accordingly, so that the height of the audio column exhibits a gradual decreasing effect, with the rate of decrease of the audio height column increasing from slow to fast.

[0051] Furthermore, amplitude values ​​at different frequencies within the same time frame can be smoothed to reduce the jaggedness of the spectrum histogram when visualizing audio data. Specifically, for amplitude values ​​at different frequencies within the same time frame, the amplitude values ​​within a preset smoothing radius can be summed and averaged to obtain the final amplitude value.

[0052] Step S120: Determine the audio column drawing area on the virtual model, and obtain the texture coordinates of the audio column drawing area based on the vertex coordinates of each target vertex in the audio column drawing area.

[0053] Among them, the virtual model refers to the object in the virtual scene to be rendered with audio visualization effects, and the audio bar drawing area refers to a part of the surface of the virtual model.

[0054] Specifically, a rectangular area can be pre-constructed, which is the rendering position of the audio visualization effect (i.e., audio column). The rectangular area is projected onto the virtual scene according to a certain projection direction and projection position, thereby forming a rectangular projection bounding box in three-dimensional space. The point where the projection direction intersects the virtual model surface for the first time is recorded as the audio column drawing area.

[0055] In one embodiment, the step of determining the audio column drawing area on the virtual model includes: obtaining the projection direction, bounding box position, and bounding box size of the projection bounding box; obtaining the projection coordinate data of each vertex on the virtual model in the projection coordinate system corresponding to the projection bounding box based on the projection direction and bounding box position; determining the target vertex within the projection bounding box among the vertices based on the bounding box position, bounding box size, and projection coordinate data of each vertex; and determining the audio column drawing area of ​​the virtual model based on the target vertex.

[0056] After obtaining the projection direction, position, and size of the bounding box, the model coordinate system corresponding to the virtual model can be rotated according to the projection direction to obtain the projection coordinate system corresponding to the projection direction of the bounding box. Then, the vertex coordinate data of each vertex on the virtual model is converted into projection coordinate data in the projection coordinate system. Finally, based on the projection coordinate data of each vertex on the virtual model, as well as the position and size of the bounding box, it is determined whether each vertex on the virtual model is inside the bounding box to identify the target vertex inside the bounding box. The face formed by the target vertex is the audio column drawing area of ​​the virtual model.

[0057] Understandably, virtual models in virtual scenes often have complex structures, with surfaces including both planes and complex curved surfaces. Due to complex display requirements, the original texture coordinates of the virtual model are often discontinuous. After defining a continuous plane on the surface of the virtual model as the audio column drawing area, it's not possible to simply use the original texture coordinates of the virtual model as the texture coordinates of the audio column drawing area. Therefore, texture coordinates can be generated programmatically for the audio column drawing area. Specifically, the texture coordinates of the audio column drawing area can be obtained based on the vertex coordinates of each target vertex within the audio column drawing area.

[0058] Furthermore, in the projection coordinate system corresponding to the projection bounding box, the coordinate data corresponding to the coordinate axes other than the coordinate axis where the projection direction of the projection bounding box is located can be regarded as planar coordinate data in a two-dimensional plane perpendicular to the projection direction. Therefore, in one embodiment, the step of obtaining the texture coordinates of the audio column drawing area based on the vertex coordinates of each target vertex in the audio column drawing area includes: converting the vertex coordinates of the target vertex into target projection coordinate data in the projection coordinate system corresponding to the projection bounding box based on the projection direction and the bounding box position; normalizing the target projection coordinate data of the target vertex according to the size of the bounding box; and determining the normalized target texture mapping coordinates of the target vertex as the texture coordinates of the audio column drawing area.

[0059] Here, texture coordinates are UV coordinates; target projection coordinate data are the coordinate data of the target vertex in the projection coordinate system corresponding to the projection bounding box. Specifically, after converting the vertex coordinates of the target vertex into target projection coordinate data in the projection coordinate system corresponding to the projection bounding box based on the projection direction and the position of the bounding box, the coordinate data corresponding to the coordinate axes other than the projection direction of the projection bounding box in the target projection coordinate data can be used as the initial texture coordinates. Then, the initial texture coordinates are normalized to the range of (0, 1) according to the size of the bounding box to obtain the texture coordinates of the audio rendering area, so that the texture coordinates can be used in subsequent steps to obtain the spectrum data from the time-spectrum map and render it onto the surface of the virtual model.

[0060] Step S130: Divide the audio bar drawing area to obtain multiple drawing areas corresponding to audio bars, and obtain the spectral data corresponding to the audio bars in different time frames in the time spectrum map according to the texture coordinates of the audio bar drawing area.

[0061] Specifically, when the audio bar chart is a bar graph displayed along a straight line, the audio bar drawing area can be divided along a horizontal or vertical straight line to obtain drawing areas corresponding to multiple audio bars. When the audio bar chart is a bar graph displayed in a ring, a ring can be defined within the audio drawing area, and the audio drawing area can be divided into drawing areas corresponding to multiple audio bars along this ring. It can be understood that one audio bar corresponds to a frequency or a frequency range in the spectral data. Correspondingly, the direction perpendicular to the aforementioned straight line or the tangent line of the ring is used to draw and display the amplitude value of the frequency or frequency range corresponding to that audio bar. After determining the drawing area for each audio bar, spectral data is sampled from the time-spectrum graph based on the texture coordinates of the audio bar drawing area to obtain the spectral data corresponding to different audio bars.

[0062] Specifically, taking an audio bar chart displayed along a straight line as an example, in one embodiment, the steps of dividing the audio bar drawing area into multiple drawing areas corresponding to audio bars, and obtaining the spectral data of the audio bars in different time frames in the time-series spectrum based on the texture coordinates of the audio bar drawing areas, include: dividing the audio bar drawing area in a first direction based on a preset number of audio bars to obtain drawing areas corresponding to each audio bar; obtaining the first texture coordinate value of the drawing sub-area corresponding to each audio bar in the first direction based on the texture coordinates of the audio bar drawing areas; and obtaining the spectral data of each audio bar in different time frames in the time-series spectrum based on the first texture coordinate value of the drawing sub-area corresponding to each audio bar.

[0063] The first direction can be either the horizontal or vertical direction of the audio bar drawing area. After splitting the audio bar drawing area along the first direction, the texture coordinates of the corresponding texture coordinates of the audio bar drawing area are further split along the first direction to obtain the first texture coordinates of the drawing sub-regions corresponding to each audio bar along the first direction. This ensures that the first texture coordinates of the drawing sub-regions corresponding to the audio bars are consistent, so that the spectral data sampled from the time-spectrum graph using the texture coordinates of the drawing sub-regions corresponding to the audio bars are consistent.

[0064] After obtaining the first texture coordinate value of the drawing sub-region corresponding to each audio bar in the first direction, the spectral data of each audio bar under different time frames is obtained from the time spectrum diagram based on the first texture coordinate value.

[0065] See Figure 3 , Figure 3 The texture coordinates corresponding to the mid-frequency bar drawing area can be represented as (u, v). Here, the direction of the texture coordinate value U (i.e., the horizontal direction of the audio bar drawing area) is used as the first direction to divide the audio bar drawing area. Simultaneously, the texture coordinate value U in the texture coordinates is segmented to obtain the drawing area corresponding to each audio bar and the texture coordinate value U corresponding to each drawing area. Then, spectral data is sampled from the time-frequency spectrogram based on the texture coordinate values ​​U of each drawing area. It can be understood that one texture coordinate value U corresponds to one frequency in the spectral data, and the amplitude value of the corresponding frequency is obtained by sampling from the time-frequency spectrogram based on the texture coordinate values ​​U of each drawing area.

[0066] Step S140: Render the drawing area corresponding to the audio bar based on the spectral data corresponding to the audio bar under different time frames to obtain the audio bar effect diagram corresponding to the audio data.

[0067] After obtaining the spectral data corresponding to the audio bars under different time frames, the corresponding drawing areas are rendered sequentially according to the spectral data of each audio bar under each time frame, in order to obtain the audio bars with high and low fluctuations under each time frame, and to obtain the audio bar effect diagram corresponding to the audio data under each time frame, thus realizing the conversion of audio data into a visual audio bar animation.

[0068] In one embodiment, the step of rendering the drawing area corresponding to the audio bar based on the spectral data of the audio bar under different time frames to obtain the audio bar effect diagram corresponding to the audio data includes: for a target time frame in different time frames, obtaining the height value corresponding to the audio bar based on the spectral data of the audio bar under the target time frame; and rendering the drawing area corresponding to the audio bar based on the height value corresponding to the audio bar to obtain the audio bar effect diagram corresponding to the audio data.

[0069] Specifically, taking a target time frame among multiple time frames as an example, the spectral data of the audio bar in the target time frame is the amplitude value of the frequency corresponding to that audio bar. After obtaining the spectral data of the audio bar, the amplitude value in the spectral data can be converted into the height value of the audio bar. Then, based on the height value, a truncation operation is performed on the drawing area corresponding to the audio bar to control the drawing height of the audio bar. See also Figure 4A As shown, Figure 4A This is a schematic diagram of the audio column effect.

[0070] Furthermore, the audio column can be controlled to, for example... Figure 4B The audio grid shown is illustrated in one embodiment. The drawing area corresponding to the audio column is divided to obtain the grid area corresponding to the audio grid on the audio column. For the grid area corresponding to the audio grid on the audio column, the grid texture coordinates corresponding to the grid area are generated. For the target time frame in different time frames, the target audio grid to be rendered on the audio column is determined according to the spectrum data of the audio column in the target time frame. Based on the grid texture coordinates of the target audio grid, the audio column effect map corresponding to the audio data is sampled from the preset audio grid texture map to generate the audio column effect map.

[0071] The audio grid map is used to store the texture pattern of the audio grid; the grid texture coordinates are used to read the audio grid map to transform the texture pattern displayed in the grid area.

[0072] Specifically, based on the drawing area corresponding to the audio column, the drawing area is further divided to obtain the grid area corresponding to the audio grid on each audio column. After determining the grid area corresponding to each audio grid in the audio column drawing area, the corresponding grid texture coordinates are generated based on the grid area; for example, for any grid area, its lower left corner is the starting point (0,0) of the grid texture coordinates, and its upper right corner is the ending point (1,1). After obtaining the spectrum data of the audio column, the amplitude value in the spectrum data can be converted into the height value of the audio column, and then the height value can be used to determine whether the audio grid on the audio column is rendered, so as to determine the target audio grid to be rendered. Then, based on the grid texture coordinates of each target audio grid, the texture pattern to be rendered is obtained from the audio grid texture map, and the texture pattern is rendered in the grid area corresponding to the target audio grid.

[0073] In the aforementioned image rendering method based on audio data, the time-spectrum map of the audio data is obtained; the audio bar drawing area on the virtual model is determined, and the texture coordinates of the audio bar drawing area are obtained based on the vertex coordinates of each target vertex in the audio bar drawing area; the audio bar drawing area is divided to obtain multiple drawing areas corresponding to the audio bars, and the spectral data corresponding to the audio bars under different time frames is obtained from the time-spectrum map based on the texture coordinates of the audio bar drawing area; the drawing area corresponding to the audio bars is rendered based on the spectral data corresponding to the audio bars under different time frames to obtain the audio bar effect map corresponding to the audio data. By converting the spectral data of the audio data into a time-spectrum map, when rendering the audio visualization effect map, after defining a continuous audio bar drawing area in any virtual model, texture coordinates are generated for the audio bar drawing area, and the spectral data of each audio bar in the audio bar drawing area is obtained through the texture coordinates to render the drawing area corresponding to the audio bars to generate the audio bar effect map, thereby realizing the visualization animation effect of rendering audio data on the surface of any model.

[0074] The following section provides a further explanation of the audio data-based image rendering method using an application scenario. Specifically, the audio data-based image rendering method includes the following steps:

[0075] Step 1: Obtain the time-spectrum diagram of the audio data.

[0076] In this step, the audio data is converted into a time-spectrum graph used in subsequent model rendering.

[0077] Specifically, the Fast Fourier Transform (FFT) is used to convert continuous audio data into discrete spectral data.

[0078] Furthermore, the audio data can be sampled based on a preset sampling window to obtain audio data segments corresponding to multiple time frames; then, a Fourier transform is performed on the audio data segments corresponding to each time frame to obtain spectral data corresponding to multiple time frames; wherein, the spectral data includes amplitude values ​​at different frequencies. See also Figure 5A , Figure 5A This is a diagram showing the effect of presenting the spectral data of a time frame in the form of a bar chart, where each bar in the bar chart represents the amplitude value of a spectrum.

[0079] from Figure 5A As can be seen, the waveform in the histogram corresponding to the spectral data exhibits strong jaggedness. To improve the aesthetics of the waveform in the histogram, further processing of the spectral data is possible. Specifically, in the histogram corresponding to the spectral data, the amplitude of the bars varies greatly between different time frames at the same frequency, making it difficult to represent the dynamic effect of audio data during playback. Therefore, for the spectral data of any target time frame, the amplitude value of the target time frame at different frequencies can be adjusted based on the amplitude value of the corresponding reference time frame at the same frequency. Specifically, the reference timing frame is the previous timing frame of the target timing frame. The spectral data of the reference timing frame is cached separately. When new spectral data (i.e., the spectral data of the target timing frame) is obtained, it is compared with the cached spectral data. If the amplitude value at a certain frequency in the new spectral data is greater than the amplitude value at the same frequency in the cached spectrum, the amplitude value at that frequency in the new spectral data is kept unchanged, so that the corresponding bar in the histogram immediately shows the spectrum growth effect. If the amplitude value at a certain frequency in the new spectral data is less than the amplitude value at the same frequency in the cached spectrum, the amplitude value at that frequency in the new spectral data can be adjusted, so that the decrease in the corresponding bar in the histogram is a gradual decrease, with the decrease gradually accelerating. The histogram corresponding to the adjusted spectral data is shown below. Figure 5B As shown.

[0080] Furthermore, the frequencies used in audio data are mainly concentrated in the low and mid-frequency regions, while the high-frequency region is less frequent. Therefore, the fluctuations in the histogram of the audio data's spectrum are primarily observed in the low and mid-frequency regions, while the high-frequency region is relatively stable; that is, the low-frequency bars are taller, and the high-frequency bars are shorter. To make the histogram exhibit a wave-like effect with consistent amplitude, the spectrum data can be normalized, transforming it to a uniform range of 0 to 1. The histogram corresponding to the normalized spectrum data is shown below. Figure 5C As shown.

[0081] Furthermore, to further reduce the jaggedness of the waveform in the histogram, the spectral data can be further smoothed. Specifically, the amplitude values ​​at different frequencies in the time frame are smoothed to obtain the adjusted amplitude values ​​at different frequencies of the target time frame. Specifically, a mean filtering algorithm can be used to smooth the spectral data. Specifically, for the amplitude value at any frequency within a time frame, the amplitude values ​​corresponding to frequencies within a specified range (i.e., the smoothing radius) are added together, and then the average value is taken as the final amplitude value. The histogram corresponding to the smoothed spectral data is shown in Figure 5D.

[0082] After processing the spectral data, the resulting histogram effectively represents the rhythmic flow of the music and provides a good waveform effect. Therefore, the spectral data for each time frame is stored as a texture map, resulting in a time-spectral graph. Specifically, the coordinate axis U of the texture map is set as a time-series variable, and the coordinate axis v is set as a frequency variable. Each pixel in the texture map stores an amplitude value ranging from 0 to 1, representing the frequency amplitude value of V in the time frame U.

[0083] For example, if a time-series frame samples 512 spectral data points, then the texture map has a resolution of 512x512 and can store spectral data for 512 frames. Assuming the game scene runs at 30fps, a single texture map can store the spectral data corresponding to 17 seconds of audio data.

[0084] Step 2: Determine the audio column drawing area on the virtual model, and obtain the texture coordinates of the audio column drawing area based on the vertex coordinates of each target vertex in the audio column drawing area.

[0085] Among them, by defining the projection direction, position, and size of the projection bounding box, the audio column drawing area can be defined at any angle and position on the surface of the virtual model.

[0086] Specifically, the projection direction, position, and size of the bounding box are obtained; then, based on the projection direction and the bounding box position, the projection coordinate data of each vertex on the virtual model in the corresponding projection coordinate system of the bounding box are obtained; and based on the bounding box position, the bounding box size, and the projection coordinate data of each vertex, a target vertex within the bounding box is determined among the vertices; and the audio column drawing area of ​​the virtual model is determined based on the target vertex.

[0087] In practice, the projection direction, position, and size of the bounding box can be obtained using graphical tools (such as Gizmos or Substance Painter in Unity). See also Figure 6A Taking a ship model as an example, the ship model in the virtual scene can be displayed in the visualization interface. A cube is defined in the virtual scene, which represents the projection bounding box. By adjusting the position and size of the cube, it is placed within the desired audio bar drawing area. The size of the cube is adjusted to match the size of the desired audio bar drawing area, and the orientation of the cube is adjusted to match the projection direction of the audio bar drawing area. Figure 6A As shown, the X-axis of the cube represents the projection direction, and thus, based on the parameters of the cube, the projection direction, bounding box position, and bounding box size can be obtained. Based on... Figure 6A The projection bounding box shown can be used to define, for example, on the ship model. Figure 6B The area where the audio bars are drawn is shown.

[0088] Understandably, the texture coordinates (i.e., UV coordinates) of a ship model are often discontinuous, making it impossible to directly use them as the texture coordinates for the audio bar drawing area. Therefore, the texture coordinates for the audio bar drawing area can be obtained based on the projection coordinates of each vertex of the ship model in the projection coordinate system corresponding to the cube's bounding box. Figure 6A It can be seen that the Y-axis and Z-axis values ​​of each target vertex in the audio column drawing area on the ship model are continuous values ​​in the projection coordinate data of the cube corresponding to the projection coordinate system. Therefore, based on the Y-axis and Z-axis values ​​of each target vertex in the audio column drawing area on the ship model in the projection coordinate data of the cube corresponding to the projection coordinate system, the texture coordinates of the audio column drawing area can be obtained.

[0089] Step 4: Divide the audio bar drawing area to obtain multiple drawing areas corresponding to the audio bars, and obtain the spectral data corresponding to the audio bars in different time frames in the time spectrum diagram according to the texture coordinates of the audio bar drawing area.

[0090] The audio bar drawing area displays a preset number of audio bars. Within this area, the U-axis represents the horizontal direction of the audio bar image, corresponding to the spectral range in the aforementioned spectrum data's histogram. The V-axis represents the vertical direction of the audio bar image, corresponding to the height range of the bars in the aforementioned spectrum data. Therefore, the audio bar drawing area is first divided along the U-axis direction of the texture coordinates to obtain drawing areas corresponding to multiple audio bars. Simultaneously, the texture coordinate values ​​U in the texture coordinates are segmented to obtain the drawing area corresponding to each audio bar and the texture coordinate value U corresponding to each drawing area. Then, spectral data is sampled from the time-frequency spectrum based on the texture coordinate values ​​U of each drawing area. It can be understood that one texture coordinate value U corresponds to one frequency in the spectrum data, and the amplitude value of the corresponding frequency is sampled from the time-frequency spectrum based on the texture coordinate values ​​U of each drawing area.

[0091] Step 5: Render the drawing area corresponding to the audio column based on the spectral data of the audio column under different time frames to obtain the audio column effect diagram corresponding to the audio data.

[0092] After obtaining the spectral data of the audio bars, the amplitude values ​​in the spectral data can be converted into the height values ​​of the audio bars. Then, based on the height values, a truncation operation can be performed on the drawing area corresponding to the audio bars, thereby controlling the drawing height of the audio bars. See also... Figure 4A As shown, Figure 4A This is a schematic diagram of the audio column effect.

[0093] Furthermore, the audio bars can be displayed as audio grids. Specifically, the drawing area of ​​the audio bars is divided along the U-axis and V-axis to obtain the grid regions corresponding to the audio grids, and the grid texture coordinates of each grid region are redefined. After obtaining the spectral data of the audio bars, the amplitude values ​​in the spectral data can be converted into the height values ​​of the audio bars. Then, based on the height values, it can be determined whether the audio grids on the audio bars are rendered to identify the target audio grids to be rendered. Then, based on the grid texture coordinates of each target audio grid, the texture pattern to be rendered is obtained from the audio grid texture map, and the texture pattern is rendered in the grid region corresponding to the target audio grid.

[0094] To better implement the image rendering method based on audio data provided in the embodiments of this application, this application also provides an image rendering apparatus based on audio data, such as... Figure 7 As shown, the image rendering apparatus 700 based on audio data includes:

[0095] The time-spectrum acquisition module 710 is used to acquire the time-spectrum of audio data;

[0096] The drawing area acquisition module 720 is used to determine the audio column drawing area on the virtual model, and to obtain the texture coordinates of the audio column drawing area according to the vertex coordinates of each target vertex in the audio column drawing area;

[0097] The spectrum data sampling module 730 is used to divide the audio bar drawing area into multiple drawing areas corresponding to the audio bars, and to obtain the spectrum data corresponding to the audio bars in different time frames in the time spectrum diagram according to the texture coordinates of the audio bar drawing area.

[0098] The rendering module 740 is used to render the drawing area corresponding to the audio column based on the spectrum data corresponding to the audio column under different time frames, so as to obtain the audio column rendering corresponding to the audio data.

[0099] In some embodiments of this application, the drawing region acquisition module is specifically used to acquire the projection direction, bounding box position, and bounding box size of the projection bounding box; acquire the projection coordinate data of each vertex on the virtual model in the projection coordinate system corresponding to the projection bounding box based on the projection direction and the bounding box position; determine the target vertex within the projection bounding box among the vertices according to the bounding box position, bounding box size, and projection coordinate data of each vertex; and determine the audio column drawing region of the virtual model based on the target vertex.

[0100] In some embodiments of this application, the drawing region acquisition module is specifically used to convert the vertex coordinates of the target vertex into target projection coordinate data in the projection coordinate system corresponding to the projection bounding box based on the projection direction and the bounding box position; normalize the target projection coordinate data of the target vertex according to the size of the bounding box, and determine the normalized target texture mapping coordinates of the target vertex as the texture coordinates of the audio column drawing region.

[0101] In some embodiments of this application, the spectrum data sampling module is specifically used to divide the audio bar drawing area in a first direction based on a preset number of audio bars to obtain the drawing area corresponding to each audio bar; obtain the first texture coordinate value of the drawing sub-area corresponding to each audio bar in the first direction according to the texture coordinates of the audio bar drawing area; and obtain the spectrum data of each audio bar in different time frames in the time spectrum diagram according to the first texture coordinate value of the drawing sub-area corresponding to each audio bar.

[0102] In some embodiments of this application, the rendering module is used to obtain the height value corresponding to the audio column based on the spectrum data of the audio column in the target time frame in different time frames; and to render the drawing area corresponding to the audio column based on the height value of the audio column to obtain the audio column rendering corresponding to the audio data.

[0103] In some embodiments of this application, the rendering module is used to divide the drawing area corresponding to the audio column to obtain the grid area corresponding to the audio grid on the audio column; generate the grid texture coordinates corresponding to the grid area for the audio grid on the audio column; determine the target audio grid to be rendered on the audio column based on the spectrum data of the audio column in the target time frame for the target time frame; and sample from a preset audio grid texture map based on the grid texture coordinates of the target audio grid to generate the audio column rendering image corresponding to the audio data.

[0104] In some embodiments of this application, the time-spectrum acquisition module is used to sample audio data based on a preset sampling window to obtain audio data segments corresponding to multiple time frames; perform Fourier transform on the audio data segments corresponding to each time frame to obtain spectrum data corresponding to multiple time frames; wherein, the spectrum data includes amplitude values ​​at different frequencies; normalize the amplitude values ​​at different frequencies in each time frame; and save the spectrum data of the audio data in multiple time frames using the frame number of the time frame as a first variable, the frequency values ​​of each frequency in the same time frame as a second variable, and the amplitude values ​​at different frequencies in different time frames as pixel data to obtain a time-spectrum.

[0105] In some embodiments of this application, the time-spectrum acquisition module is further configured to, for any target time frame in the different time frames, adjust the amplitude value of the target time frame at different frequencies according to the amplitude value of the corresponding reference time frame at the same frequency; and / or smooth the amplitude values ​​of the target time frame at different frequencies to obtain the adjusted amplitude values ​​of the target time frame at different frequencies.

[0106] Specific limitations regarding the audio data-based image rendering apparatus can be found in the above description of the audio data-based image rendering method, and will not be repeated here. Each module in the aforementioned audio data-based image rendering apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0107] In some embodiments of this application, the image rendering apparatus 700 based on audio data can be implemented as a computer program, and the computer program can be implemented in, for example, Figure 8 It runs on the computer device shown. The computer device's memory can store the various program modules that make up the audio data-based image rendering apparatus 700, for example, Figure 7 The diagram shows a time-spectrum acquisition module 710, a drawing area acquisition module 720, a spectrum data sampling module 730, and an effect drawing module 740. The computer program comprised of these modules causes the processor to execute the steps in the audio data-based image rendering methods of the various embodiments of this application described in this specification.

[0108] For example, Figure 8 The computer equipment shown can be used as follows Figure 7 The time-spectrum acquisition module 710 in the audio data-based image rendering apparatus 700 executes step S110. The computer device can execute step S120 via the drawing area acquisition module 720. The computer device can execute step S130 via the spectrum data sampling module 730. The computer device can execute step S140 via the rendering module 740. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with external computer devices via a network connection. When the computer program is executed by the processor, it implements an audio data-based image rendering method.

[0109] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0110] In some embodiments of this application, a computer device is provided, including one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor as follows:

[0111] Obtain the time-spectrum graph of the audio data;

[0112] Determine the audio column drawing area on the virtual model, and obtain the texture coordinates of the audio column drawing area based on the vertex coordinates of each target vertex in the audio column drawing area;

[0113] The audio bar drawing area is divided into multiple drawing areas corresponding to the audio bars, and the spectral data corresponding to the audio bars under different time frames is obtained in the time-spectrum graph according to the texture coordinates of the audio bar drawing area.

[0114] Based on the spectral data corresponding to the audio bars in different time frames, the drawing area corresponding to the audio bars is rendered to obtain the audio bar effect diagram corresponding to the audio data.

[0115] In some embodiments of this application, when the processor executes the computer program, it further performs the following steps: obtaining the projection direction, bounding box position, and bounding box size of the projection bounding box; obtaining the projection coordinate data of each vertex on the virtual model in the projection coordinate system corresponding to the projection bounding box based on the projection direction and the bounding box position; determining a target vertex within the projection bounding box among the vertices according to the bounding box position, the bounding box size, and the projection coordinate data of each vertex; and determining the audio column drawing area of ​​the virtual model based on the target vertex.

[0116] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: converting the vertex coordinates of the target vertex into target projection coordinate data in the projection coordinate system corresponding to the projection bounding box based on the projection direction and the bounding box position; normalizing the target projection coordinate data of the target vertex according to the size of the bounding box, and determining the normalized target texture mapping coordinates of the target vertex as the texture coordinates of the audio column drawing area.

[0117] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: dividing the audio bar drawing area in a first direction based on a preset number of audio bars to obtain a drawing area corresponding to each audio bar; obtaining a first texture coordinate value of the drawing sub-area corresponding to each audio bar in the first direction based on the texture coordinates of the audio bar drawing area; and obtaining the spectrum data of each audio bar in different time frames in the time spectrum diagram based on the first texture coordinate value of the drawing sub-area corresponding to each audio bar.

[0118] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: for a target time frame in different time frames, obtain the height value corresponding to the audio column based on the spectrum data of the audio column in the target time frame; and render the drawing area corresponding to the audio column based on the height value corresponding to the audio column to obtain the audio column effect diagram corresponding to the audio data.

[0119] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: dividing the drawing area corresponding to the audio column to obtain the grid area corresponding to the audio grid on the audio column; generating the grid texture coordinates corresponding to the grid area for the audio grid on the audio column; determining the target audio grid to be rendered on the audio column based on the spectrum data of the audio column in the target time frame for the target time frame; and sampling from a preset audio grid texture map based on the grid texture coordinates of the target audio grid to generate the audio column effect map corresponding to the audio data.

[0120] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: sampling audio data based on a preset sampling window to obtain audio data segments corresponding to multiple time frames; performing Fourier transform on the audio data segments corresponding to each time frame to obtain spectrum data corresponding to multiple time frames; wherein, the spectrum data includes amplitude values ​​at different frequencies; normalizing the amplitude values ​​at different frequencies in each time frame; and saving the spectrum data of the audio data in multiple time frames using the frame number of the time frame as a first variable, the frequency values ​​of each frequency in the same time frame as a second variable, and the amplitude values ​​at different frequencies in different time frames as pixel data to obtain a time-spectrum map.

[0121] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: for any target time frame among the different time frames, adjust the amplitude value of the target time frame at different frequencies according to the amplitude value of the corresponding reference time frame at the same frequency; and / or smooth the amplitude values ​​of the target time frame at different frequencies to obtain the adjusted amplitude values ​​of the target time frame at different frequencies.

[0122] In some embodiments of this application, a computer-readable storage medium is provided, storing a computer program that is loaded by a processor, causing the processor to perform the following steps:

[0123] Obtain the time-spectrum graph of the audio data;

[0124] Determine the audio column drawing area on the virtual model, and obtain the texture coordinates of the audio column drawing area based on the vertex coordinates of each target vertex in the audio column drawing area;

[0125] The audio bar drawing area is divided into multiple drawing areas corresponding to the audio bars, and the spectral data corresponding to the audio bars under different time frames is obtained in the time-spectrum graph according to the texture coordinates of the audio bar drawing area.

[0126] Based on the spectral data corresponding to the audio bars in different time frames, the drawing area corresponding to the audio bars is rendered to obtain the audio bar effect diagram corresponding to the audio data.

[0127] In some embodiments of this application, when the computer program is executed by the processor, it further performs the following steps: obtaining the projection direction, bounding box position, and bounding box size of the projection bounding box; obtaining the projection coordinate data of each vertex on the virtual model in the projection coordinate system corresponding to the projection bounding box based on the projection direction and the bounding box position; determining the target vertex within the projection bounding box among the vertices according to the bounding box position, the bounding box size, and the projection coordinate data of each vertex; and determining the audio column drawing area of ​​the virtual model based on the target vertex.

[0128] In some embodiments of this application, when the computer program is executed by the processor, it further implements the following steps: converting the vertex coordinates of the target vertex into target projection coordinate data in the projection coordinate system corresponding to the projection bounding box based on the projection direction and the bounding box position; normalizing the target projection coordinate data of the target vertex according to the size of the bounding box, and determining the normalized target texture mapping coordinates of the target vertex as the texture coordinates of the audio column drawing area.

[0129] In some embodiments of this application, when the computer program is executed by the processor, it further implements the following steps: dividing the audio column drawing area in a first direction based on a preset number of audio columns to obtain a drawing area corresponding to each audio column; obtaining a first texture coordinate value of the drawing sub-area corresponding to each audio column in the first direction based on the texture coordinates of the audio column drawing area; and obtaining the spectral data of each audio column in different time frames in the time spectrum based on the first texture coordinate value of the drawing sub-area corresponding to each audio column.

[0130] In some embodiments of this application, when the computer program is executed by the processor, it further implements the following steps: for a target time frame in different time frames, obtain the corresponding height value of the audio column based on the spectrum data of the audio column in the target time frame; and render the drawing area corresponding to the audio column based on the corresponding height value of the audio column to obtain the audio column effect diagram corresponding to the audio data.

[0131] In some embodiments of this application, when the computer program is executed by the processor, it further implements the following steps: dividing the drawing area corresponding to the audio column to obtain the grid area corresponding to the audio grid on the audio column; generating the grid texture coordinates corresponding to the grid area for the audio grid on the audio column; determining the target audio grid to be rendered on the audio column based on the spectrum data of the audio column in the target time frame for the target time frame in different time frames; sampling from a preset audio grid texture map based on the grid texture coordinates of the target audio grid to generate the audio column effect map corresponding to the audio data.

[0132] In some embodiments of this application, when the computer program is executed by the processor, it further implements the following steps: sampling the audio data based on a preset sampling window to obtain audio data segments corresponding to multiple time frames; performing Fourier transform on the audio data segments corresponding to each time frame to obtain spectrum data corresponding to multiple time frames; wherein, the spectrum data includes amplitude values ​​at different frequencies; normalizing the amplitude values ​​at different frequencies in each time frame; and saving the spectrum data of the audio data in multiple time frames using the frame number of the time frame as a first variable, the frequency values ​​of each frequency in the same time frame as a second variable, and the amplitude values ​​at different frequencies in different time frames as pixel data to obtain a time-spectrum map.

[0133] In some embodiments of this application, when the computer program is executed by the processor, it further implements the following steps: for any target time frame among the different time frames, adjust the amplitude value of the target time frame at different frequencies according to the amplitude value of the corresponding reference time frame at the same frequency; and / or smooth the amplitude values ​​of the target time frame at different frequencies to obtain the adjusted amplitude values ​​of the target time frame at different frequencies.

[0134] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The foregoing has provided a detailed description of an image rendering method, apparatus, computer device, and storage medium based on audio data provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image rendering method based on audio data, characterized in that, The method includes: Obtain the time-spectrum graph of the audio data; Determine the audio column drawing area on the virtual model, and obtain the texture coordinates of the audio column drawing area based on the vertex coordinates of each target vertex in the audio column drawing area; The audio bar drawing area is divided into multiple drawing areas corresponding to the audio bars, and the spectral data corresponding to the audio bars under different time frames is obtained in the time-spectrum graph according to the texture coordinates of the audio bar drawing area. Based on the spectral data corresponding to the audio bars under different time frames, the drawing area corresponding to the audio bars is rendered to obtain the audio bar effect diagram corresponding to the audio data; The step of dividing the audio bar drawing area into multiple drawing areas corresponding to audio bars, and obtaining the spectral data of the audio bars in different time frames in the time-spectrum graph based on the texture coordinates of the audio bar drawing areas, includes: dividing the audio bar drawing area in a first direction based on a preset number of audio bars to obtain drawing areas corresponding to each audio bar; obtaining the first texture coordinate value of the drawing sub-region corresponding to each audio bar in the first direction based on the texture coordinates of the audio bar drawing areas; and obtaining the spectral data of each audio bar in different time frames in the time-spectrum graph based on the first texture coordinate value of the drawing sub-region corresponding to each audio bar.

2. The method according to claim 1, characterized in that, The step of determining the audio column drawing area on the virtual model includes: Obtain the projection direction, position, and size of the projection bounding box; Based on the projection direction and the bounding box position, obtain the projection coordinate data of each vertex on the virtual model in the projection coordinate system corresponding to the projection bounding box; Based on the bounding box position, the bounding box size, and the projection coordinate data of each vertex, the target vertex within the projection bounding box is determined among each vertex; The audio column drawing area of ​​the virtual model is determined based on the target vertex.

3. The method according to claim 2, characterized in that, The step of obtaining the texture coordinates of the audio column drawing area based on the vertex coordinates of each target vertex in the audio column drawing area includes: Based on the projection direction and the bounding box position, the vertex coordinates of the target vertex are converted into target projection coordinate data in the projection coordinate system corresponding to the projection bounding box; The target projection coordinate data of the target vertex is normalized according to the bounding box size, and the normalized target texture mapping coordinates of the target vertex are determined as the texture coordinates of the audio column drawing area.

4. The method according to claim 1, characterized in that, The step of rendering the drawing area corresponding to the audio bar based on the spectral data corresponding to the audio bar under different time frames to obtain the audio bar effect diagram corresponding to the audio data includes: For a target time frame in different time frames, the height value of the audio column is obtained according to the spectral data of the audio column under the target time frame; Based on the height value corresponding to the audio column, the drawing area corresponding to the audio column is rendered to obtain the audio column effect diagram corresponding to the audio data.

5. The method according to claim 1, characterized in that, The step of rendering the drawing area corresponding to the audio bar based on the spectral data corresponding to the audio bar under different time frames to obtain the audio bar effect diagram corresponding to the audio data further includes: The drawing area corresponding to the audio column is divided to obtain the grid area corresponding to the audio grid on the audio column; For the grid region corresponding to the audio grid on the audio column, generate the grid texture coordinates corresponding to the grid region; For target time frames in different time frames, the target audio grid to be rendered on the audio column is determined based on the spectral data of the audio column under the target time frame; Based on the grid texture coordinates of the target audio grid, samples are taken from a preset audio grid texture to generate an audio bar effect image corresponding to the audio data.

6. The method according to claim 1, characterized in that, The step of obtaining the time-spectrum graph of the audio data includes: The audio data is sampled and processed based on a preset sampling window to obtain audio data segments corresponding to multiple time frames. Perform a Fourier transform on the audio data segments corresponding to each of the aforementioned time frames to obtain spectral data corresponding to multiple time frames; wherein, the spectral data includes amplitude values ​​at different frequencies; The amplitude values ​​at different frequencies in each of the aforementioned time frames are normalized. Using the frame number of the time sequence frame as the first variable, the frequency values ​​of each frequency within the same time sequence frame as the second variable, and the amplitude values ​​at different frequencies in different time sequence frames as pixel data, the spectral data of the audio data in multiple time sequence frames are saved to obtain a time-spectrum diagram.

7. The method according to claim 6, characterized in that, Before the step of saving the spectral data of the audio data in multiple time frames using the frame number of the time frame as the first variable, the frequency values ​​of each frequency within the same time frame as the second variable, and the amplitude values ​​at different frequencies in different time frames as pixel data to obtain the time-spectrum map, the method further includes: For any target time frame in the different time frames, adjust the amplitude value of the target time frame at different frequencies according to the amplitude value of the corresponding reference time frame at the same frequency; and / or The amplitude values ​​at different frequencies in the target time frame are smoothed to obtain the adjusted amplitude values ​​at different frequencies of the target time frame.

8. An image rendering apparatus based on audio data, characterized in that, The device includes: The time-spectrum acquisition module is used to acquire the time-spectrum of audio data; The drawing area acquisition module is used to determine the audio column drawing area on the virtual model, and to obtain the texture coordinates of the audio column drawing area based on the vertex coordinates of each target vertex in the audio column drawing area. The spectrum data sampling module is used to divide the audio bar drawing area into multiple drawing areas corresponding to the audio bars, and to obtain the spectrum data corresponding to the audio bars in different time frames in the time spectrum diagram according to the texture coordinates of the audio bar drawing area. The rendering module is used to render the drawing area corresponding to the audio bar based on the spectrum data corresponding to the audio bar under different time frames, so as to obtain the audio bar rendering corresponding to the audio data. The spectrum data sampling module is further configured to divide the audio bar drawing area in a first direction based on a preset number of audio bars to obtain the drawing area corresponding to each audio bar; obtain the first texture coordinate value of the drawing sub-area corresponding to each audio bar in the first direction based on the texture coordinates of the audio bar drawing area; and obtain the spectrum data of each audio bar in different time frames in the time spectrum diagram based on the first texture coordinate value of the drawing sub-area corresponding to each audio bar.

9. A computer device, characterized in that, The computer device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the image rendering method based on audio data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps in the image rendering method based on audio data as described in any one of claims 1 to 7.

Images