Audio interface rendering method and apparatus, device, and storage medium

Abstract

Embodiments of the present disclosure provide an audio interface rendering method and apparatus, a device, and a storage medium. The method comprises: acquiring an audio playback progress; acquiring an interface object texture on the basis of the audio playback progress; and rendering the interface object texture into an audio interface for display. In the audio interface rendering method provided by the embodiments of the present disclosure, the interface object texture acquired on the basis of the audio playback progress is rendered into the audio interface for display, so that the performance consumption when the audio interface is rendered can be reduced, and the rendering speed is improved.

Description

Audio interface rendering method, device, equipment and storage medium

[0001] This application claims priority to the Chinese invention patent application entitled “Rendering method, device, equipment and storage medium for audio interface” filed on October 20, 2023, with application number 202311370438.4, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present disclosure relate to the field of interface rendering technology, and in particular to a method, apparatus, device, and storage medium for rendering an audio interface. Background Art

[0003] Mobile terminals have become an important tool for users to engage in entertainment activities. Singing songs using mobile terminals is one such use case. When a user sings a song through an application on a mobile terminal, the application's interface displays lyrics and pitch information as the song progresses. In the prior art, rendering lyrics and pitch information is primarily based on a central processing unit (CPU) plus memory approach. This approach consumes a lot of CPU performance and can cause the terminal to freeze, thus affecting the user experience.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure provide a method, apparatus, device, and storage medium for rendering an audio interface, which can reduce performance consumption when rendering an audio interface and improve rendering speed.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for rendering an audio interface, comprising: obtaining an audio playback progress; obtaining an interface object texture according to the audio playback progress; and rendering the interface object texture into the audio interface for display.

[0007] In the second aspect, the embodiment of the present disclosure also provides a rendering device for an audio interface, which is characterized in that it includes: an audio playback progress acquisition module for acquiring the audio playback progress; an interface object texture acquisition module for acquiring the interface object texture according to the audio playback progress; and a rendering module for rendering the interface object texture into the audio interface for display.

[0008] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for rendering the audio interface as described in the embodiment of the present disclosure.

[0009] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the method for rendering an audio interface as described in an embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0011] FIG1 is a flow chart of a method for rendering an audio interface provided by an embodiment of the present disclosure;

[0012] FIG2 is an example diagram of cutting text texture provided by an embodiment of the present disclosure;

[0013] FIG3 is an example diagram of a first text texture provided by an embodiment of the present disclosure;

[0014] FIG4 is an example diagram of a text texture corresponding to a target text group provided by an embodiment of the present disclosure;

[0015] FIG5 is an example diagram of a pitch texture provided by an embodiment of the present disclosure;

[0016] FIG6 is an example diagram of a pitch texture provided by an embodiment of the present disclosure;

[0017] FIG7 is an exemplary diagram of a pitch evaluation texture provided by an embodiment of the present disclosure;

[0018] FIG8 is an example diagram of a pitch marker texture provided by an embodiment of the present disclosure;

[0019] FIG9 is an example diagram of an audio interface provided by an embodiment of the present disclosure;

[0020] FIG10 is a schematic structural diagram of a rendering device for an audio interface provided by an embodiment of the present disclosure;

[0021] FIG11 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0024] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0029] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0030] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0033] The present disclosure discloses a method, apparatus, device, and storage medium for rendering an audio interface. The method comprises obtaining audio playback progress; obtaining interface object textures based on the audio playback progress; and rendering the interface object textures onto the audio interface for display. The present disclosure provides an audio interface rendering method that renders interface object textures obtained based on the audio playback progress onto the audio interface for display, thereby reducing performance consumption when rendering the audio interface and increasing rendering speed.

[0034] Figure 1 is a flow chart of a method for rendering an audio interface provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the situation of rendering an audio interface. The method can be executed by a rendering device for an audio interface, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0035] As shown in FIG1 , the method includes: S110 , obtaining audio playback progress.

[0036] In some embodiments, the audio can be pre-generated or recorded audio, such as a song. The audio playback progress can be understood as the time point at which the audio is currently played or the time point at which the audio is currently adjusted by the user. For example, assuming that the total length of an audio is 3 minutes, and assuming that the audio is currently played to 1 minute and 20 seconds, the audio playback progress is 1 minute and 20 seconds. In this embodiment, for any audio playback tool, the underlying progress component can be called to obtain the audio playback progress, and the method for obtaining the audio playback progress is not limited here.

[0037] S120: Acquire the interface object texture according to the audio playback progress.

[0038] In some embodiments, the interface object texture includes at least one of the following: a text texture, a pitch texture, and a pitch texture. The interface object may be an object corresponding to the currently playing audio displayed in the audio interface, and may include at least one of text information, pitch information, and pitch information. The audio may include background audio and standard voice audio. Assuming the audio is a song, the background audio is the accompaniment, and the standard voice audio is the singing audio. The text information is the text corresponding to the standard voice audio, i.e., the lyrics; the pitch information is the height of the standard voice audio; and the pitch information represents the accuracy of the pitch of the collected voice audio.

[0039] In this application scenario, the user sings the corresponding song along with the playing audio, and the terminal will collect the user's voice audio in real time. As the audio is played and the user sings, for each frame displayed on the audio interface, the interface object to be displayed in the current frame is obtained, that is, the interface object texture corresponding to the audio playback progress is obtained.

[0040] Optionally, if the interface object texture is a text texture, the method for obtaining the interface object texture according to the audio playback progress can be: determining the target text group from the text file according to the audio playback progress; obtaining the first text texture corresponding to the first text group and the second text texture corresponding to the second text group.

[0041] In some embodiments, a text file includes multiple text groups, each text group includes multiple characters, and the characters carry a timestamp; the target text group includes a first text group currently being played and a set number of second text groups that have been played and / or not played. The text file can be understood as a lyrics file corresponding to the audio. In this embodiment, due to the characteristic of songs being sung sentence by sentence, the text will be processed into sentences, and one sentence of text corresponds to one text group. The timestamp carried by the text includes two pieces of information: the starting time and the duration of the text, which can be expressed as follows: "text <starting time, duration>". The first text group currently being played can be the text group containing the characters whose timestamp matches the audio playback progress. The timestamp matching the audio playback progress can be understood as the moment corresponding to the audio playback progress falling within the timestamp corresponding to the text. The set number can be pre-set and is not limited here.

[0042] Specifically, the method of determining the target text group from the text file according to the audio playback progress can be: obtaining the text group containing the text whose timestamp matches the audio playback progress in the text file as the first target text group; obtaining a set number of text groups before and / or after the first target text group as the second target text group. In this embodiment, since the text carries a timestamp, the text group composed of the text corresponds to a time period, and the multiple text groups contained in the text file are automatically sorted according to time. After the target text group is determined, the corresponding text texture can be obtained according to the target text group. In this embodiment, determining the text texture based on the text group can improve the accuracy of text rendering.

[0043] Optionally, the method for obtaining the first text texture corresponding to the first text group may be: obtaining the text texture of the first display style and the text texture of the second display style corresponding to the first text group; using the text whose timestamp matches the audio playback progress as the cropping point; cropping the text texture of the first display style and / or the text texture of the second display style based on the cropping point; generating the first text texture corresponding to the first text group based on the cropped text texture of the first display style and the text texture of the second display style.

[0044] In some embodiments, the first display style and the second display style are different. The display style can be color or transparency. In this application scenario, in order to highlight the playback progress of the audio, the unplayed text and the played text can be rendered in different display styles. When the audio plays to a text in the middle of the first target text group, the text texture corresponding to the first target text group needs to be divided into two display styles before and after. The first display style can be used to represent played, and the second display style can be used to represent unplayed; or, the first display style can be used to represent unplayed, and the second display style can be used to represent played, which is not limited here. Specifically, the method for obtaining the text texture of the first display style and the text texture of the second display style corresponding to the first text group can be: obtaining the text texture of the first display style and the text texture of the second display style corresponding to the first text group from a pre-created buffer area. In this embodiment, the text of the first display style and the second display style will be drawn in advance for each text group in the text file, thereby generating the text texture of the first display style and the text texture of the second display style for each text group, and storing the text texture of the first display style and the text texture of the second display style in the texture buffer area. The final text texture is generated by cropping the texts of the two display styles to highlight the audio playback progress, thereby enriching the display effect of the text.

[0045] Specifically, assuming that the first display style indicates that the video has been played, and the second display style indicates that the video has not been played, the text texture of the first display style may be processed based on the cropping point by cropping the text texture after the cropping point. The text texture of the second display style may be processed based on the cropping point by cropping the text texture before the cropping point.

[0046] In this embodiment, the first display style represents that the content has been played, and the portion of the text texture corresponding to the text that has not been played in the text texture of the first display style is cropped, and the portion of the text texture corresponding to the text that has been played in the text texture of the first display style is retained. The second display style represents that the content has not been played, and the portion of the text texture corresponding to the text that has been played in the text texture of the second display style is cropped, and the portion of the text texture corresponding to the text that has not been played in the text texture of the second display style is retained. For example, Figure 2 is an example diagram of cropping text textures in an embodiment. As shown in Figure 2, the first display style is gray and the second display style is white. Assuming that "A" is the cropping point, the texture after A in the gray text texture is cropped, and the texture before A in the white text texture is cropped. The black part in the figure represents the cropped part. In this embodiment, cropping the text textures of the two display styles based on the cropping point can improve the accuracy of cropping.

[0047] Optionally, the method for generating the first text texture corresponding to the first text group based on the cropped text texture of the first display style and the text texture of the second display style may be: if the text texture of the first display style and the text texture of the second display style are both cropped based on the cropping point, the cropped text texture of the first display style and the cropped text texture of the second display style are spliced ​​to obtain the first text texture corresponding to the first text group; if the text texture of the first display style is cropped based on the cropping point and the text texture of the second display style is not cropped, the cropped text texture of the first display style is superimposed on the corresponding part of the uncropped text texture of the second display style to obtain the first text texture corresponding to the first text group.

[0048] Specifically, if the text texture of the first display style and the text texture of the second display style are both cropped based on the cropping point, the cropped text texture of the first display style and the cropped text texture of the second display style are spliced ​​at the original cropping point. If the text texture of the first display style is cropped based on the cropping point, and the text texture of the second display style is not cropped, the cropped text texture of the first display style is aligned with the uncropped text texture of the second display style based on the cropping point and then superimposed. For example, Figure 3 is an example diagram of the first text texture in this embodiment. As shown in Figure 3, the first text texture is displayed in grayscale before the cropping point A and in white after A. In this embodiment, it is ensured that the first text texture is displayed in the first display style before the cropping point and in the second display style after the cropping point, highlighting the audio playback progress.

[0049] Optionally, the second text texture corresponding to the second text group may be obtained by obtaining a first display style text texture corresponding to the played second text group and / or a second display style text texture corresponding to the unplayed second text group.

[0050] In some embodiments, the second text group is a text group within a certain distance from the first text group. Exemplarily, assuming that the set number is 8, the first text group can be composed of the first three and last five text groups of the first text group. The first display style represents that it has been played, and the second display style represents that it has not been played. Therefore, the first display style text texture corresponding to the played second text group and / or the second display style text texture of the unplayed second text group are obtained. Exemplarily, Figure 4 is an example diagram of the text texture corresponding to the target text group in this embodiment. As shown in Figure 4, text texture 1 and text texture 2 are text textures corresponding to the played text groups, text texture 3 is the text texture corresponding to the text group currently being played, and text texture 4, text texture 5 and text texture 6 are text textures corresponding to the unplayed text groups.

[0051] Optionally, if the interface object texture is a pitch texture, the method for obtaining the interface object texture according to the audio playback progress may be: determining a time segment according to the audio playback progress; and obtaining the pitch texture whose timestamp falls within the time segment.

[0052] In some embodiments, the timestamp corresponding to the audio playback progress is in a time segment. The time segment is a period of time in the playback duration corresponding to the audio. The time segment consists of a first sub-segment before the audio playback progress and a second sub-segment after the audio playback progress. The length of the time segment can be set based on the size (width or length) of the visible area of ​​the screen of the current terminal device, and the ratio of the first sub-segment to the second sub-segment can be pre-configured. Exemplarily, the time segment is set to 10 seconds, the first sub-segment and the second sub-segment are 2:8, and the time segment consists of 2 seconds before the audio playback progress and 8 seconds after the audio playback progress. Assuming that the audio playback progress is 1 minute and 30 seconds, the time segment corresponding to the audio playback progress is: 1 minute 28 seconds to 1 minute 38 seconds.

[0053] In some embodiments, a pitch texture is pre-created based on the audio and carries a timestamp and standard pitch information. The timestamp and standard pitch information can be used to determine the location of the pitch texture in the audio interface. The pitch texture can be a long strip of a predetermined shape, which can be a rectangle, a flat ellipse, or an irregular shape. An irregular shape can be formed by two straight lines and two circular arcs. Standard pitch information is represented by a quantized value. The timestamp carried by the pitch texture can include a start time and a duration. A single pitch information can carry multiple different timestamps because different timestamps in an audio segment may correspond to the same pitch. Standard pitch information can be understood as the pitch of the speech audio in the audio data. The information carried by the pitch texture can be represented as follows: "Pitch a: <start time 1, duration 1>, <start time 2, duration 2>," meaning that pitch a carries two timestamps. Specifically, the process of creating a pitch texture can be: extracting the standard pitch information contained in the audio, generating a pitch texture of a set shape for the same standard pitch information, determining the timestamp corresponding to the pitch texture, carrying the standard pitch information and the timestamp in the pitch texture, and storing the created pitch texture in the texture cache.

[0054] In this embodiment, after determining the time segment corresponding to the audio playback progress, the pitch texture whose timestamp falls within the time segment is obtained. For example, Figure 5 is an example diagram of pitch texture in an embodiment of the present invention. As shown in Figure 5, the pitch textures falling within the time segment corresponding to the audio playback progress include: pitch texture 1, pitch texture 2, pitch texture 3, pitch texture 4 and pitch texture 5, wherein pitch texture 1 and pitch texture 2 are pitch textures before the audio playback progress, and pitch texture 3, pitch texture 4 and pitch texture 5 are pitch textures after the audio playback progress. In this embodiment, determining the pitch texture based on the time segment corresponding to the audio playback progress can improve accuracy.

[0055] Optionally, if the interface object texture is a pitch texture, the method of obtaining the interface object texture according to the audio playback progress may be: obtaining pitch evaluation information corresponding to the audio playback progress; and generating at least one pitch texture according to the pitch evaluation information.

[0056] In some embodiments, the pitch evaluation information represents the pitch evaluation information of the collected voice audio frame; the pitch texture carries a timestamp and standard pitch information. The form in which the timestamp is carried by the pitch texture is similar to that of the pitch texture and will not be repeated here. According to the timestamp and standard pitch information of the pitch texture, the position information of the pitch texture in the audio interface can be determined. In this application scenario, the user sings the corresponding song along with the background audio, the terminal collects the voice audio used in real time, and extracts the pitch information in the collected voice audio, compares the pitch information with the standard pitch information, and obtains the pitch evaluation information.

[0057] Specifically, the method for obtaining the pitch evaluation information corresponding to the audio playback progress can be: obtaining the pitch information of the voice audio frames collected within a set historical time length; determining the deviation between the pitch information and the corresponding standard pitch information; and determining the pitch evaluation information based on the deviation.

[0058] In some embodiments, the pitch evaluation information includes: matching with standard pitch information and not matching with standard pitch information. The set historical duration can be a certain duration before the audio playback progress. The deviation between the pitch information and the corresponding standard pitch information can be characterized by the difference between the quantized value of the pitch information of the collected speech audio frame and the quantized value of the corresponding standard pitch information. Specifically, the method of determining the pitch evaluation information based on the deviation can be: if the deviation between the pitch information and the corresponding standard pitch information is less than or equal to the set threshold, the pitch information of the collected speech audio frame matches the standard pitch information; if the deviation between the pitch information and the corresponding standard pitch information is greater than the set threshold, the pitch information of the collected speech audio frame does not match the standard pitch information.

[0059] Specifically, the method of generating at least one pitch texture based on the pitch evaluation information can be: extracting the timestamp of the speech audio frame whose pitch evaluation information matches the standard pitch information as the pitch timestamp; generating at least one pitch texture based on the pitch timestamp.

[0060] In some embodiments, the pitch texture can be a long strip of a set shape, and the set shape can be a rectangle, a flat ellipse or an irregular shape. The irregular shape can be a figure surrounded by two straight lines and two circular arcs. In this embodiment, the pitch texture and the intonation texture adopt the same shape, the intonation texture overlaps with part of the pitch texture in the audio interface, and the intonation texture is above the pitch texture, and the display styles (such as color or transparency) of the two are different. The method of generating at least one intonation texture according to the intonation timestamp can be: the corresponding standard pitch information is the same and the continuous intonation timestamp is divided into an intonation segment, and an intonation texture is generated according to the intonation segment. Exemplarily, Figure 6 is an example diagram of the intonation texture in this embodiment. As shown in Figure 6, the black rectangular bar is the pitch texture, and the white rectangular bar is the intonation texture. Compared with Figure 5, a part of the white rectangular bar is covered on the black rectangular bar, indicating that the pitch information of the speech audio frame collected during this period matches the standard pitch information.

[0061] Optionally, the intonation texture also includes an intonation assessment texture, where the intonation assessment information is used to represent the pitch accuracy of the collected speech audio. The intonation assessment texture can be obtained based on the audio playback progress by: obtaining the intonation assessment texture based on the deviation between the pitch information and the corresponding standard pitch information.

[0062] In some embodiments, the pitch evaluation texture includes a pitch score evaluation texture and / or a pitch grade evaluation texture. The pitch score evaluation texture may be a texture corresponding to the pitch score evaluation information, and the pitch score evaluation information may be understood as a pitch score determined based on the deviation between the pitch information of the collected speech audio frame and the corresponding standard pitch information. The audio score may be a value between 0 and 100. In this embodiment, a correspondence between the deviation and the pitch score may be established in advance, and the pitch score may be determined based on the correspondence, and then a score evaluation texture may be generated based on the pitch score. The grade evaluation texture may be a texture corresponding to the grade evaluation information, and the grade evaluation information may be a plurality of pre-set evaluation grades, for example, three grades of excellent, good, and poor. The grade evaluation texture may be pre-created and stored in a texture cache. In this embodiment, a correspondence between the deviation and the evaluation grade may be established in advance, and the grade evaluation information may be determined based on the correspondence, and finally the grade evaluation texture may be obtained based on the grade evaluation information. Exemplarily, Figure 7 is an example of a pitch evaluation texture in this embodiment. As shown in Figure 7, the audio score determined based on the deviation between the pitch information of the collected speech audio frame and the corresponding standard pitch information is 98 points, and the grade evaluation information is excellent, so a pitch score evaluation texture corresponding to 98 points is generated, and a pitch grade evaluation texture corresponding to "excellent" is obtained from the texture cache.

[0063] S130: Render the interface object texture to the audio interface for display.

[0064] In this embodiment, after obtaining at least one interface object texture among text texture, pitch texture, intonation texture, pitch identification texture and intonation evaluation texture, these interface object textures are input and output from the shader, and the graphics processing unit (GPU) completes the rendering of all interface object textures at one time.

[0065] Specifically, the method of rendering the interface object texture to the audio interface for display may be: superimposing the interface object texture on a preset empty texture to obtain a target texture; and rendering the target texture to the audio interface for display.

[0066] In some embodiments, the preset empty texture may be a pre-created empty texture without pixel content that matches the size of the terminal screen. The process of overlaying the interface object texture on the preset empty texture may include performing an affine transformation on the interface object texture and overlaying the affine transformed interface object texture on the preset empty texture.

[0067] Optionally, if the interface object texture includes a pitch texture and an intonation texture, the method of superimposing the interface object texture on the preset empty texture can be: first superimpose the pitch texture on the preset empty texture, and then superimpose the intonation texture on the preset empty texture superimposed with the pitch texture.

[0068] In this embodiment, the pitch texture is superimposed first, and then the intonation texture is superimposed, which can ensure that when displayed, the intonation texture covers the pitch texture with the same timestamp as its timestamp, so that the user can intuitively see which part of the singing voice audio has pitch information that matches the standard pitch information.

[0069] Optionally, the process of rendering the interface object texture into the audio interface for display can be: obtaining the position information of the pixel point in the audio interface; judging whether the pixel point falls into the area corresponding to the interface object texture based on the position information; if the pixel point falls into the area corresponding to the interface object texture, rendering the pixel point as the color corresponding to the interface object texture; if the pixel point does not fall into the area corresponding to the interface object texture, rendering the pixel point as a transparent color.

[0070] In this embodiment, when rendering the audio interface, the GPU processes each pixel. Therefore, during the rendering process, it determines whether each pixel in the audio interface falls into the area corresponding to the interface object texture; if the pixel falls into the area corresponding to the interface object texture, the pixel is colored as the color corresponding to the interface object texture; if the pixel does not fall into the area corresponding to the interface object texture, the pixel is colored as a transparent color.

[0071] Optionally, the pitch texture also includes a pitch identifier texture, which is used to represent the pitch of the collected speech audio. The interface object texture can be overlaid onto a preset empty texture by: obtaining pitch information of the currently collected speech audio frame for the pitch identifier texture; determining the position information of the pitch identifier texture based on the pitch information; and overlaying the pitch identifier texture onto the preset empty texture based on the position information.

[0072] In some embodiments, a pitch marker texture is used to represent the pitch information of the currently collected speech audio frame. The pitch marker texture can be a pre-created texture of a set shape, such as a circle or a rectangle. The pitch information can be represented by a quantized value, and the position information of the pitch marker texture is determined based on the quantized pitch information. The position information is then affine transformed, and the pitch marker texture after the affine transformation is superimposed on the preset empty texture. For example, Figure 8 is an example diagram of the pitch marker texture in this embodiment. As shown in Figure 8, the pitch marker texture is a circle, which moves up and down on the line corresponding to the audio playback progress according to the pitch information.

[0073] Optionally, the method of rendering the interface object texture to the audio interface for display may be: for the level evaluation texture, determining the transparency of the level evaluation texture; and rendering the evaluation texture to the audio interface for display based on the transparency.

[0074] In some embodiments, a life cycle is set when the level assessment texture is displayed in the interface, for example, 2 seconds. During this life cycle, in order to make the level assessment texture present a gradient effect, the transparency of the level assessment texture can be adjusted, for example, the transparency of the level assessment texture changes in a certain manner (for example, first increasing and then decreasing, or gradually increasing, or gradually decreasing). In this embodiment, the way the transparency changes is not limited. Specifically, the transparency of the level assessment texture at the current moment is determined; based on the transparency, the assessment texture is rendered to the audio interface for display.

[0075] Optionally, the method further includes the following steps: creating a texture cache; generating a text texture and a pitch texture based on the audio; and storing the text texture and the pitch texture in the texture cache.

[0076] In some embodiments, the process of generating text texture and pitch texture based on audio can be referred to in the above embodiments and will not be repeated here. For example, FIG9 is an example diagram of an audio interface in an embodiment of the present disclosure. As shown in FIG9, the audio interface displays text, pitch information, and intonation information.

[0077] Based on the above embodiment, this embodiment is applied to the audio interface rendering scene when the user sings along with the background audio of the target song. When the user sings the song, the terminal device obtains the playback progress of the background audio of the target song in real time, and then obtains the lyrics texture, pitch texture and intonation texture corresponding to the voice audio of the target song according to the playback progress of the background audio. Finally, the lyrics texture, pitch texture and intonation texture are rendered to the audio interface for display.

[0078] In some embodiments, the voice audio is audio of singing the target song.

[0079] The technical solution of the disclosed embodiment obtains the audio playback progress; obtains the interface object texture based on the audio playback progress; and renders the interface object texture into the audio interface for display. The audio interface rendering method provided by the disclosed embodiment renders the interface object texture obtained based on the audio playback progress into the audio interface for display, which can reduce the performance consumption when rendering the audio interface and improve the rendering speed.

[0080] Figure 10 is a schematic structural diagram of a rendering device for an audio interface provided in an embodiment of the present disclosure. As shown in Figure 10, the device includes: an audio playback progress acquisition module 210, used to obtain the audio playback progress; an interface object texture acquisition module 220, used to obtain the interface object texture according to the audio playback progress; and a rendering module 230, used to render the interface object texture into the audio interface for display.

[0081] Optionally, the interface object texture includes at least one of the following: text texture, pitch texture and intonation texture; optionally, if the interface object texture is a text texture, the interface object texture acquisition module 220 is further used to: determine the target text group from the text file according to the audio playback progress; wherein the text file includes multiple text groups, the text group includes multiple characters, and the characters carry timestamps; the target text group includes a first text group currently being played and a set number of second text groups that have been played and / or not played; obtain a first text texture corresponding to the first text group and a second text texture corresponding to the second text group.

[0082] Optionally, the interface object texture acquisition module 220 is also used to: obtain the text texture of the first display style and the text texture of the second display style corresponding to the first text group; use the text whose timestamp matches the audio playback progress as the cropping point; crop the text texture of the first display style and / or the text texture of the second display style based on the cropping point; and generate the first text texture corresponding to the first text group based on the cropped text texture of the first display style and the text texture of the second display style.

[0083] Optionally, the interface object texture acquisition module 220 is also used to: crop the text texture of the first display style that is after the cropping point; and crop the text texture of the second display style based on the cropping point, including: cropping the text texture of the second display style that is before the cropping point.

[0084] Optionally, the interface object texture acquisition module 220 is also used to: if the text texture of the first display style and the text texture of the second display style are both cropped based on the cropping point, then the cropped text texture of the first display style and the cropped text texture of the second display style are spliced ​​to obtain the first text texture corresponding to the first text group; if the text texture of the first display style is cropped based on the cropping point and the text texture of the second display style is not cropped, then the cropped text texture of the first display style is superimposed on the corresponding part of the uncropped text texture of the second display style to obtain the first text texture corresponding to the first text group.

[0085] Optionally, the interface object texture acquisition module 220 is further configured to acquire a first display style text texture corresponding to the played second text group and / or a second display style text texture corresponding to the unplayed second text group.

[0086] Optionally, if the interface object texture is a pitch texture, the interface object texture acquisition module 220 is also used to: determine the time segment according to the audio playback progress; wherein the timestamp corresponding to the audio playback progress is in the time segment; obtain the pitch texture whose timestamp falls within the time segment; wherein the pitch texture is pre-created based on the audio and carries a timestamp and standard pitch information.

[0087] Optionally, if the interface object texture is a pitch texture, the interface object texture acquisition module 220 is also used to: obtain pitch evaluation information corresponding to the audio playback progress; wherein the pitch evaluation information represents the pitch evaluation information of the collected voice audio frame; generate at least one pitch texture based on the pitch evaluation information; wherein the pitch texture carries a timestamp and standard pitch information.

[0088] Optionally, the interface object texture acquisition module 220 is also used to: obtain the pitch information of the voice audio frames collected within a set historical time length; determine the deviation between the pitch information and the corresponding standard pitch information; determine the pitch evaluation information based on the deviation; wherein the pitch evaluation information includes: matching with the standard pitch information and not matching with the standard pitch information.

[0089] Optionally, the interface object texture acquisition module 220 is further used to: extract the timestamp of the speech audio frame whose pitch evaluation information matches the standard pitch information as the pitch timestamp; and generate at least one pitch texture according to the pitch timestamp.

[0090] Optionally, the rendering module 230 is further configured to: overlay the interface object texture onto a preset empty texture; and render the preset empty texture to the audio interface for display.

[0091] Optionally, the rendering module 230 is also used to: obtain the position information of the pixel point in the audio interface; determine whether the pixel point falls into the area corresponding to the interface object texture based on the position information; if the pixel point falls into the area corresponding to the interface object texture, render the pixel point as the color corresponding to the interface object texture; if the pixel point does not fall into the area corresponding to the interface object texture, render the pixel point as a transparent color.

[0092] Optionally, it also includes: a texture generation module, which is used to: create a texture buffer; generate text texture and pitch texture according to audio; and store the text texture and pitch texture in the texture buffer.

[0093] Optionally, the audio playback progress acquisition module 210 is also used to: obtain the playback progress of the background audio of the target song; wherein the target song is the song currently being sung; the interface object texture acquisition module 220 is also used to: obtain the lyrics texture, pitch texture and intonation texture corresponding to the voice audio of the target song according to the playback progress of the background audio; wherein the voice audio is the audio of singing the target song; the rendering module 230 is also used to: render the lyrics texture, pitch texture and intonation texture into the audio interface for display.

[0094] The rendering device of the audio interface provided by the embodiment of the present disclosure can execute the rendering method of the audio interface provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0095] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0096] Figure 11 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Referring to Figure 11 below, it shows a schematic diagram of the structure of an electronic device (such as the terminal device or server in Figure 11) 500 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in Figure 11 is only an example and should not bring any limitations to the functions and scope of use of the embodiments of the present disclosure.

[0097] As shown in FIG11 , the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0098] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG11 shows the electronic device 500 with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0099] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0100] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0101] The electronic device provided by the embodiment of the present disclosure and the rendering method of the audio interface provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0102] An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio interface rendering method provided in the above embodiment.

[0103] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0104] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0105] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0106] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0107] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device is enabled to: obtain the audio playback progress; obtain the interface object texture according to the audio playback progress; wherein the interface object texture includes at least one of the following: text texture, pitch texture, intonation texture, pitch identification texture and intonation evaluation texture; and render the interface object texture into the audio interface for display.

[0108] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0111] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0112] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0114] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0115] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for rendering an audio interface, comprising: Get the audio playback progress; Acquire the interface object texture according to the audio playback progress; as well as The interface object texture is rendered into the audio interface for display. 2 . The method according to claim 1 , wherein the interface object texture comprises at least one of the following: a text texture, a pitch texture, and an intonation texture.

3. The method according to claim 2, wherein if the interface object texture is a text texture, obtaining the interface object texture according to the audio playback progress comprises: Determining a target text group from a text file according to the audio playback progress; wherein the text file includes a plurality of text groups, the text group includes a plurality of characters, and the characters carry a timestamp; the target text group includes a first text group currently being played and a set number of second text groups that have been played and / or not played; and A first text texture corresponding to the first text group and a second text texture corresponding to the second text group are obtained.

4. The method according to claim 3, wherein obtaining the first text texture corresponding to the first text group comprises: Acquire a text texture of a first display style and a text texture of a second display style corresponding to the first text group; Using the text whose timestamp matches the audio playback progress as the trimming point; Clipping the text texture of the first display style and / or the text texture of the second display style based on the clipping point; and A first text texture corresponding to the first text group is generated according to the cropped text texture of the first display style and the text texture of the second display style.

5. The method according to claim 4, wherein the text texture of the first display style based on the clipping point comprises: Cutting off the text texture after the cutting point in the text texture of the first display style; as well as The step of clipping the text texture of the second display style based on the clipping point includes: The text texture before the clipping point in the text texture of the second display style is clipped.

6. The method according to claim 4, wherein generating the first text texture corresponding to the first text group according to the cropped text texture of the first display style and the text texture of the second display style comprises: If the text texture of the first display style and the text texture of the second display style are both cropped based on the cropping point, the cropped text texture of the first display style and the cropped text texture of the second display style are spliced ​​to obtain a first text texture corresponding to the first text group; as well as If the text texture of the first display style is cropped based on the cropping point, and the text texture of the second display style is not cropped, the cropped text texture of the first display style is superimposed on the uncropped text texture of the second display style. A first text texture corresponding to the first text group is obtained on a corresponding portion of the text texture in the second display style.

7. The method according to claim 5, wherein obtaining the second text texture corresponding to the second text group comprises: The first display style text texture corresponding to the played second text group and / or the second display style text texture corresponding to the unplayed second text group are obtained.

8. The method according to claim 1, wherein if the interface object texture is a pitch texture, obtaining the interface object texture according to the audio playback progress comprises: Determine a time segment according to the audio playback progress; wherein a timestamp corresponding to the audio playback progress is within the time segment; and Obtain a pitch texture whose timestamp falls within the time segment; wherein the pitch texture is pre-created based on the audio and carries a timestamp and standard pitch information.

9. The method according to claim 1, wherein if the interface object texture is a pitch texture, obtaining the interface object texture according to the audio playback progress comprises: Acquire intonation evaluation information corresponding to the audio playback progress; wherein the intonation evaluation information represents intonation evaluation information of the collected speech audio frame; and At least one intonation texture is generated according to the intonation evaluation information; wherein the intonation texture carries a timestamp and standard pitch information.

10. The method according to claim 9, wherein obtaining the pitch evaluation information corresponding to the audio playback progress comprises: Get the pitch information of the voice audio frames collected within the set historical duration; Determining a deviation between the pitch information and corresponding standard pitch information; as well as The pitch evaluation information is determined according to the deviation; wherein the pitch evaluation information includes: matching with the standard pitch information and not matching with the standard pitch information.

11. The method according to claim 10, wherein generating at least one intonation texture according to the intonation evaluation information comprises: Extracting the timestamp of the speech audio frame whose pitch evaluation information matches the standard pitch information as the pitch timestamp; as well as At least one intonation texture is generated according to the intonation timestamp.

12. The method according to claim 1, wherein rendering the interface object texture into the audio interface for display comprises: Superimposing the interface object texture onto a preset empty texture to obtain a target texture; as well as The target texture is rendered to the audio interface for display.

13. The method according to claim 12, wherein rendering the interface object texture into the audio interface for display comprises: Get the position information of the pixel points in the audio interface; Determining whether the pixel point falls into the area corresponding to the texture of the interface object according to the position information; If the pixel point falls into the area corresponding to the interface object texture, the pixel point is rendered as the color corresponding to the interface object texture; and If the pixel point does not fall into the area corresponding to the interface object texture, the pixel point is rendered as a transparent color.

14. The method according to claim 1, further comprising: Create a texture cache; Generating a text texture and a pitch texture according to the audio; as well as The text texture and the pitch texture are stored in the texture buffer.

15. The method according to claim 2, wherein obtaining the audio playback progress comprises: Obtaining the playing progress of the background audio of a target song; wherein the target song is a song currently being sung; Acquiring the interface object texture according to the audio playback progress, including: Acquire the lyrics texture, pitch texture and intonation texture corresponding to the voice audio of the target song according to the playing progress of the background audio; wherein the voice audio is the audio of singing the target song; and Rendering the interface object texture to the audio interface for display includes: The lyrics texture, pitch texture and intonation texture are rendered into an audio interface for display.

16. A rendering device for an audio interface, comprising: Audio playback progress acquisition module, used to obtain audio playback progress; An interface object texture acquisition module, used to acquire the interface object texture according to the audio playback progress; as well as A rendering module is used to render the interface object texture into the audio interface for display.

17. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 15.

18. A storage medium comprising computer executable instructions, which when executed by a computer processor are used to perform the method according to any one of claims 1 to 15.

Images