Audio interface rendering method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an audio interface rendering method and device, equipment and a storage medium. Acquiring an audio playing progress; obtaining interface object textures according to the audio playing progress; and rendering the interface object texture to an audio interface for display. According to the audio interface rendering method provided by the embodiment of the invention, the interface object texture obtained according to the audio playing progress is rendered to the audio interface for display, so that the performance consumption during audio interface rendering can be reduced, and the rendering speed can be improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of interface rendering, and in particular, to a method, apparatus, device and storage medium for rendering an audio interface. Background Art

[0002] At present, mobile terminals have become one of the important tools for users to carry out entertainment activities. Among them, using mobile terminals to sing songs is one usage scenario. When a user sings a song through an application on a mobile terminal, the interface of the song application will display the lyrics and pitch information as the singing progresses. In the prior art, when rendering lyrics and pitch information, it is mainly based on the central processing unit (CPU) plus memory. This method consumes CPU performance and causes the terminal to become stuck, thereby affecting the user experience. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, apparatus, device and storage medium for rendering an audio interface, which can reduce the performance consumption when rendering the audio interface and improve the rendering speed.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for rendering an audio interface, comprising:

[0005] Get the audio playback progress;

[0006] Acquire the interface object texture according to the audio playback progress;

[0007] The interface object texture is rendered into the audio interface for display.

[0008] In a second aspect, an embodiment of the present disclosure further provides a rendering device for an audio interface, characterized in that it includes:

[0009] Audio playback progress acquisition module, used to obtain audio playback progress;

[0010] An interface object texture acquisition module, used to acquire the interface object texture according to the audio playback progress;

[0011] A rendering module is used to render the interface object texture into the audio interface for display.

[0012] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising: one or more processors;

[0013] a storage device for storing one or more programs,

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for rendering the audio interface as described in the embodiment of the present disclosure.

[0015] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the method for rendering an audio interface as described in the embodiments of the present disclosure.

[0016] The embodiments of the present disclosure disclose a method, device, equipment and storage medium for rendering an audio interface, which obtains the audio playback progress; obtains the interface object texture according to the audio playback progress; and renders the interface object texture to the audio interface for display. The method for rendering an audio interface provided by the embodiments of the present disclosure renders the interface object texture obtained according to the audio playback progress to the audio interface for display, which can reduce the performance consumption when rendering the audio interface and improve the rendering speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0018] Figure 1 It is a flowchart of a method for rendering an audio interface provided by an embodiment of the present disclosure;

[0019] Figure 2 is an example diagram of cutting text texture provided by an embodiment of the present disclosure;

[0020] Figure 3 is an example diagram of a first text texture provided by an embodiment of the present disclosure;

[0021] Figure 4 is an example diagram of a text texture corresponding to a target text group provided by an embodiment of the present disclosure;

[0022] Figure 5 is an example diagram of a pitch texture provided by an embodiment of the present disclosure;

[0023] Figure 6 is an example diagram of a pitch texture provided by an embodiment of the present disclosure;

[0024] Figure 7 It is an example of a pitch evaluation texture provided by an embodiment of the present disclosure;

[0025] Figure 8 is an example diagram of a pitch marker texture provided by an embodiment of the present disclosure;

[0026] Fig. 9 is an example diagram of an audio interface provided by an embodiment of the present disclosure;

[0027] Fig.10 It is a structural schematic diagram of a rendering device of an audio interface provided by an embodiment of the present disclosure;

[0028] Fig.11 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0030] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0031] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0032] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0033] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0034] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0035] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0036] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0037] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0038] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0039] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0040] Figure 1 A flow chart of a method for rendering an audio interface provided in an embodiment of the present disclosure is provided. The embodiment of the present disclosure is applicable to the situation of rendering an audio interface. The method can be executed by a rendering device for an audio interface, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0041] like Figure 1 As shown, the method includes:

[0042] S110, obtaining audio playback progress.

[0043] Among them, the audio can be pre-generated or recorded audio, such as a song. The audio playback progress can be understood as the time point at which the audio is currently played or the time point at which the audio is currently adjusted by the user. For example, assuming that the total length of an audio is 3 minutes, and assuming that the audio is currently played to 1 minute and 20 seconds, the audio playback progress is 1 minute and 20 seconds. In this embodiment, for any audio playback tool, the underlying progress component can be called to obtain the audio playback progress, and the method for obtaining the audio playback progress is not limited here.

[0044] S120, obtaining an interface object texture according to the audio playback progress.

[0045] Among them, the interface object texture includes at least one of the following: text texture, pitch texture, and intonation texture. The interface object may be an object corresponding to the currently playing audio displayed in the audio interface, and may include: at least one of text information, pitch information, and intonation information. The audio may include background audio and standard voice audio. Assuming that the audio is a song, the background audio is the accompaniment, and the standard voice audio is the singing audio. The text information is the text corresponding to the standard voice audio, that is, the lyrics; the pitch information is the height of the standard voice audio; and the intonation information represents the accuracy of the pitch of the collected voice audio.

[0046] In this application scenario, the user sings the corresponding song along with the played audio, and the terminal will collect the user's voice audio in real time. As the audio is played and the user sings, for each frame displayed in the audio interface, the interface object to be displayed in the current frame is obtained, that is, the interface object texture corresponding to the audio playback progress is obtained.

[0047] Optionally, if the interface object texture is a text texture, the method for obtaining the interface object texture according to the audio playback progress may be: determining the target text group from the text file according to the audio playback progress; obtaining the first text texture corresponding to the first text group and the second text texture corresponding to the second text group.

[0048] Among them, the text file includes multiple text groups, the text group includes multiple characters, and the characters carry timestamps; the target text group includes the first text group currently being played and a set number of second text groups that have been played and / or not played. The text file can be understood as a lyrics file corresponding to the audio. In this embodiment, due to the characteristics of the song being sung by sentence, the text will be processed in sentences, and one sentence of text corresponds to one text group. The timestamp carried by the text includes two pieces of information: the starting time and the duration of the text, which can be expressed as follows: "text <starting time, duration>". The first text group currently being played can be the text group where the timestamp matches the audio playback progress. The matching of the timestamp with the audio playback progress can be understood as the time corresponding to the audio playback progress falling within the timestamp corresponding to the text. The set number can be preset and is not limited here.

[0049] Specifically, the method of determining the target text group from the text file according to the audio playback progress can be: obtaining the text group containing the text whose timestamp matches the audio playback progress in the text file as the first target text group; obtaining a set number of text groups before and / or after the first target text group as the second target text group. In this embodiment, since the text carries a timestamp, the text group composed of the text corresponds to a time period, and the multiple text groups contained in the text file are automatically sorted according to time. After the target text group is determined, the corresponding text texture can be obtained according to the target text group. In this embodiment, determining the text texture based on the text group can improve the accuracy of text rendering.

[0050] Optionally, the method for obtaining the first text texture corresponding to the first text group may be: obtaining the text texture of the first display style and the text texture of the second display style corresponding to the first text group; using the text whose timestamp matches the audio playback progress as the cropping point; cropping the text texture of the first display style and / or the text texture of the second display style based on the cropping point; and generating the first text texture corresponding to the first text group based on the cropped text texture of the first display style and the text texture of the second display style.

[0051] Among them, the first display style and the second display style are different. The display style can be color or transparency. In this application scenario, in order to highlight the playback progress of the audio, the unplayed text and the played text can be rendered in different display styles. When the audio is played to a certain text in the middle of the first target text group, it is necessary to divide the text texture corresponding to the first target text group into two display styles before and after. The first display style can be used to represent that it has been played, and the second display style can represent that it has not been played; or, the first display style can represent that it has not been played, and the second display style can represent that it has been played, which is not limited here. Specifically, the method for obtaining the text texture of the first display style and the text texture of the second display style corresponding to the first text group can be: obtaining the text texture of the first display style and the text texture of the second display style corresponding to the first text group from a pre-created buffer area. In this embodiment, the text of the first display style and the second display style will be drawn in advance for each text group of the text file, thereby generating the text texture of the first display style and the second display style text texture of each text group, and storing the text texture of the first display style and the second display style text texture in the texture buffer area. The final text texture is generated by cropping the texts of the two display styles to highlight the playing progress of the audio, thereby enriching the display effect of the text.

[0052] Specifically, assuming that the first display style is used to represent played content, and the second display style is used to represent unplayed content, then the method of trimming the text texture of the first display style based on the trimming point may be: trimming the text texture after the trimming point in the text texture of the first display style. The method of trimming the text texture of the second display style based on the trimming point may be: trimming the text texture before the trimming point in the text texture of the second display style.

[0053] In this embodiment, the first display style indicates that the video has been played, and the part of the text texture corresponding to the text that has not been played in the text texture of the first display style is cut off, and the part of the text texture corresponding to the text that has been played in the text texture of the first display style is retained. The second display style indicates that the video has not been played, and the part of the text texture corresponding to the text that has been played in the text texture of the second display style is cut off, and the part of the text texture corresponding to the text that has not been played in the text texture of the second display style is retained. Exemplary, Figure 2 is an example diagram of cutting text texture in the embodiment, such as Figure 2 As described above, the first display style is gray, and the second display style is white. Assuming "A" is the clipping point, the texture after A in the gray text texture is clipped, and the texture before A in the white text texture is clipped. The black part in the figure represents the clipped part. In this embodiment, the text textures of the two display styles are clipped based on the clipping point, which can improve the clipping accuracy.

[0054] Optionally, a method for generating a first text texture corresponding to the first text group based on the cropped text texture of the first display style and the text texture of the second display style may be: if the text texture of the first display style and the text texture of the second display style are both cropped based on the cropping point, the cropped text texture of the first display style and the cropped text texture of the second display style are spliced ​​to obtain the first text texture corresponding to the first text group; if the text texture of the first display style is cropped based on the cropping point and the text texture of the second display style is not cropped, the cropped text texture of the first display style is superimposed on the corresponding part of the uncropped text texture of the second display style to obtain the first text texture corresponding to the first text group.

[0055] Specifically, if the text texture of the first display style and the text texture of the second display style are both cropped based on the cropping point, the cropped text texture of the first display style and the cropped text texture of the second display style are spliced ​​at the original cropping point. If the text texture of the first display style is cropped based on the cropping point, and the text texture of the second display style is not cropped, the cropped text texture of the first display style is aligned with the uncropped text texture of the second display style based on the cropping point and then superimposed. Exemplary, Figure 3 is an example diagram of the first text texture in this embodiment, such as Figure 3 As shown, the first text texture is displayed in gray before the clipping point A and in white after A. In this embodiment, the first text texture is displayed in the first display style before the clipping point and in the second display style after the clipping point, highlighting the audio playback progress.

[0056] Optionally, the second text texture corresponding to the second text group may be obtained by: obtaining a text texture of the first display style corresponding to the played second text group and / or a text texture of the second display style corresponding to the unplayed second text group.

[0057] The second text group is a text group within a certain distance from the first text group. Exemplarily, assuming that the set number is 8, the first text group can be composed of the first three and the last five text groups of the first text group. The first display style represents played, and the second display style represents unplayed, so the first display style text texture corresponding to the played second text group and / or the second display style text texture of the unplayed second text group are obtained. Exemplarily, Figure 4 is an example diagram of the text texture corresponding to the target text group in this embodiment, such as Figure 4 As shown, text texture 1 and text texture 2 are text textures corresponding to the text groups that have been played, text texture 3 is the text texture corresponding to the text group that is currently being played, and text texture 4, text texture 5 and text texture 6 are text textures corresponding to the text groups that have not been played.

[0058] Optionally, if the interface object texture is a pitch texture, the method of obtaining the interface object texture according to the audio playback progress may be: determining a time segment according to the audio playback progress; and obtaining the pitch texture whose timestamp falls within the time segment.

[0059] Among them, the timestamp corresponding to the audio playback progress is in the time segment. The time segment is a period of time in the playback duration corresponding to the audio. The time segment is composed of the first sub-segment before the audio playback progress and the second sub-segment after the audio playback progress. The length of the time segment can be set based on the size (width or length) of the visible area of ​​the screen of the current terminal device, and the ratio of the first sub-segment to the second sub-segment can be pre-configured. Exemplarily, the time segment is set to 10 seconds, the first sub-segment and the second sub-segment are 2:8, and the time segment is composed of 2 seconds before the audio playback progress and 8 seconds after the audio playback progress. Assuming that the audio playback progress is 1 minute and 30 seconds, the time segment corresponding to the audio playback progress is: 1 minute 28 seconds to 1 minute 38 seconds.

[0060] Among them, the pitch texture is pre-created based on the audio and carries a timestamp and standard pitch information. According to the timestamp and the standard pitch information, the position information of the pitch texture in the audio interface can be determined. The pitch texture can be a long strip of a set shape, and the set shape can be a rectangle, a flat ellipse or an irregular figure. The irregular figure can be a figure surrounded by two straight lines and two arcs. The standard pitch information is represented by a quantized value. The timestamp carried by the pitch texture can include two information of the starting time and the duration. One pitch information can carry multiple different timestamps. This is because in a piece of audio, different timestamps may correspond to the same pitch. The standard pitch information can be understood as the pitch of the voice audio in the audio data. The information carried by the pitch texture can be expressed as follows: "Pitch a: <starting time 1, duration 1>, <starting time 2, duration 2>", that is, pitch a carries two timestamps. Specifically, the process of creating a pitch texture can be: extracting the standard pitch information contained in the audio, generating a pitch texture of a set shape for the same standard pitch information, determining the timestamp corresponding to the pitch texture, carrying the standard pitch information and the timestamp in the pitch texture, and storing the created pitch texture in the texture cache.

[0061] In this embodiment, after determining the time segment corresponding to the audio playback progress, the pitch texture whose timestamp falls within the time segment is obtained. Figure 5 is an example diagram of pitch texture in an embodiment of the present invention, such as Figure 5As shown, the pitch textures falling into the time segment corresponding to the audio playback progress include: pitch texture 1, pitch texture 2, pitch texture 3, pitch texture 4 and pitch texture 5, wherein pitch texture 1 and pitch texture 2 are pitch textures before the audio playback progress, and pitch texture 3, pitch texture 4 and pitch texture 5 are pitch textures after the audio playback progress. In this embodiment, the pitch texture is determined based on the time segment corresponding to the audio playback progress, which can improve accuracy.

[0062] Optionally, if the interface object texture is a pitch texture, the method of obtaining the interface object texture according to the audio playback progress may be: obtaining pitch evaluation information corresponding to the audio playback progress; and generating at least one pitch texture according to the pitch evaluation information.

[0063] Among them, the pitch evaluation information represents the pitch evaluation information of the collected speech audio frame; the pitch texture carries a timestamp and standard pitch information. The form of the timestamp carried by the pitch texture is similar to that of the pitch texture, which will not be repeated here. According to the timestamp and standard pitch information of the pitch texture, the position information of the pitch texture in the audio interface can be determined. In this application scenario, the user sings the corresponding song following the background audio, the terminal collects the voice audio used in real time, extracts the pitch information in the collected voice audio, compares the pitch information with the standard pitch information, and obtains the pitch evaluation information.

[0064] Specifically, the method for obtaining the pitch evaluation information corresponding to the audio playback progress may be: obtaining the pitch information of the speech audio frames collected within a set historical duration; determining the deviation between the pitch information and the corresponding standard pitch information; and determining the pitch evaluation information based on the deviation.

[0065] Among them, the pitch evaluation information includes: matching with standard pitch information and not matching with standard pitch information. The set historical duration can be a certain duration before the audio playback progress. The deviation between the pitch information and the corresponding standard pitch information can be characterized by the difference between the quantized value of the pitch information of the collected speech audio frame and the quantized value of the corresponding standard pitch information. Specifically, the method of determining the pitch evaluation information according to the deviation can be: if the deviation between the pitch information and the corresponding standard pitch information is less than or equal to the set threshold, the pitch information of the collected speech audio frame matches the standard pitch information; if the deviation between the pitch information and the corresponding standard pitch information is greater than the set threshold, the pitch information of the collected speech audio frame does not match the standard pitch information.

[0066] Specifically, the method of generating at least one intonation texture according to the intonation evaluation information may be: extracting the timestamp of the speech audio frame whose intonation evaluation information matches the standard pitch information as the intonation timestamp; generating at least one intonation texture according to the intonation timestamp.

[0067] Among them, the pitch texture can be a long strip of a set shape, and the set shape can be a rectangle, a flat ellipse or an irregular shape. The irregular shape can be a figure surrounded by two straight lines and two arcs. In this embodiment, the pitch texture and the intonation texture have the same shape, the intonation texture overlaps with part of the pitch texture in the audio interface, and the intonation texture is above the pitch texture, and the display styles of the two (such as color or transparency) are different. The method of generating at least one intonation texture according to the intonation timestamp can be: divide the corresponding standard pitch information The same and continuous intonation timestamps are divided into an intonation segment, and generate an intonation texture according to the intonation segment. Exemplarily, Figure 6 is an example diagram of the intonation texture in this embodiment, such as Figure 6 As shown, the black rectangular bar is the pitch texture, and the white rectangular bar is the intonation texture. Figure 5 In comparison, a portion of the white rectangular bar covers the black rectangular bar, indicating that the pitch information of the speech audio frame collected during this period matches the standard pitch information.

[0068] Optionally, the intonation texture also includes an intonation evaluation texture, and the intonation evaluation information is used to characterize the accuracy of the pitch of the collected speech audio. The way to obtain the intonation evaluation texture according to the audio playback progress can be: obtain the intonation evaluation texture according to the deviation between the pitch information and the corresponding standard pitch information.

[0069] Among them, the pitch evaluation texture includes the pitch score evaluation texture and / or the pitch grade evaluation texture. The pitch score evaluation texture may be a texture corresponding to the pitch score evaluation information, and the pitch score evaluation information may be understood as the pitch score determined according to the deviation between the pitch information of the collected speech audio frame and the corresponding standard pitch information. The audio score may be a value between 0-100. In this embodiment, a correspondence between the deviation and the pitch score may be established in advance, and the pitch score may be determined based on the correspondence, and then the score evaluation texture may be generated based on the pitch score. The grade evaluation texture may be a texture corresponding to the grade evaluation information, and the grade evaluation information may be a plurality of pre-set evaluation grades, for example: three grades of excellent, good, and poor. The grade evaluation texture may be pre-created and stored in the texture cache. In this embodiment, a correspondence between the deviation and the evaluation grade may be established in advance, and the grade evaluation information may be determined based on the correspondence, and finally the grade evaluation texture may be obtained based on the grade evaluation information. Exemplarily, Figure 7 is an exemplary pitch evaluation texture in this embodiment, such as Figure 7 As shown, the audio score determined based on the deviation between the pitch information of the collected speech audio frame and the corresponding standard pitch information is 98 points, and the grade evaluation information is excellent, so a pitch score evaluation texture corresponding to 98 points is generated, and the pitch grade evaluation texture corresponding to "excellent" is obtained from the texture cache.

[0070] S130: Render the interface object texture to the audio interface for display.

[0071] In this embodiment, after obtaining at least one interface object texture among the text texture, pitch texture, intonation texture, pitch identification texture and intonation evaluation texture, these interface object textures are input and output from the shader, and the graphics processing unit (GPU) completes the rendering of all interface object textures at one time.

[0072] Specifically, the method of rendering the interface object texture into the audio interface for display may be: superimposing the interface object texture onto a preset empty texture to obtain a target texture; and rendering the target texture into the audio interface for display.

[0073] The preset empty texture may be a pre-created empty texture without pixel content that matches the size of the terminal screen. The process of superimposing the interface object texture on the preset empty texture may be: performing an affine transformation on the interface object texture, and superimposing the interface object texture after the affine transformation on the preset empty texture.

[0074] Optionally, if the interface object texture includes a pitch texture and a intonation texture, the method of superimposing the interface object texture on the preset empty texture can be: first superimpose the pitch texture on the preset empty texture, and then superimpose the intonation texture on the preset empty texture superimposed with the pitch texture.

[0075] In this embodiment, the pitch texture is superimposed first, and then the intonation texture is superimposed. This can ensure that when displayed, the intonation texture covers the pitch texture with the same timestamp as its timestamp, so that the user can intuitively see which part of the singing voice audio has pitch information that matches the standard pitch information.

[0076] Optionally, the process of rendering the interface object texture into the audio interface for display may be: obtaining the position information of the pixel in the audio interface; determining whether the pixel falls into the area corresponding to the interface object texture based on the position information; if the pixel falls into the area corresponding to the interface object texture, rendering the pixel as the color corresponding to the interface object texture; if the pixel does not fall into the area corresponding to the interface object texture, rendering the pixel as a transparent color.

[0077] In this embodiment, when the GPU renders the audio interface, it processes each pixel. Therefore, during the rendering process, it determines whether each pixel in the audio interface falls into the area corresponding to the interface object texture; if the pixel falls into the area corresponding to the interface object texture, the pixel is colored as the color corresponding to the interface object texture; if the pixel does not fall into the area corresponding to the interface object texture, the pixel is colored as a transparent color.

[0078] Optionally, the pitch texture also includes a pitch identification texture, and the pitch identification is used to characterize the pitch of the collected voice audio. The method of superimposing the interface object texture on the preset empty texture can be: for the pitch identification texture, obtaining the pitch information of the currently collected voice audio frame; determining the position information of the pitch identification texture according to the pitch information; and superimposing the pitch identification texture on the preset empty texture based on the position information.

[0079] The pitch marker texture is used to represent the pitch information of the currently collected speech audio frame. The pitch marker texture can be a pre-created texture of a set shape, such as a circle or rectangle. The pitch information can be represented by a quantized value, and the position information of the pitch marker texture is determined according to the quantized pitch information. Then, an affine transformation is performed on the position information, and the pitch marker texture after the affine transformation is superimposed on the preset empty texture. For example, Figure 8 is an example diagram of the pitch marker texture in this embodiment, such as Figure 8 As shown, the pitch mark texture is a circle, which moves up and down on the line corresponding to the audio playback progress according to the pitch information.

[0080] Optionally, the method of rendering the interface object texture to the audio interface for display may be: for the level evaluation texture, determining the transparency of the level evaluation texture; and rendering the evaluation texture to the audio interface for display based on the transparency.

[0081] Among them, a life cycle is set when the level evaluation texture is displayed in the interface, for example: 2 seconds. During this life cycle, in order to make the level evaluation texture present a gradient effect, the transparency of the level evaluation texture can be adjusted, for example, the transparency of the level evaluation texture changes in a certain way (for example: first increase and then decrease, or gradually increase, or gradually decrease). In this embodiment, the way the transparency changes is not limited. Specifically, the transparency of the level evaluation texture at the current moment is determined; based on the transparency, the evaluation texture is rendered to the audio interface for display.

[0082] Optionally, the method further includes the following steps: creating a texture cache; generating a text texture and a pitch texture based on the audio; and storing the text texture and the pitch texture in the texture cache.

[0083] The process of generating text texture and pitch texture according to audio can be referred to in the above embodiment, which will not be described in detail here. Fig. 9 is an example diagram of an audio interface in an embodiment of the present disclosure, such as Fig. 9 As shown, the audio interface displays text, pitch information and intonation information.

[0084] Based on the above embodiment, the present embodiment is applied to the audio interface rendering scene when the user sings along with the background audio of the target song. When the user sings the song, the terminal device obtains the playback progress of the background audio of the target song in real time, and then obtains the lyrics texture, pitch texture and intonation texture corresponding to the voice audio of the target song according to the playback progress of the background audio. Finally, the lyrics texture, pitch texture and intonation texture are rendered to the audio interface for display.

[0085] Among them, the voice audio is the audio of singing the target song.

[0086] The technical solution of the embodiment of the present disclosure obtains the audio playback progress; obtains the interface object texture according to the audio playback progress; and renders the interface object texture to the audio interface for display. The rendering method of the audio interface provided by the embodiment of the present disclosure renders the interface object texture obtained according to the audio playback progress to the audio interface for display, which can reduce the performance consumption when rendering the audio interface and improve the rendering speed.

[0087] Fig.10 A schematic diagram of a rendering device structure of an audio interface provided by an embodiment of the present disclosure is shown in FIG. Fig.10 As shown, the device comprises:

[0088] The audio playing progress obtaining module 210 is used to obtain the audio playing progress;

[0089] The interface object texture acquisition module 220 is used to acquire the interface object texture according to the audio playback progress; the rendering module 230 is used to render the interface object texture into the audio interface for display.

[0090] Optionally, the interface object texture includes at least one of the following: text texture, pitch texture and intonation texture;

[0091] Optionally, if the interface object texture is a text texture, the interface object texture acquisition module 220 is further used to:

[0092] Determine a target text group from a text file according to the audio playback progress; wherein the text file includes a plurality of text groups, the text group includes a plurality of characters, and the characters carry a timestamp; the target text group includes a first text group currently being played and a set number of second text groups that have been played and / or not played;

[0093] A first text texture corresponding to the first text group and a second text texture corresponding to the second text group are obtained.

[0094] Optionally, the interface object texture acquisition module 220 is further used to:

[0095] Acquire a text texture of a first display style and a text texture of a second display style corresponding to the first text group;

[0096] The text whose timestamp matches the audio playback progress is used as the trimming point;

[0097] Cropping the text texture of the first display style and / or the text texture of the second display style based on the cropping point;

[0098] A first text texture corresponding to the first text group is generated according to the cropped text texture of the first display style and the text texture of the second display style.

[0099] Optionally, the interface object texture acquisition module 220 is further used to:

[0100] The text texture after the clipping point in the text texture of the first display style is clipped;

[0101] The text texture of the second display style is clipped based on the clipping point, including:

[0102] Clip the text texture before the clipping point in the text texture of the second display style.

[0103] Optionally, the interface object texture acquisition module 220 is further used to:

[0104] If the text texture of the first display style and the text texture of the second display style are both cropped based on the cropping point, the cropped text texture of the first display style and the cropped text texture of the second display style are spliced ​​to obtain a first text texture corresponding to the first text group;

[0105] If the text texture of the first display style is cropped based on the cropping point, and the text texture of the second display style is not cropped, the cropped text texture of the first display style is superimposed on the corresponding part of the uncropped text texture of the second display style to obtain the first text texture corresponding to the first text group.

[0106] Optionally, the interface object texture acquisition module 220 is further used to:

[0107] The first display style text texture corresponding to the played second text group and / or the second display style text texture corresponding to the unplayed second text group are obtained.

[0108] Optionally, if the interface object texture is a pitch texture, the interface object texture acquisition module 220 is further used to:

[0109] Determine a time segment according to the audio playback progress; wherein the timestamp corresponding to the audio playback progress is in the time segment;

[0110] Get the pitch texture whose timestamp falls into the time segment; wherein the pitch texture is pre-created based on the audio and carries the timestamp and standard pitch information.

[0111] Optionally, if the interface object texture is a pitch texture, the interface object texture acquisition module 220 is further used to:

[0112] Acquire intonation evaluation information corresponding to the audio playback progress; wherein the intonation evaluation information represents intonation evaluation information of the collected speech audio frame;

[0113] At least one intonation texture is generated according to the intonation evaluation information; wherein the intonation texture carries a timestamp and standard pitch information.

[0114] Optionally, the interface object texture acquisition module 220 is further used to:

[0115] Get the pitch information of the voice audio frames collected within the set historical duration;

[0116] Determine the deviation between the pitch information and the corresponding standard pitch information;

[0117] The pitch evaluation information is determined according to the deviation; wherein the pitch evaluation information includes: matching with the standard pitch information and not matching with the standard pitch information.

[0118] Optionally, the interface object texture acquisition module 220 is further used to:

[0119] Extracting the timestamp of the speech audio frame whose pitch evaluation information matches the standard pitch information as the pitch timestamp;

[0120] At least one pitch texture is generated according to the pitch timestamps.

[0121] Optionally, the rendering module 230 is further configured to:

[0122] Overlay the interface object texture onto the preset empty texture;

[0123] Renders a preset empty texture to the audio interface for display.

[0124] Optionally, the rendering module 230 is further configured to:

[0125] Get the position information of the pixel points in the audio interface;

[0126] Determine whether the pixel point falls into the area corresponding to the texture of the interface object according to the position information;

[0127] If the pixel point falls into the area corresponding to the interface object texture, the pixel point is rendered as the color corresponding to the interface object texture;

[0128] If the pixel does not fall into the area corresponding to the interface object texture, the pixel will be rendered as a transparent color.

[0129] Optionally, it also includes: a texture generation module, used for:

[0130] Create a texture cache;

[0131] Generate text texture and pitch texture based on audio;

[0132] Store the text texture and pitch texture into the texture buffer.

[0133] Optionally, the audio playback progress acquisition module 210 is further used to:

[0134] Obtaining the playing progress of the background audio of a target song; wherein the target song is a song currently being sung;

[0135] The interface object texture acquisition module 220 is further used for:

[0136] Acquire the lyrics texture, pitch texture and intonation texture corresponding to the voice audio of the target song according to the playing progress of the background audio; wherein the voice audio is the audio of singing the target song;

[0137] The rendering module 230 is further used for:

[0138] The lyrics texture, pitch texture and intonation texture are rendered into an audio interface for display.

[0139] The rendering device of the audio interface provided in the embodiment of the present disclosure can execute the rendering method of the audio interface provided in any embodiment of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method.

[0140] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0141] Fig.11 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.11 , which shows an electronic device (eg, Fig.11 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0142] like Fig.11 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0143] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.11 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0144] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0145] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0146] The electronic device provided in the embodiment of the present disclosure and the method for rendering the audio interface provided in the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, reference can be made to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0147] An embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the method for rendering the audio interface provided by the above embodiment is implemented.

[0148] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0149] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0150] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0151] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0152] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the audio playback progress; obtains the interface object texture according to the audio playback progress; wherein the interface object texture includes at least one of the following: text texture, pitch texture, intonation texture, pitch identification texture and intonation evaluation texture; and renders the interface object texture into the audio interface for display.

[0153] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0154] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0155] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0156] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0157] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0159] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0160] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for rendering an audio interface, characterized in that: include: Get the audio playback progress; Acquire the interface object texture according to the audio playback progress; The interface object texture is rendered into the audio interface for display.

2. The method according to claim 1, characterized in that The interface object texture includes at least one of the following: text texture, pitch texture and intonation texture.

3. The method according to claim 2, characterized in that If the interface object texture is a text texture, obtaining the interface object texture according to the audio playback progress includes: Determine a target text group from a text file according to the audio playback progress; wherein the text file includes a plurality of text groups, the text group includes a plurality of characters, and the characters carry a timestamp; the target text group includes a first text group currently being played and a set number of second text groups that have been played and / or not played; A first text texture corresponding to the first text group and a second text texture corresponding to the second text group are obtained.

4. The method according to claim 3, characterized in that: Acquiring a first text texture corresponding to the first text group includes: Acquire a text texture of a first display style and a text texture of a second display style corresponding to the first text group; Using the text whose timestamp matches the audio playback progress as the trimming point; Cropping the text texture of the first display style and / or the text texture of the second display style based on the cropping point; A first text texture corresponding to the first text group is generated according to the cropped text texture of the first display style and the text texture of the second display style.

5. The method according to claim 4, characterized in that The text texture of the first display style based on the clipping point includes: Cutting off the text texture after the cutting point in the text texture of the first display style; The step of clipping the text texture of the second display style based on the clipping point includes: The text texture before the clipping point in the text texture of the second display style is clipped.

6. The method according to claim 4, characterized in that Generating a first text texture corresponding to the first text group according to the cropped text texture of the first display style and the text texture of the second display style includes: If the text texture of the first display style and the text texture of the second display style are both cropped based on the cropping point, the cropped text texture of the first display style and the cropped text texture of the second display style are spliced ​​to obtain a first text texture corresponding to the first text group; If the text texture of the first display style is cropped based on the cropping point, and the text texture of the second display style is not cropped, the cropped text texture of the first display style is superimposed on the corresponding part of the uncropped text texture of the second display style to obtain the first text texture corresponding to the first text group.

7. The method according to claim 5, characterized in that Acquiring a second text texture corresponding to the second text group includes: The first display style text texture corresponding to the played second text group and / or the second display style text texture corresponding to the unplayed second text group are obtained.

8. The method according to claim 1, characterized in that If the interface object texture is a pitch texture, obtaining the interface object texture according to the audio playback progress includes: Determine a time segment according to the audio playback progress; wherein the timestamp corresponding to the audio playback progress is within the time segment; Obtain a pitch texture whose timestamp falls within the time segment; wherein the pitch texture is pre-created based on the audio and carries a timestamp and standard pitch information.

9. The method according to claim 1, characterized in that: If the interface object texture is a pitch texture, obtaining the interface object texture according to the audio playback progress includes: Acquire the pitch evaluation information corresponding to the audio playback progress; wherein the pitch evaluation information represents the pitch evaluation information of the collected speech audio frame; At least one intonation texture is generated according to the intonation evaluation information; wherein the intonation texture carries a timestamp and standard pitch information.

10. The method according to claim 9, characterized in that Obtaining pitch evaluation information corresponding to the audio playback progress, including: Get the pitch information of the voice audio frames collected within the set historical duration; Determining a deviation between the pitch information and corresponding standard pitch information; The pitch evaluation information is determined according to the deviation; wherein the pitch evaluation information includes: matching with the standard pitch information and not matching with the standard pitch information.

11. The method according to claim 10, characterized in that Generating at least one intonation texture according to the intonation evaluation information includes: Extracting the timestamp of the speech audio frame whose pitch evaluation information matches the standard pitch information as the pitch timestamp; At least one intonation texture is generated according to the intonation timestamp.

12. The method according to claim 1, characterized in that Rendering the interface object texture to the audio interface for display includes: Superimposing the interface object texture onto a preset empty texture to obtain a target texture; The target texture is rendered to the audio interface for display.

13. The method according to claim 12, characterized in that Rendering the interface object texture to the audio interface for display includes: Get the position information of the pixel points in the audio interface; Determining whether the pixel point falls into the area corresponding to the texture of the interface object according to the position information; If the pixel point falls into the area corresponding to the interface object texture, the pixel point is rendered as the color corresponding to the interface object texture; If the pixel point does not fall into the area corresponding to the interface object texture, the pixel point is rendered as a transparent color.

14. The method according to claim 1, characterized in that Also includes: Create a texture cache; Generating a text texture and a pitch texture according to the audio; The text texture and the pitch texture are stored in the texture buffer.

15. The method according to claim 2, characterized in that Get the audio playback progress, including: Obtaining the playing progress of the background audio of a target song; wherein the target song is a song currently being sung; Acquiring the interface object texture according to the audio playback progress, including: Acquire the lyrics texture, pitch texture and intonation texture corresponding to the voice audio of the target song according to the playing progress of the background audio; wherein the voice audio is the audio of singing the target song; Rendering the interface object texture to the audio interface for display includes: The lyrics texture, pitch texture and intonation texture are rendered into an audio interface for display.

16. A rendering device for an audio interface, characterized in that: include: Audio playback progress acquisition module, used to obtain audio playback progress; An interface object texture acquisition module, used to acquire the interface object texture according to the audio playback progress; A rendering module is used to render the interface object texture into the audio interface for display.

17. An electronic device, characterized in that: The electronic device includes: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the audio interface rendering method as described in any one of claims 1-15.

18. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the method for rendering an audio interface as claimed in any one of claims 1 to 15 when executed by a computer processor.

Images