Text time marking method, device, electronic device and readable storage medium

Abstract

The embodiment of the present application provides a time marking method, device, electronic device and readable storage medium for text, the method comprising: receiving a marking request, obtaining the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio, and marking the play start and end time of each text unit in the text according to the play start and end time of each text unit in the target audio, obtaining a marking file, and outputting the marking file. The embodiment of the present application can identify the moment between two adjacent text units in the text corresponding to the target audio by the fundamental frequency in the target audio, that is, it can distinguish the two adjacent text units, and then align the text units. Instead of aligning the text units by the current method of identifying text units, the problem of being unable to perform time marking of the text due to inconsistency between the text units in the target audio and the text units in the text corresponding to the target audio is avoided, and the accuracy of the time marking of the text is high.

Description

Technical Field

[0001] Embodiments of the present application relate to audio processing technology, and more particularly to a method, device, electronic device, and readable storage medium for time marking of text. Background Art

[0002] When playing a song, the terminal device will display the lyrics of the song and render the words in the displayed lyrics according to the start and end time of each word in the song. Therefore, when the user listens to the song, he can see the lyrics rendered as the song progresses. It is crucial for the terminal device to obtain the start and end time of each word in the lyrics, and the accuracy of the start and end time of each word in the lyrics affects the matching of the song playback degree and the rendering of the lyrics, affecting the user experience.

[0003] In the prior art, the terminal device can use a phoneme dictionary to convert the words in the lyrics into corresponding phonemes, and recognize the phonemes in the audio, and then align the phonemes in the audio with the phonemes converted from the words in the lyrics. After the phonemes are aligned, the words in the lyrics can be marked with the start and end time of playback according to the time in the audio, and then the start and end time of playback of each word in the lyrics can be obtained.

[0004] However, if the user sings a few wrong words in the audio, the phonemes in the wrongly sung part of the audio and the phonemes converted from the corresponding words in the lyrics will not be aligned, which will also affect the alignment of the phonemes after the wrongly sung part, and thus make it impossible to obtain the accurate start and end time of each word in the lyrics. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device and readable storage medium for time marking of text, which have high accuracy in time marking of text.

[0006] In the first aspect, the embodiment of the present application provides a time annotation method for text, which can be applied to electronic devices or chips in electronic devices. Among them, the electronic device can be a server or a terminal device. The method is described below by taking application to an electronic device as an example, in which the electronic device can receive an annotation request. Among them, when the electronic device is a server, the server can receive an annotation request from a terminal device, and when the electronic device is a terminal device, the terminal device can receive an annotation request input by a user. The annotation request is used to request the start and end time of the playback of each text unit in the text corresponding to the target audio, wherein the text unit is at least one of the following: a character, a word or a phrase. After receiving the annotation request, the electronic device can obtain the start and end time of the playback of each text unit in the target audio according to the fundamental frequency of the target audio. Exemplarily, the electronic device can obtain the first moment when the change rate of the fundamental frequency in the target audio is greater than the change rate threshold, and then use the first moment as the interval between two adjacent text units in the text corresponding to the target audio, and the playback start and end time of each text unit can be obtained accordingly, such as a first moment can be used as the playback start time of the previous text unit and the playback end time of the next text unit.

[0007] When the electronic device obtains the start and end time of playing each text unit in the text, it can mark the start and end time of playing each text unit in the text according to the start and end time of playing each text unit in the target audio, and obtain a marking file. Exemplarily, in the embodiment of the present application, a correspondence between each text unit and the corresponding start and end time of playing can be established, and the correspondence can be a marking file. Alternatively, in the embodiment of the present application, the start and end time of playing each text unit can be added to the text of the target audio to obtain a marking file. The electronic device can output the marking file. Among them, when the electronic device is a server, the server can send the marking file to the terminal device; when the electronic device is a terminal device, the terminal device can play the target audio, display the text corresponding to the target audio, and according to the marking file, when playing to the target text unit in the target audio, render and display the target text unit. It should be understood that in the embodiment of the present application, the terminal device can render the text unit according to the start and end time of playing each text unit, and the method of rendering the text unit can refer to the currently existing technical solution.

[0008] In the embodiment of the present application, the time between two adjacent text units in the text corresponding to the target audio can be identified by the fundamental frequency in the target audio, that is, the two adjacent text units can be distinguished, and then the text units can be aligned. Instead of aligning the text units by using the current method of identifying text units, the problem of not being able to perform time annotation of the text when the text unit in the target audio and the text unit in the text corresponding to the target audio are avoided, and the accuracy of the time annotation of the text is high.

[0009] When the fundamental frequency of the target audio is used to determine the first moment between two adjacent text units, there may be a problem that there are multiple first moments between the two adjacent text units. Therefore, the accuracy of obtaining the start and end time of each text unit using only the fundamental frequency of the target audio is low. In order to improve the accuracy of obtaining the start and end time of each text unit in the embodiment of the present application, the start and end time of each text unit can be obtained based on the fundamental frequency of the target audio and the short-time energy of the target audio.

[0010] Among them, the electronic device can obtain the first moment when the rate of change of the fundamental frequency in the target audio is greater than the rate of change threshold, and obtain the second moment in the first moment, wherein the mean of the short-time energy between two adjacent second moments is less than the first short-time energy threshold. In the embodiment of the present application, the play start and end time of each text unit in the target audio can be obtained according to the first moment and the second moment to obtain the play start and end time of each text unit in the target audio. For example, the mean of the continuously existing second moments can be used as the play end time of the previous text unit, and the play start time of the next text unit.

[0011] The target audio may include multiple sub-audio segments, just as a song may include multiple lyrics, and each lyric corresponds to a sub-audio segment. If the entire target audio is taken as a whole, and the playback end time of the last word of a lyric is used as the playback start time of the first word of the next lyric, it may cause the problem of early rendering time of the first word of the next lyric. In order to improve the accuracy of the playback start and end time of the first text and the last text in the text corresponding to each word audio segment in the embodiment of the present application, the electronic device can divide the target audio into multiple sub-audio segments, and obtain the playback start and end time of each text unit in the text corresponding to each sub-audio segment according to the fundamental frequency and short-time energy of each sub-audio segment.

[0012] Among them, dividing the target audio into multiple sub-audio segments can be: obtaining at least one first duration in the target audio whose short-time energy is less than a second short-time energy threshold, and dividing the target audio according to the playback time corresponding to the second duration in the first duration that is greater than a preset duration, to obtain at least one sub-audio segment.

[0013] Among them, the method in which the electronic device obtains the start and end time of the playback of each text unit in the text corresponding to each sub-audio segment according to the fundamental frequency and short-time energy of each sub-audio segment is similar to the method in which the electronic device obtains the start and end time of the playback of each text unit in the text according to the fundamental frequency and short-time energy of the target audio. Specifically, the electronic device can obtain the first moment at which the change rate of the fundamental frequency in the sub-audio segment is greater than the change rate threshold according to the fundamental frequency of the sub-audio segment. If the number of first moments is greater than or equal to the number of text units in the text corresponding to the sub-audio segment, then according to the short-time energy of the sub-audio segment, the second moment is obtained in the first moment, and the average of the short-time energy between two adjacent second moments is less than the first short-time energy threshold. The electronic device can obtain the start and end time of the playback of each text unit in the sub-audio segment according to the first moment and the second moment to obtain the start and end time of the playback of each text unit in the target audio.

[0014] Wherein, when the electronic device obtains the start and end time of playing each text unit in the sub-audio segment according to the first moment and the second moment, at least one second moment group of the sub-audio segment can be obtained in the order of the third moment and the second moment from early to late, and the second moment group is composed of at least two consecutive second moments, and the moment adjacent to the second moment group is the third moment. Wherein, the third moment is the moment other than the second moment in the first moment. In the embodiment of the present application, each third moment and each second moment group can be regarded as a moment unit to obtain multiple moment units. Wherein, if the moment in the i-th moment unit is the third moment, the third moment in the i-th moment unit is used as the end time of playing the i-th text unit in the sub-audio segment and the start time of playing the i+1-th text unit; if the moment in the i-th moment unit is the second moment group, the earliest second moment in the second moment group in the i-th moment unit is used as the end time of playing the i-th text unit in the sub-audio segment, and the latest second moment in the second moment group in the i-th moment unit is used as the start time of playing the i+1-th text unit in the sub-audio segment. Correspondingly, when obtaining the play start and end time of each text unit in each sub-audio segment, the electronic device can annotate the play start and end time of each text unit in the sub-audio segment to each text unit in the text corresponding to the sub-audio segment to obtain an annotation file. Wherein, i is an integer greater than or equal to 1.

[0015] At present, the lyrics of the songs that are online are pre-edited and stored and uploaded manually, and the lyrics database generally includes the lyrics of the songs. For original songs, poems or other vocal audio, the database usually does not include the text corresponding to the audio. If the lyrics are manually edited and uploaded, the efficiency is low and prone to errors. Therefore, on the basis of the above-mentioned embodiment, in the time marking method of the text provided in the embodiment of the present application, the text corresponding to the target audio can also be identified based on the target audio to improve efficiency and accuracy. Among them, when the text corresponding to the target audio is not included in the database, the electronic device can use speech recognition technology based on the target audio to identify the target audio. This method can avoid the problems of low efficiency and low accuracy in manually editing lyrics.

[0016] It should be understood that in the embodiment of the present application, the user can upload the target audio through the terminal device, wherein the target audio can be human voice audio. When the audio uploaded by the user is the initial audio, the initial audio includes non-human voice audio and human voice audio. In order to ensure that the fundamental frequency and short-time energy of the target audio can be accurately obtained, and the lyrics in the target audio can be accurately identified, in the embodiment of the present application, the electronic device can extract the human voice audio from the audio uploaded by the user as the target audio. Among them, the way in which the electronic device extracts the human voice audio can be: the electronic device can input the audio uploaded by the user into the human voice recognition model to obtain the human voice audio with the non-human voice audio removed. It should be noted that in the embodiment of the present application, for the audio uploaded by the user, whether it is non-human voice audio or human voice audio, the audio can be output to the human voice recognition model to remove the non-human voice audio.

[0017] In a second aspect, an embodiment of the present application provides a time-marking device for text, which can be a server or terminal device in the above embodiment. The time-marking device for text can include: a transceiver module, a processing module, and an output module.

[0018] The transceiver module is used to receive a marking request, wherein the marking request is used to request marking the play start and end time of each text unit in the text corresponding to the target audio, and the text unit is at least one of the following: a character, a word or a phrase.

[0019] A processing module is used to obtain the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio, and to mark the play start and end time of each text unit in the text according to the play start and end time of each text unit in the target audio to obtain a marking file.

[0020] An output module is used to output the annotation file.

[0021] In a possible implementation, the processing module is specifically configured to obtain the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio and the short-time energy of the target audio.

[0022] In a possible implementation, the target audio is composed of at least one sub-audio segment. The processing module is specifically used to obtain, based on the fundamental frequency of the sub-audio segment, the first moment when the change rate of the fundamental frequency in the sub-audio segment is greater than the change rate threshold, if the number of the first moments is greater than or equal to the number of text units in the text corresponding to the sub-audio segment, then according to the short-time energy of the sub-audio segment, obtain the second moment in the first moment, the average of the short-time energy between two adjacent second moments is less than the first short-time energy threshold, and according to the first moment and the second moment, obtain the play start and end time of each text unit in the sub-audio segment to obtain the play start and end time of each text unit in the target audio.

[0023] In a possible implementation, the third moment is a moment other than the second moment in the first moment. The processing module is specifically used to obtain at least one second moment group of the sub-audio segment in the order of the third moment and the second moment from early to late, wherein the second moment group is composed of at least two consecutive second moments, and the adjacent moments of the second moment group are the third moments; each third moment and each second moment group is taken as a moment unit to obtain a plurality of moment units; and the play start and end time of each text unit in the sub-audio segment in the order of the moment units from early to late.

[0024] In a possible implementation, the processing module is specifically used to, in the order of the time units from early to late, if the time in the i-th time unit is the third time, use the third time in the i-th time unit as the playback end time of the i-th text unit in the sub-audio segment and the playback start time of the i+1-th text unit; if the time in the i-th time unit is a second time group, use the earliest second time in the second time group in the i-th time unit as the playback end time of the i-th text unit in the sub-audio segment, and use the latest second time in the second time group in the i-th time unit as the playback start time of the i+1-th text unit in the sub-audio segment.

[0025] In a possible implementation, the tagging request includes the target audio, and the target audio is human voice audio; or the tagging request includes initial audio, and the initial audio includes non-human voice audio and human voice audio. The processing module is further used to extract human voice audio from the initial audio as the target audio.

[0026] In a possible implementation, the processing module is specifically configured to input the initial audio into a human voice recognition model to obtain human voice audio from which the non-human voice audio is removed.

[0027] In one possible implementation, the processing module is further used to, if the tagging request includes the identifier of the target audio, query whether the database contains text corresponding to the target audio based on the identifier of the target audio; or, if the tagging request does not include the identifier of the target audio, query whether the database contains text corresponding to the target audio based on the audio fingerprint of the target audio; if the database does not contain the text corresponding to the target audio, use speech recognition technology to identify the target audio and obtain the text corresponding to the target audio.

[0028] In a possible implementation, the target audio is composed of at least one sub-audio segment, and the text corresponding to the target audio includes the text corresponding to the at least one sub-audio segment. The processing module is specifically configured to annotate the play start and end time of each text unit in the sub-audio segment to each text unit in the text corresponding to the sub-audio segment to obtain the annotated file.

[0029] In a possible implementation, the processing module is further used to obtain at least one first duration in the target audio whose short-time energy is less than a second short-time energy threshold; and to divide the target audio according to the playback time corresponding to the second duration in the first duration that is greater than a preset duration, to obtain the at least one sub-audio segment.

[0030] In a possible implementation, the electronic device is a server, and the transceiver module is specifically used to receive the annotation request from the terminal device. The output module is specifically used to send the annotation file to the terminal device.

[0031] In one possible implementation, the electronic device is a terminal device, and the transceiver module is specifically used to receive a playback request for the target audio input by a user, the playback request is used to request to mark the playback start and end time of each text unit in the text corresponding to the target audio, and to play the target audio, and when playing the target audio, each text unit in the text is rendered and displayed according to the playback start and end time of each text unit in the text corresponding to the target audio.

[0032] The output module is specifically used to play the target audio and display the text corresponding to the target audio; according to the annotation file, when the target text unit in the target audio is played, the target text unit is rendered and displayed.

[0033] The implementation principle and technical effect of the text time marking device provided in the embodiment of the present application are similar to those of the server or terminal device in the above-mentioned embodiment, and will not be repeated here.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, which may be a terminal device or a server in the first aspect. The electronic device comprises: a processor, a memory, and a transceiver; the transceiver is coupled to the processor, and the processor controls the transceiver's transceiver action; wherein the memory is used to store computer executable program code, and the program code comprises instructions; when the processor executes the instructions, the instructions cause the electronic device to perform the method provided in the first aspect.

[0035] In a fourth aspect, an embodiment of the present application provides an electronic device, including a unit, module or circuit for executing the method provided by the possible design of the first aspect above. The electronic device may be a terminal device or a server, or may be a module applied to a terminal device or a server, for example, may be a chip applied to a terminal device or a server.

[0036] In a fifth aspect, an embodiment of the present application provides a chip having a computer program stored thereon, and when the computer program is executed by the chip, the method provided in the first aspect is implemented.

[0037] In a sixth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the first aspect above.

[0038] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the method in the above-mentioned first aspect.

[0039] The embodiment of the present application provides a method, device, electronic device and readable storage medium for time annotation of a text, the method comprising: receiving an annotation request, obtaining the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio, and annotating the play start and end time of each text unit in the text according to the play start and end time of each text unit in the target audio, obtaining an annotation file, and outputting the annotation file. In the embodiment of the present application, the moment between two adjacent text units in the text corresponding to the target audio can be identified by the fundamental frequency in the target audio, that is, the two adjacent text units can be distinguished, and then the text units can be aligned. Instead of aligning the text units by the current method of identifying text units, the problem of being unable to perform time annotation of the text due to inconsistency between the text units in the target audio and the text corresponding to the target audio is avoided, and the accuracy of the time annotation of the text is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of an interface for displaying lyrics;

[0041] Figure 2 This is a schematic diagram of another interface for displaying lyrics;

[0042] Figure 3 A schematic diagram for notating a phoneme;

[0043] Figure 4 a schematic diagram notating another phoneme;

[0044] Figure 5 Schematic diagram of time marking of text provided in the embodiment of the present application Figure 1 ;

[0045] Figure 6 A schematic diagram of a scenario applicable to the time marking method of text provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of a flow chart of an embodiment of a method for time marking of text provided in an embodiment of the present application;

[0047] Figure 8 Schematic diagram of the interface changes of the terminal device provided in the embodiment of the present application Figure 1 ;

[0048] Fig. 9 A schematic diagram of the baseband provided in an embodiment of the present application;

[0049] Fig.10 Schematic diagram of time marking of text provided in the embodiment of the present application Figure 2 ;

[0050] Fig.11 A schematic diagram of the fundamental frequency and short-time energy provided in an embodiment of the present application;

[0051] Fig.12 A schematic diagram of a flow chart of another embodiment of a method for time marking of text provided in an embodiment of the present application;

[0052] Fig.13 A schematic diagram of a flow chart of another embodiment of a method for time-marking a text provided in an embodiment of the present application;

[0053] Fig.14 A schematic diagram of a flow chart of another embodiment of a method for time-marking a text provided in an embodiment of the present application;

[0054] Fig.15 A schematic diagram of a flow chart of another embodiment of a method for time-marking a text provided in an embodiment of the present application;

[0055] Fig.16Schematic diagram of interface change of the terminal device provided in the embodiment of the present application Figure 2 ;

[0056] Fig.17 Flow schematic diagram of another embodiment of the method for time annotation of text provided in the embodiment of the present application;

[0057] Fig.18 Schematic diagram of the structure of the text time annotation device provided in the embodiment of the present application;

[0058] Fig.19 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Detailed implementation manners

[0059] When a user uses a terminal device to play a song, the terminal device can display the lyrics of the song on the interface. Figure 1 It is a schematic diagram of an interface for lyrics display. Figure 1 As shown, the terminal device displays the lyrics line by line. Among them, the terminal device can store the start and end times of playing each line of lyrics, and then flip the lyrics line by line according to the start and end times of playing each line of lyrics when displaying the lyrics. As Figure 1 shown in interface 101, the currently played lyrics are "not a moment can be separated", and then this line of lyrics can be displayed within the start and end times of playing "not a moment can be separated". In interface 101, the lyrics before and after the currently played lyrics are exemplarily displayed in gray to distinguish them from the currently played lyrics. When playing to the line "wherever I go", the terminal device can flip the previous line of lyrics up, as shown in interface 102. The way of displaying lyrics line by line cannot well match the playing progress of the song, and the user experience is poor. It should be understood that the start and end times of playing in the embodiment of the present application may include the start time of playing and the end time of playing.

[0060] The terminal device can render the lyrics word by word according to the playing progress of the song, thereby improving the user experience. Figure 2 It is a schematic diagram of another interface for lyrics display. As Figure 2 shown, the terminal device can display the lyrics of the song. For example, when the currently played song plays to the character "moment" in "not a moment can be separated", the terminal device can render the character "moment", such as deepening the color of the character "moment" to represent the playing progress of the current song. Among them, the terminal device needs to pre-obtain the start and end times of playing each character in the lyrics, and then render each character according to the start and end times of playing each character when starting to play this song.

[0061] At present, annotators can manually annotate the start and end time of each word in the lyrics. For example, annotators can record the start and end time of each word in the lyrics while listening to the song. In order to improve the annotation accuracy of the annotators, the annotators can use the time domain diagram of the song to annotate the start and end time of each word in the lyrics. However, the manual annotation method requires a lot of manpower, is time-consuming, and has low accuracy. Among them, the time domain diagram of the song includes an audio waveform diagram of the song that changes with time.

[0062] Currently, a method for automatically marking the start and end time of each word in the lyrics is also provided. In this method, the marking device (such as a terminal device) can identify the phonemes in the audio according to the audio of the song being played, and the terminal device uses a phoneme dictionary to convert the words in the lyrics into corresponding phonemes. The terminal device can convert the phonemes in the audio and the words in the lyrics into corresponding phonemes and align them, and then mark the start and end time of the words in the lyrics according to the time in the audio, so as to obtain the start and end time of each word in the lyrics. Exemplary, Figure 3 A schematic diagram of a phoneme notation. Figure 3 As shown, the terminal device converts the words in the lyrics "not even a moment can be separated" into corresponding phonemes as shown in A, and the terminal device recognizes the phonemes in the audio "not even a moment can be separated" as shown in B. The terminal device can align the phonemes one by one, and then mark the start and end time of the playback of the words in the lyrics according to the time in the audio. However, if the user sings the wrong lyrics in the audio, such as singing "not even a moment can be separated" as "we cannot be separated". Figure 4 A diagram for another phoneme notation. Figure 4 As shown, the phonemes in the audio recognized by the terminal device and the corresponding phonemes in the lyrics cannot be aligned, so the playback time of the words in the lyrics cannot be marked. In addition, this method requires accurate phonemes in the audio. Once the phoneme recognition is inaccurate, the phonemes in the audio and the corresponding phonemes in the lyrics cannot be aligned.

[0063] In order to solve the above problems, the present application provides a method for time annotation of text, which distinguishes two adjacent text units in the audio by the fundamental frequency of the audio, and then aligns the playback start and end time of the distinguished text units with the text units in the text to obtain the time annotation of the text. Because the focus of the embodiments of the present application is on how to identify the interval between two text units, rather than how to identify text units in the current prior art, the problem of text misalignment caused by the user singing incorrectly can be ignored, thereby improving the annotation accuracy. For example, Figure 5 Schematic diagram of time marking of text provided in the embodiment of the present application Figure 1 .like Figure 5As shown, in the embodiment of the present application, the intervals between two adjacent characters such as "yi" and "ke", "ke" and "ye" in the audio "yi ke ye bu neng fen ge" are obtained based on the fundamental frequency of the audio. Indirectly, each character in the audio "yi ke ye bu neng fen ge" can be aligned with the corresponding character in the lyrics "yi ke ye bu neng fen ge", and then the start and end times of each character in the lyrics can be accurately marked. It should be understood that the text time annotation method provided in the embodiment of the present application is not only applicable to the time annotation of lyrics, but also applicable to the time annotation of corresponding texts such as poems, human voices, and electronic human voices. The electronic human voice can be a synthesized human voice, such as the audio output during the interaction between a voice assistant and a user.

[0064] Figure 6 FIG. is a schematic diagram of a scenario applicable to the text time annotation method provided in the embodiment of the present application. As Figure 6 shown, this scenario may include a terminal device and a server. Among them, the terminal device can send audio to the server, and the server can perform time annotation on the text corresponding to the audio according to the audio. The terminal device in the embodiment of the present application can be referred to as a terminal Terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The terminal device can be a mobile phone, a tablet computer (pad), a computer with wireless transceiver function, a personal digital assistant (PDA), a handheld device with wireless communication function, a computer or other processing device, a wearable device (watch, sports bracelet, sports ankle ring, etc.), a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in a smart home, a terminal device in a 5G network, or a terminal device in a future evolved public land mobile network (PLMN), etc. The embodiment of the present application does not limit this.

[0065] The text time annotation method in the embodiment of the present application will be described in detail below with reference to specific embodiments. It should be understood that the execution subject of the text time annotation method in the embodiment of the present application can be a server or a terminal device. In the following embodiments, an electronic device (server or terminal device) is used as an example for description. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Figure 7 FIG. is a flowchart of an embodiment of the text time annotation method provided in the embodiment of the present application. As Figure 7As shown, the time marking method of the text provided in the embodiment of the present application may include:

[0066] S701, receiving a marking request, where the marking request is used to request marking the play start and end time of each text unit in the text corresponding to the target audio, where the text unit is any of the following: a character, a word, or a phrase.

[0067] S702, obtaining the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio.

[0068] S703, annotating the play start and end time of each text unit in the text according to the play start and end time of each text unit in the target audio, and obtaining an annotation file.

[0069] S704, output the annotation file.

[0070] In the above S701, it should be understood that when the electronic device is a server, the server can receive a labeling request from a terminal device or a labeling request input by a user. When the electronic device is a terminal device, the terminal device can receive a labeling request input by a user or a labeling request from other terminal devices. Exemplarily, a labeling control may be provided on the terminal device, and the user selects the labeling control (such as selecting the labeling control by clicking, sliding or pressing), which can trigger the input of a labeling request to the terminal device. The terminal device can receive the labeling request, or the terminal device can send a labeling request to the server (or other terminal device) so that the server (or other terminal device) receives the labeling request from the terminal device.

[0071] The marking request is used to request the start and end time of playing each text unit in the text corresponding to the target audio. Among them, the target audio can be a network audio or a local audio in a terminal device. When the target audio is a network audio, the marking request can include the identifier of the target audio, such as the title of a song, and the electronic device can download the target audio according to the identifier of the target audio. Alternatively, the target audio can be the audio recorded by the terminal device. It should be understood that because the interval of the text unit in the audio needs to be identified in the embodiment of the present application, in order to improve the accuracy of the time marking of the text, the target audio can only contain human voice audio, and the human voice can also be called dry voice, which refers to pure human voice without music. It should be understood that the human voice audio in the embodiment of the present application can be a sound emitted by a real person, or it can also be a synthesized human voice, such as a synthesized human voice played by a voice assistant. Or the human voice audio can also include non-human audio that can be converted into text, such as "meow" emitted by a kitten and "woof" emitted by a puppy. It should be understood that if the network audio or local audio is an audio containing non-human voice, the electronic device can also separate the non-human voice audio and the human voice audio to obtain a target audio containing only human voice audio.

[0072] The text corresponding to the target audio. Exemplarily, when the target audio is a song, the text corresponding to the target audio is the lyrics of the song; when the target audio is a poem, the text corresponding to the target audio is the poem; when the target audio is a hosting audio, the text corresponding to the target audio is the hosting words. The text unit can be at least one of the following: a Chinese character, a word, or a phrase. Exemplarily, when the text is a Chinese text, the text unit can be a Chinese character, such as "刻" (kè) or a phrase "一刻" (yī kè). When the text is a non-Chinese text, the text unit can be a word, such as "Sunday". In the embodiments of the present application, whether the text unit includes Chinese characters, phrases, or words can be preset. It should be understood that the smaller the composition of the text unit, such as a Chinese character being smaller than a phrase, the finer the text annotation granularity, that is, the higher the fineness. In the following embodiments, the text unit is taken as an example of a Chinese character for illustration.

[0073] The playing start and end times of the text unit include the playing start time and the playing end time. Exemplarily, for example, the playing start time and the playing end time of "刻" (kè) are [00:11.12] and [00:11.14] respectively, where "00" in 00:11.12 represents the minutes of the playing time, "11" represents the seconds of the time, and "12" represents the milliseconds of the time. That is to say, when the audio is played to [00:11.12], the terminal device can start rendering the lyrics "刻" (kè), and at [00:11.12], the rendering of "刻" (kè) is completed. Figure 8 Schematic diagram of the interface change of the terminal device provided by the embodiments of the present application Figure 1 . Such as Figure 8 As shown in interface 801 in, at this time, the playing time of the audio is [00:11.12], then the rendering of "刻" (kè) starts at this time. Figure 8 In, taking the rendering of the Chinese character "刻" (kè) from gray to black as an example, at this time, the left side of the Chinese character "刻" (kè) is rendered. When the playing time of the audio is [00:11.14], the interface can be as shown in 802, and at this time, the Chinese character "刻" (kè) is completely rendered black. In the embodiments of the present application, the rendering method of the terminal device is not limited. Figure 8 One example in.

[0074] In the above S702, the fundamental tone is the pure tone with the lowest frequency in each musical tone, and its intensity is the greatest. The frequency of the fundamental tone is the fundamental frequency, which determines the pitch of the entire tone. The fundamental frequency of the target audio can include the fundamental frequencies of each musical tone in the target audio. Exemplarily, the fundamental tones of "one", "moment", "also", "not", "can", "divide", "cut" can be regarded as "do", "mi", "si", "la", "mi", "so" respectively. Then, the fundamental frequency of each text unit in the target audio can be obtained. It should be understood that even for the same fundamental tone, the audio of each fundamental tone can be different. Exemplarily, the fundamental tones of "moment" and "divide" are both "mi", but the frequencies (i.e., the fundamental frequencies) of their fundamental tones can be different. Therefore, the fundamental frequency can be used to distinguish the intervals between adjacent two text units in the target audio.

[0075] In the embodiments of the present application, the start and end playback times of each text unit in the target audio can be obtained according to the fundamental frequency of the target audio. Specifically, the moment between adjacent two text units in the target audio can be obtained according to the change rate of the fundamental frequency of the target audio, and then the start and end playback times of each text unit in the target audio can be obtained. Optionally, in the embodiments of the present application, the moment when the change rate of the fundamental frequency is greater than the change rate threshold can be obtained, and this moment can be used as the moment to divide adjacent two text units in the target audio, and this moment can be used as the end playback time of the previous text unit in the target audio and the start playback time of the next text unit in the target audio. Fig. 9 It is a schematic diagram of the fundamental frequency provided by the embodiments of the present application. As Fig. 9 shown, the fundamental frequency corresponding to the waveform of the target audio is represented by the gray line in Fig. 9 . In the embodiments of the present application, the moment when the change rate of the fundamental frequency is greater than the change rate threshold can be marked, as shown by the black vertical line in the figure. Exemplarily, if the text corresponding to the target audio is "The cabbages are rolled up", then Fig. 9 the 5 black vertical lines in it are used to divide the 6 text units in the text. That is to say, the first black vertical line is used to divide "cab" and "bage", and the moment corresponding to the first black vertical line is the end playback time of "cab" and the start playback time of "bage". By analogy, the start and end playback times of each text unit in "The cabbages are rolled up" can be obtained. It should be understood that the start playback time of "cab" can be the end playback time of the previous text unit of "cab". If there is no previous text unit before "cab", the start playback time of the target audio can be used as the start playback time of "cab", as Fig. 9 shown.

[0076] In the above S703, in the embodiments of the present application, after obtaining the start and end playback times of each text unit in the target audio in the above S702 manner, the start and end playback times of each text unit in the text can be marked, and then a marked file can be obtained. Exemplarily, Fig.10 Schematic diagram of time marking of text provided in the embodiment of the present application Figure 2 . Fig.10 A in can refer to the above Figure 5 According to the relevant description of , in the embodiment of the present application, the time between two adjacent text units obtained from the target audio can be used to align the text corresponding to the target audio word by word, and then the play start and end time of each text unit in the target audio is mapped one by one to each text unit, such as Fig.10 As shown in B.

[0077] It should be understood that the annotation file can be a mapping relationship between each text unit in the text corresponding to the target audio and its corresponding playback start and end time. The embodiment of the present application does not limit the format of the annotation file, as long as the annotation text can represent the playback start and end time of each text unit in the text corresponding to the target audio.

[0078] In the above S704, in the embodiment of the present application, after the electronic device obtains the annotation file, the annotation file can be output. Exemplarily, when the electronic device is a server, the server can send the annotation file to the terminal device. When the electronic device is a terminal device, the terminal device can render the text corresponding to the target audio according to the annotation file, as described above. Figure 8 shown.

[0079] The time marking method of the text provided in the embodiment of the present application includes: receiving a marking request, obtaining the start and end time of playing each text unit in the target audio according to the fundamental frequency of the target audio, marking the start and end time of playing each text unit in the text according to the start and end time of playing each text unit in the target audio, obtaining a marking file, and outputting the marking file. Among them, the marking request is used to request the marking of the start and end time of playing each text unit in the text corresponding to the target audio, and the text unit is at least one of the following: a character, a word or a phrase. In the embodiment of the present application, the moment between two adjacent text units in the text corresponding to the target audio can be identified through the fundamental frequency in the target audio, that is, the two adjacent text units can be distinguished, and then the text units can be aligned. Instead of aligning the text units by using the current method of identifying text units, the problem of not being able to perform time marking of the text due to the inconsistency between the text unit in the target audio and the text unit in the text corresponding to the target audio is avoided. In the embodiment of the present application, the accuracy of time marking of the text can be improved.

[0080] The above embodiment describes a method for obtaining the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio. Fig.11 Schematic diagram of base frequency and short-time energy provided in the embodiment of the present application. Fig.11As shown, the text corresponding to the target audio is "The cabbages rolled up their hearts". According to the method of determining the time between two adjacent text units based on the fundamental frequency as described above, there are two moments between "heart" and "room" where the change rate of the fundamental frequency is greater than the change rate threshold, as shown by A and B in Fig.11 . However, there should be only one interval between "heart" and "room". Therefore, if only the fundamental frequency is used to obtain the playback start and end times of each text unit in the target audio, the accuracy may not be high. Accordingly, in order to improve the accuracy of obtaining the playback start and end times of each text unit, in the embodiments of the present application, the fundamental frequency of the target audio and the short-time energy of the target audio can also be combined to obtain the playback start and end times of each text unit. Fig.12 FIG. is a schematic flowchart of another embodiment of the method for annotating the time of text provided by the embodiments of the present application. As Fig.12 shown, S702 in the above embodiment can be replaced by S702': Obtain the playback start and end times of each text unit in the target audio according to the fundamental frequency of the target audio and the short-time energy of the target audio.

[0081] In S702', in the embodiments of the present application, the first moment when the change rate of the fundamental frequency in the target audio is greater than the change rate threshold can be obtained according to the fundamental frequency of the target audio. Among them, if the number of the first moments is equal to the number of text units in the text corresponding to the target audio minus 1, then it can be determined that the first moment is the moment between the text units in the text corresponding to the target audio. As described above Fig. 9 shown, the number of the first moments of the target audio "The cabbages rolled up" is 5, and the 5 first moments are the moments between the text units in the text corresponding to the target audio.

[0082] It should be noted that in the embodiments of the present application, if the number of the first moments is greater than or equal to the number of text units in the text corresponding to the target audio, it can be determined that there are at least two first moments between two text units in the text (such as the first text unit and the second text unit). For this case, in the embodiments of the present application, the short-time energy of the target audio can be used to determine the playback end time of the first text unit and the playback start time of the second text unit among the at least two first moments.

[0083] Among them, in the embodiments of the present application, the second moment can be obtained from the first moments according to the short-time energy. The average value of the short-time energy between two adjacent second moments is less than the first short-time energy threshold. That is to say, in the embodiments of the present application, the average value of the short-time energy between two adjacent first moments can be obtained. If the average value is less than the first short-time energy threshold, then both of the two adjacent first moments are used as the second moments. Exemplarily, as described above Fig.11There are 8 first moments in it. There are two first moments between "heart" and "room". If the mean value of the short-time energy between these two first moments is less than the first short-time energy threshold, then these two first moments can be used as the second moments.

[0084] In the embodiments of the present application, the start and end times of playing each text unit in the target audio can be obtained according to the first moment and the second moment, so as to obtain the start and end times of playing each text unit in the target audio. A possible implementation is: the mean value of continuously existing second moments can be used as the end time of playing the previous text unit and the start time of playing the next text unit. Exemplarily, the mean value of two second moments in Fig.11 can be used as the end time of playing "heart" and the start time of playing "room".

[0085] It should be understood that a song may contain multiple lines of lyrics. There may be a large interlude between one line of lyrics and another line of lyrics. There is no human voice during this interlude, and it may be a pure music interlude. If the entire target audio is taken as a whole and the end time of playing the last word of one line of lyrics is used as the start time of playing the first word of the next line of lyrics, then it may cause the problem that the rendering time of the first word of the next line of lyrics is too early. Exemplarily, the next line of lyrics after "Cabbages rolled up their hearts" is "There is fresh orange juice flowing in the white mist". If there is a large music interlude between these two lines and the end time of playing "heart" is used as the start time of playing "white", then it will cause the target audio not to play up to the word "white" yet, but the rendering of the word "white" has already started. Since the target audio can be composed of at least one sub-audio segment, in the embodiments of the present application, the time of each text unit in the text corresponding to the sub-audio segment can be obtained with one sub-audio segment as a whole, so as to obtain the start and end times of playing each text unit in the target audio. Exemplarily, the sub-audio segment can be an audio segment corresponding to one line of lyrics.

[0086] Fig.13 It is a schematic flowchart of another embodiment of the text time annotation method provided by the embodiments of the present application. As Fig.13 shown, S702 in the above embodiment can be replaced by S702”-S704”, and correspondingly, S703 in the above embodiment can be replaced by S705”.

[0087] S702”, according to the fundamental frequency of the sub-audio segment, obtain the first moments in the sub-audio segment where the change rate of the fundamental frequency is greater than the change rate threshold.

[0088] S703”, if the number of first moments is greater than or equal to the number of text units in the text corresponding to the sub-audio segment, then according to the short-time energy of the sub-audio segment, obtain the second moments among the first moments, and the mean value of the short-time energy between two adjacent second moments is less than the first short-time energy threshold.

[0089] S704”, according to the first moment and the second moment, the play start and end time of each text unit in the sub-audio segment is obtained to obtain the play start and end time of each text unit in the target audio.

[0090] S705”, annotate the play start and end time of each text unit in the sub-audio segment to each text unit in the text corresponding to the sub-audio segment to obtain an annotation file.

[0091] It should be understood that in the embodiment of the present application, the target audio can be divided into multiple sub-audio segments. One way to divide the sub-audio segments can be: if there is no human voice audio within a preset time after a human voice audio segment ends, the human voice audio can be used as a sub-audio segment.

[0092] In order to more accurately obtain the start time and end time of at least one sub-audio segment in the target audio, that is, to more accurately obtain the start time and end time of the first text unit in the text corresponding to the sub-audio segment, the short-time energy of the target audio can also be used in the present application to divide the target audio into at least one sub-audio segment. Among them, in the embodiment of the present application, at least one first duration in which the short-time energy in the target audio is less than the second short-time energy threshold can be obtained, and the target audio can be divided according to the playback time corresponding to the second duration in the first duration that is greater than the preset duration to obtain at least one sub-audio segment.

[0093] Exemplarily, the first duration in which the short-time energy of the audio after the playback of "The cabbages rolled up their atria" in the target audio is less than the second short-time energy threshold is 3s, and the preset duration is 2s, then the target audio can be segmented according to the playback time in the target audio corresponding to the first duration. If the moment corresponding to the first duration is [3:00.12] to [6:00.12], then in the embodiment of the present application, [3:00.12] can be used as the playback end time of the previous sub-audio segment, and [6:00.12] can be used as the playback start time of the next sub-audio segment. According to this method, the target audio can be divided into at least one sub-audio segment, and the playback start and end time of each sub-audio segment can be accurately obtained, thereby avoiding the problem of the last text unit in the text corresponding to each sub-audio segment being rendered too late, and the last text unit being rendered too early, so that the playback start and end time of each text unit in the text is more accurate.

[0094] The implementation of S702"-S703" may refer to the relevant description of S702', that is, the target audio in S702' may be replaced by the sub-audio segment.

[0095] In the above "S704", in the above embodiment, the average value of at least two second moments is used as the playback end time of the previous text unit and the playback start time of the next text unit. In fact, the accuracy is still relatively low. Because these two second moments may be the moments corresponding to the voiceless audio, which still delays the playback end time of the previous text unit and the playback start time of the next text unit.

[0096] In the embodiment of the present application, in order to more accurately obtain the playback start and end times of each text unit, when obtaining the playback start and end times of each text unit in the sub-audio segment according to the first moment and the second moment, at least one second moment group of the sub-audio segment can be obtained in the order of the third moment and the second moment from earliest to latest. Wherein, the second moment group is composed of at least two consecutive second moments, and the adjacent moments of the second moment group are the third moment. Wherein, the third moment is the moment in the first moment other than the second moment. Exemplarily, Fig.11 Two consecutive second moments between the "heart" and the "atrium" in can be used as a second moment group, and the other first moments are the third moments. Wherein, if there are three second moments between the "package" and the "dish", then in the embodiment of the present application, these three second moments can be used as a second moment group. It should be understood that the second moments in each second moment group in the embodiment of the present application are consecutive second moments in the first moment.

[0097] In the embodiment of the present application, each third moment and each second moment group can be used as a moment unit to obtain a plurality of moment units, and then, according to the order of the moment units from earliest to latest, the playback start and end times of each text unit in the sub-audio segment can be obtained. Exemplarily, as Fig.11 shown, in the embodiment of the present application, 6 third moments and 1 second moment group are used as a moment unit to obtain 7 moment units. Among them, according to the order of the moment units from earliest to latest, if the moment in the i-th moment unit is the third moment, then the third moment in the i-th moment unit is used as the playback end time of the i-th text unit in the sub-audio segment and the playback start time of the (i + 1)-th text unit; if the moment in the i-th moment unit is the second moment group, then the earliest second moment in the second moment group in the i-th moment unit is used as the playback end time of the i-th text unit in the sub-audio segment, and the latest second moment in the second moment group in the i-th moment unit is used as the playback start time of the (i + 1)-th text unit in the sub-audio segment. Wherein, i is an integer greater than or equal to 1.

[0098] Exemplarily, if the first moment unit (the first third moment) is used as the playback end time of "bao" and the playback start time of "cai", the second moment unit (the second third moment) is used as the playback end time of "cai" and the playback start time of "men", and so on. The sixth moment unit (the sixth third moment) is used as the playback end time of "le" and the playback start time of "xin"; the earliest second moment in the seventh moment unit (the first second moment group) is used as the playback end time of "xin", and the latest second moment in the seventh moment unit (the first second moment group) is used as the playback start time of "fang". It should be understood that the playback start time of "bao" and the playback end time of "fang" can be obtained when determining the playback start and end times of the sub-audio segment "bao cai men juan qi le xin fang". The playback start time of the sub-audio segment "bao cai men juan qi le xin fang" is the playback start time of "bao", and the playback end time of the sub-audio segment "bao cai men juan qi le xin fang" is the playback end time of "fang".

[0099] In the above "S705", the target audio in the embodiments of the present application is composed of at least one sub-audio segment, and the text corresponding to the target audio includes the text corresponding to the at least one sub-audio segment. In the embodiments of the present application, when annotating the text corresponding to the target audio, each text unit in the sub-audio segment is aligned with each text unit in the text corresponding to the sub-audio segment. Furthermore, the playback start and end times of each text unit in the sub-audio segment can be annotated to each text unit in the text corresponding to the sub-audio segment to obtain an annotation file.

[0100] In the embodiments of the present application, the playback start and end times of each text unit in the text corresponding to the target audio can be obtained according to the fundamental frequency and short-time energy of the target audio, which can improve the problem of being unable to accurately determine the interval between two text units caused by the fundamental frequency. Moreover, in the embodiments of the present application, the target audio can also be segmented into sub-audio segments, and thus the playback start time of the first text unit and the playback end time of the last text unit in each sub-audio segment can be determined more accurately, avoiding the problem of low user experience caused by premature or late rendering of text units.

[0101] Currently, the lyrics of the songs on the line are pre-edited and stored and uploaded manually, and the lyrics database generally includes the lyrics of the songs. For original songs, poems, or other human voice audios, etc., the database usually does not include the text corresponding to the audio. If the method of manually editing and uploading lyrics is adopted, the efficiency is low and it is easy to make mistakes. Therefore, based on the above embodiments, in the text time annotation method provided by the embodiments of the present application, the text corresponding to the target audio can also be recognized according to the target audio to improve the efficiency and accuracy. Fig.14 It is a schematic flowchart of another embodiment of the text time annotation method provided by the embodiments of the present application. As Fig.14As shown, the time marking method of the text provided in the embodiment of the present application may include:

[0102] S1401, receiving a labeling request.

[0103] S1402, obtaining the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio.

[0104] S1403, if the tagging request includes the identifier of the target audio, then based on the identifier of the target audio, query whether the database contains the text corresponding to the target audio; or, if the tagging request does not include the identifier of the target audio, then based on the audio fingerprint of the target audio, query whether the database contains the text corresponding to the target audio.

[0105] S1404: If the database does not contain the text corresponding to the target audio, speech recognition technology is used to recognize the target audio to obtain the text corresponding to the target audio.

[0106] S1405, annotate the play start and end time of each text unit in the text according to the play start and end time of each text unit in the target audio, and obtain an annotation file.

[0107] S1406, output the annotation file.

[0108] It should be understood that the implementation of S1401-S1402 and S1405-S1406 in the embodiment of the present application can refer to the relevant description of S701-S702 and S703-S704 in the above embodiment. S1402 and "S1403-S1404" in the embodiment of the present application can be executed simultaneously, and there is no distinction between the two in order.

[0109] In the above S1403, the markup request may include the identifier of the target audio. If the target audio is a song, the identifier of the target audio may be the title of the song. Among them, the user can enter the identifier of the target audio on the interface of the terminal device, and then carry the identifier of the target audio in the markup request. Among them, if the identifier of the target audio is included in the markup request, the electronic device can query whether the database contains the text corresponding to the target audio according to the identifier of the target audio. It should be understood that the database may be different according to the different scenarios of the application of the embodiment of the present application. If the scenario of the application of the embodiment of the present application is the time markup of lyrics, the database may be a lyrics database, and a large number of lyrics may be stored in the lyrics database, and each lyric corresponds to a song title. If the scenario of the application of the embodiment of the present application is the time markup of poems, the database may be a poetry database, and each lyric corresponds to a poem name. Optionally, the database in the embodiment of the present application may also be a database containing lyrics, poems and other texts.

[0110] Among them, when the user triggers the labeling request, if the song title is entered in the electronic device, the lyrics of the song, that is, the text corresponding to the target audio, can be queried according to the song title in the embodiment of the present application. Among them, if the labeling request does not include the identification of the target audio, the electronic device can query whether the database contains the text corresponding to the target audio based on the audio fingerprint of the target audio. Exemplarily, in the embodiment of the present application, the song of the target audio can be identified based on the audio fingerprint of the target audio, and then the lyrics of the song can be obtained from the database. It should be understood that the principle of using audio fingerprints to identify songs can refer to the relevant description of the prior art.

[0111] In the above S1404, it should be understood that if the database contains text corresponding to the target audio, the text corresponding to the target audio can be obtained. If the database does not contain the text corresponding to the target audio, the embodiment of the present application adopts speech recognition technology (automatic speech recognition, ASR) to recognize the target audio and obtain the text corresponding to the target audio.

[0112] For different versions of target audio, or target audio that is about to be launched, or target audio without text, etc., in an embodiment of the present application, manual editing of text can be avoided, and automatic recognition of the text corresponding to the target audio can be used to achieve high efficiency and high accuracy.

[0113] Based on the description of the above embodiment, the time marking process of the text in the embodiment of the present application is explained below for two application scenarios when the electronic device is a server or a terminal device. Fig.15 This is a flow chart of another embodiment of the method for time-marking text provided in the present application. It should be understood that Fig.15 The electronic device in is a server. Fig.15 As shown, the time marking method of the text provided in the embodiment of the present application may include:

[0114] S1501: The terminal device receives a play request input by a user.

[0115] S1502, the terminal device sends a marking request to the server.

[0116] Correspondingly, the server receives the annotation request from the terminal device.

[0117] S1503, if the marking request includes target audio, and the target is frequently human voice audio, then execute the following S1504; or, if the marking request includes initial audio, and the initial audio includes non-human voice audio and human voice audio, then extract the human voice audio from the initial audio as the target audio.

[0118] S1504: The server obtains the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio.

[0119] S1505, the server marks the play start and end time of each text unit in the text according to the play start and end time of each text unit in the target audio, and obtains a marked file.

[0120] S1506, the server sends the annotation file to the terminal device.

[0121] S1507, the terminal device plays the target audio, displays the text corresponding to the target audio, and renders and displays the target text unit when the target text unit in the target audio is played according to the annotation file.

[0122] It should be understood that the implementation of S1504-S1505 in the embodiments of the present application can refer to the relevant descriptions in S702-S703 in the above embodiments, and is not limited here.

[0123] In the above S1501, the play request is used to request the marking of the play start and end time of each text unit in the text corresponding to the target audio, and to play the target audio, and when playing the target audio, each text unit in the text is rendered and displayed according to the play start and end time of each text unit in the text corresponding to the target audio. That is to say, after the play request is used to request the marking of the play start and end time of each text unit in the text corresponding to the target audio, when the terminal device plays the target audio, it renders the text corresponding to the target audio according to the play start and end time of each text unit in the text corresponding to the target audio, and then displays the above-mentioned Figure 8 Interface 801 is shown.

[0124] Exemplarily, a “play” control may be displayed on the interface of the terminal device, and after the user triggers and selects the “play” control, a play request may be input to the terminal device. Fig.16 Schematic diagram of the interface changes of the terminal device provided in the embodiment of the present application Figure 2 . Fig.16 A "play" control may be displayed on interface 1601. After the user triggers the "play" control, interface 1601 may jump to interface 1602. A control for uploading audio may be displayed on interface 1602. The user may select the "upload" control to upload the audio. Alternatively, in an embodiment of the present application, the user may also select the "play" control after selecting the audio. In an embodiment of the present application, there is no restriction on the way in which the user uploads the audio and how to trigger the input of the play request. Optionally, an input box for inputting the audio identifier may also be displayed on interface 1601. If the audio uploaded by the user is an adapted song A, the user may enter the song title "A" in the input box. Optionally, the user may not enter the audio identifier.

[0125] In the above S1503, it should be noted that if the audio uploaded by the user is the target audio (audio containing only human voice audio), that is, the audio carried in the above play request is the target audio, the server can execute S1504. If the audio uploaded by the user is the initial audio, the initial audio includes non-human voice audio and human voice audio (that is, non-pure human voice audio), the server can extract the human voice audio from the initial audio as the target audio.

[0126] In an embodiment of the present application, a voice recognition model may be pre-stored in the server, and the human voice audio with non-human voice audio removed may be obtained by inputting the initial audio into the human voice recognition model. The human voice recognition model is used to separate the human voice audio and non-human voice audio in the initial audio. Optionally, the human voice recognition model in the embodiment of the present application is a neural network model, and the server may pre-train the human voice recognition model with training parameters of pure human voice audio and accompaniment audio synthesized training audio as training parameters. Optionally, the neural network model in the embodiment of the present application may be, but is not limited to, a U-net network model.

[0127] In the above S1502 and S1506, after receiving the play request, the terminal device can send a marking request to the server to trigger the server to execute the time marking method of the text in the above embodiment to obtain a marking file. After the server obtains the marking file, it can send the marking file to the terminal device.

[0128] In the above S1507, the terminal device can play the target audio, and display the text corresponding to the target audio when playing the target audio. In the embodiment of the present application, when displaying the text corresponding to the target audio, the terminal device can render and display the target text unit according to the annotation file when the target text unit in the target audio is played. That is to say, the terminal device can start rendering the target text unit at the start time of the target text unit according to the time of playing the target audio and the start and end time of the playback corresponding to each text unit in the annotation file, and complete the rendering of the target text unit at the end time of the playback of the target text unit. Exemplarily, the above interface 1602 can jump to interface 1603, which is the same as the above interface 1603. Figure 8 It should be noted that in order to illustrate the schematic diagram of the terminal device rendering the text unit in the embodiment of the present application, the interface 1603 is taken as an example to illustrate the interface in which the target audio has been played for a period of time.

[0129] In the embodiment of the present application, the terminal device can interact with the server, and the server can time-mark the text corresponding to the target audio, thereby obtaining an accurate marking file. Other technical effects in the embodiment of the present application can refer to the relevant description in the above embodiment.

[0130] Fig.17 This is a flow chart of another embodiment of the method for time-marking text provided in the present application. It should be understood that Fig.17 The electronic device in is the terminal device. Fig.17 As shown, the time marking method of the text provided in the embodiment of the present application may include:

[0131] S1701, receiving a play request input by a user.

[0132] S1702, if the marking request includes target audio, and the target is frequently human voice audio, then execute the following S1703; if the marking request includes initial audio, and the initial audio includes non-human voice audio and human voice audio, then extract the human voice audio from the initial audio as the target audio.

[0133] S1703, obtaining the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio.

[0134] S1704: annotate the play start and end time of each text unit in the text according to the play start and end time of each text unit in the target audio to obtain an annotation file.

[0135] S1705, the terminal device plays the target audio, displays the text corresponding to the target audio, and renders and displays the target text unit when the target text unit in the target audio is played according to the annotation file.

[0136] It should be understood that S1701-S1705 in the embodiments of the present application can refer to the relevant descriptions in S1501, S1503-S1505 and S1507 in the above embodiments, and will not be repeated here.

[0137] In the embodiment of the present application, the terminal device can time-mark the text corresponding to the target audio, thereby obtaining an accurate marking file. Other technical effects in the embodiment of the present application can refer to the relevant description in the above embodiment.

[0138] Fig.18 The schematic diagram of the structure of the time marking device of the text provided in the embodiment of the present application. The time marking device of the text can be the server or terminal device in the above embodiment. The time marking device of the text is used to execute the action of the server or terminal device in the above method embodiment. Fig.18 As shown, the text time tagging device 1800 may include: a transceiver module 1801, a processing module 1802 and an output module 1803.

[0139] The transceiver module 1801 is used to receive a marking request, wherein the marking request is used to request marking the play start and end time of each text unit in the text corresponding to the target audio, and the text unit is at least one of the following: a character, a word or a phrase.

[0140] Processing module 1802 is used to obtain the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio, and to mark the play start and end time of each text unit in the text according to the play start and end time of each text unit in the target audio to obtain a marking file.

[0141] The output module 1803 is used to output the annotation file.

[0142] In a possible implementation, the processing module 1802 is specifically configured to obtain the play start and end time of each text unit in the target audio according to the fundamental frequency of the target audio and the short-time energy of the target audio.

[0143] In a possible implementation, the target audio is composed of at least one sub-audio segment. The processing module 1802 is specifically configured to obtain, based on the fundamental frequency of the sub-audio segment, a first moment at which the change rate of the fundamental frequency in the sub-audio segment is greater than a change rate threshold, and if the number of the first moments is greater than or equal to the number of text units in the text corresponding to the sub-audio segment, then, based on the short-time energy of the sub-audio segment, obtain a second moment in the first moment, the average of the short-time energy between two adjacent second moments is less than a first short-time energy threshold, and obtain the play start and end time of each text unit in the sub-audio segment based on the first moment and the second moment, so as to obtain the play start and end time of each text unit in the target audio.

[0144] In a possible implementation, the third moment is a moment other than the second moment in the first moment. The processing module 1802 is specifically configured to obtain at least one second moment group of the sub-audio segment in the order of the third moment and the second moment from early to late, wherein the second moment group is composed of at least two consecutive second moments, and the adjacent moments of the second moment group are the third moments; each third moment and each second moment group is taken as a moment unit to obtain a plurality of moment units; and the play start and end time of each text unit in the sub-audio segment in the order of the moment units from early to late.

[0145] In a possible implementation, the processing module 1802 is specifically used to, in the order of the time units from early to late, if the time in the i-th time unit is the third time, then the third time in the i-th time unit is used as the end time of playing the i-th text unit in the sub-audio segment and the start time of playing the i+1-th text unit; if the time in the i-th time unit is a second time group, then the earliest second time in the second time group in the i-th time unit is used as the end time of playing the i-th text unit in the sub-audio segment, and the latest second time in the second time group in the i-th time unit is used as the start time of playing the i+1-th text unit in the sub-audio segment. Wherein, i is an integer greater than or equal to 1.

[0146] In a possible implementation, the tagging request includes the target audio, and the target audio is human voice audio; or the tagging request includes initial audio, and the initial audio includes non-human voice audio and human voice audio. The processing module 1802 is further configured to extract human voice audio from the initial audio as the target audio.

[0147] In a possible implementation, the processing module 1802 is specifically configured to input the initial audio into a human voice recognition model to obtain human voice audio from which the non-human voice audio is removed.

[0148] In one possible implementation, the processing module 1802 is also used to query whether the database contains text corresponding to the target audio based on the identifier of the target audio if the annotation request includes the identifier of the target audio; or, if the annotation request does not include the identifier of the target audio, query whether the database contains text corresponding to the target audio based on the audio fingerprint of the target audio; if the database does not contain text corresponding to the target audio, use speech recognition technology to identify the target audio and obtain the text corresponding to the target audio.

[0149] In a possible implementation, the target audio is composed of at least one sub-audio segment, and the text corresponding to the target audio includes the text corresponding to the at least one sub-audio segment. The processing module 1802 is specifically configured to annotate the play start and end time of each text unit in the sub-audio segment to each text unit in the text corresponding to the sub-audio segment to obtain the annotated file.

[0150] In a possible implementation, the processing module 1802 is further used to obtain at least one first duration in the target audio whose short-time energy is less than a second short-time energy threshold; and to divide the target audio according to the playback time corresponding to the second duration in the first duration that is greater than a preset duration, to obtain the at least one sub-audio segment.

[0151] In a possible implementation, the electronic device is a server, and the transceiver module 1801 is specifically configured to receive the annotation request from the terminal device. The output module 1803 is specifically configured to send the annotation file to the terminal device.

[0152] In one possible implementation, the electronic device is a terminal device, and the transceiver module 1801 is specifically used to receive a playback request for the target audio input by a user, wherein the playback request is used to request marking of the playback start and end time of each text unit in the text corresponding to the target audio, and to play the target audio, and when playing the target audio, each text unit in the text is rendered and displayed according to the playback start and end time of each text unit in the text corresponding to the target audio.

[0153] The output module 1803 is specifically used to play the target audio and display the text corresponding to the target audio; according to the annotation file, when the target text unit in the target audio is played, the target text unit is rendered and displayed.

[0154] The implementation principle and technical effect of the text time marking device provided in the embodiment of the present application are similar to those of the server or terminal device in the above-mentioned embodiment, and will not be repeated here.

[0155] It should be noted that the above processing unit can be implemented in the form of software calling through a processing element; it can also be implemented in the form of hardware. For example, the processing unit can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The functions of the above processing module. In addition, all or part of the above units can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by an integrated logic circuit of hardware in the processor element or an instruction in the form of software. For example, the above units can be one or more integrated circuits configured to implement the above method, such as: one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For example, when a certain module above is implemented in the form of a processing element scheduling program code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0156] Fig.19 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.19 The electronic device shown may be the device to be charged in the above embodiment. Fig.19As shown, the electronic device may include: a processor 1901, a memory 1902 and a transceiver 1903; the memory 1902 may include a high-speed random access memory (random-access memory, RAM), and may also include a non-volatile memory (non-volatile memory, NVM), such as at least one disk storage, and the memory 1902 may store various instructions for completing various processing functions and implementing the method steps of the present application. The transceiver 1903 can be coupled to the processor 1901, and the processor 1901 can control the transmission and reception of information between other electronic devices. When the processor 1901 executes the instruction, the instruction causes the processor 1901 of the electronic device to perform the processing action of the electronic device in the above method embodiment, and causes the transceiver 1903 to perform the transmission and reception action of the electronic device in the above method embodiment. Optionally, the electronic device involved in the present application may also include: a power supply 1904, a communication bus 1905, a communication port 1906 and a display 1907. The communication bus 1905 is used to realize the communication connection between components. The display 1907 is used to display the user interface of the electronic device. The communication port 1906 is used to realize the connection and communication between the electronic device and other peripherals. In the embodiment of the present application, the memory 1902 is used to store computer executable program codes, and the program codes include instructions.

[0157] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integration. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state hard disk Solid State Disk (SSD)), etc.

[0158] The term "plurality" in this article refers to two or more than two. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship; in a formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship.

[0159] It is to be understood that the various digital numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. It is to be understood that in the embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto, and any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A time tagging method for text, characterized in that: Applied to electronic equipment, the method comprises: Receiving a marking request, the marking request is used to request marking the play start and end time of each text unit in the text corresponding to the target audio, the text unit being at least one of the following: a character, a word, or a phrase; Obtaining the start and end time of playing each text unit in the target audio according to the change rate of the fundamental frequency of the target audio; The playback end time of the previous text unit and the playback start time of the next text unit in the target audio are determined according to the moment when the change rate of the fundamental frequency is greater than the change rate threshold; according to the playback start and end time of each text unit in the target audio, the playback start and end time of each text unit in the text is marked to obtain a marked file; The annotation file is output.

2. The method according to claim 1, characterized in that The step of obtaining the start and end time of playing each text unit in the target audio according to the change rate of the fundamental frequency of the target audio comprises: The play start and end time of each text unit in the target audio is obtained according to the change rate of the fundamental frequency of the target audio and the short-time energy of the target audio.

3. The method according to claim 2, characterized in that The target audio is composed of at least one sub-audio segment; and obtaining the play start and end time of each text unit in the target audio according to the change rate of the fundamental frequency of the target audio and the short-time energy of the target audio includes: According to the change rate of the fundamental frequency of the sub-audio segment, acquiring a first moment when the change rate of the fundamental frequency in the sub-audio segment is greater than a change rate threshold; If the number of the first moments is greater than or equal to the number of text units in the text corresponding to the sub-audio segment, obtaining a second moment in the first moment according to the short-time energy of the sub-audio segment, and an average of the short-time energies between two adjacent second moments is less than a first short-time energy threshold; According to the first moment and the second moment, the play start and end time of each text unit in the sub-audio segment is acquired to obtain the play start and end time of each text unit in the target audio.

4. The method according to claim 3, characterized in that The third moment is a moment other than the second moment in the first moment, and acquiring the play start and end time of each text unit in the sub-audio segment according to the first moment and the second moment includes: Acquire at least one second time group of the sub-audio segment in a descending order of the third time and the second time, wherein the second time group consists of at least two consecutive second time periods, and adjacent time periods of the second time group are third time periods; Taking each third moment and each second moment group as a moment unit, a plurality of moment units are obtained; The play start and end time of each text unit in the sub-audio segment is obtained according to the order of the time units from early to late.

5. The method according to claim 4, characterized in that The step of obtaining the play start and end time of each text unit in the sub-audio segment in the order of the time units from early to late includes: According to the order of the time units from early to late, if the time in the i-th time unit is the third time, the third time in the i-th time unit is used as the playback end time of the i-th text unit in the sub-audio segment and the playback start time of the i+1-th text unit, where i is an integer greater than or equal to 1; If the moment in the i-th moment unit belongs to the second moment group, the earliest second moment in the second moment group in the i-th moment unit is used as the playback end time of the i-th text unit in the sub-audio segment, and the latest second moment in the second moment group in the i-th moment unit is used as the playback start time of the i+1-th text unit in the sub-audio segment.

6. The method according to any one of claims 1 to 5, characterized in that The marking request includes the target audio, and the target is frequently human voice audio; or; The annotation request includes initial audio, and the initial audio includes non-human voice audio and human voice audio; the method further includes: The human voice audio is extracted from the initial audio as the target audio.

7. The method according to claim 6, characterized in that The extracting human voice audio from the initial audio comprises: The initial audio is input into a human voice recognition model to obtain human voice audio with the non-human voice audio removed.

8. The method according to any one of claims 1 to 5 and 7, characterized in that: Before obtaining the play start and end time of each text unit in the target audio according to the change rate of the fundamental frequency of the target audio, the method further includes: If the tagging request includes the identifier of the target audio, query whether the database contains the text corresponding to the target audio according to the identifier of the target audio; or, if the tagging request does not include the identifier of the target audio, query whether the database contains the text corresponding to the target audio according to the audio fingerprint of the target audio; If the database does not contain the text corresponding to the target audio, speech recognition technology is used to recognize the target audio to obtain the text corresponding to the target audio.

9. The method according to any one of claims 1 to 5 and 7, characterized in that: The target audio is composed of at least one sub-audio segment, and the text corresponding to the target audio includes the text corresponding to the at least one sub-audio segment; and the marking of the playing start and end time of each text unit in the text according to the playing start and end time of each text unit in the target audio to obtain the marking file includes: The play start and end time of each text unit in the sub-audio segment is marked to each text unit in the text corresponding to the sub-audio segment to obtain the marking file.

10. The method according to claim 9, characterized in that The method further comprises: Acquire at least one first duration in which the short-time energy in the target audio is less than a second short-time energy threshold; The target audio is segmented by the playback time corresponding to the second duration greater than the preset duration in the first duration to obtain the at least one sub-audio segment.

11. The method according to any one of claims 1 to 5, 7 and 10, characterized in that The electronic device is a server, and the receiving of the annotation request includes: Receiving the annotation request from a terminal device; The outputting the annotation file comprises: The annotation file is sent to the terminal device.

12. The method according to any one of claims 1 to 5, 7 and 10, characterized in that The electronic device is a terminal device, and the receiving of the labeling request includes: Receive a play request for the target audio input by a user, the play request being used to request marking the play start and end time of each text unit in the text corresponding to the target audio, and play the target audio, and when playing the target audio, render and display each text unit in the text according to the play start and end time of each text unit in the text corresponding to the target audio; The outputting the annotation file comprises: Play the target audio and display the text corresponding to the target audio; According to the annotation file, when the target text unit in the target audio is played, the target text unit is rendered and displayed.

13. A time marking device for text, characterized in that: include: A transceiver module, used for receiving a marking request, wherein the marking request is used for requesting to mark the play start and end time of each text unit in the text corresponding to the target audio, wherein the text unit is at least one of the following: a character, a word or a phrase; A processing module, used to obtain the play start and end time of each text unit in the target audio according to the change rate of the fundamental frequency of the target audio; the play end time of the previous text unit and the play start time of the next text unit in two adjacent text units in the target audio are determined according to the moment when the change rate of the fundamental frequency is greater than the change rate threshold; According to the play start and end time of each text unit in the target audio, mark the play start and end time of each text unit in the text to obtain a marked file; An output module is used to output the annotation file.

14. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is used to couple with the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 12; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

15. A computer-readable storage medium, characterized in that: The computer storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 12.

16. A computer program product comprising: A computer program, characterized in that when the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 12.

Images