TTS voice corresponding subtitle scrolling method, device and storage medium

Abstract

The present application discloses a method, device and storage medium for scrolling subtitles corresponding to TTS voice. The method includes: determining a first percentage value of a scrolling start position of subtitles according to the total height of all subtitles corresponding to an audio file and the maximum display height of subtitles displayed in a screen image to be drawn, wherein the audio file is an audio file obtained through a text-to-speech technology; determining a second percentage value of the current playback progress of the audio file when drawing the screen image of the current frame; determining a scrolling position of a scroll bar corresponding to the screen image of the current frame according to the total height of all subtitles, the scrolling start position and the current playback progress when determining that the second percentage value is greater than the first percentage value, wherein the scroll bar is used to scroll and display all subtitles in the screen image; and setting the position of the scroll bar according to the scrolling position.

Description

Technical Field

[0001] The present application relates to the technical field of subtitle processing, and in particular to a method, device and storage medium for scrolling subtitles corresponding to TTS speech. Background Art

[0002] TTS is the abbreviation of Text To Speech, which is a part of human-computer dialogue that enables machines to speak. TTS speech, in layman's terms, is speech obtained through text-to-speech technology. In some special scenarios, when playing such TTS voice files, it is necessary to scroll and display the corresponding subtitles. However, since the TTS voice files in these scenarios are temporarily generated based on text conversion, it is impossible to set the scrolling time of the subtitles in advance like songs, that is, it is impossible to display the corresponding subtitles according to the preset display time. Therefore, it is impossible to scroll the subtitles corresponding to the TTS voice files smoothly and accurately.

[0003] With regard to the technical problem in the above-mentioned prior art that the TTS voice file is temporarily generated according to text conversion, the display time of the subtitles corresponding to the TTS voice file cannot be set in advance, and thus the subtitles corresponding to the TTS voice file cannot be scrolled smoothly and accurately, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device and storage medium for scrolling subtitles corresponding to TTS voice, so as to at least solve the technical problem in the prior art that the display time of the subtitles corresponding to the TTS voice file cannot be set in advance because the TTS voice file is temporarily generated according to the text conversion, and thus the subtitles corresponding to the TTS voice file cannot be scrolled smoothly and accurately.

[0005] According to one aspect of an embodiment of the present disclosure, a method for scrolling subtitles corresponding to TTS voice is provided, comprising: determining a first percentage value of a scrolling start position of subtitles according to a total height of all subtitles corresponding to an audio file and a maximum display height of subtitles displayed in a screen image to be drawn, wherein the audio file is an audio file obtained via a text-to-speech technology; determining a second percentage value of a current playback progress of the audio file when drawing a screen image of a current frame; determining a scrolling position of a scroll bar corresponding to a screen image of a current frame according to a total height of all subtitles, a scrolling start position, and a current playback progress when it is determined that the second percentage value is greater than the first percentage value, wherein the scroll bar is used to scroll and display all subtitles in the screen image; and setting the position of the scroll bar according to the scrolling position.

[0006] According to another aspect of an embodiment of the present disclosure, a storage medium is further provided, the storage medium including a stored program, wherein when the program is running, a processor executes any one of the methods described above.

[0007] According to another aspect of an embodiment of the present disclosure, a scrolling device for subtitles corresponding to TTS voice is also provided, including: a first determination module, used to determine a first percentage value of a scrolling start position of subtitles according to a total height of all subtitles corresponding to an audio file and a maximum display height of subtitles displayed in a screen image to be drawn, wherein the audio file is an audio file obtained through text-to-speech technology; a second determination module, used to determine a second percentage value of a current playback progress of the audio file when drawing a screen image of a current frame; a third determination module, used to determine a scrolling position of a scroll bar corresponding to a screen image of a current frame according to the total height of all subtitles, a scrolling start position and a current playback progress when it is determined that the second percentage value is greater than the first percentage value, wherein the scroll bar is used to scroll and display all subtitles in the screen image; and a setting module, used to set the position of the scroll bar according to the scrolling position.

[0008] According to another aspect of an embodiment of the present disclosure, a scrolling device for subtitles corresponding to TTS voice is also provided, including: a processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: determining a first percentage value of a scrolling start position of the subtitles according to a total height of all subtitles corresponding to an audio file and a maximum display height of subtitles displayed in a screen image to be drawn, wherein the audio file is an audio file obtained via text-to-speech technology; determining a second percentage value of a current playback progress of the audio file when drawing a screen image of a current frame; when determining that the second percentage value is greater than the first percentage value, determining a scrolling position of a scroll bar corresponding to the screen image of the current frame according to the total height of all subtitles, the scrolling start position and the current playback progress, wherein the scroll bar is used to scroll and display all subtitles in the screen image; and setting the position of the scroll bar according to the scrolling position.

[0009] In the disclosed embodiment, firstly, the value of the scrolling start position of the subtitles is determined by the total height of all subtitles corresponding to the audio file obtained by the text-to-speech technology and the maximum display height of the subtitles displayed in the screen image to be drawn, then the current playback progress of the audio file obtained by the text-to-speech technology is determined, and finally the scrolling position of the scroll bar corresponding to the screen image of the current frame is determined according to the total height of all subtitles, the scrolling start position and the current playback progress, so as to accurately set the position of the scroll bar in the screen image of the current frame. By connecting the screen images of all frames in which the accurate setting of the scroll bar position is completed, the technical effect of smoothly and accurately scrolling the subtitles displayed on the screen according to the playback progress of the TTS voice is achieved. In addition, the technical problem existing in the prior art that the display time of the subtitles corresponding to the TTS voice file cannot be set in advance because the TTS voice file is temporarily generated according to the text conversion, and thus the subtitles corresponding to the TTS voice file cannot be smoothly and accurately scrolled is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0011] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to Embodiment 1 of the present disclosure;

[0012] Figure 2 It is a flowchart of the method for scrolling subtitles corresponding to TTS speech according to Embodiment 1 of the present disclosure;

[0013] Figure 3a is a schematic diagram of the maximum display height of subtitles displayed in the screen image to be drawn according to Embodiment 1 of the present disclosure;

[0014] Figure 3b is a schematic diagram of the total height of all subtitles corresponding to the audio file according to Embodiment 1 of the present disclosure;

[0015] Figure 3c is a schematic diagram of subtitles displayed after the position of the scroll bar is set according to the calculated scroll position according to Embodiment 1 of the present disclosure;

[0016] Figure 3d is another schematic diagram of subtitles displayed after the position of the scroll bar is set according to the calculated scroll position according to Embodiment 1 of the present disclosure;

[0017] Figure 3eis another schematic diagram of subtitles displayed after the position of the scroll bar is set according to the calculated scroll position according to Embodiment 1 of the present disclosure;

[0018] Figure 4 is a schematic diagram of the audio clip corresponding to each line of subtitles and the audio playback progress according to Embodiment 1 of the present disclosure;

[0019] Figure 5 It is a schematic diagram of the overall process of the method for scrolling subtitles corresponding to TTS voice according to Embodiment 1 of the present disclosure;

[0020] Figure 6 is a schematic diagram of a scrolling device for TTS voice corresponding to subtitles according to Embodiment 2 of the present disclosure; and

[0021] Figure 7 It is a schematic diagram of a scrolling device for TTS voice corresponding to subtitles according to Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0024] First, some nouns or terms that appear in the process of describing the embodiments of the present disclosure are subject to the following explanations:

[0025] Term 1: TTS is the abbreviation of Text To Speech, which means "from text to speech". It is a part of human-computer dialogue that enables machines to speak;

[0026] Term 2: TTS voice, in layman's terms, is the voice obtained through text-to-speech technology.

[0027] Example 1

[0028] According to this embodiment, an embodiment of a method for scrolling subtitles corresponding to TTS speech is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] The method embodiment provided in this embodiment can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Figure 1 The hardware structure block diagram of a computing device for a method of scrolling subtitles corresponding to TTS speech is shown. Figure 1 As shown, the computing device may include one or more processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0030] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computing device. As involved in the embodiments of the present disclosure, the data processing circuitry acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0031] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the scrolling method of TTS voice corresponding to subtitles in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the scrolling method of TTS voice corresponding to subtitles of the above-mentioned application program is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the computing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computing device. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0033] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of the computing device.

[0034] It should be noted that, in some optional embodiments, the above Figure 1 The computing device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computing devices described above.

[0035] In the above operating environment, according to the first aspect of this embodiment, a method for scrolling subtitles corresponding to TTS speech is provided. Figure 2 A schematic diagram showing the process of the method is shown in FIG. Figure 2 As shown, the method includes:

[0036] S202: determining a first percentage value of a scrolling start position of the subtitles according to a total height of all subtitles corresponding to the audio file and a maximum display height of the subtitles displayed in the screen image to be drawn, wherein the audio file is an audio file obtained by a text-to-speech technology;

[0037] S204: in the case of drawing the screen image of the current frame, determining a second percentage value of the current playing progress of the audio file;

[0038] S206: if it is determined that the second percentage value is greater than the first percentage value, determining a scroll position of a scroll bar corresponding to the screen image of the current frame according to the total height of all subtitles, the scrolling start position, and the current playback progress, wherein the scroll bar is used to scroll and display all subtitles in the screen image; and

[0039] S208: Setting the position of the scroll bar according to the scroll position.

[0040] As mentioned in the background above, TTS is the abbreviation of Text To Speech, which is "from text to speech". It is a part of human-computer dialogue and enables machines to speak. TTS speech, in layman's terms, is speech obtained through text-to-speech technology. In some special scenarios, when playing such TTS voice files, it is necessary to scroll and display the corresponding subtitles. However, since the TTS voice files in these scenarios are temporarily generated based on text conversion, it is impossible to set the scrolling time of the subtitles in advance like songs, that is, it is impossible to display the corresponding subtitles according to the preset display time. Therefore, it is impossible to scroll the subtitles corresponding to the TTS voice files smoothly and accurately.

[0041] In view of the technical problems existing in the background technology, the technical solution proposed in this embodiment, in the process of scrolling the subtitles corresponding to the TTS voice file, first determines the first percentage value of the scrolling start position of the subtitles according to the total height of all subtitles corresponding to the audio file and the maximum display height of the subtitles displayed in the screen image to be drawn. Figure 3a and Figure 3b As shown, the maximum display height of the subtitles displayed in the screen to be drawn is a, the total height of all subtitles corresponding to the audio file is b, and the width of all subtitles corresponding to the audio file is consistent with the width of the subtitles displayed in the screen to be drawn. At this time, the first percentage value of the scrolling start position of the subtitles is, for example, d, and the calculation formula is, for example, d=a / b / 2 or d=a / b / 3, which is not specifically limited here. The scrolling start position is used to indicate the scrolling start position of the subtitles, that is, the subtitles start to scroll when the start position is reached.

[0042] Furthermore, since the screen is refreshed frame by frame to produce the effect of continuous animation according to the phenomenon of visual persistence, everything (including subtitles) needs to be redrawn when drawing each frame of the screen. In this application scenario, considering that the TTS voice file is different from the song, there is no transition sound effect, and it is a compact and coherent voice playback, so the subtitles can be scrolled by the playback progress of the TTS voice file. Therefore, in the case of drawing the screen of the current frame, it is necessary to determine the second percentage value (for example, f) of the current playback progress of the audio file. Then, in the case of determining that the second percentage value is greater than the first percentage value, according to the total height of all subtitles, the scrolling start position and the current playback progress, the scrolling position of the scroll bar corresponding to the screen of the current frame is determined, wherein the scroll bar is used to scroll and display all subtitles in the screen. Among them, the second percentage value is greater than the first percentage value, which means that the current playback progress of the audio file is greater than the scrolling start position of the subtitles, and the subtitles need to be scrolled accordingly. Among them, the scroll bar is a system View control, which is used to scroll and display all subtitles on the screen, that is, the total height of the scroll bar is equal to the total height of all subtitles, and the display height of the scroll bar is equal to the maximum display height of the displayed subtitles. The subtitles corresponding to the remaining heights are hidden. By setting the position of the scroll bar to display subtitles corresponding to different heights, all playbacks are connected, and the scroll position g is continuously calculated and set on the screen to realize the scrolling display of subtitles along with the playback progress of the TTS voice.

[0043] In a specific embodiment, the scroll position of the scroll bar is g, and the scroll range is 0-b (b is equal to the total height of the TTS subtitles, such as in this example: 713mm). Since the playback progress f is continuously notified when the TTS audio is played (for example, the progress is notified every 100 milliseconds, where the audio playback progress f ranges from 0-1), the scroll position of the scroll bar can be changed synchronously with the notification of the audio playback progress. The calculation formula for the scroll position is g=b*(fd). Exemplarily, in this embodiment, a=330mm, b=713mm, and according to the calculation formula d=a / b / 2, d=330 / 713 / 2=23% is obtained. Therefore, the calculated scroll starting point position is 23%. When the playback progress f of the TTS audio is 0~0.23, since the current playback progress f of the TTS voice is not greater than the calculated scroll starting position d, the subtitles are not scrolled and displayed. At this time, the scroll position g of the scroll bar is 0, and the scroll position does not change (refer to Figure 3aWhen the playback progress f=25%, since the current playback progress f of the TTS voice is greater than the calculated scrolling start position d, the scrolling position of the scroll bar needs to be calculated. At this time, according to the calculation formula g=b*(fd), g=713*(25%-23%)=14mm is calculated. At this time, the scroll bar position needs to be set according to the calculated scrolling position g=14mm (refer to Figure 3c When the playback progress f=50%, since the current playback progress f of the TTS voice is greater than the calculated scrolling start position d, it is necessary to calculate the scrolling position of the scroll bar. At this time, according to the calculation formula g=b*(fd), g=713*(50%-23%)=227mm is calculated. At this time, the position of the scroll bar needs to be set according to the calculated scrolling position g=227mm (refer to Figure 3d When the playback progress f=1, since the current playback progress f of the TTS voice is greater than the calculated scrolling start position d, it is necessary to calculate the scrolling position of the scroll bar. At this time, according to the calculation formula g=b*(fd), g=713*(100%-23%)=549mm is calculated. At this time, it is necessary to set the position of the scroll bar according to the calculated scrolling position g=549mm (refer to Figure 3e shown).

[0044] Thus, in this way, the present embodiment first determines the value of the scrolling start position of the subtitles by the total height of all subtitles corresponding to the audio file obtained by the text-to-speech technology and the maximum display height of the subtitles displayed in the screen image to be drawn, and then determines the current playback progress of the audio file obtained by the text-to-speech technology, and finally determines the scrolling position of the scroll bar corresponding to the screen image of the current frame according to the total height of all subtitles, the scrolling start position and the current playback progress, so as to accurately set the position of the scroll bar in the screen image of the current frame. By connecting the screen images of all frames in which the accurate setting of the scroll bar position is completed, the technical effect of smoothly and accurately scrolling the subtitles displayed on the screen according to the playback progress of the TTS voice is achieved. In addition, the technical problem existing in the prior art that the display time of the subtitles corresponding to the TTS voice file cannot be set in advance because the TTS voice file is temporarily generated according to the text conversion, and thus the subtitles corresponding to the TTS voice file cannot be smoothly and accurately scrolled is solved.

[0045] Optionally, the operation of determining a first percentage value of a scrolling start position of subtitles based on a total height of all subtitles corresponding to an audio file and a maximum display height of subtitles displayed in a screen image to be drawn includes: calculating a ratio of a maximum display height of subtitles to a total height of all subtitles; and determining half of the ratio as a first percentage value.

[0046] Specifically, in the process of calculating the first percentage value of the scrolling start position of the subtitles, the ratio of the maximum display height of the subtitles to the total height of all subtitles is first calculated, that is, the percentage of a in the total height c=a / b is first calculated, and then in order to ensure that the subtitles start to scroll when they are played to the middle position, half of the percentage c is determined as the percentage value of the scrolling start position (corresponding to the first percentage value): d=c / 2=a / b / 2. In the above manner, it is ensured that the subtitles start to scroll when they are played to the middle position.

[0047] Optionally, the operation of determining the scrolling position of the scroll bar corresponding to the screen image of the current frame based on the total height of all subtitles, the scrolling starting position and the current playback progress includes: obtaining the difference between the current playback progress and the scrolling starting position; and determining the scrolling position of the scroll bar corresponding to the screen image of the current frame based on the difference and the total height of all subtitles.

[0048] Specifically, in the process of determining the scrolling position of the scroll bar corresponding to the screen image of the current frame, firstly, the difference between the current playback progress and the scrolling start position is obtained, that is, (fd) is calculated, and then (fd)*the total height b of all subtitles is used to determine the scrolling position g of the scroll bar corresponding to the screen image of the current frame. In the above manner, the scrolling position of the scroll bar corresponding to the screen image of the current frame can be accurately calculated according to the current playback progress of the TTS voice, which lays a foundation for subsequently connecting the screen images of all frames that have completed the accurate setting of the scroll bar position so as to realize smooth and accurate scrolling of the subtitles displayed on the screen according to the playback progress of the TTS voice.

[0049] Optionally, when drawing the screen image of the current frame, the operation of determining the current playback progress of the audio file includes: obtaining the current playback length of the currently played audio file and the total length of the audio file from a preset system API tool class for playing audio files; and determining the current playback progress of the audio file based on the obtained current playback length and total length.

[0050] Specifically, when using the system API tool class to play an audio file, the total length and current playback length of the loaded audio file can be read. For example, the Android system API tool class MediaPlayer can obtain the playback length and total length of the currently playing audio through mediaPlayer.getCurrentPosition() / mediaPlayer.getDuration(). Therefore, when drawing the screen image of the current frame, in the process of determining the current playback progress of the audio file, the current playback length of the currently playing audio file and the total length of the audio file can be directly obtained from the system API tool class, and then the current playback progress of the audio file can be determined based on the current playback length and total length obtained, that is, the ratio of the current playback length to the total length is determined as the current playback progress of the audio file. In the above manner, the current playback progress of the audio file can be quickly and accurately obtained.

[0051] Optionally, the method further includes: determining an audio segment corresponding to each line of subtitles from an audio file according to a ratio of a line width of each line of subtitles to a total line width of all subtitles; determining an audio playback progress of the audio file at the moment when display of each line of subtitles is completed according to an audio position of the audio segment corresponding to each line of subtitles in the audio file; determining a coloring range of the subtitles displayed on the screen of the current frame according to the audio playback progress and the current playback progress; and coloring the subtitles displayed on the screen of the current frame according to the determined coloring range.

[0052] In some special scenarios, when playing such TTS audio files, the corresponding subtitles need to be displayed. However, since the TTS audio files in these scenarios are temporarily generated based on text conversion, it is impossible to set the progress time of subtitle coloring in advance like songs. Therefore, it is impossible to accurately color the subtitles corresponding to the TTS audio files.

[0053] In view of the technical problems existing in the background technology, the technical solution proposed in this embodiment, in the process of coloring the subtitles corresponding to the TTS voice file, first determines the audio segment corresponding to each line of subtitles from the audio file (corresponding to the above-mentioned TTS voice file) obtained through the text-to-speech technology according to the ratio of the line width of each line of subtitles to the total line width of all subtitles. Among them, the line width of each line of subtitles can represent the text contained in this line of subtitles to a certain extent. The wider the line width, the more text contained in this line of subtitles, and the longer the corresponding audio segment, so that the line width of each line of subtitles is proportional to the length of the corresponding audio segment. The accuracy of the line width of each line of subtitles is to the pixel value, and the line width of each line of subtitles is different. For example, the blank space of symbols and the mixed arrangement of Chinese and English will cause the line width of each line of subtitles to be inconsistent. Therefore, the position of the audio segment corresponding to each line of subtitles in the entire audio file can be calculated by the ratio of the line width of each line of subtitles to the total line width of all subtitles.

[0054] Among them, refer to Figure 4 As shown, the entire audio file corresponds to three lines of subtitles, the first line of subtitles is AAAAA, the second line of subtitles is BBB., and the third line of subtitles is CC. According to the ratio of the line width of the first line of subtitles to the total line width of the total subtitles is 50%, the ratio of the line width of the second line of subtitles to the total line width of the total subtitles is 36%, and the ratio of the line width of the third line of subtitles to the total line width of the total subtitles is 14%, the audio clip corresponding to the first line of subtitles is calculated as Figure 4 The audio clip 1 in the second line of subtitles corresponds to Figure 4 The audio clip 2 in the third line of subtitles is Figure 4 At this time, it is necessary to determine the audio playback progress of the audio file at the moment of completing the display of each line of subtitles according to the audio position of the audio segment corresponding to each line of subtitles in the audio file, that is, the audio playback progress of the audio file at the moment of completing the display of the first line of subtitles (for example, 50%), the audio playback progress of the audio file at the moment of completing the display of the second line of subtitles (for example, 86%), and the audio playback progress of the audio file at the moment of completing the display of the third line of subtitles (for example, 100%).

[0055] Furthermore, since the screen is refreshed frame by frame to produce the effect of continuous animation based on the phenomenon of visual persistence, everything (including subtitles) needs to be redrawn when drawing each frame of the screen. In this application scenario, considering that TTS voice files are different from songs and have no transition sound effects, they are compact and coherent voice playback, so the subtitles can be dyed by the playback progress of the TTS voice file. Therefore, when drawing the screen of the current frame, it is necessary to determine the current playback progress of the audio file, that is, to determine the ratio of the current playback length of the audio file to the total length. Then, according to the determined audio playback progress and current playback progress of the audio file at the moment of completing the display of each line of subtitles, the dyeing range of the subtitles displayed in the screen of the current frame is determined. Finally, according to the determined dyeing range, the subtitles displayed in the screen of the current frame are dyed. For example: the current playback progress is determined to be 50%. Based on the audio playback progress of the audio file determined above when the display of each line of subtitles is completed, it can be determined that the dyeing range of the subtitles displayed on the screen of the current frame is all subtitles including the first line of subtitles. Therefore, the first line of subtitles in the subtitles displayed on the screen of the current frame needs to be completely dyed, and other lines of subtitles do not need to be dyed.

[0056] Thus, through the above method, this embodiment can accurately color the subtitles displayed in the screen image of the current frame through the current playback progress of the audio file obtained through the text-to-speech technology. By connecting the screen images of all frames that have completed accurate coloring, the technical effect of accurately coloring the subtitles displayed on the screen according to the playback progress of the TTS voice is achieved. This solves the technical problem in the prior art that the subtitles corresponding to the TTS voice file cannot be accurately colored because the TTS voice file is temporarily generated based on text conversion, and the coloring progress time of the subtitles corresponding to the TTS voice file cannot be set in advance.

[0057] Optionally, the operation of determining the coloring range of the subtitles displayed on the screen of the current frame based on the audio playback progress and the current playback progress includes: comparing the current playback progress with the audio playback progress to determine whether the current playback progress exceeds the target audio playback progress of the audio file at the moment of completing the display of the target row of subtitles; and determining the coloring range of the subtitles displayed on the screen of the current frame based on the result of the determination.

[0058] Specifically, in the process of determining the coloring range of the subtitles displayed in the current frame of the screen image according to the audio playback progress and the current playback progress, the current playback progress and the audio playback progress are first compared, and it is determined according to the comparison result whether the current playback progress exceeds the target audio playback progress of the audio file at the moment when the target line of subtitles is displayed. For example, if the current playback progress is 51%, refer to Figure 4 As shown, 51% is compared with 50%, 86% and 100% respectively to determine whether 51% exceeds the target audio playback progress (e.g., 50%, 86% and 100%) of the audio file at the moment of completing the display of the target line of subtitles (e.g., the first line of subtitles, the second line of subtitles and the third line of subtitles). Finally, according to the result of the determination, the dyeing range of the subtitles displayed in the screen picture of the current frame is determined. That is, since the current playback progress "51%" exceeds the target audio playback progress "50%" of the audio file at the moment of completing the display of the first line of subtitles, but does not exceed the target audio playback progress "86%" and "100%" of the audio file at the moment of completing the display of the second line of subtitles and the third line of subtitles, according to the result of the determination, the dyeing range of the subtitles displayed in the screen picture of the current frame is determined to be all the subtitles of the first line of subtitles plus part of the subtitles of the second line of subtitles. In this way, the dyeing range of the subtitles displayed in the screen picture of the current frame can be quickly and accurately determined.

[0059] Optionally, based on the result of the determination, the operation of determining the coloring range of the subtitles displayed on the screen of the current frame includes: when the result of the determination is that the current playback progress exceeds the target audio playback progress of the audio file at the moment of completing the display of the target row of subtitles, all the target row of subtitles are included in the coloring range of the subtitles displayed on the screen of the current frame.

[0060] Specifically, according to the above, when the result of the determination is that the current playback progress (i.e., 51%) exceeds the target audio playback progress (i.e., 50%) of the audio file at the moment when the display of the target line of subtitles (e.g., the first line of subtitles) is completed, the first line of subtitles is completely included in the coloring range of the subtitles displayed in the screen image of the current frame, that is, all the subtitles in the first line of subtitles are dyed. In this way, the subtitles corresponding to the audio clip that has been played can be accurately dyed, so that the user can more intuitively check whether the audio corresponding to the target subtitle has been played, which effectively improves the user's experience.

[0061] Optionally, based on the result of the determination, an operation of determining the coloring range of the subtitles displayed on the screen of the current frame includes: when the result of the determination is that the current playback progress does not exceed the target audio playback progress of the audio file at the moment of completing the display of the target row of subtitles, incorporating the target row of subtitles into the coloring range of the subtitles displayed on the screen of the current frame according to the current playback progress.

[0062] Specifically, according to the foregoing, the current playback progress is, for example, 51%. Since 51% does not exceed the target audio playback progress of the audio file when the display of the second line of subtitles is completed (i.e., 86%), in this application scenario, it is necessary to include the second line of subtitles in the coloring range of the subtitles displayed on the screen of the current frame according to the current playback progress "51%" to achieve accurate coloring of the subtitles.

[0063] In a preferred embodiment, in the operation process of including the target line subtitle part into the dyeing range of the subtitles displayed in the screen picture of the current frame according to the current playback progress, the audio playback progress (i.e., 50%) of the audio file at the moment of completing the display of the previous line of subtitles (i.e., the first line of subtitles) of the second line of subtitles can be determined first, and then the difference between the current playback progress (51%) and the determined audio playback progress (50%) is obtained, and the difference is 1%. At this time, it is necessary to determine the dyeing width represented by pixels according to the difference and the sum of the line widths of all subtitles. Among them, 1% can be multiplied by the sum of the line widths of all subtitles to obtain a dyeing width of 3 pixels. Finally, according to the dyeing width (i.e., 3 pixels), the subtitles of the corresponding width in the second line of subtitles are included in the dyeing range, that is, the width of 3 pixels in the second line of subtitles is included in the dyeing range. Therefore, in the process of dyeing, not only the first line of subtitles are dyed in full, but also the width of 3 pixels in the second line of subtitles needs to be dyed, and the display is, for example, a part of the first subtitle B is dyed red. In the above manner, accurate dyeing of subtitles corresponding to TTS speech is achieved.

[0064] Optionally, the method also includes: simulating the playback scenario of the audio file obtained through text-to-speech technology to generate a simulated scenario; and determining the audio segment corresponding to each line of subtitles and the audio playback progress of the audio file at the moment when each line of subtitles is displayed under the simulated scenario.

[0065] Specifically, before coloring the subtitles corresponding to the TTS voice, a scene simulation is first performed, that is, a scene related to TTS voice playback is simulated, and a general TTS solid color subtitle display is adopted. Then, an algorithm is written, that is, the audio clip corresponding to each line of subtitles and the audio playback progress of the audio file at the moment when each line of subtitles is displayed are determined in the simulated scene. The audio clips and audio playback progress obtained in the above manner can provide a strong comparison basis for the above-mentioned coloring of the subtitles corresponding to the TTS voice, thereby accurately determining the coloring range of the subtitles.

[0066] also, Figure 5 The overall flow chart of the method for scrolling subtitles corresponding to TTS speech provided in this embodiment is shown. Figure 5 As shown, the scrolling method of TTS voice corresponding subtitles provided in this embodiment includes the following steps:

[0067] 1) Simulate TTS playback-related scenarios and use general TTS solid-color subtitle display.

[0068] 2) Write an algorithm. First, calculate the percentage starting point of TTS subtitle scrolling. Before calculating the percentage starting point of subtitles, you need to get several key points (Note: except for g, which is of int type and cannot have decimal points, it will be rounded off. Others need to be calculated in floating point expression):

[0069] 1. The maximum display height of TTS subtitles: a;

[0070] 2. The total height of TTS subtitles: b;

[0071] 3. The scrolling start position of TTS subtitles: d = a / b / 2; In order to ensure that the TTS subtitles start to scroll when they are played to the middle of the TTS subtitles you see, first calculate the percentage of a in the total height c = a / b, and then in order to ensure that the scrolling is in the middle, the percentage c / 2 is the scrolling start position: d = c / 2 = a / b / 2;

[0072] 4. The TTS audio calculates the scroll position of the scroll bar as the playback progresses: g = b*(fd).

[0073] Among them, g is a numeric type and will be rounded off, and no decimal point is allowed. With the percentage starting point d of the scroll position of the scroll bar corresponding to the TTS subtitle, we need to obtain the current playback progress value f of the voice file. In order to ensure smooth scrolling, we need to do a simple calculation. When f is greater than d, the scroll calculation is performed: g = b*(fd). The maximum scroll value of g is equal to b (the total height of the TTS subtitle), and the scroll range of g is 0 to b. When g is greater than 0, the position of the scroll bar is set. All playbacks are connected. By continuously calculating the g value and setting it to the view, the purpose of scrolling the subtitles following the playback progress of the TTS voice is achieved.

[0074] In summary, the above method provided in this embodiment achieves the technical effect of smoothly and accurately scrolling the subtitles displayed on the screen according to the playback progress of the TTS voice, which can be used in dual recording scenarios, greatly improves the user experience, and has a friendly display prompt effect on subsequent dual recording business review and playback.

[0075] In addition, reference Figure 1 As shown, according to the second aspect of this embodiment, a storage medium is provided. The storage medium includes a stored program, wherein when the program is running, a processor executes any one of the above methods.

[0076] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0078] Example 2

[0079] Figure 6The device 600 for scrolling subtitles corresponding to TTS speech according to this embodiment is shown. The device 600 corresponds to the method according to the first aspect of embodiment 1. Figure 6 As shown, the device 600 includes: a first determination module 610, which is used to determine a first percentage value of a scrolling start position of a subtitle according to a total height of all subtitles corresponding to an audio file and a maximum display height of subtitles displayed in a screen image to be drawn, wherein the audio file is an audio file obtained through a text-to-speech technology; a second determination module 620, which is used to determine a second percentage value of a current playback progress of the audio file when drawing a screen image of a current frame; a third determination module 630, which is used to determine a scrolling position of a scroll bar corresponding to a screen image of a current frame according to a total height of all subtitles, a scrolling start position and a current playback progress when it is determined that the second percentage value is greater than the first percentage value, wherein the scroll bar is used to scroll and display all subtitles in the screen image; and a setting module 640, which is used to set the position of the scroll bar according to the scrolling position.

[0080] Optionally, the operation of determining a first percentage value of a scrolling start position of subtitles based on a total height of all subtitles corresponding to an audio file and a maximum display height of subtitles displayed in a screen image to be drawn includes: calculating a ratio of a maximum display height of subtitles to a total height of all subtitles; and determining half of the ratio as a first percentage value.

[0081] Optionally, the operation of determining the scrolling position of the scroll bar corresponding to the screen image of the current frame based on the total height of all subtitles, the scrolling starting position and the current playback progress includes: obtaining the difference between the current playback progress and the scrolling starting position; and determining the scrolling position of the scroll bar corresponding to the screen image of the current frame based on the difference and the total height of all subtitles.

[0082] Optionally, when drawing the screen image of the current frame, the operation of determining the current playback progress of the audio file includes: obtaining the current playback length of the currently played audio file and the total length of the audio file from a preset system API tool class for playing audio files; and determining the current playback progress of the audio file based on the obtained current playback length and total length.

[0083] Optionally, the device 600 also includes: determining an audio segment corresponding to each line of subtitles from the audio file according to the ratio of the line width of each line of subtitles to the total line width of all subtitles; determining an audio playback progress of the audio file at the moment when each line of subtitles is displayed according to the audio position of the audio segment corresponding to each line of subtitles in the audio file; determining a coloring range of the subtitles displayed on the screen of the current frame according to the audio playback progress and the current playback progress; and coloring the subtitles displayed on the screen of the current frame according to the determined coloring range.

[0084] Optionally, the operation of determining the coloring range of the subtitles displayed on the screen of the current frame based on the audio playback progress and the current playback progress includes: comparing the current playback progress with the audio playback progress to determine whether the current playback progress exceeds the target audio playback progress of the audio file at the moment of completing the display of the target row of subtitles; and determining the coloring range of the subtitles displayed on the screen of the current frame based on the result of the determination.

[0085] Optionally, based on the result of the determination, the operation of determining the coloring range of the subtitles displayed on the screen of the current frame includes: when the result of the determination is that the current playback progress exceeds the target audio playback progress of the audio file at the moment of completing the display of the target row of subtitles, all the target row of subtitles are included in the coloring range of the subtitles displayed on the screen of the current frame.

[0086] Therefore, according to this embodiment,

[0087] Example 3

[0088] Figure 7 The device 700 for scrolling subtitles corresponding to TTS speech according to this embodiment is shown. The device 700 corresponds to the method according to the first aspect of embodiment 1. Figure 7 As shown, the device 700 includes: a processor 710; and a memory 720, which is connected to the processor 710 and is used to provide the processor 710 with instructions for processing the following processing steps: determining a first percentage value of a scrolling start position of a subtitle according to a total height of all subtitles corresponding to an audio file and a maximum display height of subtitles displayed in a screen image to be drawn, wherein the audio file is an audio file obtained through a text-to-speech technology; determining a second percentage value of a current playback progress of the audio file when drawing a screen image of a current frame; determining a scrolling position of a scroll bar corresponding to a screen image of a current frame according to a total height of all subtitles, a scrolling start position, and a current playback progress when it is determined that the second percentage value is greater than the first percentage value, wherein the scroll bar is used to scroll and display all subtitles in the screen image; and setting the position of the scroll bar according to the scrolling position.

[0089] Optionally, the operation of determining a first percentage value of a scrolling start position of subtitles based on a total height of all subtitles corresponding to an audio file and a maximum display height of subtitles displayed in a screen image to be drawn includes: calculating a ratio of a maximum display height of subtitles to a total height of all subtitles; and determining half of the ratio as a first percentage value.

[0090] Optionally, the operation of determining the scrolling position of the scroll bar corresponding to the screen image of the current frame based on the total height of all subtitles, the scrolling starting position and the current playback progress includes: obtaining the difference between the current playback progress and the scrolling starting position; and determining the scrolling position of the scroll bar corresponding to the screen image of the current frame based on the difference and the total height of all subtitles.

[0091] Optionally, when drawing the screen image of the current frame, the operation of determining the current playback progress of the audio file includes: obtaining the current playback length of the currently played audio file and the total length of the audio file from a preset system API tool class for playing audio files; and determining the current playback progress of the audio file based on the obtained current playback length and total length.

[0092] Optionally, the memory 720 is also used to provide the processor 710 with instructions for processing the following processing steps: determining an audio segment corresponding to each line of subtitles from the audio file according to the ratio of the line width of each line of subtitles to the total line width of all subtitles; determining an audio playback progress of the audio file at the moment when each line of subtitles is displayed according to the audio position of the audio segment corresponding to each line of subtitles in the audio file; determining a coloring range of the subtitles displayed on the screen of the current frame according to the audio playback progress and the current playback progress; and coloring the subtitles displayed on the screen of the current frame according to the determined coloring range.

[0093] Optionally, the operation of determining the coloring range of the subtitles displayed on the screen of the current frame based on the audio playback progress and the current playback progress includes: comparing the current playback progress with the audio playback progress to determine whether the current playback progress exceeds the target audio playback progress of the audio file at the moment of completing the display of the target row of subtitles; and determining the coloring range of the subtitles displayed on the screen of the current frame based on the result of the determination.

[0094] Optionally, based on the result of the determination, the operation of determining the coloring range of the subtitles displayed on the screen of the current frame includes: when the result of the determination is that the current playback progress exceeds the target audio playback progress of the audio file at the moment of completing the display of the target row of subtitles, all the target row of subtitles are included in the coloring range of the subtitles displayed on the screen of the current frame.

[0095] Therefore, according to this embodiment,

[0096] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0097] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0102] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for scrolling subtitles corresponding to TTS speech, characterized in that: include: Determining a first percentage value of a scrolling start position of the subtitles according to a total height of all subtitles corresponding to the audio file and a maximum display height of the subtitles displayed in the screen image to be drawn, wherein the audio file is an audio file obtained by a text-to-speech technology; In the case of drawing the screen image of the current frame, determining a second percentage value of the current playing progress of the audio file; If it is determined that the second percentage value is greater than the first percentage value, determining a scroll position of a scroll bar corresponding to the screen image of the current frame according to the total height of all the subtitles, the scrolling start position, and the current playback progress, wherein the scroll bar is used to scroll and display all the subtitles in the screen image; as well as According to the scroll position, setting the position of the scroll bar; Among them, the calculation formula of the scrolling position g is g=b*(fd), where the first percentage value of the scrolling starting position of the subtitle is d, the total height of all subtitles corresponding to the audio file is b, and the second percentage value of the current playback progress of the audio file is f.

2. The method according to claim 1, characterized in that: The operation of determining a first percentage value of a scrolling start position of the subtitles according to a total height of all subtitles corresponding to the audio file and a maximum display height of the subtitles displayed in the screen image to be drawn includes: Calculating the ratio of the maximum display height of the subtitle to the total height of all the subtitles; and A value half of the ratio is determined as the first percentage value.

3. The method according to claim 1, characterized in that The operation of determining the scrolling position of the scroll bar corresponding to the screen image of the current frame according to the total height of all subtitles, the scrolling start position and the current playing progress includes: Calculating the difference between the current playback progress and the scrolling start position; and The scroll position of the scroll bar corresponding to the screen image of the current frame is determined according to the difference and the total height of all the subtitles.

4. The method according to claim 1, characterized in that: In the case of drawing the screen image of the current frame, the operation of determining the current playing progress of the audio file includes: Obtaining the current playing length of the currently playing audio file and the total length of the audio file from a preset system API tool class for playing the audio file; and The current playing progress of the audio file is determined according to the acquired current playing length and the total length.

5. The method according to claim 1, characterized in that: Also includes: Determining the audio segment corresponding to each line of subtitles from the audio file according to the ratio of the line width of each line of subtitles to the total line width of all subtitles; Determining the audio playback progress of the audio file at the moment when each line of subtitles is displayed according to the audio position of the audio segment corresponding to each line of subtitles in the audio file; Determining a coloring range of subtitles displayed in the screen image of the current frame according to the audio playback progress and the current playback progress; and The subtitles displayed in the screen image of the current frame are dyed according to the determined dyeing range.

6. The method according to claim 5, characterized in that The operation of determining the coloring range of the subtitles displayed in the screen image of the current frame according to the audio playback progress and the current playback progress includes: Comparing the current playback progress with the audio playback progress to determine whether the current playback progress exceeds the target audio playback progress of the audio file at the moment when the display of the target line of subtitles is completed; and According to the result of the determination, a coloring range of the subtitles displayed in the screen image of the current frame is determined.

7. The method according to claim 6, characterized in that The operation of determining the coloring range of the subtitles displayed in the screen image of the current frame according to the result of the determination includes: When the result of the determination is that the current playback progress exceeds the target audio playback progress of the audio file at the moment when the display of the target row of subtitles is completed, the target row of subtitles is completely included in the coloring range of the subtitles displayed on the screen of the current frame.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the processor executes the method according to any one of claims 1 to 7.

9. A scrolling device for TTS voice corresponding to subtitles, characterized in that: include: A first determination module is used to determine a first percentage value of a scrolling start position of the subtitles according to a total height of all subtitles corresponding to the audio file and a maximum display height of the subtitles displayed in the screen image to be drawn, wherein the audio file is an audio file obtained by a text-to-speech technology; A second determination module is used to determine a second percentage value of the current playback progress of the audio file when drawing the screen image of the current frame; a third determining module, configured to determine, when it is determined that the second percentage value is greater than the first percentage value, a scrolling position of a scroll bar corresponding to the screen image of the current frame according to the total height of all the subtitles, the scrolling start position, and the current playback progress, wherein the scroll bar is used to scroll and display all the subtitles in the screen image; as well as A setting module, used for setting the position of the scroll bar according to the scroll position; Among them, the calculation formula of the scrolling position g is g=b*(fd), where the first percentage value of the scrolling starting position of the subtitle is d, the total height of all subtitles corresponding to the audio file is b, and the second percentage value of the current playback progress of the audio file is f.

10. A scrolling device for TTS voice corresponding to subtitles, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Determining a first percentage value of a scrolling start position of the subtitles according to a total height of all subtitles corresponding to the audio file and a maximum display height of the subtitles displayed in the screen image to be drawn, wherein the audio file is an audio file obtained by a text-to-speech technology; In the case of drawing the screen image of the current frame, determining a second percentage value of the current playing progress of the audio file; If it is determined that the second percentage value is greater than the first percentage value, determining a scroll position of a scroll bar corresponding to the screen image of the current frame according to the total height of all the subtitles, the scrolling start position, and the current playback progress, wherein the scroll bar is used to scroll and display all the subtitles in the screen image; as well as According to the scroll position, setting the position of the scroll bar; Among them, the calculation formula of the scrolling position g is g=b*(fd), where the first percentage value of the scrolling starting position of the subtitle is d, the total height of all subtitles corresponding to the audio file is b, and the second percentage value of the current playback progress of the audio file is f.

Images