Live broadcast data processing method and system

Abstract

An embodiment of the present application provides a live broadcast data processing method and system, including: decoding a received initial live stream to generate an audio stream and a first video stream; performing speech recognition on the audio stream to generate corresponding recognized text and determine time interval information between the generation time of the recognized text and the reception time of the audio stream; using the recognized text as subtitle information, adding the subtitle information and the time interval information to the first video stream to generate a second video stream; encoding the second video stream and the audio stream to generate a live stream to be pushed, and returning the live stream to be pushed to a client.

Description

[Technical Field]

[0001] This application claims priority to Chinese Patent Application No. 202211311544.0, entitled "Live Broadcast Data Processing Method and System," filed on October 25, 2022, the entire contents of which are incorporated herein by reference.

[0002] FIELD Embodiments of the present application relate to the field of computers, and in particular to a live broadcast data processing method. One or more embodiments of the present application also relate to a live broadcast data processing system, a computing device, and a computer-readable storage medium. [Background technology]

[0003] With the rapid development of the audio-video live broadcasting industry, the requirements of high definition, low latency, and audio-video synchronization have been optimized to the utmost using data stream transmission technology, but this alone does not satisfy the needs of users.

[0004] Special scenarios, such as large-scale sporting events, large-scale conference reports, and online education and training, require real-time translation and the addition of language subtitles to live broadcasts. To display subtitles, the live stream must first be recorded, and then the audio stream must be extracted, manually or machine-translated, and then burned into the video, allowing the subtitles to be displayed during retransmission. However, this processing method does not provide the same live streaming experience for viewers who do not understand the language or are hearing impaired. The inventors have noticed that while technologies for generating subtitles in real time during live broadcasts, such as live barrages, have been developed, they have several drawbacks. For example, the audio and subtitles are not synchronized, and the subtitles may sometimes be ahead or behind, resulting in an extremely poor user experience and failing to satisfy the needs of viewers. Therefore, an effective method for solving these problems is urgently needed. Summary of the Invention

[0005] In view of this, embodiments of the present application provide a live broadcast data processing method, and one or more embodiments of the present application also relate to a live broadcast data processing device, a live broadcast data processing system, a computing device, and a computer-readable storage medium, thereby solving the technical deficiencies of high cost, low efficiency, and delayed subtitle generation in related art live subtitling.

[0006] According to a first aspect of the embodiment of the present application, decoding the received initial live stream to generate an audio stream and a first video stream; performing speech recognition on the audio stream to generate a corresponding recognized text and determine time interval information between a generation time of the recognized text and a reception time of the audio stream; using the recognized text as subtitle information, and adding the subtitle information and the time interval information to the first video stream to generate a second video stream; encoding the second video stream and the audio stream to generate a live stream to be pushed, and returning the live stream to a client.

[0007] According to a second aspect of the embodiment of the present application, a decoding module configured to decode the received initial live stream to generate an audio stream and a first video stream; a recognition module configured to perform speech recognition on the audio stream, generate a corresponding recognized text, and determine time interval information between a generation time of the recognized text and a reception time of the audio stream; an additional module configured to use the recognized text as subtitle information, add the subtitle information and the time interval information to the first video stream, and generate a second video stream; an encoding module configured to encode the second video stream and the audio stream to generate a live stream to be pushed, and return the live stream to a client.

[0008] According to a third aspect of the embodiment of the present application, receiving and caching the live stream to be pushed returned from the live streaming server; Decoding the live stream to be pushed and generating a corresponding audio stream, a video stream, subtitle information, and time interval information corresponding to the subtitle information, wherein the time interval information is determined by the live distribution server based on a generation time of the subtitle information and a reception time of the audio stream; determining a display time of the subtitle information based on the time interval information; When it is determined that the playback conditions for the live stream to be pushed are satisfied, the video stream and the audio stream are synchronously played back, and the subtitle information is displayed based on the display time.

[0009] According to a fourth aspect of the embodiment of the present application, a receiving module configured to receive and cache the to-be-pushed live stream returned from the live streaming server; a decoding module configured to decode the live stream to be pushed and generate a corresponding audio stream, a video stream, subtitle information, and time interval information corresponding to the subtitle information, the time interval information being determined by the live distribution server based on a generation time of the subtitle information and a reception time of the audio stream; a determination module configured to determine a display time of the subtitle information based on the time interval information; and a display module configured to synchronously play back the video stream and the audio stream and display the subtitle information based on the display time when it is determined that the playback condition for the live stream to be pushed is satisfied.

[0010] According to a fifth aspect of the embodiment of the present application, Includes live streaming server and client, the live streaming server is used to decode the received initial live stream, generate an audio stream and a first video stream, perform speech recognition on the audio stream, generate corresponding recognized text, and determine time interval information between a generation time of the recognized text and a receiving time of the audio stream, use the recognized text as subtitle information, add the subtitle information and the time interval information to the first video stream, generate a second video stream, and encode the second video stream and the audio stream, generate a live stream to be pushed, and return the live stream to the client; The client provides a live broadcast data processing system used to receive, cache, and decode the live stream to be pushed, obtain the audio stream, the second video stream, the subtitle information, and the time interval information, determine a display time for the subtitle information based on the time interval information, and, when it is determined that a playback condition for the live stream to be pushed is satisfied, synchronously play back the second video stream and the audio stream, and display the subtitle information based on the display time.

[0011] According to a sixth aspect of the embodiment of the present application, It has memory and a processor, The memory is used to store computer-executable instructions, the processor is used to execute the computer-executable instructions, and when the processor executes the computer-executable instructions, the steps of the live broadcast data processing method are realized.

[0012] According to a seventh aspect of an embodiment of the present application, there is provided a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, realize the steps of the live broadcast data processing method.

[0013] An embodiment of the present application realizes a live broadcast data processing method and system, which includes: decoding a received initial live stream to generate an audio stream and a first video stream; performing speech recognition on the audio stream to generate corresponding recognized text and determine time interval information between the generation time of the recognized text and the reception time of the audio stream; using the recognized text as subtitle information, adding the subtitle information and the time interval information to the first video stream to generate a second video stream; encoding the second video stream and the audio stream to generate a live stream to be pushed, and returning the live stream to be pushed to a client.

[0014] In an embodiment of the present application, the live streaming server performs speech recognition on the audio stream, generates corresponding recognized text, and records the time interval between the generation of the recognized text and the reception of the audio stream. This time interval can be used to characterize the length of time spent by the live streaming server to perform speech recognition on the audio stream in the initial live stream after receiving the initial live stream. The recognized text and time interval information are added to the video stream and returned to the client. The client then pre-analyzes to obtain the subtitle information to be carried in the live stream to be pushed, and determines the display time of the subtitle information based on the time interval information between the generation of the subtitle information and the reception of the audio stream by the live streaming server. That is, it can determine the display time of the entire subtitles corresponding to the live stream to be pushed. Pre-displaying the entire subtitles based on this display time not only reduces the cost of subtitle generation and improves the efficiency of subtitle generation, but also helps to avoid asynchronization between the subtitles and the video screen or audio, thereby meeting users' needs for viewing live subtitles while watching a live broadcast and improving users' live broadcast viewing experience. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is an architecture diagram of a live broadcast data processing system provided by an embodiment of the present application; [Figure 2] 1 is a flowchart of a live broadcast data processing method provided by an embodiment of the present application; [Figure 3] 4 is a flowchart of another live broadcast data processing method provided by an embodiment of the present application; [Figure 4] FIG. 2 is an interactive schematic diagram of the application of the live broadcast data processing method provided by an embodiment of the present application to the live broadcast field; [Figure 5] FIG. 1 is a schematic diagram showing the structure of a live broadcast data processing device provided by an embodiment of the present application; [Figure 6]FIG. 10 is a schematic diagram showing the structure of another live broadcast data processing device provided by an embodiment of the present application; [Figure 7] 1 is a block diagram illustrating the structure of a computing device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0016] Numerous specific details are set forth below to facilitate a thorough understanding of the present application, however, the present application may be implemented in other ways than those described herein, and those skilled in the art may make similar extensions without departing from the spirit of the present application, and therefore the present application is not limited by the specific examples disclosed below.

[0017] Terms used in one or more examples of the present application are used only to describe a particular example and are not intended to limit one or more examples of the present application. As used in one or more examples of the present application and in the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context indicates otherwise. In addition, the term "and / or" used in one or more examples of the present application should be understood to mean and include any and all possible combinations of one or more associated listed items.

[0018] It should be understood that, although one or more embodiments of the present application may use terms such as "first," "second," etc. to describe various pieces of information, such information should not be limited to these terms. These terms are used solely to distinguish between pieces of information of the same type. For example, a first may be referred to as a second, and similarly, a second may be referred to as a first, without departing from the scope of one or more embodiments of the present application. Depending on the context, the terms "when," "when," "upon," or "depending on the decision" as used herein may be interpreted to mean the same thing.

[0019] First, we will interpret the noun terms associated with one or more embodiments of this application.

[0020] Live broadcast: In a broad sense, this term also includes live television broadcasts, but here it generally refers to live video streaming over a network. The audio and video to be streamed live are pushed to the server as a media stream (push streaming). When there are viewers watching the live broadcast, the server receives a request from the user, then transmits the video to the website, app, or client player, and plays the video in real time. H264 encoding: H264 generally refers to H.264, a highly compressed digital video codec standard developed by the Joint Video Team (JVT), which is comprised of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG).

[0021] H265 encoding: H.265 is a new video encoding standard established by ITU-T VCEG following H.264.

[0022] The H.265 standard is based on the video encoding standard H.264, and improves on some of the related technologies while retaining some of the conventional technologies.

[0023] SEI: Supplemental Enhancement Information, which belongs to the codestream category and provides a way to add additional information to the codestream, and is one of the features of video compression standards such as H.264 / H.265.

[0024] Speech recognition technology: A technology in which a machine converts voice signals into corresponding text or instructions through a process of recognition and understanding.

[0025] GRPC: A type of RPC (abbreviation for Remote Procedure Call) framework, it is a high-performance, open-source, general-purpose RPC framework developed based on the ProtoBuf (Protocol Buffers) serialization protocol and supports many development languages.

[0026] Transcoding: A video transcoding technique that converts video signals from one format to another.

[0027] The present application provides a live broadcast data processing method. One or more embodiments of the present application also relate to a live broadcast data processing device, a live broadcast data processing system, a computing device, and a computer-readable storage medium, each of which will be described in detail in the following embodiments.

[0028] In specific implementations, the subtitle information according to the embodiments of the present application can be presented to clients such as large-screen video players, game consoles, desktop computers, smartphones, tablet computers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, e-book readers, and other display terminals.

[0029] Furthermore, the subtitle information according to the embodiments of the present application can be applied to any video or audio for which subtitles can be displayed, for example, to display subtitles on video such as live video or recorded video, or to display subtitles on audio for listening to songs, books, etc. online or offline.

[0030] Please refer to FIG. 1, which is an architecture diagram of a live broadcast data processing system provided by one embodiment of the present application; A live streaming server 102 and a client 104 are included. the live streaming server 102 is used to decode the received initial live stream, generate an audio stream and a first video stream, perform speech recognition on the audio stream, generate corresponding recognized text, and determine time interval information between the generation time of the recognized text and the reception time of the audio stream, use the recognized text as subtitle information, add the subtitle information and the time interval information to the first video stream, generate a second video stream, and encode the second video stream and the audio stream, generate a live stream to be pushed, and return the live stream to be pushed to the client 104; The client 104 receives, caches, and decodes the live stream to be pushed, obtains the audio stream, the second video stream, the subtitle information, and the time interval information, determines a display time for the subtitle information based on the time interval information, and, if it determines that the playback conditions for the live stream to be pushed are met, plays the second video stream and the audio stream synchronously and displays the subtitle information based on the display time.

[0031] Specifically, in FIG. 1, user U1 broadcasts live through an intelligent terminal and pushes the generated initial live stream to the live streaming server 102. The live streaming server 102 decodes the received initial live stream to generate an audio stream and a first video stream. Then, it performs speech recognition on the audio stream to generate a corresponding recognized text and determine the time interval information between the generation time of the recognized text and the reception time of the audio stream. Then, it uses the recognized text as subtitle information, adds the subtitle information and the time interval information to the first video stream, generates a second video stream, and further encodes the second video stream and the audio stream to generate a live stream to be pushed. While users U2 and U3 are watching user U1's live broadcast, the live streaming server can push the live stream to users U2 and U3's clients 104. In the process of playing a live stream for a user, the client 104 can pull and cache the live stream to be pushed from the live distribution server in advance for a predetermined time, thereby pre-decoding the cached live stream to be pushed to obtain the subtitle information contained in the live stream to be pushed, and then determine the display time of the subtitle information based on the time interval information between the generation time of the subtitle information carried in the live stream to be pushed and the reception time of the audio stream by the live distribution server 102, and if it determines that the playback conditions of the live stream to be pushed are met, it can synchronously play the video stream and audio stream obtained by decoding and display the subtitle information based on the display time.

[0032] In an embodiment of the present application, the above processing method allows the client to analyze in advance to obtain the subtitle information carried in the live stream to be pushed, and determine the display time of the subtitle information based on the time interval information between the generation time of the subtitle information and the reception time of the audio stream by the live streaming server, i.e., determine the display time of the entire subtitles corresponding to the live stream to be pushed. Pre-displaying the entire subtitles based on the display time not only reduces the cost of subtitle generation and improves the efficiency of subtitle generation, but also helps to avoid asynchronization between the subtitles and the video screen or audio, thereby meeting the needs of users who want to view live subtitles while watching live broadcasts and improving users' live broadcast viewing experience.

[0033] The above is a general solution of the live broadcast data processing system according to this embodiment. Note that the technical solution of this live broadcast data processing system is based on the same concept as the technical solution of the live broadcast data processing method described later. For details not detailed in the technical solution of this live broadcast data processing system, please refer to the description of the technical solution of the live broadcast data processing method described later.

[0034] Referring to FIG. 2, which is a flowchart of a live broadcast data processing method provided by an embodiment of the present application, the method includes the following steps 202, 204, 206, and 208.

[0035] In step 202, the received initial live stream is decoded to generate an audio stream and a first video stream.

[0036] Specifically, the live broadcast data processing method provided by the embodiment of the present application is applied to a live broadcast server, and the initial live stream is the live stream that the broadcaster pushes to the live broadcast server during the live broadcast.

[0037] In the process of live broadcasting through its intelligent terminal, the broadcaster can push the live stream generated during the live broadcast to a live broadcasting server by the intelligent terminal, so that if there are other users who want to watch the broadcaster's live broadcast, the live broadcasting server can push the live stream pushed by the broadcaster to the user terminals (clients) of the other users.

[0038] Currently, most live broadcasts are viewed without subtitles. However, special scenarios, such as large-scale sporting events, large-scale conference reports, and online education and training, require real-time translation of live broadcasts or the addition of language subtitles. To display subtitles, the live stream must first be recorded, and then the audio stream must be extracted, manually or machine-translated, and then burned into the video, allowing the subtitles to be displayed during retransmission. However, this processing method cannot provide the same live broadcast effect to viewers who do not speak the language or who are hearing impaired.

[0039] In addition, technologies have been developed to generate subtitles in real time during live broadcasts, such as live barrages, but there are often problems with the subtitles not being synchronized with the video or audio, which creates an extremely poor experience for users watching the live broadcast and fails to meet their needs.

[0040] Based on this, in an embodiment of the present application, after receiving an initial live stream pushed by a broadcaster, the live streaming server can decode the initial live stream to obtain an audio stream and a first video stream, and perform speech recognition on the audio stream to obtain corresponding recognized text. Then, the recognized text can be added to the first video stream as subtitle information to generate a second video stream. Thus, after the encoding results of the audio stream and the second video stream are pushed to the user's client, the client can decode and obtain the subtitle information, and simultaneously play the audio stream and the second video stream for the user, displaying the subtitle information. In this way, the problem of live subtitles and live video screen or audio being out of sync while the user is watching a live broadcast in real time can be avoided, the user's need to view live subtitles while watching a live broadcast can be met, and the user's live broadcast viewing experience can be improved.

[0041] In a specific implementation, decoding the received initial live stream specifically includes: determining live streams to be played that are cached by the client and determining generation times corresponding to the live streams to be played; This can be achieved by obtaining an initial live stream corresponding to the live stream identifier at a predetermined time interval after the generation time based on a live stream identifier corresponding to the live stream to be played back and the generation time, and decoding the initial live stream.

[0042] Furthermore, the client decodes the to-be-played live stream to generate a corresponding to-be-played audio stream, a to-be-played video stream, a to-be-displayed subtitle, and a display time corresponding to the to-be-displayed subtitle; If it is determined that the playback condition for the live stream to be played back is satisfied, the video stream to be played back and the audio stream to be played back are played back in synchronization, and the subtitles to be displayed are displayed based on the display time.

[0043] Specifically, when playing a live stream, the client caches the live stream scheduled to be played from the current playback time onwards for a predetermined period of time in advance, analyzes this portion of the live stream scheduled to be played in advance, obtains the video stream scheduled to be played, the audio stream scheduled to be played, the subtitles scheduled to be displayed and the display times corresponding to the subtitles scheduled to be displayed which are included in the live stream scheduled to be played, and then, if it determines that the playback conditions for the live stream scheduled to be played are met, it can synchronously play back the decoded video stream and audio stream scheduled to be played and display the subtitles scheduled to be displayed based on the display times.

[0044] For example, if the time for pre-caching the live stream to be played is 5 seconds and the current playback time is t, the client will pre-cache and analyze the live stream to be played from t to t+5 seconds, and determine whether the subtitles to be displayed need to be displayed in advance based on the display time of the subtitles to be displayed in the analysis result, thereby reducing the delay between the live subtitles and the live video screen or audio while the user is watching the live broadcast in real time.

[0045] Furthermore, because the client has a limited time to pre-cache the live stream to be played, once this portion of the live stream to be played is played, the client needs to further cache the new live stream to be played. For example, the client pre-caches the live stream to be played from t to t+5s. Once the live stream to be played from t to t+3s is played, the client needs to cache the live stream to be played from t+5s to t+8s. That is, the client needs to obtain the live stream to be played from t+5s to t+8s from the live distribution server.

[0046] Therefore, the live distribution server can predetermine the live streams to be cached by the client and determine the generation time (playback time) corresponding to the cached live streams to be played, and then, based on the live stream identifier and generation time corresponding to the cached live streams to be played, obtain an initial live stream corresponding to the live stream identifier for a predetermined period from the generation time, process the initial live stream, and generate a live stream to be pushed including subtitle information, and push it to the client.

[0047] Based on this, while a user is watching a live broadcast in real time through a client, the client will cache the live stream to be played from the current playback time onwards for a predetermined period of time in advance, and will analyze this portion of the live stream to be played in advance. Similarly, the live streaming server can determine the live stream to be cached by the client in advance, and determine and analyze an initial live stream based on the cached live stream to be played. Both the analysis process of the initial live stream by the live streaming server and the analysis process of the live stream to be played by the client take a predetermined amount of time, which will result in a certain degree of live delay. However, in the embodiment of the present application, the analysis process of the initial live stream by the live streaming server and the analysis process of the live stream to be played by the client are performed in parallel, and the client can determine whether the subtitles to be displayed need to be displayed in advance based on the display time of the subtitles to be displayed in the analysis result, thereby reducing the delay between the live subtitles and the live-streamed video or audio while the user is watching a live broadcast in real time.

[0048] In step 204, speech recognition is performed on the audio stream to generate a corresponding recognized text, and time interval information between the generation time of the recognized text and the reception time of the audio stream is determined.

[0049] Specifically, the live streaming server decodes the initial live stream and obtains the audio stream and the first video stream, and then performs speech recognition on the audio stream to generate corresponding recognized text. Then, the server adds the recognized text to the first video stream as subtitle information to generate a second video stream. Thus, a client for watching the live broadcast can obtain the second video stream and then display subtitle information to a user while the second video stream is being played.

[0050] However, in actual applications, after the live streaming server obtains the audio stream by decoding, it often needs to spend a certain amount of time performing speech recognition on the audio stream. In this case, there will be a time difference between the time when the recognized text is generated and the time when the audio stream, i.e., the initial live stream, is received. If only the recognized text and the initial live stream are pushed to the client without taking this time difference into consideration, there is a possibility that when the client displays the recognized text, the recognized text will not be synchronized with the video screen or audio.

[0051] In an embodiment of the present application, after recognition and obtaining the complete recognized text, in order to avoid asynchronization between the recognized text and the video screen or audio, it is necessary to determine the time spent on generating the recognized text, i.e., the time interval between the time the recognized text is generated and the time the audio stream is received by the live streaming server, so that the client can determine how soon after obtaining the recognized text it needs to be displayed based on the time interval.

[0052] In particular, the live streaming server may obtain an audio stream by decoding, and then split the audio stream according to spectral information corresponding to the audio stream to generate at least two audio clips; Accordingly, performing speech recognition on the audio stream, generating a corresponding recognized text, and determining time interval information between a generation time of the recognized text and a reception time of the audio stream as described above may include: performing speech recognition on a target audio clip, the target audio clip being one of the at least two audio clips, to generate corresponding recognized text; determining a time of generation of the recognized text and determining time interval information between the time of generation and a time of receipt of the target audio clip.

[0053] Specifically, when performing speech recognition on an audio stream, the accuracy of the recognition result can be ensured as long as the recognized audio stream is a complete lexical stream. Therefore, in an embodiment of the present application, the audio stream is first divided based on spectral information corresponding to the audio stream to generate at least two audio clips. For example, based on the spectral information, the audio stream between any two adjacent points where the spectral value is 0 (indicating a pause) can be used as one audio clip. Next, speech recognition is performed on each audio clip to generate corresponding recognized text, determine the generation time of the recognized text, and determine time interval information between the generation time and the reception time of each audio clip (the reception time of the audio stream or the initial live stream).

[0054] Alternatively, performing speech recognition on the audio stream, generating a corresponding recognized text, and determining time interval information between a generation time of the recognized text and a receiving time of the audio stream may include: Dividing the audio stream according to a predetermined identification window to generate at least one audio clip; performing speech recognition on a target audio clip, the target audio clip being one of the at least one audio clip, to generate corresponding recognized text; determining a time of generation of the recognized text and determining time interval information between the time of generation and a time of receipt of the audio stream.

[0055] Specifically, in the process of performing speech recognition on an audio stream, a preset identification window is usually used, and the length of the preset identification window may be 0.5 seconds to 1 second. When the preset identification window is used to perform speech recognition on an audio stream, a single character contained in the audio stream can be recognized. Alternatively, the length of the preset identification window may be 1 second to 5 seconds. When the preset identification window is used to perform speech recognition on an audio stream, a complete sentence contained in the audio stream can be recognized. The specific length of the window can be determined according to actual needs and is not limited herein.

[0056] Speech recognition is performed on the audio stream according to a predetermined identification window, specifically, the audio stream is divided according to the predetermined identification window to generate at least one audio clip, speech recognition is performed on each audio clip, corresponding recognized text is generated, and then the generation time of the recognized text is determined, and time interval information between the generation time and the reception time of the audio stream is determined.

[0057] In addition, in an embodiment of the present application, the live streaming server includes a transcoding module and a speech recognition service module, whereby decoding the received initial live stream and generating an audio stream and a first video stream specifically means decoding the received initial live stream by the transcoding module and generating an audio stream and a first video stream, and performing speech recognition on the audio stream and generating corresponding recognized text specifically means performing speech recognition on the audio stream by the speech recognition service module and generating corresponding recognized text.

[0058] Here, the transcoding module transmits the audio stream to the voice recognition service module via a data transmission path.

[0059] Specifically, the data transmission channel may be a GRPC.

[0060] Furthermore, performing speech recognition on the audio stream by a speech recognition service module and generating a corresponding recognized text as described above, Dividing the audio stream by a speech recognition service module according to a preset identification window to generate at least one audio clip;

[0061] performing speech recognition on a first audio clip, the first audio clip being one of the at least one audio clip, generating a corresponding first recognized text, and returning the first recognized text to the transcoding module.

[0062] Specifically, as mentioned above, a preset identification window is usually used in the process of performing speech recognition on an audio stream. Similarly, a preset identification window may be used in the speech recognition on an audio stream by a speech recognition service module. Performing speech recognition on an audio stream according to the preset identification window specifically involves dividing the audio stream according to the preset identification window, generating at least one audio clip, performing speech recognition on each audio clip, generating corresponding recognized text, and then determining the generation time of the recognized text and determining the time interval information between the generation time and the receiving time of the audio stream.

[0063] Here, the length of the preset identification window may be 0.5 seconds to 1 second, and when speech recognition of the audio stream is performed using the preset identification window, it can recognize one character contained in the audio stream, or the length of the preset identification window may be 1 second to 5 seconds, and when speech recognition of the audio stream is performed using the preset identification window, it can recognize a complete sentence contained in the audio stream. The specific length of the window can be determined according to actual needs and is not limited herein.

[0064] In step 206, the recognized text is used as subtitle information, and the subtitle information and the time interval information are added to the first video stream to generate a second video stream.

[0065] Specifically, after generating a recognition text and determining the time interval information between the generation time of the recognition text and the reception time of the audio stream, the recognition text can be used as subtitle information, and the subtitle information and the time interval information can be added to the first video stream to generate a second video stream.

[0066] Here, subtitle information in the form of SEI can be written into the primary video stream to generate the secondary video stream.

[0067] In a specific implementation, the text type of the recognized text can be further determined based on the text length and / or text semantics of the recognized text; Accordingly, using the above-mentioned recognized text as subtitle information and adding the subtitle information and the time interval information to the first video stream, determining a target video frame to be included in the first video stream based on the generation time;

[0068] using the recognized text as subtitle information, and adding the subtitle information, the time interval information and the text type to the first video stream as video frame information of the target video frame.

[0069] Specifically, as mentioned above, a preset identification window is usually used in the process of performing speech recognition on an audio stream, and the length of the preset identification window may be 0.5 seconds to 1 second. Using the preset identification window to perform speech recognition on an audio stream can recognize one character contained in the audio stream, or the length of the preset identification window may be 1 second to 5 seconds. Using the preset identification window to perform speech recognition on an audio stream can recognize a complete sentence contained in the audio stream.

[0070] Therefore, in the embodiment of the present application, after generating the recognized text, the text type of the recognized text can be further determined based on the text length and / or text semantics of the recognized text, and in practical applications, the text type includes, but is not limited to, character, word, sentence, etc. The text semantics is used to determine whether the recognized text can express the complete meaning, and if so, the text type of the recognized text can be determined as a sentence type; otherwise, if the text length of the recognized text is two or more characters, the text type of the recognized text can be determined as a word type, and if the text length is one, the text type can be determined as a character type.

[0071] After determining the text type of the recognized text, a target video frame included in the first video stream can be determined based on the generation time of the recognized text, the recognized text can be used as subtitle information, and the subtitle information, time interval information and text type can be added to the first video stream as video frame information of the target video frame.

[0072] In practical applications, the last video frame in the video clip corresponding to the target audio clip is usually used as the target video frame, and subtitle information, time interval information, and text type are added to the first video stream as its video frame information to generate a second video stream. After obtaining the second video stream, the client can determine the subtitle information to be displayed according to the text type. Typically, sentence-type subtitle information is preferentially selected and displayed to ensure the subtitle viewing effect of live broadcasts.

[0073] In addition, in an embodiment of the present application, the audio stream is divided by a speech recognition service module according to a preset identification window, and at least one audio clip is generated; and speech recognition is performed on the first audio clip, and a corresponding first recognized text is generated; and then the above-mentioned recognized text is used as subtitle information; and the subtitle information and the time interval information are added to the first video stream. determining, by the transcoding module, a first target video frame included in the first video stream based on a generation time of the first recognized text; using the first recognition text as first subtitle information, and adding the first subtitle information and time interval information between the generation time of the first recognition text and the reception time of the audio stream to the first video stream as video frame information of the first target video frame.

[0074] Specifically, when the speech recognition service module divides the audio stream into one or at least two audio clips, it can perform speech recognition on each audio clip in turn, generate a recognized text corresponding to any audio clip, and then return the recognized text to the transcoding module. The transcoding module can determine a target video frame (usually the last video frame of a video clip corresponding to any audio clip) included in the first video stream based on the generation time of the recognized text, use the recognized text as subtitle information, and add the subtitle information, the time interval information between the generation time of the recognized text and the receiving time of the audio stream, to the first video stream as video frame information of the target video frame.

[0075] Furthermore, performing speech recognition on the audio stream by a speech recognition service module and generating a corresponding recognized text as described above, The method includes performing speech recognition on a second audio clip of the at least two audio clips that is adjacent to the first audio clip, generating a corresponding second recognized text, and returning the first recognized text and the second recognized text to the transcoding module. Accordingly, using the above-mentioned recognized text as subtitle information and adding the subtitle information and the time interval information to the first video stream, determining, by the transcoding module, a second target video frame included in the first video stream based on a generation time of the second recognized text; using the first recognition text and the second recognition text as second subtitle information, and adding the second subtitle information and time interval information between the generation time of the second recognition text and the reception time of the audio stream to the first video stream as video frame information of the second target video frame.

[0076] Specifically, as described above, when the speech recognition service module divides the audio stream into at least two audio clips, it can first perform speech recognition on a first audio clip of the at least two audio clips and generate a corresponding first recognized text, and the transcoding module can use the first recognized text as subtitle information and add the subtitle information and the time interval information between the generation time of the first recognized text and the reception time of the audio stream to the first video stream as video frame information of a first target video frame (usually the last video frame of the video clip corresponding to the first audio clip).

[0077] Next, speech recognition can be performed on a second audio clip of the at least two audio clips adjacent to the first audio clip to generate a corresponding second recognized text, and the transcoding module can use the first recognized text and the second recognized text as subtitle information, and add the subtitle information and time interval information between the generation time of the second recognized text and the reception time of the audio stream to the first video stream as video frame information of a second target video frame (usually the last video frame of the video clip corresponding to the second audio clip), and so on.

[0078] After performing speech recognition to obtain a first recognized text, the speech recognition service module can temporarily store the first recognized text; after performing recognition to obtain a second recognized text, because the first audio clip and the second audio clip are adjacent, the speech recognition service module can return both the first recognized text and the second recognized text as subtitle information for the video stream, thereby enabling the speech recognition service module to multiplex caches and improve the accuracy of the recognition results of the subtitle information.

[0079] Step 208 encodes the second video stream and the audio stream to generate a live stream to be pushed, and returns the live stream to be pushed to the client.

[0080] Specifically, after generating the second video stream, the live streaming server can encode the second video stream and the audio stream to generate a live stream to be pushed, and when there is a demand from a user to watch the live broadcast, the live stream to be pushed can be pushed to the user's client.

[0081] Specifically, in implementation, the client decodes the live stream to be pushed, and generates a corresponding audio stream, a video stream, and video frame information of a target video frame included in the video stream, where the video frame information includes the subtitle information, the time interval information, and the text type; If the text type is determined as the target type, determine a display time of the subtitle information based on the playback time of the target video frame and the time interval information; determining at least two video frames in the video stream for displaying the subtitle information based on the display time, where a playback time of the at least two video frames is before a playback time of the target video frame; If it is determined that the playback conditions for the live stream to be pushed are satisfied, the video stream and the audio stream are played back synchronously, and the subtitle information is displayed on the at least two video frames and the target video frame based on the display time.

[0082] Specifically, as described above, when playing a live stream, the client can pull and cache the live stream to be pushed from the live distribution server for a predetermined time period after the current playback time, and then pre-decode the cached live stream to be pushed to obtain subtitle information corresponding to a target video frame included in the live stream to be pushed, the text type of the subtitle information, and the time interval information between the generation time of the subtitle information and the reception time of the audio stream by the live distribution server. If the subtitle information belongs to the target text type according to the text type, i.e., belongs to a sentence type, the client can determine the display time of the subtitle information based on the time interval information between the generation time of the subtitle information carried in the live stream to be pushed and the reception time of the audio stream by the live distribution server, and further in combination with the playback time of the target video frame. Then, based on the display time, the client can determine another video frame in the live stream to be pushed that is before the target video frame and that is to be used to display the subtitle information. If it is determined that the playback condition of the live stream to be pushed is met, the client can synchronously play the decoded and obtained video stream and audio stream, and display the subtitle information in the determined video frame and target video frame based on the display time.

[0083] For example, the current time is t. In the process of playing the live stream, the client pre-caches the live stream to be pushed from t to t+5s, and then decodes it to obtain the subtitle information carried in the live stream to be pushed. If the decoding result includes recognized texts corresponding to five video frames at t+1s, t+2s, t+3s, t+4s, and t+5s, and the text type of the recognized text corresponding to the time t+5s is a sentence type, the recognized text can be preferentially displayed. In this case, the time interval information corresponding to the recognized text can be determined, and the generated time of the recognized text and the audio stream can be calculated. is 4 seconds, and the time interval between the generation time of the recognized text and the reception time of the video frame at time t+5s is 1 second, it indicates that the subtitle information (recognized text) needs to be displayed 3 seconds in advance, which can also indicate that the broadcaster will express a complete sentence from t+3s to t+5s. Therefore, the subtitle information can be displayed simultaneously with the start of playback of the scheduled push live stream at t+3s, and the display can be completed at t+5s, thereby realizing the advance display of the entire subtitle, avoiding a delay between the subtitle and the video screen or audio, and suspending the display of the subtitle information until it is determined that other subtitle information needs to be displayed.

[0084] An embodiment of the present application realizes a live broadcast data processing method and system, which includes: decoding a received initial live stream to generate an audio stream and a first video stream; performing speech recognition on the audio stream to generate corresponding recognized text and determine time interval information between the generation time of the recognized text and the reception time of the audio stream; using the recognized text as subtitle information, adding the subtitle information and the time interval information to the first video stream to generate a second video stream; encoding the second video stream and the audio stream to generate a live stream to be pushed, and returning the live stream to be pushed to a client.

[0085] In an embodiment of the present application, the live streaming server performs speech recognition on the audio stream, generates corresponding recognized text, and records the time interval between the generation of the recognized text and the reception of the audio stream. This time interval can be used to characterize the length of time spent by the live streaming server to perform speech recognition on the audio stream in the initial live stream after receiving the initial live stream. The recognized text and time interval information are added to the video stream and returned to the client. The client then pre-analyzes to obtain the subtitle information to be carried in the live stream to be pushed, and determines the display time of the subtitle information based on the time interval information between the generation of the subtitle information and the reception of the audio stream by the live streaming server. That is, it can determine the display time of the entire subtitles corresponding to the live stream to be pushed. Pre-displaying the entire subtitles based on this display time not only reduces the cost of subtitle generation and improves the efficiency of subtitle generation, but also helps to avoid asynchronization between the subtitles and the video screen or audio, thereby meeting users' needs for viewing live subtitles while watching a live broadcast and improving users' live broadcast viewing experience.

[0086] Referring to FIG. 3, which is a flowchart of another live broadcast data processing method provided by an embodiment of the present application, the method includes the following steps 302, 304, 306, and 308.

[0087] In step 302, the live stream to be pushed returned from the live distribution server is received and cached.

[0088] In step 304, the live stream to be pushed is decoded to generate a corresponding audio stream, a video stream, subtitle information, and time interval information corresponding to the subtitle information, where the time interval information is determined by the live distribution server based on the generation time of the subtitle information and the reception time of the audio stream.

[0089] In step 306, the display time of the subtitle information is determined based on the time interval information.

[0090] In step 308, if it is determined that the playback condition of the live stream to be pushed is satisfied, the video stream and the audio stream are synchronously played back, and the subtitle information is displayed based on the display time.

[0091] Specifically, a broadcaster broadcasts live through an intelligent terminal and pushes the generated initial live stream to a live streaming server. The live streaming server decodes the received initial live stream to generate an audio stream and a first video stream, then performs speech recognition on the audio stream to generate corresponding recognized text, and determines time interval information between the generation time of the recognized text and the reception time of the audio stream. Subsequently, the recognized text is used as subtitle information, and the subtitle information and the time interval information are added to the first video stream to generate a second video stream. Furthermore, the second video stream and the audio stream are encoded to generate a live stream to be pushed. While a user is watching the broadcaster's live broadcast, the live streaming server can push the live stream to the user's client.

[0092] In the process of playing a live stream for a user, the client can pull and cache the live stream to be pushed from the live distribution server in advance for a predetermined time, so that the client can pre-decode the cached live stream to be pushed to obtain the subtitle information contained in the live stream to be pushed, and then determine the display time of the subtitle information based on the time interval information between the generation time of the subtitle information carried in the live stream to be pushed and the reception time of the audio stream by the live stream server, and if it determines that the playback condition of the live stream to be pushed is met, it can synchronously play the video stream and audio stream obtained by decoding and display the subtitle information based on the display time.

[0093] In an embodiment of the present application, the above processing method allows the client to analyze in advance to obtain the subtitle information carried in the live stream to be pushed, and determine the display time of the subtitle information based on the time interval information between the generation time of the subtitle information and the reception time of the audio stream by the live streaming server, i.e., determine the display time of the entire subtitles corresponding to the live stream to be pushed. Pre-displaying the entire subtitles based on the display time not only reduces the cost of subtitle generation and improves the efficiency of subtitle generation, but also helps to avoid asynchronization between the subtitles and the video screen or audio, thereby meeting the needs of users who want to view live subtitles while watching live broadcasts and improving users' live broadcast viewing experience.

[0094] The above is a schematic solution of another live broadcast data processing method according to this embodiment. Note that the technical solution of this live broadcast data processing method belongs to the same concept as the technical solution of the live broadcast data processing method described above, and for details not detailed in the technical solution of this live broadcast data processing method, please refer to the description of the technical solution of the live broadcast data processing method described above.

[0095] The live broadcast data processing method provided by an embodiment of the present application will be further described by taking the application of the live broadcast data processing method provided by an embodiment of the present application to the live broadcast field as an example, with reference to Figure 4. Here, Figure 4 is a diagram illustrating an interaction of the application of the live broadcast data processing method provided by an embodiment of the present application to the live broadcast field, specifically including steps 402 to 420.

[0096] In step 402, the initial live stream of the broadcaster is received by a transcoding module.

[0097] In step 404, the initial live stream is decoded by a transcoding module to generate an audio stream and a first video stream.

[0098] In step 406, the transcoding module transmits the audio stream to the speech recognition service module via the GRPC.

[0099] In step 408, the audio stream is speech recognized by a speech recognition service module to generate corresponding recognized text.

[0100] In step 410, the speech recognition service module determines the generation time of the recognized text and determines the time interval information between the generation time and the reception time of the audio stream, and determines the text type of the recognized text based on the text length and / or text semantics of the recognized text.

[0101] In step 412, the speech recognition service module transmits the recognized text, text type, and time interval information to the transcoding module via GRPC.

[0102] In step 414, the transcoding module uses the recognized text as subtitle information, adds subtitle information, time interval information and text type to the first video stream, and generates a second video stream.

[0103] In step 416, the transcoding module encodes the secondary video stream and audio stream to generate the to-be-pushed live stream.

[0104] In step 418, the client pulls the to-be-pushed live stream from the live delivery server.

[0105] The live streaming server includes a transcoding module and a voice recognition service module.

[0106] In step 420, the client decodes the live stream to be pushed, generates a corresponding audio stream, a second video stream, subtitle information and time interval information, determines a display time of the subtitle information based on the time interval information, and if it determines that the playback conditions of the live stream to be pushed are met, synchronously plays the second video stream and the audio stream, and displays the subtitle information based on the display time.

[0107] In an embodiment of the present application, the above processing method allows the client to analyze in advance to obtain the subtitle information carried in the live stream to be pushed, and determine the display time of the subtitle information based on the time interval information between the generation time of the subtitle information and the reception time of the audio stream by the live streaming server, i.e., determine the display time of the entire subtitles corresponding to the live stream to be pushed. Pre-displaying the entire subtitles based on the display time not only reduces the cost of subtitle generation and improves the efficiency of subtitle generation, but also helps to avoid asynchronization between the subtitles and the video screen or audio, thereby meeting the needs of users who want to view live subtitles while watching live broadcasts and improving users' live broadcast viewing experience.

[0108] Corresponding to the above method embodiment, the present application further provides an embodiment of a live broadcast data processing device, and Figure 5 is a schematic diagram showing the structure of a live broadcast data processing device provided by an embodiment of the present application. As shown in Figure 5, the device includes: a decoding module 502 configured to decode the received initial live stream to generate an audio stream and a first video stream; a recognition module 504 configured to perform speech recognition on the audio stream, generate a corresponding recognized text, and determine time interval information between a generation time of the recognized text and a reception time of the audio stream; an additional module 506 configured to use the recognized text as subtitle information, add the subtitle information and the time interval information to the first video stream, and generate a second video stream; an encoding module 508 configured to encode the second video stream and the audio stream to generate a live stream to be pushed, and return the live stream to a client.

[0109] Optionally, the decoding module 502 further comprises: determining live streams to be played that are cached by the client and determining generation times corresponding to the live streams to be played; The system is configured to obtain an initial live stream corresponding to the live stream identifier at a predetermined time interval after the generation time based on a live stream identifier corresponding to the live stream to be played back and the generation time, and to decode the initial live stream.

[0110] Optionally, the client decodes the to-be-played live stream to generate a corresponding to-be-played audio stream, a to-be-played video stream, subtitles to be displayed, and display times corresponding to the to-be-displayed subtitles; If it is determined that the playback condition for the live stream to be played back is satisfied, the video stream to be played back and the audio stream to be played back are played back in synchronization, and the subtitles to be displayed are displayed based on the display time.

[0111] Optionally, the live broadcast data processing device: a determination module configured to determine a text type of the recognized text based on a text length and / or a text semantic of the recognized text; Accordingly, the additional module 506 further: determining a target video frame to be included in the first video stream based on the generation time; The video stream is configured to use the recognized text as subtitle information, and add the subtitle information, the time interval information and the text type to the first video stream as video frame information of the target video frame.

[0112] Optionally, the client decodes the to-be-pushed live stream and generates a corresponding audio stream, a video stream, and video frame information of a target video frame included in the video stream, wherein the video frame information includes the subtitle information, the time interval information, and the text type; If the text type is determined as the target type, determine a display time of the subtitle information based on the playback time of the target video frame and the time interval information; determining at least two video frames in the video stream for displaying the subtitle information based on the display time, where a playback time of the at least two video frames is before a playback time of the target video frame; If it is determined that the playback conditions for the live stream to be pushed are satisfied, the video stream and the audio stream are played back synchronously, and the subtitle information is displayed on the at least two video frames and the target video frame based on the display time.

[0113] Optionally, the live broadcast data processing device: a segmentation module configured to segment the audio stream based on spectral information corresponding to the audio stream to generate at least two audio clips; In response, the recognition module 504 further: performing speech recognition on a target audio clip, the target audio clip being one of the at least two audio clips, to generate corresponding recognized text; The apparatus is configured to determine a time of generation of the recognized text and to determine time interval information between the time of generation and a time of receipt of the target audio clip.

[0114] Optionally, the recognition module 504 further comprises: Dividing the audio stream according to a predetermined identification window to generate at least one audio clip; performing speech recognition on a target audio clip, the target audio clip being one of the at least one audio clip, to generate corresponding recognized text; The apparatus is configured to determine a time of generation of the recognized text and to determine time interval information between the time of generation and a time of receipt of the audio stream.

[0115] Optionally, the decoding module 502 further comprises: configured to decode the received initial live stream by a transcoding module to generate an audio stream and a first video stream; In response, the recognition module 504 further: The audio stream is configured to be speech-recognized by a speech recognition service module to generate a corresponding recognized text.

[0116] Optionally, the recognition module 504 further comprises: Dividing the audio stream by a speech recognition service module according to a preset identification window to generate at least one audio clip; The audio clip processing module is configured to perform speech recognition on a first audio clip, the first audio clip being one of the at least one audio clip, generate a corresponding first recognized text, and return the first recognized text to the transcoding module.

[0117] Optionally, the additional module 506 further comprises: determining, by the transcoding module, a first target video frame included in the first video stream based on a generation time of the first recognized text; The first recognition text is used as first subtitle information, and the first subtitle information and time interval information between the generation time of the first recognition text and the reception time of the audio stream are configured to be added to the first video stream as video frame information of the first target video frame.

[0118] Optionally, the recognition module 504 further comprises: The audio clip processing module is configured to perform speech recognition on a second audio clip of the at least two audio clips that is adjacent to the first audio clip, generate a corresponding second recognized text, and return the first recognized text and the second recognized text to the transcoding module.

[0119] Optionally, the additional module 506 further comprises: determining, by the transcoding module, a second target video frame included in the first video stream based on a generation time of the second recognized text; The first recognition text and the second recognition text are used as second subtitle information, and the second subtitle information and time interval information between the generation time of the second recognition text and the reception time of the audio stream are added to the first video stream as video frame information of the second target video frame.

[0120] Optionally, the live broadcast data processing device: The audio system further includes a transmission module configured to transmit the audio stream via the transcoding module to the speech recognition service module via a data transmission channel.

[0121] The above is a general solution of the live broadcast data processing device according to this embodiment. Note that the technical solution of the live broadcast data processing device belongs to the same concept as the technical solution of the live broadcast data processing method described above, and for details not detailed in the technical solution of the live broadcast data processing device, please refer to the description of the technical solution of the live broadcast data processing method described above.

[0122] Corresponding to the above method embodiment, the present application further provides an embodiment of a live broadcast data processing device, and Figure 6 is a schematic diagram showing the structure of another live broadcast data processing device provided by an embodiment of the present application. As shown in Figure 6, the device includes: a receiving module 602 configured to receive and cache the to-be-pushed live stream returned from the live streaming server; a decoding module 604 configured to decode the live stream to be pushed and generate a corresponding audio stream, a video stream, closed caption information, and time interval information corresponding to the closed caption information, the time interval information being determined by the live distribution server based on a generation time of the closed caption information and a reception time of the audio stream; a determination module 606 configured to determine a display time of the subtitle information based on the time interval information; and a display module 608 configured to synchronously play the video stream and the audio stream and display the subtitle information based on the display time when it is determined that the playback condition of the live stream to be pushed is met.

[0123] The above is a schematic solution of another live broadcast data processing device according to this embodiment. Note that the technical solution of this live broadcast data processing device belongs to the same concept as the technical solution of the other live broadcast data processing method described above, and for details not detailed in the technical solution of the live broadcast data processing device, please refer to the description of the technical solution of the other live broadcast data processing method described above.

[0124] 7 is a block diagram illustrating the structure of a computing device 700 provided by one embodiment of the present application. Components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0125] Computing device 700 further includes an access device 740 that enables computing device 700 to communicate over one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks, such as the Internet. Access device 740 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as, for example, an IEEE 802.11 wireless local area network (WLAN) radio interface, a global microwave interconnect access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, etc.

[0126] In one embodiment of the present application, the above components of computing device 700, and other components not shown in Figure 7, may be connected to each other, for example, via a bus. It should be understood that the block diagram of the structure of a computing device shown in Figure 7 is for illustrative purposes only and is not intended to limit the scope of the present application. Those skilled in the art can add or replace other components as needed.

[0127] Computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., smartphone), a wearable computing device (e.g., smart watch, smart glasses, etc.) or other type of mobile equipment, or a desktop computer or PC stationary computing device. Computing device 700 may be a mobile server or a fixed server.

[0128] Here, the processor 720 is used to execute the following computer-executable instructions, and the processor is used to execute the computer-executable instructions, where, when the processor executes the computer-executable instructions, the steps of the live broadcast data processing method are realized.

[0129] The above is a schematic solution of the computing device according to this embodiment. Note that the technical solution of the computing device belongs to the same concept as the technical solution of the live broadcast data processing method described above, and for details not detailed in the technical solution of the computing device, please refer to the description of the technical solution of the live broadcast data processing method described above.

[0130] An embodiment of the present application further provides a computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, perform the following steps: performing speech recognition on the audio stream to generate a corresponding recognized text and determining time interval information between a generation time of the recognized text and a reception time of the audio stream; using the recognized text as subtitle information, and adding the subtitle information and the time interval information to the first video stream to generate a second video stream; encoding the second video stream and the audio stream to generate a live stream to be pushed, and returning the live stream to the client; Or the following steps: receiving and caching the live stream to be pushed returned from the live streaming server; a step of decoding the live stream to be pushed and generating a corresponding audio stream, a video stream, subtitle information, and time interval information corresponding to the subtitle information, the time interval information being determined by the live distribution server based on a generation time of the subtitle information and a reception time of the audio stream; determining a display time of the subtitle information based on the time interval information; If it is determined that the playback condition for the live stream to be pushed is satisfied, a step of synchronously playing back the video stream and the audio stream and displaying the subtitle information based on the display time is realized.

[0131] The above is a schematic solution of a computer-readable storage medium according to this embodiment. Note that the technical solution of the storage medium belongs to the same concept as the technical solution of the live broadcast data processing method described above, and for details not detailed in the technical solution of the storage medium, please refer to the description of the technical solution of the live broadcast data processing method described above.

[0132] The above describes specific examples of the present application. Other examples are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the examples and still achieve desirable results. Furthermore, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve desirable results. In some embodiments, multitasking or parallel processing is also possible and may be advantageous.

[0133] The computer instructions include computer program code, which may be in the form of source code, object code, an executable file, or any intermediate form. The computer-readable medium may include any entity or device on which the computer program code can be carried, such as a recording medium, a USB flash drive, a removable hard disk, a magnetic disk, a compact disk, computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier wave signal, a telecommunications signal, and a software distribution medium. It should be noted that what is included on the computer-readable medium may be increased or decreased as appropriate, as required by the legislative and patent practices of any jurisdiction; for example, in some jurisdictions, the legislative and patent practices may require that the computer-readable medium exclude electrical carrier wave signals and telecommunications signals.

[0134] Although the above-described embodiments of the methods are all expressed as a combination of a series of operations for ease of explanation, it should be noted that the embodiments of the present application are not limited by the order of operations described, since certain steps can be performed in other orders or simultaneously according to the embodiments of the present application as will be understood by those skilled in the art. It should also be understood by those skilled in the art that the embodiments described herein are all preferred embodiments, and that not all of the related operations and modules are necessarily required for the embodiments of the present application.

[0135] In the above embodiments, each embodiment has been described with emphasis, but for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0136] The preferred embodiments of the present application disclosed above are merely intended to assist in the description of the present application. The selectable embodiments are not exhaustive and do not limit the present invention to the specific embodiments described. Of course, many modifications and variations are possible based on the content of the embodiments of the present application. In order to better understand the principles and practical applications of the embodiments of the present application, these embodiments are selected and described in detail in the present application so that those skilled in the art can better understand and utilize the present application. The present application is limited only by the claims, their full scope, and equivalents.

Claims

1. decoding the received initial live stream to generate an audio stream and a first video stream; performing speech recognition on the audio stream to generate a corresponding recognized text and determine time interval information between a generation time of the recognized text and a reception time of the audio stream; using the recognized text as subtitle information, and adding the subtitle information and the time interval information to the first video stream to generate a second video stream; encoding the second video stream and the audio stream to generate a live stream to be pushed, and returning the live stream to a client.

2. Decoding the received initial live stream as described above includes: determining live streams to be played that are cached by the client and determining generation times corresponding to the live streams to be played; The live broadcast data processing method of claim 1, further comprising: obtaining an initial live stream corresponding to the live stream identifier at a predetermined time interval after the generation time based on the live stream identifier corresponding to the live stream to be played back and the generation time; and decoding the initial live stream.

3. the client decodes the to-be-played live stream to generate a corresponding to-be-played audio stream, a to-be-played video stream, subtitles to-be-displayed, and a display time corresponding to the to-be-displayed subtitles; The live broadcast data processing method according to claim 2, wherein, when it is determined that the playback conditions of the live stream to be played back are satisfied, the video stream to be played back and the audio stream to be played back are played back synchronously, and the subtitles to be displayed are displayed based on the display time.

4. determining a text type of the recognized text based on a text length and / or a text semantic of the recognized text; Accordingly, using the above-mentioned recognized text as subtitle information and adding the subtitle information and the time interval information to the first video stream, determining a target video frame to be included in the first video stream based on the generation time; 2. The live broadcast data processing method of claim 1, further comprising: using the recognized text as subtitle information; and adding the subtitle information, the time interval information, and the text type to the first video stream as video frame information of the target video frame.

5. the client decodes the to-be-pushed live stream and generates a corresponding audio stream, a video stream, and video frame information of a target video frame included in the video stream, wherein the video frame information includes the subtitle information, the time interval information, and the text type; If the text type is determined as the target type, determine a display time of the subtitle information based on the playback time of the target video frame and the time interval information; determining at least two video frames in the video stream for displaying the subtitle information based on the display time, where a playback time of the at least two video frames is before a playback time of the target video frame; 5. The live broadcast data processing method of claim 4, wherein, when it is determined that the playback conditions for the live stream to be pushed are met, the video stream and the audio stream are played back synchronously, and the subtitle information is displayed on the at least two video frames and the target video frame based on the display time.

6. further comprising splitting the audio stream to generate at least two audio clips based on spectral information corresponding to the audio stream; Accordingly, performing speech recognition on the audio stream, generating a corresponding recognized text, and determining time interval information between a generation time of the recognized text and a reception time of the audio stream as described above may include: performing speech recognition on a target audio clip, the target audio clip being one of the at least two audio clips, to generate corresponding recognized text; 2. The live broadcast data processing method of claim 1, further comprising: determining a generation time of the recognized text; and determining time interval information between the generation time and a receiving time of the target audio clip.

7. performing speech recognition on the audio stream, generating a corresponding recognized text, and determining time interval information between a generation time of the recognized text and a reception time of the audio stream, Dividing the audio stream according to a predetermined identification window to generate at least one audio clip; performing speech recognition on a target audio clip, the target audio clip being one of the at least one audio clip, to generate corresponding recognized text; The live broadcast data processing method of claim 1 , further comprising: determining a generation time of the recognized text; and determining time interval information between the generation time and a receiving time of the audio stream.

8. Decoding the received initial live stream to generate an audio stream and a first video stream includes: decoding the received initial live stream by a transcoding module to generate an audio stream and a first video stream; Accordingly, performing speech recognition on the audio stream and generating a corresponding recognized text as described above includes: The live broadcast data processing method according to claim 1 , further comprising: performing speech recognition of the audio stream by a speech recognition service module to generate corresponding recognized text.

9. The step of performing speech recognition on the audio stream by a speech recognition service module and generating a corresponding recognized text includes: Dividing the audio stream by a speech recognition service module according to a preset identification window to generate at least one audio clip; 9. The live broadcast data processing method of claim 8, further comprising: performing speech recognition on a first audio clip that is one of the at least one audio clip, generating a corresponding first recognized text, and returning the first recognized text to the transcoding module.

10. The above-mentioned method of using the recognized text as subtitle information and adding the subtitle information and the time interval information to the first video stream includes: determining, by the transcoding module, a first target video frame included in the first video stream based on a generation time of the first recognized text; 10. The live broadcast data processing method of claim 9, further comprising: using the first recognition text as first subtitle information; and adding the first subtitle information and time interval information between the generation time of the first recognition text and the reception time of the audio stream to the first video stream as video frame information of the first target video frame.

11. The step of performing speech recognition on the audio stream by a speech recognition service module and generating a corresponding recognized text includes:

11. The live broadcast data processing method of claim 10, further comprising: performing speech recognition on a second audio clip of the at least two audio clips that is adjacent to the first audio clip, generating a corresponding second recognized text, and returning the first recognized text and the second recognized text to the transcoding module.

12. The above-mentioned method of using the recognized text as subtitle information and adding the subtitle information and the time interval information to the first video stream includes: determining, by the transcoding module, a second target video frame included in the first video stream based on a generation time of the second recognized text; 12. The live broadcast data processing method of claim 11, further comprising: using the first recognition text and the second recognition text as second subtitle information; and adding the second subtitle information and time interval information between the generation time of the second recognition text and the reception time of the audio stream to the first video stream as video frame information of the second target video frame.

13. The live broadcast data processing method of claim 8 , further comprising transmitting the audio stream by the transcoding module to the voice recognition service module via a data transmission channel.

14. receiving and caching the live stream to be pushed returned from the live streaming server; Decoding the live stream to be pushed and generating a corresponding audio stream, a video stream, subtitle information, and time interval information corresponding to the subtitle information, wherein the time interval information is determined by the live distribution server based on a generation time of the subtitle information and a reception time of the audio stream; determining a display time of the subtitle information based on the time interval information; When it is determined that the playback conditions for the live stream to be pushed are satisfied, the live broadcast data processing method includes: synchronously playing the video stream and the audio stream, and displaying the subtitle information based on the display time.

15. Includes live streaming server and client, the live streaming server is used to decode the received initial live stream, generate an audio stream and a first video stream, perform speech recognition on the audio stream, generate corresponding recognized text, and determine time interval information between a generation time of the recognized text and a receiving time of the audio stream, use the recognized text as subtitle information, add the subtitle information and the time interval information to the first video stream, generate a second video stream, and encode the second video stream and the audio stream, generate a live stream to be pushed, and return the live stream to the client; the client receives and caches the live stream to be pushed, decodes it, obtains the audio stream, the second video stream, the subtitle information, and the time interval information, determines a display time for the subtitle information based on the time interval information, and, when it determines that a playback condition for the live stream to be pushed is satisfied, synchronously plays back the second video stream and the audio stream, and displays the subtitle information based on the display time.

16. Decoding the received initial live stream as described above includes: determining live streams to be played that are cached by the client and determining generation times corresponding to the live streams to be played; The live broadcast data processing system of claim 15 further comprising: obtaining an initial live stream corresponding to the live stream identifier at a predetermined time interval after the generation time based on the live stream identifier corresponding to the live stream to be played back and the generation time; and decoding the initial live stream.

17. the client decodes the to-be-played live stream to generate a corresponding to-be-played audio stream, a to-be-played video stream, subtitles to-be-displayed, and a display time corresponding to the to-be-displayed subtitles; 17. The live broadcast data processing system of claim 16, wherein, when it is determined that the playback conditions of the live stream to be played back are satisfied, the video stream to be played back and the audio stream to be played back are played back synchronously, and the subtitles to be displayed are displayed based on the display time.

18. The live streaming server further determining a text type of the recognized text based on a text length and / or a text semantic of the recognized text; Accordingly, using the above-mentioned recognized text as subtitle information and adding the subtitle information and the time interval information to the first video stream, determining a target video frame to be included in the first video stream based on the generation time; 16. The live broadcast data processing system of claim 15, further comprising: using the recognized text as subtitle information; and adding the subtitle information, the time interval information, and the text type to the first video stream as video frame information of the target video frame.

19. It has memory and a processor, A computing device, wherein the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, and when the processor executes the computer-executable instructions, the steps of the live broadcast data processing method described in any one of claims 1 to 14 are realized.

20. A computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the live broadcast data processing method according to any one of claims 1 to 14.

Images