A method and device for generating subtitles by real-time speech recognition in mobile live broadcast

Abstract

The present application provides a method and device for generating subtitles through real-time speech recognition in mobile live broadcast. The method collects audio and video data streams in the mobile device of the anchor in real time. The audio and video data streams include audio and video sources and carry the timestamp of the current time; the audio stream format is transcoded to the audio and video data streams in a preset format to obtain audio data to be recognized in a specific format; speech recognition is performed on the audio data to be recognized to obtain subtitle data in a corresponding text format, and the subtitle data carries the corresponding timestamp in the audio and video data streams; the audio and video data streams and subtitle data are synchronously pushed to the live content distribution network. The present application solves the problems of delay in audio and video live content introduced in the existing live subtitle solutions, and the difficulty in displaying subtitles on the audience side.

Description

Technical Field

[0001] The present application relates to the field of mobile live broadcast technology, and in particular, to a method and device for generating subtitles through real-time speech recognition in mobile live broadcast. Background Art

[0002] Mobile live broadcast is a live broadcast method in which the host uses a mobile device to push audio and video data, and the audience uses a playback device to watch the received audio and video content. For live broadcasts, due to the on-site environment, different languages, and other reasons, some live broadcasts need to add text subtitles in real time, adding live real-time subtitles. However, there are currently few real-time subtitles for mobile live broadcasts. Therefore, the demand for live subtitles is becoming more and more urgent. On the one hand, it can satisfy the audience's ability to better understand the video content when watching the live broadcast. On the other hand, the live subtitle product has a real-time transcription function with voice transcription as the core, which can help media workers improve their work efficiency to a certain extent.

[0003] The traditional principle of implementing live subtitles is to add a 5-10 minute delay to the original live signal. During this delay, the simultaneous interpreter matches the stenographer to output text. Then the subtitle machine receives the text and matches the subtitles according to the subtitle director's requirements. The subtitle signal is then superimposed through the stream encoder to complete the live streaming. This traditional implementation requires more personnel, more equipment, and higher costs.

[0004] With the development of technology, a kind of intelligent speech recognition and subtitle synthesis technology has emerged. By deploying an intelligent speech recognition program in a device, the audio content is automatically transcribed into subtitle content, that is, the acquisition module and the push module on the anchor side push the live content to the live CDN (Content Delivery Network), and the device deploying this technology pulls the live audio and video stream from the live CDN and transmits it to the speech recognition program for speech recognition. The push module then merges the subtitle content recognized by the speech recognition module into the audio and video stream and pushes it back to the live CDN. Compared with the traditional solution, this solution has greatly improved the delay, and has greatly saved personnel and equipment. However, due to the implementation method of pulling, merging and pushing, there will still be a live delay of about 5 seconds; and because the subtitles are still merged into the audio and video stream, the audience end cannot freely display the subtitles. Summary of the invention

[0005] In view of the problems mentioned above, the present application provides a method and device for generating subtitles through real-time speech recognition in mobile live broadcast, so as to solve the problems of delay in audio and video live broadcast content introduced in existing live subtitle solutions, and the lack of freedom in displaying subtitles on the audience side.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A method for generating subtitles by real-time speech recognition in mobile live broadcast, comprising:

[0008] Collecting audio and video data streams in the host's mobile device in real time, wherein the audio and video data streams include audio and video sources and carry a timestamp of the current time;

[0009] Performing audio stream format transcoding on the audio and video data stream according to a preset format to obtain audio data to be identified in a specific format;

[0010] Performing speech recognition on the audio data to be recognized to obtain subtitle data in a corresponding text format, wherein the subtitle data carries a corresponding timestamp in the audio and video data stream;

[0011] The audio and video data stream and the subtitle data are synchronously pushed to a live content distribution network.

[0012] Furthermore, the step of synchronously pushing the audio and video data stream and the subtitle data to a live content distribution network includes:

[0013] Obtaining a timestamp corresponding to the audio and video data stream and a timestamp corresponding to the audio and video data stream in the subtitle data;

[0014] The audio and video progress is adjusted by coordinating the timestamp corresponding to the audio and video data stream and the timestamp corresponding to the audio and video data stream in the subtitle data, so as to achieve synchronization between the audio and video data stream and the subtitle data;

[0015] The synchronized audio and video data stream and the subtitle data are pushed to the live content distribution network.

[0016] Furthermore, it also includes:

[0017] Receiving audio and video data from an edge node of the live content distribution network, and decoding the audio and video data into audio and video content through a decoder;

[0018] The timestamp in the audio and video data stream is extracted, and the audio and video content is displayed on the display screen of the mobile device at the viewer end according to a preset display mode.

[0019] A device for generating subtitles by real-time speech recognition in mobile live broadcast, comprising:

[0020] The first processing unit is used to collect the audio and video data stream in the mobile device of the anchor end in real time, wherein the audio and video data stream includes audio and video sources and carries a timestamp of the current time;

[0021] A second processing unit is used to perform audio stream format transcoding on the audio and video data stream according to a preset format to obtain audio data to be identified in a specific format;

[0022] A third processing unit, configured to perform speech recognition on the audio data to be recognized, and obtain subtitle data in a corresponding text format, wherein the subtitle data carries a corresponding timestamp in the audio and video data stream;

[0023] The fourth processing unit is used to synchronously push the audio and video data stream and the subtitle data to the live content distribution network.

[0024] Furthermore, the fourth processing unit is used for:

[0025] Obtaining a timestamp corresponding to the audio and video data stream and a timestamp corresponding to the audio and video data stream in the subtitle data;

[0026] The audio and video progress is adjusted by coordinating the timestamp corresponding to the audio and video data stream and the timestamp corresponding to the audio and video data stream in the subtitle data, so as to achieve synchronization between the audio and video data stream and the subtitle data;

[0027] The synchronized audio and video data stream and the subtitle data are pushed to the live content distribution network.

[0028] Furthermore, it also includes:

[0029] A fifth processing unit, configured to receive audio and video data from an edge node of the live content distribution network, and decode the audio and video data into audio and video content through a decoder;

[0030] The sixth processing unit is used to extract the timestamp in the audio and video data stream, and display the audio and video content on the display screen of the mobile device at the viewer end according to a preset display method.

[0031] A storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the method for generating subtitles by real-time speech recognition in mobile live broadcast as described above.

[0032] An electronic device comprises at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; and the processor is used to call program instructions in the memory to execute the method for generating subtitles by real-time speech recognition in mobile live broadcast as described above.

[0033] The method and device for generating subtitles by real-time speech recognition in mobile live broadcast described in the present application, the method collects audio and video data streams in the mobile device of the anchor in real time, the audio and video data streams include audio and video sources, and carry the timestamp of the current time; the audio stream format is transcoded to the audio and video data streams in a preset format to obtain audio data to be recognized in a specific format; speech recognition is performed on the audio data to be recognized to obtain subtitle data in a corresponding text format, the subtitle data carries the corresponding timestamp in the audio and video data stream; the audio and video data streams and the subtitle data are synchronously pushed to the live content distribution network. Through this application, the anchor who broadcasts on the mobile device can convert the voice content into subtitles in real time during the live broadcast and distribute them to the audience watching the live broadcast, and the audience can freely choose the way to display the subtitles, which solves the problems of delay in audio and video live content introduced in the existing live subtitle solutions, and the lack of freedom to display subtitles on the audience side. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A flowchart of a method for generating subtitles by real-time speech recognition for mobile live broadcast disclosed in an embodiment of the present application;

[0036] Figure 2 This is a schematic diagram of the structure of the host terminal and the audience terminal disclosed in the embodiment of the present application;

[0037] Figure 3 A schematic diagram of the structure of a device for generating subtitles by real-time speech recognition for mobile live broadcast disclosed in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0039] Mobile live broadcast is a highly interactive form of live broadcast. The interaction between the anchor and the audience can maximize the audience's sense of presence, participation and enthusiasm, thereby achieving the ultimate goal of platform traffic diversion and monetization. Therefore, the live broadcast is required to achieve low latency of live subtitles as much as possible. Traditional live subtitle technology requires an additional 5-10 minutes of delay. Although the intelligent voice recognition and synthesis subtitle technology has greatly improved the delay, there is still a live broadcast delay of about 5 seconds, which will reduce the audience's experience of watching live broadcasts. At the same time, traditional live subtitle technology and intelligent voice synthesis subtitle technology both synthesize subtitles into audio and video streams. The audience cannot display subtitles freely, and other components and props in the live broadcast room will block the subtitles. Not only will the audience's experience of watching live subtitles be bad, but it will also affect the audience's use of other functions in the live broadcast room.

[0040] The present application provides a method and device for generating subtitles by real-time speech recognition in mobile live broadcast, so as to solve the problems of delay in audio and video live broadcast content introduced in existing live broadcast subtitle solutions, and the problem of limited freedom in displaying subtitles on the audience end.

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] Please see attached Figure 1 , is a flow chart of a method for generating subtitles by real-time speech recognition in mobile live broadcast provided by an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides a method for generating subtitles by real-time speech recognition in mobile live broadcast, and the method comprises the following steps:

[0043] S101: Collecting audio and video data streams in a mobile device of a host in real time, wherein the audio and video data streams include audio and video sources and carry a timestamp of the current time.

[0044] In an embodiment of the present application, the host terminal collects real-time audio and video streams and synthesizes audio and video data streams in a specific format through an encoder, wherein the audio and video data streams carry a timestamp of the current time.

[0045] S102: performing audio stream format transcoding on the audio and video data stream using a preset format to obtain audio data to be identified in a specific format.

[0046] S103: Perform speech recognition on the audio data to be recognized to obtain subtitle data in a corresponding text format, wherein the subtitle data carries a corresponding timestamp in the audio and video data stream.

[0047] S104: synchronously push the audio and video data stream and the subtitle data to a live content distribution network.

[0048] In this step, the anchor can push the collected audio and video data streams to the nodes of the live CDN. By utilizing the load balancing, content distribution, and scheduling capabilities of the live CDN central platform, the audience can receive audio and video data nearby, reducing network congestion and improving user access response speed and hit rate.

[0049] In the embodiment of the present application, the above-mentioned synchronously pushing the audio and video data stream and the subtitle data to the live content distribution network includes:

[0050] Obtaining a timestamp corresponding to the audio and video data stream and a timestamp corresponding to the audio and video data stream in the subtitle data;

[0051] The audio and video progress is adjusted by coordinating the timestamp corresponding to the audio and video data stream and the timestamp corresponding to the audio and video data stream in the subtitle data, so as to achieve synchronization between the audio and video data stream and the subtitle data;

[0052] The synchronized audio and video data stream and the subtitle data are pushed to the live content distribution network.

[0053] In this step, the timestamp corresponding to the audio and video data stream and the timestamp corresponding to the audio and video data stream in the subtitle data are obtained in real time; the progress is adjusted by coordinating the above two timestamps. When the audio and video data is played, the timestamp of the audio and video data is obtained, and the subtitle data with the corresponding equal timestamp is found in the received subtitle data. The system displays the subtitle data according to the audience's operation requirements. In this way, the synchronization module matches the audio and video data with the subtitle data, thereby achieving the purpose of synchronizing the audio and video data and the subtitle data.

[0054] S105: receiving audio and video data from an edge node of the live content distribution network, and decoding the audio and video data into audio and video content through a decoder.

[0055] S106: extracting the timestamp in the audio and video data stream, and displaying the audio and video content on the display screen of the mobile device at the viewer end according to a preset display mode.

[0056] In this step, the audience can receive the audio and video data from the edge node of the live content distribution network nearby, decode the audio and video data into audio and video content through the decoder, and extract the timestamp encoded by the anchor, and finally display the audio and video content on the display screen of the audience's mobile device according to the preset display method.

[0057] It should be noted that if Figure 2 As shown, the embodiment of the present application integrates an acquisition module, a streaming module, a transcoding module, an identification module, and a communication module at the anchor end. The anchor end acquires real-time audio and video streams through the acquisition module, and directly inputs the audio and video data into the transcoding module. After the transcoding module converts it into the audio stream format required by the identification module, the transcoded audio stream is input into the identification module, and the corresponding subtitle content is obtained through the identification module. While the streaming module pushes the audio and video streams, the identified subtitle content is uploaded through the instant messaging module. The audience end includes a streaming module, a playback module, a communication module, a subtitle module, and a synchronization module. The audience end receives audio and video data and subtitle data through the streaming module and the communication module, respectively, and after synchronization by the synchronization module, the playback module and the subtitle module display them to the audience.

[0058] like Figure 2 As shown in the figure, the process indicated by the dotted one-way arrow is the process of pushing and pulling streams of audio and video data through the live CDN: the acquisition module on the host side collects the audio and video sources in the mobile device on the host side, and synthesizes the audio and video data stream in a specific format through the encoder, where the audio and video data stream will carry the timestamp of the current time. The push stream module on the host side pushes the data stream in the acquisition module to the node of the live CDN. By utilizing the load balancing, content distribution, and scheduling capabilities of the central platform of the live CDN, the audience can receive audio and video data nearby, reduce network congestion, and improve user access response speed and hit rate. The pull stream module on the audience side receives the audio and video data from the edge node of the live CDN nearby, decodes the audio and video data into audio and video content through the decoder, and extracts the timestamp encoded by the host side. The playback module on the audience side displays the audio and video content on the display screen of the audience's mobile device.

[0059] like Figure 2 As shown, the process shown by the solid one-way arrow in the figure is the process of real-time voice recognition and sending and receiving subtitle data through the communication module: the acquisition module on the host side transmits the audio and video data in step A to the transcoding module locally at the same time. The transcoding module on the host side transcodes the original audio and video data into an audio data format in a specific format required by the voice recognition module and inputs it into the voice recognition module. The voice recognition module on the host side performs real-time voice recognition on the transcoded audio and video data and converts it into subtitle data in text format, where the subtitle data will carry the corresponding timestamp in the audio and video data. The communication module on the host side distributes the subtitle data in text format to the communication modules of all communication devices on the audience side through the communication service. The subtitle module on the audience side provides the subtitle data to the audience, and the audience is free to choose how to display it.

[0060] like Figure 2As shown in the figure, the process indicated by the solid bidirectional arrows in the dotted box is the process of receiving real-time audio and video data and subtitle data respectively and synchronizing them: the synchronization module obtains in real time the timestamp of the audio and video data when the streaming module decodes the audio and video data in A (3) above. The synchronization module obtains in real time the timestamp of the corresponding audio and video data in the subtitle data mentioned in B (3) above. The synchronization module adjusts the progress by coordinating the above two timestamps. When playing the audio and video data, the timestamp of the audio and video data is obtained, and the subtitle data with the corresponding equal timestamp is found in the received subtitle data. The system displays the subtitle data according to the viewer's operation requirements. In this way, the synchronization module matches the audio and video data with the subtitle data, thereby achieving the purpose of synchronizing the audio and video data with the subtitle data.

[0061] In the embodiment of the present application, the audio data source of the live subtitles is synchronously obtained from the audio and video data stream collected by the anchor end, and the subsequent transcoding identification and sending process is immediately executed on the anchor end, avoiding the CDN distribution delay caused by the subsequent process after pulling the audio and video stream from the CDN, making the generation of subtitles more efficient. At the same time, the subtitle generation process will not affect the process of the audio and video data stream, so it will not introduce delays in the audio and video live content.

[0062] In addition, the embodiment of the present application is a method for receiving real-time audio and video data and subtitle data separately at the audience end and performing synchronization and free display. Since the subtitle data is received separately through the communication module, the subtitle data can be displayed independently according to the audience's wishes, solving the problem of the audience end being unable to display subtitles freely.

[0063] The method for generating subtitles by real-time speech recognition in mobile live broadcast provided by the embodiment of the present application, the method collects the audio and video data stream in the mobile device of the anchor end in real time, the audio and video data stream includes audio and video sources, and carries the timestamp of the current time; the audio stream format of the audio and video data stream is transcoded by a preset format to obtain audio data to be recognized in a specific format; speech recognition is performed on the audio data to be recognized to obtain subtitle data in a corresponding text format, the subtitle data carries the corresponding timestamp in the audio and video data stream; the audio and video data stream and the subtitle data are synchronously pushed to the live content distribution network. Through the embodiment of the present application, the anchor who starts broadcasting on the mobile device can convert the voice content into subtitles in real time during the live broadcast and distribute it to the audience watching the live broadcast, and the audience can freely choose the way to display the subtitles, which solves the problems of delay in audio and video live content introduced in the existing live subtitle solution, and the lack of freedom to display subtitles on the audience end.

[0064] See also Figure 3Based on the method for generating subtitles by real-time speech recognition in mobile live broadcast disclosed in the above embodiment, this embodiment correspondingly discloses a device for generating subtitles by real-time speech recognition in mobile live broadcast, and the device includes:

[0065] The first processing unit 31 is used to collect the audio and video data stream in the mobile device of the anchor end in real time, wherein the audio and video data stream includes audio and video sources and carries a timestamp of the current time;

[0066] The second processing unit 32 is used to perform audio stream format transcoding on the audio and video data stream according to a preset format to obtain audio data to be identified in a specific format;

[0067] The third processing unit 33 is used to perform speech recognition on the audio data to be recognized, and obtain subtitle data in a corresponding text format, wherein the subtitle data carries a corresponding timestamp in the audio and video data stream;

[0068] The fourth processing unit 34 is used to synchronously push the audio and video data stream and the subtitle data to the live content distribution network.

[0069] Furthermore, the fourth processing unit 34 is used for:

[0070] Obtaining a timestamp corresponding to the audio and video data stream and a timestamp corresponding to the audio and video data stream in the subtitle data;

[0071] The audio and video progress is adjusted by coordinating the timestamp corresponding to the audio and video data stream and the timestamp corresponding to the audio and video data stream in the subtitle data, so as to achieve synchronization between the audio and video data stream and the subtitle data;

[0072] The synchronized audio and video data stream and the subtitle data are pushed to the live content distribution network.

[0073] Furthermore, it also includes:

[0074] A fifth processing unit 35 is used to receive audio and video data from an edge node of the live content distribution network, and decode the audio and video data into audio and video content through a decoder;

[0075] The sixth processing unit 36 ​​is used to extract the timestamp in the audio and video data stream, and display the audio and video content on the display screen of the mobile device at the viewer end according to a preset display method.

[0076] The device for generating subtitles by real-time speech recognition in mobile live broadcast includes a processor and a memory. The first processing unit, the second processing unit, the third processing unit and the fourth processing unit are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

[0077] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to improve the ability to identify misleading advertisements and improve the efficiency and accuracy of review.

[0078] An embodiment of the present application provides a storage medium on which a program is stored. When the program is executed by a processor, the method for generating subtitles through real-time speech recognition in mobile live broadcast is implemented.

[0079] An embodiment of the present application provides a processor, which is used to run a program, wherein the method for generating subtitles by real-time speech recognition in mobile live broadcast is executed when the program is run.

[0080] The present application embodiment provides an electronic device, such as Figure 4 As shown, the electronic device 40 includes at least one processor 401, and at least one memory 402 and a bus 403 connected to the processor; wherein the processor 401 and the memory 402 communicate with each other through the bus 403; the processor 401 is used to call the program instructions in the memory 402 to execute the above-mentioned method of generating subtitles by real-time speech recognition in mobile live broadcast.

[0081] The electronic devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0082] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing the following method steps:

[0083] Collecting audio and video data streams in the host's mobile device in real time, wherein the audio and video data streams include audio and video sources and carry a timestamp of the current time;

[0084] Performing audio stream format transcoding on the audio and video data stream according to a preset format to obtain audio data to be identified in a specific format;

[0085] Performing speech recognition on the audio data to be recognized to obtain subtitle data in a corresponding text format, wherein the subtitle data carries a corresponding timestamp in the audio and video data stream;

[0086] The audio and video data stream and the subtitle data are synchronously pushed to a live content distribution network.

[0087] Furthermore, the step of synchronously pushing the audio and video data stream and the subtitle data to a live content distribution network includes:

[0088] Obtaining a timestamp corresponding to the audio and video data stream and a timestamp corresponding to the audio and video data stream in the subtitle data;

[0089] The audio and video progress is adjusted by coordinating the timestamp corresponding to the audio and video data stream and the timestamp corresponding to the audio and video data stream in the subtitle data, so as to achieve synchronization between the audio and video data stream and the subtitle data;

[0090] The synchronized audio and video data stream and the subtitle data are pushed to the live content distribution network.

[0091] Furthermore, it also includes:

[0092] Receiving audio and video data from an edge node of the live content distribution network, and decoding the audio and video data into audio and video content through a decoder;

[0093] The timestamp in the audio and video data stream is extracted, and the audio and video content is displayed on the display screen of the mobile device at the viewer end according to a preset display mode.

[0094] The present application is described by flowcharts and / or block diagrams of the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0095] In a typical configuration, the device includes one or more processors (CPU), memory and bus. The device may also include input / output interface, network interface and the like.

[0096] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0097] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0098] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0099] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0100] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for generating subtitles by real-time speech recognition in mobile live broadcast, characterized in that: include: The host terminal collects the audio and video data stream in the host terminal's mobile device in real time through the host terminal, wherein the audio and video data stream includes audio and video sources and carries a timestamp of the current time; The host terminal performs audio stream format transcoding on the audio and video data stream in a preset format to obtain audio data to be identified in a specific format; The host terminal performs speech recognition on the audio data to be recognized to obtain subtitle data in a corresponding text format, wherein the subtitle data carries a corresponding timestamp in the audio and video data stream; The audio and video data stream and the subtitle data are synchronously pushed to the live content distribution network through the host end, wherein while the audio and video data stream is pushed through the streaming module of the host end, the subtitle data is distributed to the communication module of the communication device of the audience end through the communication service through the communication module of the host end.

2. The method according to claim 1, characterized in that: The step of synchronously pushing the audio and video data stream and the subtitle data to a live content distribution network includes: Obtaining a timestamp corresponding to the audio and video data stream and a timestamp corresponding to the audio and video data stream in the subtitle data; The audio and video progress is adjusted by coordinating the timestamp corresponding to the audio and video data stream and the timestamp corresponding to the audio and video data stream in the subtitle data, so as to achieve synchronization between the audio and video data stream and the subtitle data; The synchronized audio and video data stream and the subtitle data are pushed to the live content distribution network.

3. The method according to claim 1, characterized in that Also includes: Receiving audio and video data from an edge node of the live content distribution network, and decoding the audio and video data into audio and video content through a decoder; The timestamp in the audio and video data stream is extracted, and the audio and video content is displayed on the display screen of the mobile device at the viewer end according to a preset display mode.

4. A device for generating subtitles by real-time speech recognition in mobile live broadcast, characterized in that: include: The first processing unit is used to collect the audio and video data stream in the mobile device of the anchor terminal in real time through the anchor terminal, wherein the audio and video data stream includes audio and video sources and carries a timestamp of the current time; The second processing unit is used to perform audio stream format transcoding on the audio and video data stream in a preset format through the anchor terminal to obtain audio data to be identified in a specific format; A third processing unit, configured to perform speech recognition on the audio data to be recognized through the anchor terminal, and obtain subtitle data in a corresponding text format, wherein the subtitle data carries a corresponding timestamp in the audio and video data stream; The fourth processing unit is used to synchronously push the audio and video data stream and the subtitle data to the live content distribution network through the anchor end, wherein while the audio and video data stream is pushed through the streaming module of the anchor end, the subtitle data is distributed to the communication module of the communication device of the audience end through the communication service through the communication module of the anchor end.

5. The device according to claim 4, characterized in that The fourth processing unit is used for: Obtaining a timestamp corresponding to the audio and video data stream and a timestamp corresponding to the audio and video data stream in the subtitle data; The audio and video progress is adjusted by coordinating the timestamp corresponding to the audio and video data stream and the timestamp corresponding to the audio and video data stream in the subtitle data, so as to achieve synchronization between the audio and video data stream and the subtitle data; The synchronized audio and video data stream and the subtitle data are pushed to the live content distribution network.

6. The device according to claim 4, characterized in that Also includes: A fifth processing unit, configured to receive audio and video data from an edge node of the live content distribution network, and decode the audio and video data into audio and video content through a decoder; The sixth processing unit is used to extract the timestamp in the audio and video data stream, and display the audio and video content on the display screen of the mobile device at the viewer end according to a preset display method.

7. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the method for generating subtitles by mobile live broadcast real-time speech recognition as described in any one of claims 1 to 3.

8. An electronic device, characterized in that: The electronic device includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the method for generating subtitles by mobile live broadcast real-time speech recognition as described in any one of claims 1 to 3.

Images