Live streaming translation method and apparatus, storage medium and computer device

Abstract

The present application discloses a live streaming translation method, comprising: acquiring a candidate live stream from cached live streams; performing translation processing on a voice recognition content corresponding to the candidate live stream, so as to obtain a translation result corresponding to the candidate live stream; on the basis of the translation result corresponding to the candidate live stream and a target end timestamp of a target live stream to be pushed, determining a target translation result to be pushed; on the basis of the target translation result, re-encoding the target live stream to be pushed, so as to obtain a re-encoded live stream; and pushing the re-encoded live stream to display the target translation result at a viewer end. The present application sets a duration threshold and determines, on the basis of the translation result corresponding to the candidate live stream, the target translation result to be pushed, so that the target translation result can be acquired and pushed within a time period corresponding to the duration threshold, so as to improve the accuracy of a live streaming translation result.

Description

Live translation method, device, storage medium and computer equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 5, 2023, with application number 2023111468669 and application name “Live Translation Method, Device, Storage Medium and Computer Equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and more specifically, to a live translation method, apparatus, storage medium, and computer equipment. Background Art

[0003] Simultaneous interpretation is a method of interpretation in which the interpreter interprets the content to the audience without interrupting the speaker. Relying on the development of computer and information technology, simultaneous interpretation and communication technology can now be closely integrated.

[0004] For example, automatic translation is performed in scenarios such as live broadcasts, webcasts, or real-time calls. When the relevant technology automatically translates the speaker's words in the scene, it usually provides a translation result corresponding to the speaker's expression after the speaker finishes speaking.

[0005] This automatic translation method can translate the speaker's content, but it often requires a certain amount of time to obtain accurate translation results. Furthermore, if the translation results corresponding to the live stream to be pushed are sent to the audience in real time, the translation results may be inaccurate. In short, the live translation effect is poor, affecting the user's viewing experience.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a live broadcast translation method, apparatus, storage medium, and computer device to solve the problem in the related art that live broadcast translation results are inaccurate, resulting in poor live broadcast translation effects.

[0008] On the one hand, an embodiment of the present application provides a live translation method, the method comprising: obtaining a candidate live stream to be translated from a cached live stream; the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp, the target timestamp refers to the end time of the live stream corresponding to the last stable translation result, the live stream corresponding to the last stable translation result refers to a translated live stream, the translated live stream is the last live stream of the candidate live stream, and the last stable translation result refers to a stable translation result obtained after translating the translated live stream; translating the speech recognition content corresponding to the candidate live stream to obtain a candidate live stream. The target translation result of the stream to be pushed is determined based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the target translation result of the stream to be pushed refers to the translation result to be pushed along with the target live stream, and the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a preset duration threshold, the duration threshold is a positive number, and the duration threshold is used to represent the maximum delayed pushing duration of the target live stream; based on the target translation result, the target live stream to be pushed is re-encoded to obtain a re-encoded live stream; the re-encoded live stream is pushed to display the target translation result on the audience end.

[0009] On the other hand, an embodiment of the present application further provides a live translation device, which includes: a live stream acquisition module, configured to acquire a candidate live stream to be translated from a cached live stream; the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp, the target timestamp refers to the end time of the live stream corresponding to the last stable translation result, the live stream corresponding to the last stable translation result refers to a translated live stream, the translated live stream is the previous live stream of the candidate live stream, and the last stable translation result refers to a stable translation result obtained after translation processing of the translated live stream; a translation processing module, configured to determine a target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the target translation result of the stream to be pushed refers to the translation result to be pushed along with the target live stream, and the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a preset duration threshold, the duration threshold is a positive number, and the duration threshold is used to represent the maximum delayed push duration of the target live stream; The re-encoding module is used to re-encode the target live stream to be pushed based on the target translation result to obtain the re-encoded live stream; the streaming module is used to push the re-encoded live stream to display the target translation result on the audience end.

[0010] On the other hand, an embodiment of the present application further provides a computer device, which includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are called by the processor, the above-mentioned live broadcast translation method is executed.

[0011] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores program code, wherein the above-mentioned live broadcast translation method is executed when the program code is executed by a processor.

[0012] In another aspect, embodiments of the present application further provide a computer program product or computer program, comprising computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium and executes the computer instructions, causing the computer to perform the steps of the live broadcast translation method described above.

[0013] The live broadcast translation method provided by the present application can obtain a candidate live broadcast stream from a cached live broadcast stream, wherein the candidate live broadcast stream refers to a live broadcast stream whose start time is a target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live broadcast stream corresponding to the last stable translation result; the live broadcast stream corresponding to the last stable translation result here refers to the translated live broadcast stream, and the translated live broadcast stream is the last live broadcast stream of the candidate live broadcast stream; the last stable translation result refers to the stable translation result obtained after translating the translated live broadcast stream; thereafter, the embodiment of the present application can translate the speech recognition content corresponding to the candidate live broadcast stream to obtain the translation result corresponding to the candidate live broadcast stream, and The target translation result of the stream to be pushed can be determined based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed, wherein the target translation result of the stream to be pushed refers to the translation result to be pushed along with the target live stream, and the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold, where the duration threshold is a positive number, and the duration threshold is used to represent the maximum delayed pushing duration of the target live stream; thereafter, the embodiment of the present application can re-encode the target live stream to be pushed based on the target translation result, obtain the re-encoded live stream, and push the re-encoded live stream to display the translation result in the target translation result on the audience end. It can be seen from this that since the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold, and the start time of the candidate live stream is the end time of the live stream corresponding to the previous stable translation result, the target translation result is equivalent to being obtained by translating a live stream with a duration longer than the target live stream to be pushed. Therefore, in the live broadcast scenario, when using this live translation method for live translation, the live stream on which it is based is longer in duration, so a more complete and accurate target translation result can be obtained by translating a live stream with a longer duration.

[0014] Therefore, in the present application, compared to directly translating the speech recognition content corresponding to the target live stream to be pushed, and recoding the translation result corresponding to the target live stream into the target live stream, the accuracy of the target coding result finally obtained by the live stream with a longer duration involved in the embodiment of the present application is higher, so when the relatively accurate target translation result obtained by translation is pushed, it is possible to ensure that the live translation can ensure accurate translation quality. In addition, since the end time of the live stream corresponding to the target translation result does not exceed the duration threshold value relative to the time difference between the target end timestamp, like this, under the premise of ensuring that the duration of the target live stream is delayed at most (i.e., the maximum delayed push duration) is the above-mentioned duration threshold value, it is possible to ensure the accuracy of the live translation carried out in real time under the live broadcast scenario by sacrificing the real-time of part live broadcast, so that the target translation result obtained by translation can be used to improve the user's live viewing experience in the entire live broadcast scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0016] FIG1 shows a schematic diagram of the architecture of a live translation system provided in an embodiment of the present application.

[0017] FIG2 shows an application scenario diagram of a live broadcast translation method provided in an embodiment of the present application.

[0018] FIG3 shows a flow chart of a live broadcast translation method provided in one embodiment of the present application.

[0019] FIG4 shows a schematic diagram of a process for determining a target translation result provided by an embodiment of the present application.

[0020] FIG5 shows a flowchart of a translation process provided by an embodiment of the present application.

[0021] FIG6 shows a flow chart of a live broadcast translation method provided in another embodiment of the present application.

[0022] FIG7 shows a flowchart of a live broadcast translation provided by an embodiment of the present application.

[0023] FIG8 shows a schematic diagram of a code stream format of a network abstraction layer data unit provided in an embodiment of the present application.

[0024] FIG9 shows a schematic diagram of a data format of additional enhancement information provided in an embodiment of the present application.

[0025] FIG10 shows a module block diagram of a live broadcast translation device provided in an embodiment of the present application.

[0026] FIG11 is a module block diagram of a computer device provided in an embodiment of the present application.

[0027] FIG12 is a module block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] It should be noted that in the specific implementation of this application, when it comes to user audio, video, live streaming and other related data, when applied to the specific products or technologies of the embodiments of this application, it is necessary to obtain user permission or consent, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and subsequent data use and processing must be carried out within the scope of authorization of laws, regulations and personal information subjects.

[0029] At present, different languages ​​have their own specific grammars and expressions, but all of them need to translate the audio of the complete sentence according to the translation method of the complete sentence (i.e., offline translation method) after obtaining an audio expressing a complete sentence, so as to obtain the translation result corresponding to the audio expressing the complete sentence. However, in a live broadcast scenario, since a live stream needs to be pushed to the audience end in real time, and the duration of each pushed live stream is short, if a live broadcast translation method is directly adopted to translate the speech recognition content of the audio in each pushed live stream, and the translation result is pushed to the audience end together with the currently pushed live stream, this means that once the audio in the live stream cannot fully express the speaker's (i.e., the host's) speech content, the translation result of the live stream obtained by the real-time translation using the live broadcast translation method will be inaccurate. In this way, when the live stream and the translation result of the live stream are pushed to the audience end together, the translation result of the live stream displayed on the audience end will also be inaccurate, so as to affect the live viewing experience of the audience in the live broadcast scenario.

[0030] However, if the speaker (i.e., the host) waits for the speaker to finish speaking one or more sentences before translating the speech recognition content of the audio in the live stream, and then sending the translation results and the live stream to the audience, this can easily lead to serious delays in the live broadcast. Moreover, the real-time performance of the live broadcast translation is poor, which may cause the translation results displayed to the audience to be out of sync with the audio and video images decoded from the live broadcast stream, affecting the audience's live viewing experience and also affecting the interaction between the host and the audience. In order to solve the above problems, the inventors have proposed the live broadcast translation method of the present application after research.

[0031] The live translation method of the present application involves streaming media technology, which specifically refers to a technology and process that compresses a series of media data, sends the data in segments over the Internet, and transmits audio and video online for viewing in real time. Streaming can transmit live audio and video or videos pre-stored on a server. When the live audio and video or the audio and video data of the video are delivered to the viewer's computer device, the received audio and video data (i.e., audio and video data) can be immediately played by specific playback software on the computer device, so that the viewer can watch the live audio and video or the video pre-stored on the server. For example, the live application related to the live translation method provided in the present application.

[0032] The live broadcast translation method referred to in this application is a method for transmitting and playing live audio and video using streaming media technology in a live broadcast scenario. The live broadcast translation method can be applied to live broadcasts and online video conferencing. For example, during a live broadcast, when the host terminal corresponding to the host is integrated with the above-mentioned live broadcast application, the live broadcast application can be used to call a camera to capture video frames associated with the host, and the captured video frames can be encoded using a video encoding protocol (such as the H.264 encoding protocol) to obtain video encoding content for the host (i.e., a video stream or a video encoding stream); at the same time, the host terminal corresponding to the host can also call a microphone through the live broadcast application to capture audio data associated with the host, and can use an audio encoding protocol to perform audio encoding on the captured audio data to obtain audio encoding content for the host (i.e., an audio stream or an audio encoding stream). Then, the host terminal can use the streaming format indicated by the streaming technology to perform streaming encapsulation processing (e.g., perform the above-mentioned media data compression processing) on ​​the currently obtained video encoding content (i.e., video stream) and audio encoding content (i.e., audio stream) to obtain a live broadcast data stream for pushing to the server. Specifically, the H.264 coding framework consists of the Video Coding Layer (VCL) and the Network Abstraction Layer (NAL). The VCL is used to efficiently represent video content, while the NAL formats data and provides header information to ensure efficient transmission across various channels and storage media.

[0033] The video content representation includes I-frames generated by intra-frame compression of video frames (i.e., the aforementioned video frames) and P and / or B-frames generated by inter-frame compression of video frames. I-frames are fully encoded keyframes, P-frames are forward-predictive coded frames, and B-frames are bidirectionally predicted interpolated coded frames. In the network abstraction layer, a Network Abstract Layer Unit (NALU) is the basic unit for encoding, storage, or transmission using the H.264 encoding protocol.

[0034] Each NALU contains a header structure and a body. The header structure occupies 1 byte (8 bits). The header structure indicates whether the corresponding network abstraction layer unit in the NAL (i.e., the NALU where the header structure is located) can be discarded, the importance indication, and the NALU type. In the H.264 bitstream, each frame of data is a NALU. For example, in an embodiment of the present application, the auxiliary enhancement frame generated based on the target translation result is a custom data field in the Supplemental Enhancement Information (SEI) that adds the target translation result to the target translation result and encapsulates it into a specific type of NALU.

[0035] To prevent the problem of excessive pressure on the server due to excessive number of users and slow download speeds for concentrated users. The live broadcast scenario can use a content delivery network (CDN) to distribute data. The content distribution network consists of two layers: the center and the edge. The edge servers of the edge layer are deployed in various places and across major operators, and the physical distance between the edge servers and the users is the shortest. The streaming media service cluster of the center layer is responsible for forwarding the content. For example, based on the user's geographic location information, the nearest edge server is selected to provide push / pull streaming services to the user. It should be understood that in the embodiment of the present application, the push streaming service provided by the edge server to the user refers to the service that the edge server can intelligently push the acquired live stream of the anchor to the audience end after obtaining the live stream of the above-mentioned anchor; similarly, the pull streaming service provided by the edge server to the user refers to the service that the edge server can pull the live stream of the anchor from the anchor end.

[0036] The following first introduces the system architecture of the live broadcast translation method involved in this application.

[0037] Please refer to Figure 1, which shows a schematic diagram of the architecture of a live translation system provided in an embodiment of the present application. As shown in Figure 1, the live translation system 100 may include a content distribution network 110, a host terminal 130, and an audience terminal 150. The host terminal 130 and the audience terminal 150 may be terminal devices such as smartphones, desktop computers, tablet computers, laptops, smart TVs, in-vehicle devices, augmented reality (AR), virtual reality (VR), etc., without limitation herein.

[0038] The content distribution network 110 may include edge servers and a streaming service cluster, which may include multiple streaming servers. The edge servers and streaming servers may be independent physical servers, or they may be server clusters or distributed systems composed of multiple physical servers. They may also be cloud servers that provide basic cloud computing services such as cloud services, cloud computing, cloud functions, cloud storage, cloud communications, network services, domain name services, middleware services, security services, blockchain, and big data and artificial intelligence platforms, without limitation.

[0039] In some embodiments of the present application, after collecting the live stream, the host end 130 can translate the live stream based on the live translation method involved in the present application to obtain a live stream carrying the translation result (i.e., the re-encoded live stream in the present application). Further, the host end 130 can push the live stream carrying the translation result to the content distribution network 110, and then the audience end 150 can pull the stream and decode it from the content distribution network 110 to obtain audio and video carrying subtitle content (the subtitle content at least includes the translation result, and can also include the translation result and the speech recognition content corresponding to the audio). Among them, the speech recognition content corresponding to the audio (i.e., audio data) can be the speech text data (e.g., speech conversion text data) corresponding to the audio data of the host recognized when the host speaks in a certain language (e.g., a first language), and the translation result here refers to the translation text data (e.g., translated subtitles) expressed in another language (e.g., a second language) obtained by performing simultaneous translation on the recognized audio data of the host. Among them, the first language and the second language here can include but are not limited to English, Chinese and other voices, and the specific language types of the first language and the second language will not be limited here.

[0040] In other words, stream pulling and decoding here specifically means that the viewer terminal 150 can pull the live stream carrying the translation results (i.e., the re-encoded live stream in this application) from the content distribution network 110, and decode the pulled re-encoded live stream to obtain the audio and video carrying the translation results (i.e., the audio and video data carrying the translation results). In this way, the viewer terminal 150 can also display the decoded subtitle content (e.g., the translation results, i.e., translated subtitles) on the screen while playing the audio and video.

[0041] It should be understood that Figure 1 is merely a schematic diagram of the architecture of a system provided by an embodiment of the present application. The system architecture described in the embodiment of the present application is intended to more clearly illustrate the technical solution of the embodiment of the present application and does not constitute a limitation on the technical solution provided by the embodiment of the present application. For example, the process of translating the live stream in Figure 1 is that the host translates the live stream and then pushes the stream to the content distribution network.

[0042] Optionally, in other cases, the host end can also push the live stream to the live broadcast server end, and the live broadcast server end can translate the live stream according to the live broadcast translation method involved in this application to obtain the translation result of the live stream. Then, the live broadcast server end can upload the live stream carrying the translation result (i.e., the re-encoded live stream) to the content distribution network. In this way, after pulling the stream from the content distribution network and decoding it, the audience end can obtain audio and video with the translation result, so that the audio and video with the translation result can be played on the screen of the audience end. For example, the player called by the live broadcast application can display the host's video frame on the screen of the audience end, and synchronously play the host's audio data and display the translation result corresponding to the audio data.

[0043] Please refer to Figure 2, which shows an application scenario diagram of a live broadcast translation method provided by an embodiment of the present application. As shown in Figure 2, the live broadcast translation method can be applied to a computer system 200, and the computer system 200 can be applied to a live broadcast scenario. The computer system 200 may include a live broadcast service end (or backend) 220 of a live broadcast service provider, a live broadcast end 210 on the user side, and an audience end 230. The content distribution network includes a streaming service cluster 260, a first edge server 240 corresponding to the live broadcast end 210, and a second edge server 280 corresponding to the audience end 230.

[0044] In a live broadcast scenario, when the anchor uses the live broadcast terminal 210 to broadcast live on the Internet, the live broadcast terminal 210 can encode the audio and video collected in real time into a live stream, and then push the live stream to the live broadcast server 220. Then, after receiving the live stream, the live broadcast server 220 can further cache the live stream received in real time, and perform live translation processing (i.e., translation processing) on ​​the cached live stream according to the live translation method involved in this application to obtain a re-encoded live stream, and can further push the re-encoded live stream to the first edge server 240 in the content distribution network.

[0045] Furthermore, as shown in FIG2 , the first edge server 240 can push the re-encoded live stream to the streaming service cluster 260. After receiving the re-encoded live stream, the streaming service cluster 260 can perform certain processing on the re-encoded live stream, such as transcoding, that is, converting it from one encoding format to another encoding format. This is because different viewers use different clients and it is necessary to ensure that they can all watch the live broadcast normally. The re-encoded live stream (or the re-encoded live stream after transcoding) is then pre-loaded to an edge server closest to the viewer terminal, such as the second edge server 280. At this time, the viewer terminal 230 can pull the re-encoded live stream (or the re-encoded live stream after transcoding) from the second edge server 280 and decode the re-encoded live stream (or the re-encoded live stream after transcoding) to obtain the host's audio and video and the corresponding translation results of the audio and video. Then, while playing the decoded audio and video, the corresponding translation results (e.g., subtitles) of the corresponding audio can be displayed on the host's video screen (i.e., the above-mentioned video frame screen).

[0046] It should be noted that Figure 2 is only an application scenario diagram of a system provided by an embodiment of the present application. The system architecture and application scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided by the embodiment of the present application. For example, the first edge server 240 can generally refer to one of multiple edge servers deployed in the content distribution network, and the second edge server 280 can also generally refer to one of multiple edge servers deployed in the content distribution network. This embodiment only uses the first edge server 240 and the second edge server 280 as examples. It is known to those skilled in the art that with the evolution of the system architecture, the technical solution provided by the embodiment of the present invention is also applicable to similar technical problems.

[0047] Please refer to Figure 3, which shows a flow chart of a live broadcast translation method provided by an embodiment of the present application. In the embodiment of the present application, the live broadcast translation method can be executed by a computer device, and the computer device here can be a live broadcast server and / or an anchor terminal. In other words, the live broadcast translation method involved in the embodiment of the present application can be specifically executed by the live broadcast server, or by the anchor terminal, or by the live broadcast server and the anchor terminal interacting with each other, which is not specifically limited here. For ease of understanding, as shown in Figure 3, taking the live broadcast translation method executed by the live broadcast server as an example, the live broadcast translation method can specifically include the following steps S110-step S150:

[0048] Step S110: Obtain a candidate live stream from the cached live stream; the candidate live stream refers to a live stream whose start time is the target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation result.

[0049] Typically, the host side can perform real-time audio and video acquisition when the host initiates a live broadcast. Audio acquisition refers to when the host side is integrated with a live broadcast application, the live broadcast application can call the corresponding audio acquisition device (for example, the microphone carried by the host side itself or the microphone connected to the host side, etc.), and capture or collect the host's audio signal (i.e., an analog signal) in real time according to a preset audio sampling rate, and perform audio processing on the audio signal captured or collected in real time (for example, time-spectrum conversion processing can be performed, etc.), and then the audio signal after audio processing can be collectively referred to as the real-time collected audio data of the host. Similarly, video acquisition refers to when the host side is integrated with a live broadcast application, the live broadcast application can call the corresponding video acquisition device (for example, the camera carried by the host side itself or the camera connected to the host side, etc.) to collect the host's video frames in real time, and the real-time collected video frames of the host can be collectively referred to as collected video data. In other words, in an embodiment of the present application, the anchor terminal can collect the anchor's audio data and video data in real time when the anchor initiates a live broadcast, and the collected audio data and video data can be collectively referred to as audio and video data.

[0050] Afterwards, the host end can encode the audio and video data collected in real time (i.e., the collected audio data and video data), and refer to the encoded audio and video data as the audio and video stream obtained by encoding (or audio and video code stream, i.e., the above-mentioned live stream). Optionally, in an embodiment of the present application, the host end can also pre-process the audio and video data (i.e., audio data and video data) currently collected in real time in the process of collecting audio and video data (e.g., beautify, filter or special effects, etc., for the collected video data, and perform echo cancellation or noise reduction on the collected audio data), and then encode the pre-processed audio and video data to obtain an audio and video stream (or audio and video code stream, i.e., the above-mentioned live stream) that can be transmitted. It should be understood that in an embodiment of the present application, the audio and video stream (i.e., the above-mentioned live stream) can specifically include: an audio stream obtained by encoding the audio data and a video stream obtained by encoding the video data.

[0051] Furthermore, the host can push the encoded audio and video stream to the backend corresponding to the live broadcast application (i.e., the live broadcast server). Specifically, the host can push the encoded live stream to the live broadcast server based on a streaming media protocol. It should be understood that in the embodiment of the present application, in the live broadcast scenario, the live broadcast server refers to the server for providing live broadcast services. The streaming media protocol may include the Real-Time Messaging Protocol (RTMP) or the HTTP-based adaptive bitrate streaming media transmission protocol (HTTP Live Streaming, HLS), etc., which are not limited here.

[0052] As an implementation, a computer device (e.g., a live streaming server) can cache a live stream pushed by a host to facilitate subsequent processing such as translation. For example, the live streaming server can cache the received live stream in a stream buffer pool and then retrieve a segment of the live stream to be translated from the live streams cached in the stream buffer pool, also known as a candidate live stream. Specifically, the live streaming server can obtain the end time of the live stream corresponding to the last stable translation result, also known as the target timestamp.

[0053] Furthermore, a computer device (e.g., a live streaming server) can obtain a live streaming segment from the cached live streaming stream with the target timestamp as the start time as a candidate live streaming stream. A stable translation result refers to a translation result in a steady state, that is, a stable translation result refers to a translation result that will not change based on the translation processing of a subsequent live streaming stream (e.g., the candidate live streaming stream here), or the stable translation result is a translation result corresponding to a segment of audio with basically complete semantics (e.g., audio corresponding to a sentence), and the content of the translation result (i.e., the aforementioned audio with basically complete semantics) will not change with the translation processing of a subsequent live streaming stream (e.g., the candidate live streaming stream here).

[0054] For example, when the host initiates a live broadcast, the host terminal 210 shown in FIG2 can collect audio (i.e., audio data) and video (video data) during the live broadcast process, and encode the audio data and video data collected in real time to encode a live stream. After the host terminal 210 pushes (i.e., pushes) the encoded live stream to the live broadcast server 220, the live broadcast server 220 can store the live stream pushed by the host terminal in a stream buffer pool. Furthermore, the live broadcast server 220 can obtain the target timestamp corresponding to the last stable translation result, for example, the target timestamp is 08:34.17, and then select the live stream starting at 08:34.17 from the stream buffer pool as a candidate live stream.

[0055] It should be understood that in the embodiments of the present application, if the computer device used to perform the live broadcast translation method is a live broadcast server, the stream buffer pool can be used to cache in real time the various live broadcast streams encoded and uploaded by the host. A live broadcast stream refers to the audio and video stream obtained by encoding the audio and video data collected in real time based on the sampling duration indicated by the corresponding sampling rate.

[0056] Based on this, any live stream cached in the bitstream cache pool corresponds to a duration (for example, a sampling duration). The sampling durations corresponding to different live streams each correspond to a start timestamp corresponding to the start time and an end timestamp corresponding to the end time. To ensure the continuity and uninterrupted nature of live data in a live network broadcast scenario, an embodiment of the present application proposes that, upon obtaining a live stream to be translated from the bitstream cache pool, the live streaming server can quickly obtain the end time corresponding to the live stream of the last stable translation result (i.e., the most recently obtained stable translation result) obtained using the live translation method, and use the end time corresponding to the live stream of the last stable translation result (i.e., the most recently obtained stable translation result) as the start time of the live stream to be translated. This allows the live stream to be searched in the bitstream cache pool for a live stream whose start time is the end time corresponding to the live stream of the last stable translation result (i.e., the most recently obtained stable translation result), and use the found live stream as a candidate live stream to be translated.

[0057] It should be understood that in an embodiment of the present application, the candidate live stream is the current live stream to be translated, and the live stream corresponding to the last stable translation result is the currently translated live stream. Since the end timestamp of the currently translated live stream is the start timestamp of the current live stream to be translated, the currently translated live stream can be regarded as a live stream that has been translated and has obtained a stable translation result before the candidate live stream. For example, in one implementable method, the live stream of the last stable translation result can specifically be the last live stream of the candidate live stream (for example, relative to the candidate live stream obtained this time, it is essentially the last live stream obtained from the code stream cache pool), and the translation result obtained after the live translation method is used to translate the previous live stream is collectively referred to as the last stable translation result. In other words, the translation result of the previous live stream can be the stable translation result obtained last time (i.e., the last stable translation result).

[0058] More specifically, a live stream with a start time equal to the target timestamp and an end time equal to the end time of the most recently cached live stream in the stream cache can be retrieved from the stream cache pool as a candidate live stream. For example, if the end time of the most recently cached live stream in the stream cache pool is 08:34.23 and the target timestamp is 08:34.17, the live stream from 08:34.17 to 08:34.23 can be selected as a candidate live stream. It is understood that in this case, because the stream cache pool caches new live streams in real time, the end times of the candidate live streams retrieved from the stream cache pool twice are different. It should be understood that the candidate live streams retrieved from the stream cache pool twice refer specifically to the live stream currently retrieved from the stream cache pool (i.e., the candidate live stream to be translated here) and the live stream previously retrieved from the stream cache pool (e.g., the translated live stream mentioned above). In other words, the two consecutive candidate live streams obtained from the code stream buffer pool specifically refer to the candidate live stream to be translated and the previous live stream of the candidate live stream to be translated (ie, the translated live stream mentioned above).

[0059] Optionally, in another possible implementation, the candidate live streams obtained from the code stream cache pool twice adjacently may specifically refer to the live stream obtained from the code stream cache pool this time (i.e., the candidate live stream to be translated here) and the new live stream obtained from the code stream cache pool next time. It is understood that, in the embodiment of the present application, the start time of the next new live stream (e.g., a new candidate live stream) obtained from the stream cache pool can be the same as the start time of the candidate live stream obtained from the stream cache pool this time (both are target timestamps). For example, after performing live translation on the candidate live stream obtained from the stream cache pool this time, if there is no stable translation result in the translation result of the candidate live stream obtained this time, the target timestamp will not be updated. This means that the start time of the next new live stream (e.g., a new candidate live stream) obtained from the stream cache pool will still be the target timestamp, but the duration of the new live stream (e.g., a new candidate live stream) will be longer than the duration of the old live stream (e.g., the current candidate live stream). Similarly, the start time of the next new live stream (e.g., a new candidate live stream) obtained from the stream cache pool can also be different from the start time of the candidate live stream obtained from the stream cache pool this time. For example, after performing live translation on the candidate live stream obtained from the code stream cache pool this time, if there is a stable translation result in the translation results of the candidate live stream obtained this time, then the start time of the new live stream (for example, the new candidate live stream) obtained from the code stream cache pool next time can be the new target timestamp, and the new target timestamp is the end time of the candidate live stream obtained from the code stream cache pool this time.

[0060] Step S120: Translate the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream.

[0061] Specifically, when a computer device (for example, a live broadcast server) obtains a candidate live broadcast stream, it can decode the candidate live broadcast stream to obtain audio and video data corresponding to the candidate live broadcast stream, obtain audio data to be voice recognized from the audio and video data, perform voice recognition on the audio data to obtain corresponding voice recognition content, and then translate the voice recognition content to obtain a translation result corresponding to the candidate live broadcast stream. For example, the voice recognition content expressed in English (the above-mentioned first language) can be translated (for example, English-to-Chinese processing) to obtain translated text data (that is, translated text content, for example, the above-mentioned translated subtitles) expressed in Chinese (the above-mentioned second language).

[0062] Optionally, it is understandable that the embodiment of the present application may also perform translation processing (for example, Chinese to English processing) on ​​the recognized speech recognition content expressed in Chinese to obtain translated text data expressed in English.

[0063] Among them, it can be understood that the first language refers to the language used by the anchor when speaking during the live broadcast when the anchor initiates the live broadcast through the anchor terminal, and the second language here refers to the preset language flexibly set by the audience for translating the first language when the audience watches the live broadcast through the audience terminal. In other words, in the embodiment of the present application, it is necessary to identify the language used by the anchor when speaking (i.e., the first language) from the decoded audio data during the speech recognition process, and obtain the second language preset by the audience currently accessing the live broadcast room where the anchor is located, so that when the speech recognition content corresponding to the audio data is obtained by speech recognition, the language recognition content expressed in the first language can be translated into translated text data expressed in the second language (i.e., translated text content, for example, the above-mentioned translated subtitles).

[0064] In some embodiments, a speech recognition model and a text translation model may be deployed, wherein the speech recognition model and the text translation model may be models constructed using a neural network. It should be understood that the candidate live stream is the live stream to be translated. Since the live stream refers to the audio and video stream obtained by encoding the audio and video data collected in real time by the host end, the audio and video stream here may specifically include an audio stream obtained by audio encoding the audio data and a video stream obtained by video encoding the video data.

[0065] In this way, after obtaining the candidate live stream, the live broadcast service end can decode the audio stream in the candidate live stream (for ease of distinction, the audio stream is referred to as the candidate audio stream) to input the audio data corresponding to the decoded audio stream into the speech recognition model, so as to perform speech recognition on the audio data through the speech recognition model, and output the speech recognition content corresponding to the candidate audio stream through the speech recognition model, that is, the speech recognition content corresponding to the candidate live stream. Afterwards, the live broadcast service end can input the speech recognition content corresponding to the candidate live stream into the text translation model for text translation to output the translation result corresponding to the speech recognition content, that is, the translation result corresponding to the candidate live stream includes the translation result corresponding to the speech recognition content corresponding to the candidate live stream.

[0066] In some embodiments, a speech recognition model for performing speech recognition on audio in a first language can be deployed according to a first language (the first language is the language used by the anchor when broadcasting live through the anchor terminal), and a text translation model for translating text in the first language into text in a second language can be deployed according to the first language and a second language (the second language is the language of the translated text pre-specified by the audience when watching the live broadcast), for example, a text translation model for translating Mandarin Chinese into English, or another example, a text translation model for translating English into German.

[0067] As an implementation method, the live broadcast server can combine the translation result corresponding to the speech recognition content corresponding to the candidate live broadcast stream (for example, the above-mentioned translated subtitles), the end time and start time corresponding to the candidate live broadcast stream, and the target timestamp (that is, the end time of the live broadcast stream corresponding to the previous stable translation result) to obtain the translation result corresponding to the candidate live broadcast stream.

[0068] In some embodiments, after the text translation model translates and obtains the translation result corresponding to the speech recognition content corresponding to the candidate live stream (for example, the above-mentioned translated subtitles), the translation result corresponding to the speech recognition content corresponding to the candidate live stream (for example, the above-mentioned translated subtitles), the speech recognition content corresponding to the candidate live stream, the end time and start time corresponding to the candidate live stream, and the target timestamp (i.e., the end time of the live stream corresponding to the previous stable translation result) can be combined to obtain the translation result corresponding to the candidate live stream. Of course, in other embodiments, the translation result corresponding to the speech recognition content corresponding to the candidate live stream and the speech recognition content corresponding to the candidate live stream can also be combined as the translation result corresponding to the candidate live stream.

[0069] In other words, in an embodiment of the present application, the translation result may at least include the translation result corresponding to the voice recognition content corresponding to the candidate live stream. Optionally, in one or more implementable methods, the translation result may also include one or more of the voice recognition content corresponding to the candidate live stream, the start time and end time corresponding to the candidate live stream, and the end time of the live stream corresponding to the currently acquired stable content (for example, the previous stable translation result).

[0070] It should be understood that in the embodiment of the present application, stable content (i.e., the last stable translation result) refers to a sentence that the translator considers to be complete and will not change its translation result with the content of the subsequent speech. After obtaining the end time of the stable content (i.e., the last stable translation result), the end time of the stable content (i.e., the last stable translation result) can be used as the above-mentioned target timestamp (i.e., recorded as stamp), so that when the translation content selector subsequently obtains the candidate live stream to be translated from the code stream cache pool, the live stream with a start time before the stamp can be directly skipped, and the live stream with a start time of at least the target timestamp and an end time after the target timestamp can be selected as the candidate live stream. In other words, in the embodiment of the present application, the live broadcast service end can select the audio data in the live stream with a start time at least after the stamp from the code stream cache pool through the translation content selector (i.e., the live stream or audio data in the live stream with a start time exceeding the stamp can be selected) and sent to the translator for translation. In this way, not only can the amount of translation data of the translator be reduced, but also the translation speed of the translator through the above-mentioned text translation model can be improved.

[0071] Among them, it can be understood that a translator refers to a tool that can be used to perform speech recognition and translation on an audio stream or audio data in an audio stream. For example, an embodiment of the present application can input an audio stream or audio data decoded from an audio stream into a translator, and then the speech recognition model in the translator can identify the speech recognition content expressed in the first language in the audio data, and then the identified speech recognition content expressed in the first language can be quickly translated through the text translation model in the translator to obtain a translation result corresponding to the speech recognition content corresponding to the candidate live stream (for example, the above-mentioned translated subtitles expressed in the second language).

[0072] Step S130: Determine the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold.

[0073] The target live stream refers to the live stream to be pushed. In this application, for the sake of distinction, the timestamp representing the end time of the target live stream is called the target end timestamp.

[0074] In some embodiments, the duration of each live stream pushed can be set, assuming it is a first duration. If the end time of the last live stream pushed is the first time, then the live stream with a start time of the first time and an end time of the second time (second time = first time + first duration) can be obtained from the cached live stream (for example, the live stream cached in the stream cache pool described above) as the target live stream to be pushed. It is worth mentioning that in a live broadcast scenario, live streams in the same time period will not be pushed repeatedly. Therefore, the live streams pushed twice are different. The end time of the live stream pushed once can usually be the start time of the live stream pushed next time, thereby ensuring that the live streams pushed twice are different.

[0075] In the present application, since the candidate live stream is a live stream selected by the translation content selector from the code stream cache pool and its end time is after the target timestamp (that is, the end time of the live stream corresponding to the previous stable translation result), that is, the start time of the candidate live stream is the target timestamp, and the end time is after the target timestamp, therefore, the start time of the candidate live stream is not later than the start time of the target live stream to be pushed, that is, the start time of the candidate live stream can be earlier than the start time of the target live stream. If the translation result of the previous pushed stream is a stable translation result, the start time of the current candidate live stream can be equal to the start time of the target live stream.

[0076] Moreover, in this application, the end time of the candidate live stream is no earlier than the end time of the target live stream. That is, the end time of the candidate live stream can be the same as the end time of the target live stream, or it can be later than the end time of the target live stream. In other words, the live broadcast time period corresponding to the target live stream falls within the live broadcast time period corresponding to the candidate live stream, or the live broadcast time period corresponding to the target live stream is the same as the live broadcast time period corresponding to the candidate live stream.

[0077] Among them, the target translation result to be pushed refers to the translation result to be pushed along with the target live stream. In this application, the duration threshold is a positive number. Since the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold, if the end time of the live stream corresponding to the target translation result is not later than (i.e., earlier than) the target end timestamp (i.e., the end time of the target live stream), and the difference between the end time of the live stream corresponding to the target translation result and the target end timestamp is greater than or equal to the duration threshold, then it means that before the target end timestamp, stable content that can be pushed along with the target live stream has been obtained (i.e., the translation result corresponding to the candidate live stream obtained by this translation currently already has a stable translation result obtained by the latest translation).

[0078] In a typical live streaming scenario, once the target live stream of the current stream to be pushed is read, the target live stream of the current stream to be pushed is pushed in real time, generally without any additional waiting. However, in this application, if the end time of the live stream corresponding to the target translation result is later than the target end timestamp, it means that after reading the target live stream of the current stream to be pushed, it is necessary to wait for a period of time to obtain a complete and stable translation result that will not change with the subsequent speech content.

[0079] It is understandable that during the waiting period, if a new live stream is cached, then in this case, the currently obtained live stream (i.e. the above-mentioned target live stream, in this case, the target live stream here is the above-mentioned candidate live stream) and the live stream newly obtained during the waiting period can be combined as a new candidate live stream with a longer duration obtained from the cached live stream, and then the new candidate live stream can be used as the data basis for the translator to translate. In this way, compared with only using the speech recognition content corresponding to the target live stream as the data basis for translation, in this case of the present application, more data basis for translation can be obtained. In the live broadcast scenario, the longer the duration corresponding to the live stream that serves as the data basis for translation, the higher the probability that the live stream includes an audio stream of a complete sentence, and correspondingly, the higher the accuracy of the translation result obtained by translating the speech recognition content of the live stream.

[0080] As an implementation method, semantic analysis can be performed on the translation results corresponding to the candidate live stream to determine the target translation result. Specifically, please refer to Figure 4, which shows a flow chart of determining the target translation result provided by an embodiment of the present application. As shown in Figure 4, step S130 may specifically include:

[0081] Step S131: performing semantic analysis on the translation result corresponding to the candidate live stream to obtain a semantic analysis result corresponding to the translation result.

[0082] The semantic analysis result is used to indicate whether there is a stable translation result in the translation result.

[0083] As an implementation method, a semantic analysis can be performed on the translation results corresponding to the candidate live streams (e.g., subtitles in the translation results) based on natural language processing (NLP) technology. For example, a lexical-level semantic analysis and / or sentence-level semantic analysis can be performed on the sentence content of the subtitles in the translation results to determine whether a stable translation result exists in the translation results based on the contextual semantics of the words, phrases, or short sentences in the sentence content.

[0084] Step S132, obtaining the end time corresponding to the candidate live stream;

[0085] Specifically, if the translation result corresponding to the candidate live stream includes the end time corresponding to the candidate live stream, the end time corresponding to the candidate live stream is obtained from the translation result corresponding to the candidate live stream.

[0086] Step S133: Obtain the target end timestamp of the target live stream to be pushed and the preset duration threshold for the target live stream, and use the sum of the target end timestamp and the duration threshold as the reference time;

[0087] Step S134: compare the end time corresponding to the candidate live stream with the reference time to obtain a comparison result.

[0088] It should be understood that the duration threshold here may be a preset threshold for assuming that stable content can be obtained, and the duration threshold may be denoted as N or Tthr;

[0089] It can be understood that in the embodiment of the present application, the comparison result obtained by comparison can determine whether the end time corresponding to the candidate live stream is less than the reference time, or whether it is equal to the reference time.

[0090] Step S135: Based on the comparison result, determine whether the end time corresponding to the candidate live stream is less than the reference time;

[0091] Step S136: Based on the semantic analysis result corresponding to the translation result, determine whether there is a stable translation result in the translation result.

[0092] Specifically, if the comparison result indicates that the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result indicates that there is a stable translation result in the translation results corresponding to the candidate live stream, step S137 can be further executed: the translation result corresponding to the candidate live stream is used as the target translation result of the stream to be pushed.

[0093] For example, the duration threshold T thr = 5 seconds and the target end timestamp T of the target live stream to be pushed stamp =14:32.15, then the reference time is T ref =T stamp +T thr =14:38.15. If the translation result corresponding to the candidate live stream includes a translation result (e.g., the above-mentioned translated subtitle) of a sentence expressed by an audio with a duration (e.g., the audio duration of the audio data collected within the above-mentioned sampling duration) of 3.23 seconds, and the translation result (e.g., the above-mentioned translated subtitle) of the sentence expressed by the audio is a sentence, then after performing semantic analysis on the translation result (e.g., the above-mentioned translated subtitle) of the sentence expressed by the audio, the translation result (e.g., the above-mentioned translated subtitle) can be determined as a stable translation result, and the end time T corresponding to the candidate live stream isend = stamp + 3.23s = 14:35.38, not exceeding the reference time T ref Therefore, the translation result corresponding to the candidate live stream can be used as the target translation result of the stream to be pushed.

[0094] Optionally, if the end time corresponding to the candidate live stream is less than the reference time, and there is no stable translation result in the translation results corresponding to the candidate live stream, execute step S138: obtain a new candidate live stream from the cached live stream based on the target timestamp; then, return to execute step S120.

[0095] In step S138, the new candidate live stream obtained from the cached live stream based on the target timestamp still has a start time as the target timestamp and an end time equal to the end time of the most recently cached live stream. It is understood that since the new live stream is cached in real time, while the target timestamp remains unchanged, the duration of the newly obtained candidate live stream is longer than that of the previously obtained candidate live stream. The audio stream in the newly obtained candidate live stream is also longer, and contains more content. Therefore, the probability of obtaining a stable translation result when translating based on the audio stream in the longer candidate live stream is higher. It is understood that based on the obtained new candidate live stream, translation processing is then performed according to step S120, followed by step S131 and subsequent steps.

[0096] It is understandable that when the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result of the candidate live stream indicates that there is no stable translation result, the possible reason for the absence of a stable translation result is that the duration corresponding to the candidate live stream is short, and the audio stream in the candidate live stream does not fully present the host's sentence. In this case, if the translation result of the candidate live stream is pushed to the audience end along with the target live stream, the translation result displayed on the audience end may be inaccurate. In the subsequent process, the translation result displayed in the live broadcast screen may be modified many times, thus causing the translation result seen by the audience to be frequently modified, resulting in a poor live viewing experience for the audience and affecting the live broadcast effect. Therefore, in this case, the embodiment of the present application proposes that when there is no stable translation result in the translation result corresponding to the candidate live stream, the translation result of the candidate live stream obtained by this translation needs to be discarded, and a new candidate live stream needs to be obtained from the cached live stream for translation. It should be understood that the duration corresponding to the audio data in the new candidate live stream (i.e., the candidate live stream obtained from the code stream cache pool this time) will be longer than the duration corresponding to the audio data in the old candidate live stream (i.e., the candidate live stream obtained from the code stream cache pool last time). This means that at this time, in the process of translating the audio data in the new candidate live stream (i.e., the candidate live stream obtained from the code stream cache pool this time) through the translator, the audio data in the old candidate live stream (i.e., the candidate live stream obtained from the code stream cache pool last time) contained therein will also be translated again by the translator together with other newly obtained audio data.

[0097] Specifically, when the end time corresponding to a candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result of the candidate live stream indicates that a stable translation result does not exist, considering that the lack of a stable translation result may be due to the short duration of the candidate live stream, since the stream buffer pool is constantly caching newly generated live streams in real time, in this case, a longer candidate live stream can be obtained from the live streams cached in the stream buffer pool. In this way, a new candidate live stream with a longer duration can be provided for translation. Subsequently, after the new candidate live stream is translated by the translator, the probability of obtaining a stable translation result may be higher. It is understandable that when the end time corresponding to a candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result of the candidate live stream indicates that a stable translation result does not exist, a new candidate live stream can be obtained for translation. Since the target translation result has not been determined, the target translation result is actually in a state of waiting to be pushed.

[0098] In some embodiments, a first interval duration between two adjacent acquisitions of candidate live streams can be set. When the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result indicates that there is no stable translation result, if the interval duration between the current time and the last acquisition of the candidate live stream reaches the first interval duration, a new candidate live stream is read from the cache.

[0099] Optionally, in an embodiment of the present application, if based on the comparison result, it is determined that the end time corresponding to the candidate live stream is not less than (for example, greater than or equal to) the reference time, step S139 can be further executed to determine whether the end time corresponding to the candidate live stream is equal to the reference time.

[0100] Specifically, if the determination is yes, that is, if the end time corresponding to the candidate live stream is equal to the reference time, step S137 may be further executed: the translation result corresponding to the candidate live stream is used as the target translation result of the stream to be pushed.

[0101] For example, the duration threshold T thr = 5 seconds and the target end timestamp T stamp =14:32.15, then the reference time is T ref =T stamp +T thr =14:38.15, if the translation result of the candidate live stream is already 5 seconds long, then the end time T of the candidate live stream is end = stamp + 5s = 14:35.15, equal to the reference time T ref , the translation result corresponding to the candidate live stream can be determined as the target translation result of the stream to be pushed. Wherein, stamp refers to the start time of the candidate live stream, that is, the above-mentioned target timestamp.

[0102] Optionally, if the end time of the candidate live stream is equal to the reference time, the target live stream has been delayed for a time period that reaches the time threshold (i.e., T thr =5 seconds). If the target live stream continues to be pushed with a delay, the real-time performance of the live broadcast will be greatly reduced for users. Therefore, when the end time of the candidate live stream is equal to the reference time, regardless of whether there is a stable translation result among the translation results corresponding to the candidate live stream, the translation result corresponding to the candidate live stream needs to be determined as the target translation result to be pushed, so that the target translation result can be pushed along with the target live stream to avoid excessive live broadcast delays in live broadcast scenarios.

[0103] In the above embodiment, a compromise is made between ensuring the real-time performance of the live broadcast and ensuring the accuracy of the translation results. If the translation results of the candidate live stream obtained do not include a stable translation result, and the end time of the candidate live stream is less than the reference time, a new candidate live stream with a longer duration can be obtained for translation, and the target live stream can be delayed and pushed. In the case where the end time of the candidate live stream is equal to the reference time, regardless of whether the translation results of the candidate live stream include a stable translation result, the translation result of the candidate live stream needs to be used as the target translation result of the stream to be pushed, and the target translation result is pushed along with the target live stream. In this way, for the target live stream, the maximum delay in pushing the stream is the duration threshold (for example, the above T thr = 5 seconds). In other words, in order to improve the accuracy of live translation, the embodiment of the present application can appropriately sacrifice the real-time performance of some live broadcasts. In this way, it can not only avoid the situation where the translation result is inaccurate due to the short duration of the audio stream in the live broadcast, which causes frequent modifications to the translation result on the viewer side, but also avoid the situation where the audio stream in the live broadcast is too long and the translation result needs to be adjusted within the preset duration threshold (for example, the above T thr = 5 seconds), directly intercept the translation results of the candidate live streams that are too long by the time threshold (for example, the above T thr =5 seconds) is used as the target translation result for final streaming along with the target live stream, so as to ensure that the target live stream and the target translation result in the live streaming process can ensure the relative timeliness of the live broadcast within a certain delay time.

[0104] Furthermore, the method further includes: if the semantic analysis result corresponding to the translation result of the candidate live stream indicates that there is a stable translation result in the translation result, step S141 can be further performed: storing the end time of the candidate live stream as the new target timestamp. In this case, regardless of whether the end time corresponding to the candidate live stream is less than the reference time or equal to the reference time, the end time of the candidate live stream can be stored as the new target timestamp.

[0105] In the above embodiment, if the translation result of the candidate live stream includes a stable translation result, the end time of the candidate live stream can be used as the new target timestamp, and the candidate live stream can be retrieved from the above-mentioned code stream cache pool again according to the new target timestamp next time. In this way, it can be ensured that before a stable translation result appears, the audio stream in the live stream from the target timestamp to the start time of the target live stream is used for auxiliary translation, so as to provide more data for translation, thereby ensuring the accuracy of the live translation. Optionally, it is understandable that, in an embodiment of the present application, when the translator believes that the translation result of the candidate live stream includes a stable translation result, the controller may further determine that the translated text content in the translation result of the candidate live stream translated by the translator is generally accurate. In this way, the controller may use the translation result of the candidate live stream as the above-mentioned stable translation result in the subsequent processing process, and further use the end time of the candidate live stream as the new target timestamp to obtain a new candidate live stream for translation according to the new target timestamp (i.e., the end time of the candidate live stream). It should be understood that, in the process of obtaining a new candidate live stream for translation from the bitstream cache pool according to the new target time, the computer device (e.g., the live broadcast server) involved in the embodiment of the present application may select a live stream with a start time of the new target timestamp and an end time after the new target timestamp from the cached live streams through the above-mentioned translation content selector as the new candidate live stream. In other words, the embodiment of the present application may skip the new target timestamp and select the live stream after the new target timestamp for translation. In this way, the translation workload can be reduced and the translation speed can be improved.

[0106] As an implementation method, after determining the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed, the end time of the candidate live stream can be used as the new target timestamp.

[0107] When semantic analysis is performed on the translation results corresponding to the candidate live streams, the stable translation results in the translation results can be used as the target translation results for streaming. Moreover, when the end time corresponding to the candidate live stream is equal to the reference time, the translation results corresponding to the candidate live stream can be directly used as the target translation results for streaming. In other words, the embodiment of the present application can make it possible to achieve the goal of achieving ... thr=5 seconds) within the corresponding time period (i.e., the time period from the target timestamp to the reference time), the target translation result can be obtained for streaming. For example, assuming that the duration threshold is 5 seconds, and the candidate live stream is 10 seconds long and includes a sentence, when the candidate live stream is translated to the 5th second, the complete semantics is still unknown. That is, in the time period corresponding to the duration threshold of 5 seconds, although a stable translation result cannot be obtained, the translated first 5 seconds still has certain semantics. For this reason, the translation result of the first 5 seconds can be used as the target translation result for streaming, and then the translation of the remaining live stream is continued to avoid the audience waiting for a long time and then suddenly watching a large translation result. And because the stable translation result is a translation result in a steady state, it can ensure that the translation result of the live translation has a high accuracy. In this way, it can effectively avoid the long translation waiting time caused by translating the complete sentence for a long time. In this way, while ensuring the accuracy of the live translation, it can also improve the relative real-time performance of the live translation to a certain extent.

[0108] In the above embodiment, when the end time corresponding to the candidate live stream is equal to the reference time and the translation results corresponding to the candidate live stream do not include a stable translation result, the translation result corresponding to the candidate live stream can still be determined as the target translation result of the stream to be pushed. In this case, the target timestamp is not updated. In this way, the next time the candidate live stream is obtained, it is still based on the target timestamp based on the last time the candidate live stream was obtained. In this way, compared with the candidate live stream obtained last time, the newly obtained candidate live stream has a longer duration, and the start time of both is the target timestamp. In this way, if the translation result corresponding to the pushed candidate live stream is inaccurate, a relatively accurate translation result can be obtained by translating based on the newly obtained candidate live stream with a longer duration. In this way, the more accurate translation result can be used to overwrite the inaccurate translation result of the previous push. Therefore, the new more accurate translation result can be used to replace the previously inaccurate translation result. For example, assuming that the duration threshold is 5 seconds, if there is no stable translation result in the translation results of the candidate live streams translated within the translation duration corresponding to these 5 seconds (e.g., the reference time), but since the end time of the candidate live stream is currently equal to the reference time calculated when the duration threshold is 5 seconds, it is necessary to use the translation result of the candidate live stream currently translated as the target translation result for phased streaming. In this way, when a new candidate live stream with a longer duration (e.g., 10 seconds) is obtained from the live streams cached in the above-mentioned code stream cache pool based on the same target timestamp, the new candidate live stream currently obtained can be translated, and the old translation result obtained by the previous translation can be replaced by the translation result of the new candidate live stream with a longer duration (e.g., 10 seconds), so that the new, more accurate translation result can be transmitted to the audience.

[0109] Alternatively, when the end time corresponding to the candidate live stream is equal to the reference time, and when the translation result corresponding to the candidate live stream does not include a stable translation result, the translation result corresponding to the candidate live stream can still be determined as the target translation result of the stream to be pushed. In other words, the embodiment of the present application can also, when the end time corresponding to the candidate live stream is equal to the reference time, first stage-by-stage transmit the old translation result (i.e., the translation result corresponding to the currently obtained candidate live stream) to the audience, so as to be displayed on the audience's screen in real time, and subsequently, a new candidate live stream with a longer duration can be obtained from the live stream of the cache based on the target timestamp for translation, and then the old translation result can be revised according to the new translation result obtained, such as, the text (i.e., the old translated subtitles) that has been displayed can be modified by the text in the new translation result. In addition, the embodiment of the present application can also, when a new translation result is obtained and a stable translation result is present in the new translation result, when the new translation result is pushed to the audience end for display, the voice content of the host can also be displayed together with the new translation result through the identified speech recognition content.

[0110] Optionally, in some embodiments, the speech recognition content corresponding to the candidate live stream and the translation result corresponding to the candidate live stream can be combined to determine whether there is a stable translation result in the translation result corresponding to the candidate live stream. Specifically, the speech recognition content corresponding to the candidate live stream can be semantically analyzed to obtain a first semantic analysis result, which is used to indicate whether the speech recognition content corresponding to the candidate live stream is a complete sentence; if the speech recognition content corresponding to the candidate live stream is a complete sentence, it can be determined that the probability of the translation result corresponding to the candidate live stream being a stable translation result is high; conversely, if the speech recognition content corresponding to the candidate live stream is not a complete sentence, then the probability of the translation result corresponding to the candidate live stream being a stable translation result is low. Based on this principle, when the first semantic analysis result indicates that the speech recognition content corresponding to the candidate live stream is a complete sentence, the translation result corresponding to the candidate live stream can be further semantically analyzed to determine whether there is a stable translation result in the translation result; conversely, if the first semantic analysis result indicates that the speech recognition content corresponding to the candidate live stream is not a complete sentence, step 138 can be executed, that is, a new candidate live stream can be obtained from the cached live stream for translation based on the target timestamp.

[0111] It should be understood that in the live broadcast scenario, the accuracy and real-time performance of live translation are mutually exclusive. For example, the longer the live stream used for translation, the higher the accuracy of the translation. Correspondingly, the longer the waiting time for obtaining the live stream to be translated, the lower the real-time performance of the live translation. In this application, by setting a duration threshold, that is, the duration threshold is the maximum delayed push duration of the target live stream, in other words, the duration threshold can be used to characterize the maximum delayed push duration of the target live stream in the process of obtaining the target translation result. This means that within the maximum delayed push duration, the timeliness and reliability of the target translation content obtained in the live translation process can still be ensured. Therefore, the size of the duration threshold can reflect the level of live real-time performance.

[0112] The duration threshold can be set based on the real-time requirements of the live broadcast. It's understandable that for live broadcast scenarios where the host and audience interact, the real-time requirements are higher. In this case, a smaller duration threshold can be set to avoid excessive delays in the target live stream. On the other hand, for live broadcast scenarios with little or no interaction between the host and audience, the real-time requirements are relatively low. A larger duration threshold can be set to fully ensure the accuracy of the target translation results pushed to the audience.

[0113] In some embodiments, scenario application information may be obtained, indicating a target live broadcast interaction level. The target live broadcast interaction level is the live broadcast interaction level of the current live broadcast scenario (or the current target live broadcast room). The target live broadcast interaction level may be pre-set by a user (e.g., a user who is a live streamer). Subsequently, based on the correspondence between the live broadcast interaction level and the duration, the duration corresponding to the target live broadcast interaction level is determined as the duration threshold.

[0114] Among them, the target live broadcast interaction level is a quantitative expression of the real-time requirements of the current live broadcast scene. Scenario application information refers to information that can reflect the real-time requirements of the current live broadcast scene. The correspondence between the live broadcast interaction level and the duration can be pre-stored in a relational database. Optionally, the scenario application information can be pre-specified by the user at the terminal. For example, the anchor terminal can send the required size value for the real-time performance of the live broadcast translation directly set by the anchor to the server terminal as scenario application information. The scenario application information can also be information obtained by the server terminal from the anchor terminal to represent the amount of audience interaction, such as the barrage sending rate, and then the duration threshold is determined based on the barrage sending rate. The higher the sending rate, the more interactions there are, the higher the real-time requirements for the live broadcast translation, and the smaller the duration threshold can be.

[0115] In other embodiments, the duration of the target audience's interaction data can be predicted by a duration prediction model to obtain a corresponding duration threshold. The duration prediction model can be obtained by supervised learning of a preset classification network based on sample interaction data and label duration threshold. Among them, interaction data refers to data on the audience's participation in live broadcast interaction when watching the live broadcast, and may include the average barrage sending rate or average like rate of the audience when participating in different live broadcasts, etc., which are not limited here. Optionally, for different sample interaction data in the training set, the corresponding label duration threshold (Ground Truth) can be pre-calculated based on experiments. The preset classification network can be a deep neural network, for example, a combination of an embedding layer (Embedding Layer) and a multilayer perceptron (MLP).

[0116] Specifically, a duration prediction model is used to predict the duration of sample interaction data, obtaining a predicted duration threshold. A target loss is then calculated based on the predicted duration threshold and the label duration threshold corresponding to the sample interaction data. This target loss is then used to iteratively train a preset classification network for supervised learning until the preset classification network meets the training termination criteria, thereby obtaining a duration prediction model. In this way, the duration prediction model can be used to perform personalized duration predictions for the target audience, obtaining a time threshold that matches the target audience's tolerance for delay. This ensures that the target translation results pushed to the audience are more accurate, and the real-time display of the target translation results better meets the audience's personalized needs.

[0117] In other embodiments, the setting component of duration threshold value can be displayed at the audience end, and then the audience can, when watching live broadcast, according to the actual experience demand of the live broadcast itself, set the time threshold value of different durations.For example, the setting button of duration threshold value is displayed on the live broadcast interface of the audience end, when the audience clicks on the setting button, the slide bar for adjusting the duration threshold value size is displayed, and the audience can adjust the duration threshold value required when the audience broadcasts by sliding the slide bar.Like this, by allowing the audience to directly set the duration threshold value themselves, the real-time of the target translation result display can more meet the current viewing demand of the audience, making the target translation result of being pushed to the audience end more accurate relative to the target audience, while also improving the efficiency of adjusting the real-time and accuracy of live translation.

[0118] Step S140: Based on the target translation result, the target live stream to be pushed is re-encoded to obtain a re-encoded live stream.

[0119] It should be understood that when the embodiment of the present application translates the candidate live stream, since the candidate live stream has been decoded in advance, the corresponding audio and video data and other decoding information can be quickly obtained. Therefore, after the target translation result obtained by translation is added to the audio and video data, the audio and video data with the translation result is obtained. To this end, the audio and video data with the translation result can be re-encoded to obtain a new audio and video stream (i.e., a new audio and video code stream, or the above-mentioned re-encoded live stream) for streaming.

[0120] As an implementation method, the target translation result can be added to the auxiliary enhancement frame as custom data of the frame corresponding to the additional enhancement information (i.e., the auxiliary enhancement frame). Then, the auxiliary enhancement frame can be inserted into the video stream of the target live stream to be pushed before the I frame or P frame, and then encoded together with the target live stream into a new audio and video stream, that is, a re-encoded live stream.

[0121] Step S150: Push the re-encoded live stream to display the target translation result on the viewer side.

[0122] It can be understood that since the re-encoded live stream includes the video stream and the audio stream in the target live stream, as well as the target translation result, after the re-encoded live stream is pushed to the audience end, the audience end can display the screen based on the video stream in the target live stream, and display the translated text content in the target translation result as subtitles (for example, the above-mentioned translated subtitles) on the live screen, and play the corresponding audio stream synchronously while playing the video stream.

[0123] In some embodiments, the target translation result may also include the speech recognition content of the corresponding live stream, so that both the speech recognition content and the translation result may be displayed as subtitles on the live screen at the viewer end.

[0124] In some embodiments, because different target live streams may be delayed for different lengths of time, if the re-encoded live stream is directly pushed after re-encoding, the push may be discontinuous, that is, there may be periods of time when there is no re-encoded live stream available for push. For example, taking a candidate live stream with a duration of 30 seconds, if a stable translation result is obtained after translating the first 3 seconds and inserting it into the audio and video stream, it is assumed that another stable translation result can only be obtained after waiting until the end of the candidate live stream.

[0125] If the live stream sent to the viewer is only the re-encoded live stream corresponding to the stable translation result, then for a period of time after the re-encoded live stream of the stable translation result of the first 3 seconds is sent, there will be no live stream of the translation result to be sent. This will cause the viewer to experience screen freezes and affect the viewer's viewing experience. To this end, in some embodiments of the application, the re-encoded live stream can be cached before being pushed.

[0126] As an implementation method, if the time difference between the current time and the end time corresponding to the re-encoded live stream reaches a duration threshold, the target live stream is pushed. Specifically, the target live stream can be stored in a re-encoding cache pool. If the time difference between the current time and the end time corresponding to the re-encoded live stream reaches a duration threshold, the audio and video stream stored in the re-encoding cache pool can be pushed to the viewer end in real time. The end time corresponding to the re-encoded live stream is the end time of the target live stream corresponding to the re-encoded live stream. In this embodiment, it can be ensured that the stream is pushed while the re-encoded live stream is cached for a certain length of time, thereby ensuring the continuity and stability of the push stream.

[0127] For example, taking a candidate live stream with a duration of 30 seconds as an example, when the first 3 seconds of the stable translation result are re-encoded into the live stream, the re-encoded live stream is stored in the re-encoding cache pool, and the re-encoded live stream is continued to be accumulated in the re-encoding cache pool until the time difference between the current time and the end time corresponding to the re-encoded live stream reaches the duration threshold, that is, the re-encoding cache pool has accumulated live streams of at least the duration corresponding to the duration threshold (that is, the entire encoding duration, for example, 30 seconds here), and then the live stream stored in the re-encoding cache pool is pushed to the audience end in real time.

[0128] In this way, by setting a specific duration (i.e., reaching the duration threshold) at which a phased translation result must be given, and then by setting a certain duration (i.e., reaching the duration threshold) of delayed sending, it is ensured that the re-encoding cache pool always has new re-encoded live streams replenished in real time, thereby achieving continuous streaming and avoiding the situation where the live broadcast screen on the audience side is stuck due to discontinuous streaming.

[0129] Exemplarily, please refer to Figure 5, which shows a flow chart of a translation process provided by an embodiment of the present application. As shown in Figure 5, a code stream buffer pool, a translation content selector, a translator, a controller, an uploader, and a recoding buffer pool may be included in the live translation process. Among them, the code stream buffer pool can be used to cache the live stream pushed by the anchor end. For example, if the code stream buffer pool is deployed on the live service end, then, after the anchor end pushes the live stream obtained by real-time encoding to the background (i.e., the live service end), the live service end can add the received live stream to the code stream buffer pool so that the live stream continuously pushed by the anchor end can be cached through the code stream buffer pool. Among them, the translation content selector can be used to quickly select the live stream that currently needs to be translated as the candidate live stream to be translated from the live stream cached by the code stream buffer pool by the target timestamp. Further, as shown in Figure 5, the translator can be used to translate the candidate live stream. Then, the controller can be used to determine the target translation result of the stream to be pushed. Finally, the re-encoding buffer pool can cache the re-encoded live stream waiting to be pushed to the viewer end, where the re-encoded live stream is the live stream obtained by re-encoding and carrying the target translation result.

[0130] Specifically, as an implementation method, the host end can collect audio and video data in real time when the host initiates a live broadcast, and obtain the audio and video code stream corresponding to the audio and video data after audio and video encoding (i.e., the above-mentioned audio encoding and video encoding), and then push the encoded audio and video code stream (i.e., the above-mentioned live stream) to the live broadcast server end, and then the live broadcast server end can cache the received encoded live stream into the code stream cache pool. Furthermore, the live broadcast server end can use the translation content selector to obtain a live stream (i.e., a live stream with a start time of the target timestamp and an end time after the target timestamp) from the code stream cache pool as a candidate live stream to be translated with the target timestamp (i.e., the end time of the live stream corresponding to the last stable translation result) as the start time. Furthermore, the live broadcast server end can use the translator to translate the speech recognition content corresponding to the candidate live stream to output the translation result corresponding to the candidate live stream.

[0131] Furthermore, the live broadcast service end can perform semantic analysis on the translation result corresponding to the candidate live stream through the controller, and then determine whether there is a stable translation result in the translation result corresponding to the candidate live stream or whether the end time corresponding to the candidate live stream is equal to the reference time based on the semantic analysis result. If the end time corresponding to the candidate live stream does not exceed (for example, less than) the reference time, and the semantic analysis result corresponding to the translation result indicates that there is a stable translation result, the translation result corresponding to the candidate live stream is used as the target translation result of the stream to be pushed. Alternatively, if the end time corresponding to the candidate live stream is equal to the reference time, regardless of whether there is a stable translation result in the translation result corresponding to the candidate live stream, the translation result corresponding to the candidate live stream needs to be determined as the target translation result of the stream to be pushed. Optionally, if the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result indicates that there is no stable translation result, the controller needs to notify the translation content selector to continue to obtain a new candidate live stream from the cached live stream based on the target timestamp (that is, the next live stream to be translated obtained from the code stream cache pool, such as the next candidate live stream) to continue executing the step of translating the new candidate live stream. It should be understood that the specific process of translating the new candidate live stream and obtaining a new translation result in the embodiment of the present application can be described in the description of the specific process of translating the above-mentioned candidate live stream, and will not be further described here.

[0132] Furthermore, the live broadcast service end can re-encode the target live broadcast stream to be pushed based on the target translation result to obtain a re-encoded live broadcast stream. Specifically, as shown in Figure 5, when the live broadcast service end determines the target translation result of the stream to be pushed through the controller, it can generate a corresponding auxiliary enhancement frame based on the target translation result, and insert the auxiliary enhancement frame. For example, the auxiliary enhancement frame can be inserted into the video data of the video stream of the target live broadcast stream to be pushed (for example, during the re-encoding process, the auxiliary enhancement frame can be first inserted before the I frame and / or P frame in the video stream), and then re-mixed and encoded together with the audio data in the audio stream of the target live broadcast stream (i.e., the above-mentioned audio and video encoding) to obtain a re-encoded live broadcast stream (i.e., the above-mentioned live broadcast stream carrying the target translation result can be re-encoded). Further, the live broadcast service end can cache the re-encoded live broadcast stream in the re-encoding cache pool.

[0133] Among them, it can be understood that in an embodiment of the present application, the live broadcast service end can generate an auxiliary enhancement frame based on the target translation result, that is, a custom data field in which the target translation result is added to the supplemental enhancement information (SEI) and encapsulated into a specific type of network abstract layer unit (NALU). In addition, it can be understood that the re-encoding involved in the embodiment of the present application refers to decoding the original code stream (i.e., the live stream to be pushed) to decode the video data and audio data in the live stream, and then encode it according to the new encoding rules and encoding parameters to generate a new code stream, and then the new code stream can be called a re-encoded live stream. Among them, the new encoding rules here mainly refer to monitoring the encoded I frames and / or P frames appearing in the decoded video data during re-encoding, and inserting the SEI frame (i.e., the above-mentioned auxiliary enhancement frame) constructed by the above-mentioned SEI content before the I frame / P frame. Among them, the above-mentioned SEI content refers to the content generated when the above-mentioned target translation result is added to the custom data field in the SEI data format.

[0134] Furthermore, the live broadcast server can upload the re-encoded live stream in the re-encoding cache pool to the content distribution network through the uploader when it determines that the duration of the re-encoded live stream in the re-encoding cache pool is not less than the duration threshold. Then, the audience can pull the stream from the content distribution network to obtain the re-encoded live stream, so that the re-encoded live stream can be further decoded and played later.

[0135] In this embodiment, a candidate live stream to be translated can be obtained from a cached live stream, where the candidate live stream refers to a live stream whose start time is the target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation result. It should be understood that in the embodiment of the present application, the live stream corresponding to the last stable translation result refers to the translated live stream, and the translated live stream is the previous live stream of the candidate live stream. The last stable translation result refers to the stable translation result obtained after the translated live stream is translated. Furthermore, the embodiment of the present application can translate the speech recognition content corresponding to the candidate live stream, obtain the translation result corresponding to the candidate live stream, and determine the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed. Among them, the target translation result of the stream to be pushed refers to the translation result to be pushed along with the target live stream, and the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold. And then, this embodiment of the present application can be based on target translation result, the target live streaming of the target to be pushed stream is recoded, obtain recoded live streaming, and push stream to the recoded live streaming, to display target translation result at audience end (such as, specifically can display the translation text content in target translation result, i.e. above-mentioned translation subtitles).Thus, the embodiment of the present application can be under the condition that the end time of the live streaming corresponding to target translation result relative to the time difference between the target end timestamp does not exceed the duration threshold, can be based on the translation result corresponding to candidate live streaming in fixed duration (i.e., no more than the reference duration corresponding to duration threshold), determine that the target translation result of the target to be pushed stream is pushed first, avoid the translation waiting time caused by translating a complete sentence for a long time too long, improve the real-time of live translation.Enable translation result and speaker's video screen to be displayed synchronously.

[0136] Please refer to Figure 6, which shows a flow chart of a live broadcast translation method provided by another embodiment of the present application. In this embodiment, the live broadcast translation method can be executed by the live broadcast service end corresponding to the live broadcast service provider. The live broadcast translation method shown in Figure 6 can be applied to the live broadcast translation process shown in Figure 7. Further, please refer to Figure 7, which shows a flow chart of a live broadcast translation provided by an embodiment of the present application. Specifically, the live broadcast translation process shown in Figure 7 includes the host end pushing the stream to the backend (i.e., the above-mentioned live broadcast service end), and then the backend (i.e., the above-mentioned live broadcast service end) translates the currently obtained candidate live stream according to the live broadcast translation method involved in Figure 6, and the live broadcast service end uploads the translated code stream with translated subtitles (i.e., the above-mentioned re-encoded live stream) to the CDN. Then, the audience end can go to the CDN to pull the stream to the re-encoded live stream, and after decoding the re-encoded live stream, obtain the audio and video (i.e., the above-mentioned audio and video data) and the translated subtitles (i.e., the above-mentioned translated text content), and then can display the translated subtitles on the screen while playing the audio and video (i.e., the audio and video data). At the same time, the audience end can also adjust the translated subtitles. Specifically, as shown in FIG6 , the live broadcast translation method may include the following steps S210 to S260:

[0137] Step S210: Obtain a candidate live stream to be translated from the cached live stream; the candidate live stream refers to a live stream whose start time is the target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation result, the live stream corresponding to the last stable translation result refers to the translated live stream, the translated live stream is the previous live stream of the candidate live stream, and the last stable translation result refers to the stable translation result obtained after translating the translated live stream.

[0138] Step S220: Translate the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream.

[0139] Step S230: Determine the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the target translation result of the stream to be pushed refers to the translation result to be pushed along with the target live stream, and the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a duration threshold, the duration threshold is a positive number, and the duration threshold is used to represent the maximum delayed push duration of the target live stream.

[0140] Specifically, the specific implementation of steps S210 to S230 can refer to the description of steps S110 to S130 in the aforementioned embodiment, and will not be repeated here.

[0141] Step S240: Generate an auxiliary enhanced frame based on the target translation result.

[0142] Among them, the auxiliary enhancement frame refers to the additional enhancement information (SEI) frame. The additional enhancement information frame is a type of network abstraction layer data unit (NALU) in the audio and video coding stream that encapsulates auxiliary information into a specific type. It is also a data unit in video coding that meets network transmission. For example, in the H.264 code stream, images are organized in sequence units. A sequence is composed of multiple frames of images (i.e., the multiple video frames collected above). The embodiment of the present application can encode the corresponding sequence as a single sequence to obtain a coded data stream for pushing the stream (for example, in a live broadcast scenario, a live stream can be obtained). During the encoding process, the live stream corresponding to a sequence starts with an I frame and ends with the next I frame. Among them, a frame of image (i.e., a video frame) can be divided into one or more slices (Slice), and a slice is composed of a macroblock (Macroblock). Macroblock is the basic unit of encoding processing. After the slice is encoded, it will be packaged into a NALU, that is, one frame of image corresponds to one NALU.

[0143] Please refer to Figure 8, which shows a schematic diagram of the code stream format of a network abstraction layer data unit provided in an embodiment of the present application. As shown in Figure 8, the network abstraction layer data unit includes a start code (Start Code), a unit header (NALU Header) and a unit body (NALU Payload). Among them, the Start Code is used to separate two adjacent network abstraction layer data units (i.e., two adjacent NALUs), and the start code (Start Code) can be "0X 00 00 00 01" in hexadecimal or "00 00 01" in binary.

[0144] The NALU Header is used to indicate the type of data and other information contained in it (ie, the network abstraction layer data unit).

[0145] For example, if the NALU Header is 0X06, the NALU Payload can be filled in according to the data format requirements of SEI encoding. Among them, the NALU Header occupies 1 byte, and the detection of the sequence parameter set (SPS), picture parameter set (PPS), and I / P / B frame in H.264 is realized by the NALU type in the NALU Header, wherein the NALU type may include the data type (for example, SEI data) with the data being 0X06SEI as shown in Figure 8, the data type (for example, I frame) with the data being 0X25 / 65, and the data type (for example, P frame) with the data being 0X21 / 61.

[0146] As an implementation, a custom data field corresponding to the additional enhancement information can be determined based on the data encoding format (i.e., the abstraction layer data unit), and the target translation result can be added to the custom data field to obtain an auxiliary enhancement frame. For example, refer to Figure 9, which shows a schematic diagram of the data format of additional enhancement information provided in an embodiment of the present application.

[0147] As shown in Figure 8, the header data (NRI) of the additional enhancement information is 0X06, the main type Payload type is 0X05, which indicates that it complies with the H264 standard format, and UUID is a business custom identification code. The custom data length is calculated by the custom data, and the custom data is a type format specified by the business party itself, and the end is aligned with 0X80 (i.e., the end alignment code is 0X80). Based on this, the embodiment of the present application can add the target translation result to the field of the custom data in the SEI data format to obtain an auxiliary enhancement frame by encoding. And then, after pulling the stream, the audience end can be decoded according to the data format of the SEI, so as to obtain the translated subtitles in the target translation result for display from the field corresponding to the custom data obtained by decoding (i.e., the custom data field).

[0148] As another implementation, the present application embodiment can obtain the speech recognition content of the live stream corresponding to the target translation result, and then add the speech recognition content and the target translation result together to the field corresponding to the custom data (i.e., the custom data field) to obtain an auxiliary enhancement frame. In this way, after pulling the stream, the viewer can decode it according to the SEI data format to obtain the translation result and the speech recognition content corresponding to the live stream for bilingual subtitle display.

[0149] Step S250: Encode the auxiliary enhanced frame into the target live stream to be pushed to obtain a re-encoded live stream.

[0150] As an implementation method, the embodiment of the present application can decode the target live stream to be pushed, obtain the audio data of the audio stream and the video data of the video stream in the target live stream to be pushed, and insert the auxiliary enhancement frame before the specified video frame (for example, the above-mentioned I frame and / or P frame) in the video data of the video stream to obtain reference video data. Furthermore, the embodiment of the present application can re-encode the audio data of the audio stream and the reference video data to obtain the above-mentioned re-encoded live stream.

[0151] The designated video frame is a key frame (I frame) or a forward predictive coding frame (P frame) in the video data of the video stream. Taking into account the requirements of the coding protocol, the auxiliary enhancement frame can be inserted at the beginning of the I frame / P frame during encoding without affecting the decoding order of the re-encoded live stream.

[0152] Step S260: Push the re-encoded live stream to display the target translation result on the viewer side.

[0153] As an implementation method, when the live broadcast service end pushes the re-encoded live stream, if it determines that there is a translation result that needs to be adjusted, it can send a translation adjustment instruction to the viewer end. The translation adjustment instruction is used to control the viewer end to adjust the previously sent re-encoded live stream based on the newly sent re-encoded live stream. For example, for a candidate live stream, the i-th translation result in the previous 10 seconds is I{X1X3}, and the i+1-th translation result is II{X1X2X3X4X5}. To this end, it is necessary to generate a translation adjustment instruction and send it to the viewer end, so that the viewer end replaces X3 in the second position of the translation result I{X1X3} with X2X3, and fills X4X5 after X3.

[0154] In this way, the translated results can be adjusted again, which improves the real-time performance of the live translation while also improving the accuracy of the translation results. Specifically, the specific implementation of step S260 can refer to the description of step S150 in the above embodiment, and will not be repeated here.

[0155] In this embodiment, a candidate live stream to be translated can be obtained from a cached live stream, wherein the candidate live stream refers to a live stream whose start time is the target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation result. In addition, the embodiment of the present application can translate the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream, and then determine the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; wherein the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold.

[0156] Furthermore, the embodiment of the present application can generate an auxiliary enhancement frame based on the target translation result. For example, it can specifically include determining the custom data field (i.e., the field of the above-mentioned custom data) corresponding to the additional enhancement information based on the data encoding format, and adding the target translation result to the custom data field to obtain an auxiliary enhancement frame. The auxiliary enhancement frame is then encoded into the target live stream to be pushed, and a re-encoded live stream is obtained, and the re-encoded live stream is pushed to display the translated subtitles in the target translation result at the audience end. In other words, the embodiment of the present application can determine the target translation result of the stream to be pushed first within a fixed time (e.g., the above-mentioned reference time) based on the translation result corresponding to the candidate live stream, under the condition that the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold, so as to improve the real-time performance of the live translation. At the same time, the embodiment of the present application can also insert the auxiliary enhancement frame in the leading position of the key frame or the forward prediction coding frame during encoding, so as to avoid affecting the order in which the audience end decodes the audio and video streams, so as to ensure that the translation result and the video screen can be displayed synchronously.

[0157] Please refer to Figure 10, which shows a module block diagram of a live broadcast translation device provided by an embodiment of the present application. The live broadcast translation device 300 includes: a live broadcast stream acquisition module 310, which is used to obtain a candidate live broadcast stream to be translated from a cached live broadcast stream; the candidate live broadcast stream refers to a live broadcast stream whose start time is a target timestamp and whose end time is after the target timestamp, and the target timestamp refers to the end time of the live broadcast stream corresponding to the last stable translation result, and the live broadcast stream corresponding to the last stable translation result refers to the translated live broadcast stream, and the translated live broadcast stream is the last live broadcast stream of the candidate live broadcast stream, and the last stable translation result refers to the stable translation result obtained after the translated live broadcast stream is translated; the translation processing module 320 is used to translate the speech recognition content corresponding to the candidate live broadcast stream to obtain the translation result corresponding to the candidate live broadcast stream; The result determination module 330 is used to determine the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the target translation result of the stream to be pushed refers to the translation result to be pushed along with the target live stream, and the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a preset duration threshold, the duration threshold is a positive number, and the duration threshold is used to represent the maximum delayed pushing duration of the target live stream; the re-encoding module 340 is used to re-encode the target live stream to be pushed based on the target translation result to obtain a re-encoded live stream; the streaming module 350 is used to stream the re-encoded live stream to display the target translation result on the audience end.

[0158] In some embodiments, the result determination module 330 may include: a semantic analysis unit and a first determination unit. The semantic analysis unit is configured to perform semantic analysis on the translation result corresponding to the candidate live stream to obtain a semantic analysis result corresponding to the translation result; the semantic analysis result is used to indicate whether there is a stable translation result in the translation result; the first determination unit is configured to obtain the end time corresponding to the candidate live stream, obtain the target end timestamp of the target live stream to be pushed, and a preset duration threshold for the target live stream, use the sum of the target end timestamp and the duration threshold as a reference time, and compare the end time corresponding to the candidate live stream with the reference time to obtain a comparison result; the comparison result is used to determine whether the end time corresponding to the candidate live stream is less than the reference time. In this way, the first determination unit is further configured to use the translation result corresponding to the candidate live stream as the target translation result for the stream to be pushed if the comparison result indicates that the end time corresponding to the candidate live stream is less than the reference time and the semantic analysis result corresponding to the translation result indicates that there is a stable translation result in the translation result.

[0159] In some embodiments, the result determination module 330 may further include a second determination unit configured to determine the translation result corresponding to the candidate live stream as the target translation result of the stream to be pushed if the end time corresponding to the candidate live stream is equal to the reference time.

[0160] In some embodiments, the result determination module 330 may further include a return execution unit. The return execution unit is configured to, if the end time corresponding to the candidate live stream is less than the reference time and the semantic analysis result corresponding to the translation result indicates that there is no stable translation result, obtain a new candidate live stream from the cached live stream based on the target timestamp, and return to the step of translating the speech recognition content corresponding to the candidate live stream to obtain the translation result corresponding to the candidate live stream.

[0161] In some embodiments, the result determination module 330 may further include a timestamp determination unit configured to store the end time of the candidate live stream as a new target timestamp.

[0162] In some embodiments, the re-encoding module 340 may include: an enhancement frame generation unit and a re-encoding unit. The enhancement frame generation unit is configured to generate an auxiliary enhancement frame based on the target translation result; and the re-encoding unit is configured to encode the auxiliary enhancement frame into the target live stream to be pushed, thereby obtaining a re-encoded live stream.

[0163] In some embodiments, the enhanced frame generation unit may include: a determination subunit configured to determine, based on a data encoding format, a custom data field corresponding to the additional enhancement information; and an addition subunit configured to add the target translation result to the custom data field to obtain an auxiliary enhanced frame.

[0164] In some embodiments, the adding subunit may be specifically configured to: obtain speech recognition content of a live stream corresponding to a target translation result; and add the speech recognition content and the target translation result to a custom data field to obtain an auxiliary enhanced frame.

[0165] In some embodiments, the re-encoding unit can be specifically used to: decode the target live stream to be pushed, obtain the audio data of the audio stream and the video data of the video stream in the target live stream to be pushed; insert the auxiliary enhancement frame before the specified video frame in the video data of the video stream to obtain the reference video data; the specified video frame refers to the key frame and / or forward prediction coding frame in the video data of the video stream; re-encode the audio data and reference video data of the audio stream to obtain the re-encoded live stream.

[0166] In some embodiments, the streaming module 350 may be specifically configured to: if the time difference between the current time and the end time corresponding to the re-encoded live stream reaches a duration threshold, stream the target live stream.

[0167] In some embodiments, the translation processing module 320 can be specifically used for: performing translation processing on the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the speech recognition content corresponding to the candidate live stream; combining the translation result corresponding to the speech recognition content corresponding to the candidate live stream, the end time corresponding to the candidate live stream, and the target timestamp to obtain a translation result corresponding to the candidate live stream.

[0168] In some embodiments, the live broadcast translation device 300 may further include an application information acquisition module and a duration threshold determination module. The application information acquisition module is configured to acquire scenario application information indicating a target live broadcast interaction level; and the duration threshold determination module is configured to determine, based on the correspondence between the live broadcast interaction level and duration, the duration corresponding to the target live broadcast interaction level as the duration threshold.

[0169] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] In several embodiments provided in this application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0171] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0172] The solution provided by this application can obtain a candidate live stream from a cached live stream, wherein the candidate live stream refers to a live stream whose start time is the target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation result. The speech recognition content corresponding to the candidate live stream is then translated to obtain the translation result corresponding to the candidate live stream, and based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed, the target translation result of the stream to be pushed is determined, wherein the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold.

[0173] Based on the target translation result, the target live stream to be pushed is then re-encoded to obtain a re-encoded live stream, which is then pushed to display the translated subtitles in the target translation result on the viewer's end. In this way, by limiting the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp to no more than a duration threshold, the target translation result to be pushed can be determined based on the translation results corresponding to the candidate live streams. This allows the target translation result to be obtained and pushed within the time period corresponding to the duration threshold, improving the real-time performance of live translation and ensuring that the translation result is displayed synchronously with the video screen of the speaker.

[0174] Further, please refer to Figure 11, which is a module block diagram of a computer device provided in an embodiment of the present application. As shown in Figure 11, the computer device 400 includes a processor 410, a memory 420, a power supply 430 and an input unit 440. The memory 420 stores computer program instructions. When the computer program instructions are called by the processor 410, the various method steps provided in the above-mentioned embodiments can be implemented. It can be understood by those skilled in the art that the structure of the computer device shown in the figure does not constitute a limitation on the computer device, and it can include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0175] The processor 410 may include one or more processing cores. The processor 410 utilizes various interfaces and circuits to connect various components within the battery management system. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 420 and accessing data stored in the memory 420, the processor 410 performs various functions of the battery management system and processes data, as well as various functions of the computer device and processes data, thereby providing overall control of the computer device. Optionally, the processor 410 may be implemented in the form of at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 410 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 410 and may be implemented separately via a communication chip.

[0176] The memory 420 may include a random access memory 420 (Random Access Memory, RAM), and may also include a read-only memory 420 (Read-Only Memory). The memory 420 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 420 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the various method embodiments described below, etc. The data storage area may also store data created by the computer device during use (such as a phone book and audio and video data), etc. Accordingly, the memory 420 may also include a memory controller to provide the processor 410 with access to the memory 420.

[0177] The power supply 430 can be logically connected to the processor 410 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. The power supply 430 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0178] The input unit 440 can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0179] Although not shown, the computer device 400 may further include a display unit, etc., which will not be described in detail herein. Specifically, in this embodiment, the processor 410 in the computer device loads the executable files corresponding to one or more application processes into the memory 420 according to the following instructions, and the processor 410 runs the application stored in the memory 420, thereby implementing the various method steps provided in the aforementioned embodiments.

[0180] Further, please refer to Figure 12, which is a block diagram of a computer-readable storage medium provided in an embodiment of the present application. As shown in Figure 12, the computer-readable storage medium 500 stores program code 510 for executing the steps of the method embodiment of the present application. The program code 510 may be a computer program instruction that can be called by a processor to execute the method described in the above embodiment.

[0181] The computer-readable storage medium can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium includes a non-volatile computer-readable storage medium (Non-Transitory Computer-Readable Storage Medium). The computer-readable storage medium 500 has storage space for program codes for executing any of the method steps in the above method. These program codes can be read from or written into one or more computer program products. The program codes can be compressed, for example, in an appropriate form.

[0182] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations provided in the above embodiments.

[0183] The above is only a preferred embodiment of the present application and does not constitute any form of limitation to the present application. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present application. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A live broadcast translation method, characterized in that: The method comprises: A candidate live stream to be translated is obtained from the cached live stream; the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp, the target timestamp refers to the end time of the live stream corresponding to the last stable translation result, the live stream corresponding to the last stable translation result refers to a translated live stream, the translated live stream is the last live stream of the candidate live stream, and the last stable translation result refers to a stable translation result obtained after translating the translated live stream; Translate the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream; Based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed, determine the target translation result of the stream to be pushed; the target translation result of the stream to be pushed refers to the translation result to be pushed along with the target live stream, and the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a preset duration threshold, the duration threshold is a positive number, and the duration threshold is used to characterize the maximum delayed streaming duration of the target live stream; Based on the target translation result, re-encoding the target live stream to be pushed to obtain a re-encoded live stream; The re-encoded live stream is pushed to display the target translation result on the viewer side.

2. The method according to claim 1, characterized in that The determining the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed includes: Performing semantic analysis on the translation result corresponding to the candidate live stream to obtain a semantic analysis result corresponding to the translation result; the semantic analysis result is used to indicate whether there is a stable translation result in the translation result; Obtain the end time corresponding to the candidate live stream, obtain the target end timestamp of the target live stream to be pushed and a preset duration threshold for the target live stream, and use the sum of the target end timestamp and the duration threshold as the reference time; Compare the end time corresponding to the candidate live stream with the reference time to obtain a comparison result; the comparison result is used to determine whether the end time corresponding to the candidate live stream is less than the reference time; If the comparison result indicates that the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result indicates that there is a stable translation result in the translation result, the translation result corresponding to the candidate live stream is used as the target translation result of the stream to be pushed.

3. The method according to any one of claims 1 to 2, characterized in that: The step of determining a target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed further includes: If the end time corresponding to the candidate live stream is equal to the reference time, the translation result corresponding to the candidate live stream is determined as the target translation result of the stream to be pushed.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: If the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result indicates that there is no stable translation result, a new candidate live stream is obtained from the cached live stream based on the target timestamp, and the step of translating the speech recognition content corresponding to the candidate live stream is returned to obtain the translation result corresponding to the candidate live stream.

5. The method according to any one of claims 1 to 4, characterized in that: If the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result indicates that there is a stable translation result, after taking the translation result corresponding to the candidate live stream as the target translation result of the stream to be pushed, the method further includes: The end time of the candidate live stream is stored as a new target timestamp.

6. The method according to any one of claims 1 to 5, characterized in that: The step of re-encoding the target live stream to be pushed based on the target translation result to obtain the re-encoded live stream comprises: Based on the target translation result, generating an auxiliary enhanced frame; The auxiliary enhanced frame is encoded into the target live stream of the stream to be pushed to obtain a re-encoded live stream.

7. The method according to any one of claims 1 to 6, characterized in that: The generating an auxiliary enhanced frame based on the target translation result includes: Based on the data encoding format, determine the custom data field corresponding to the additional enhanced information; The target translation result is added to the custom data field to obtain an auxiliary enhanced frame.

8. The method according to any one of claims 1 to 7, characterized in that: The step of adding the target translation result to the custom data field to obtain an auxiliary enhanced frame includes: Obtaining the speech recognition content of the live stream corresponding to the target translation result; The speech recognition content and the target translation result are added to the custom data field to obtain an auxiliary enhanced frame.

9. The method according to any one of claims 1 to 8, characterized in that: The step of encoding the auxiliary enhanced frame into the target live stream of the stream to be pushed to obtain a re-encoded live stream includes: Decoding the target live stream of the stream to be pushed, and obtaining audio data of the audio stream and video data of the video stream in the target live stream of the stream to be pushed; Inserting the auxiliary enhancement frame before a specified video frame in the video data of the video stream to obtain reference video data; the specified video frame refers to a key frame and / or a forward prediction coding frame in the video data of the video stream; The audio data of the audio stream and the reference video data are re-encoded to obtain a re-encoded live stream.

10. The method according to any one of claims 1 to 9, characterized in that: The pushing of the re-encoded live stream includes: If the time difference between the current time and the end time corresponding to the re-encoded live stream reaches the duration threshold, the target live stream is pushed.

11. The method according to any one of claims 1 to 10, characterized in that: The translating process of the speech recognition content corresponding to the candidate live stream to obtain the translation result corresponding to the candidate live stream includes: Translate the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the speech recognition content corresponding to the candidate live stream; The translation result corresponding to the speech recognition content corresponding to the candidate live stream, the end time corresponding to the candidate live stream, and the target timestamp are combined to obtain the translation result corresponding to the candidate live stream.

12. The method according to any one of claims 1 to 11, characterized in that: The method further comprises: Acquire scenario application information, where the scenario application information indicates a target live broadcast interaction level; Based on the correspondence between the live broadcast interaction level and the duration, the duration corresponding to the target live broadcast interaction level is determined as the duration threshold.

13. A live translation device, characterized in that: The device comprises: A live stream acquisition module, used to acquire a candidate live stream from the cached live stream; the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation result; A translation processing module, used to translate the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream; A result determination module is used to determine a target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a duration threshold; A re-encoding module, used for re-encoding the target live stream to be pushed based on the target translation result to obtain a re-encoded live stream; A streaming module is used to stream the re-encoded live stream to display the translation result in the target translation result on the audience end.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 12.

15. A computer device, characterized in that: include: Memory; One or more processors coupled to the memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to execute the method according to any one of claims 1 to 12.

16. A computer program product or a computer program, characterized in that The computer program product or computer program includes computer instructions, which are stored in a storage medium; a processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions, so that the computer executes the method according to any one of claims 1 to 12.

Images