Live translation method and device, storage medium and computer equipment

Abstract

The invention discloses a live broadcast translation method. The method comprises the following steps: acquiring candidate live broadcast streams from cached live broadcast streams; performing translation processing on the voice recognition content corresponding to the candidate live broadcast stream to obtain a translation result corresponding to the candidate live broadcast stream; determining a target translation result of the to-be-pushed stream based on the translation result corresponding to the candidate live stream and a target end timestamp of the to-be-pushed target live stream; based on the target translation result, recoding the to-be-pushed target live broadcast stream to obtain a recoded live broadcast stream; and pushing the recoding live stream to display the translation content in the target translation result at the audience side. According to the method and the device, the duration threshold is set, and the target translation result of the to-be-pushed stream is determined according to the translation result corresponding to the candidate live broadcast stream, so that the target translation result can be obtained in the time period corresponding to the duration threshold for stream pushing, and the accuracy of the live broadcast translation result is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a live translation method, apparatus, storage medium and computer equipment. Background Art

[0002] Simultaneous interpretation is a translation method in which the interpreter interprets the content to the audience without interrupting the speaker. Relying on the development of computer and information technology, simultaneous interpretation and communication technology can be closely combined.

[0003] For example, automatic translation is performed in live broadcast, webcast or real-time call scenarios. When the relevant technology automatically translates the words of the speaker in the scenario, it usually gives the translation result corresponding to the expression content after the speaker completes the expression.

[0004] This automatic translation method can translate the speaker's expression, but in order to obtain accurate translation content, it usually takes a certain amount of time to translate. If the translation content corresponding to the live stream to be pushed is sent to the audience in real time, the translation content may be inaccurate. In short, the effect of live translation is not good, which affects the user's live viewing experience. Summary of the invention

[0005] The embodiments of the present application provide a live broadcast translation method, apparatus, storage medium and computer equipment to solve the problem in the related art that the live broadcast translation content is inaccurate, resulting in poor effect of the live broadcast translation.

[0006] On the one hand, an embodiment of the present application provides a live translation method, which includes: obtaining a candidate live stream from a cached live stream; the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the previous stable translation content; translating the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream; determining a target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a duration threshold; based on the target translation result, re-encoding the target live stream to be pushed to obtain a re-encoded live stream; pushing the re-encoded live stream to display the translated content in the target translation result on the audience end.

[0007] On the other hand, an embodiment of the present application also provides a live translation device, which includes: a live stream acquisition module, which is used to acquire a candidate live stream from a cached live stream; the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the previous stable translation content; a translation processing module, which is used to translate and process the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream; a result determination module, which is used to determine the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a duration threshold; a re-encoding module, which is used to re-encode the target live stream to be pushed based on the target translation result to obtain a re-encoded live stream; and a streaming module, which is used to stream the re-encoded live stream to display the translated content in the target translation result on the audience end.

[0008] Optionally, the result determination module may include: a semantic analysis unit and a first determination unit. The semantic analysis unit is used to perform semantic analysis on the translation result corresponding to the candidate live stream to obtain a semantic analysis result corresponding to the translation result; the semantic analysis result is used to indicate whether there is stable translation content in the translation result; the first determination unit is used to use the translation result corresponding to the candidate live stream as the target translation result of the stream to be pushed if the end time corresponding to the candidate live stream is less than the reference time and the semantic analysis result corresponding to the translation result indicates the existence of stable translation content; the reference time is equal to the sum of the target end timestamp and the duration threshold.

[0009] Optionally, the result determination module may further include a second determination unit. The second determination unit is configured to determine the translation result corresponding to the candidate live stream as the target translation result of the stream to be pushed if the end time corresponding to the candidate live stream is equal to the reference time.

[0010] Optionally, the result determination module may further include a return execution unit. The return execution unit is used to obtain a new candidate live stream from the cached live stream based on the target timestamp if the end time corresponding to the candidate live stream is less than the reference time and the semantic analysis result corresponding to the translation result indicates that there is no stable translation content, and return to the step of performing translation processing on the speech recognition content corresponding to the candidate live stream to obtain the translation result corresponding to the candidate live stream.

[0011] Optionally, the result determination module may further include a timestamp determination unit. The timestamp determination unit is configured to store the end time of the candidate live stream as a new target timestamp.

[0012] Optionally, the re-encoding module may include: an enhanced frame generation unit and a re-encoding unit. The enhanced frame generation unit is used to generate an auxiliary enhanced frame based on the target translation result; the re-encoding unit is used to encode the auxiliary enhanced frame into the target live stream to be pushed to obtain the re-encoded live stream.

[0013] Optionally, the enhanced frame generation unit may include: a determination subunit and an addition subunit. The determination subunit is used to determine the custom data field corresponding to the additional enhancement information based on the data encoding format; the addition subunit is used to add the target translation result to the custom data field to obtain an auxiliary enhanced frame.

[0014] Optionally, the adding subunit may be specifically used to: obtain the speech recognition content of the live stream corresponding to the target translation result; and add the speech recognition content and the target translation result to a custom data field to obtain an auxiliary enhanced frame.

[0015] Optionally, the re-encoding unit can be specifically used to: decode the target live stream to be pushed, obtain the audio stream and video stream in the target live stream to be pushed; insert the auxiliary enhancement frame before the specified video frame in the video stream to obtain the reference video stream; the specified video frame refers to the key frame or forward prediction coding frame in the video stream; re-encode the audio stream and the reference video stream to obtain the re-encoded live stream.

[0016] Optionally, the streaming module may be specifically used to: if the time difference between the current time and the end time corresponding to the re-encoded live stream reaches a duration threshold, push the target live stream.

[0017] Optionally, the translation processing module can be specifically used to: translate the speech recognition content corresponding to the candidate live stream to obtain the translation content corresponding to the candidate live stream; combine the translation content corresponding to the candidate live stream, the end time corresponding to the speech recognition content corresponding to the candidate live stream, and the target timestamp to obtain the translation result corresponding to the candidate live stream.

[0018] Optionally, the live broadcast translation device may further include: an application information acquisition module and a duration threshold determination module. The application information acquisition module is used to acquire scene application information, which indicates the target live broadcast interaction level; the duration threshold determination module is used to determine the duration corresponding to the target live broadcast interaction level as the duration threshold based on the corresponding relationship between the live broadcast interaction level and the duration.

[0019] On the other hand, an embodiment of the present application further provides a computer device, which includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are called by the processor, the above-mentioned live broadcast translation method is executed.

[0020] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores a program code, wherein the above-mentioned live broadcast translation method is executed when the program code is executed by a processor.

[0021] On the other hand, the embodiment of the present application further provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a storage medium. The processor of the computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions, so that the computer performs the steps in the above-mentioned live translation method.

[0022] The live translation method provided by the present application can obtain a candidate live stream from a cached live stream, wherein the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the previous stable translation content, and then the speech recognition content corresponding to the candidate live stream is translated to obtain a translation result corresponding to the candidate live stream, and based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed, the target translation result of the stream to be pushed is determined, wherein the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a duration threshold; then, based on the target translation result, the target live stream to be pushed is re-encoded to obtain the re-encoded live stream, and the re-encoded live stream is pushed to display the translated content in the target translation result on the audience end.

[0023] Since the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold, and the start time of the candidate live stream is the end time of the live stream corresponding to the previous stable translation content, the target translation result is equivalent to being obtained by translating a live stream with a duration longer than the target live stream to be pushed. In the live broadcast scenario, the longer the duration of the live stream on which the translation is based, the higher the accuracy of the translation result.

[0024] Therefore, in the present application, compared with directly translating the speech recognition content corresponding to the target live stream to be pushed, and re-encoding the translation result corresponding to the target live stream into the target live stream, the accuracy of the target encoding result is higher, and the relatively accurate target translation result is pushed, so that the live translation can ensure accurate translation quality. Since the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold, the target live stream is guaranteed to be delayed for a maximum of the duration threshold, thereby achieving the sacrifice of the real-time performance of part of the live broadcast to ensure the accuracy of the live translation, thereby improving the user's live viewing experience in the live translation scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A schematic diagram of the architecture of a live broadcast translation system provided in an embodiment of the present application is shown.

[0027] Figure 2 An application scenario diagram of a live broadcast translation provided by an embodiment of the present application is shown.

[0028] Figure 3 A flow chart of a live broadcast translation method provided in an embodiment of the present application is shown.

[0029] Figure 4 A schematic diagram of a process for determining a target translation result provided in an embodiment of the present application is shown.

[0030] Figure 5 A flowchart of a translation process provided by an embodiment of the present application is shown.

[0031] Figure 6 A flow chart of another live broadcast translation method provided in an embodiment of the present application is shown.

[0032] Figure 7 A flowchart of a live broadcast translation provided by an embodiment of the present application is shown.

[0033] Figure 8 A schematic diagram of a code stream format of a network abstraction layer data unit provided in an embodiment of the present application is shown.

[0034] Fig. 9 A schematic diagram of the data format of additional enhancement information provided in an embodiment of the present application is shown.

[0035] Fig.10 A module block diagram of a live broadcast translation device provided in an embodiment of the present application is shown.

[0036] Fig.11 It is a module block diagram of a computer device provided in an embodiment of the present application.

[0037] Fig.12 It is a module block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0039] In some processes described in the specification, claims and the above drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, the descriptions such as "first" and "second" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0040] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0041] It should be noted that in the specific implementation of this application, when it comes to user audio, video, live stream and other related data, when applied to the specific products or technologies of the embodiments of this application, it is necessary to obtain user permission or consent, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and subsequent data use and processing must be carried out within the scope of authorization of laws, regulations and personal information subjects.

[0042] At present, different languages ​​have their own specific grammar and expressions. Ideally, if an audio that expresses a complete sentence is translated, its accurate translation content can be obtained. For this reason, the translation of offline videos can be based on the translation method of complete sentences. However, in the live broadcast scenario, since the live stream needs to be pushed to the audience in real time, and the duration of each live stream is short, if the speech recognition content of the audio in the live stream is translated each time, and the translation result is pushed to the audience with the current live stream, the audio in the live stream may not be able to fully express the speaker's speech content, and it is easy for the translation content of the live stream to be inaccurate. In this way, this kind of live translation method is prone to inaccurate translation content displayed on the audience side, which affects the audience's live viewing experience in the live broadcast scenario.

[0043] However, if the speech recognition content of the audio in the live stream is translated only after the host has finished speaking one or more sentences, and then the translation content and the audio stream are sent to the audience, this will easily lead to serious delays in the live broadcast. In addition, the real-time performance of the live broadcast translation is poor, which may cause the translation results displayed to the audience to be out of sync with the audio and video images, affecting the audience's live viewing experience and the interaction between the host and the audience. In order to solve the above problems, the inventor has proposed a live broadcast translation method of the present application after research.

[0044] The live translation method of the present application involves streaming media technology, which specifically refers to a technology and process of compressing a series of media data, sending the data in segments over the Internet, and transmitting audio and video online for viewing. Streaming can transmit live audio and video or videos pre-stored on a server. When viewers watch these audio and video files, the audio and video data is immediately played by specific playback software after being sent to the viewer's computer. For example, the live broadcast application related to the live translation method provided in the present application.

[0045] The live broadcast referred to in this application is based on the transmission and playback of live audio and video using streaming media technology, which can be applied to live broadcasts on the Internet and online video conferencing. During the live broadcast process, a video coding protocol (such as the H.264 coding protocol) is used to encode the video frames. Specifically, the H.264 coding framework includes a video coding layer (Video Coding Layer, VCL) and a network abstraction layer (Network Abstraction Layer, NAL). The video coding layer is used to efficiently represent video content, and the network abstraction layer is used to format data and provide header information to ensure that the data is suitable for effective transmission on various channels and storage media.

[0046] The video content representation includes intra-frame compression of video frames, i.e., generating I frames, and inter-frame compression of video frames, i.e., generating P / B frames. I frames are fully encoded key frames, P frames are forward predictive encoded frames, and B frames are bidirectional predictive interpolated encoded frames. In the network abstraction layer, the Network Abstract Layer Unit (NALU) is the basic unit of H.264 encoding storage or transmission.

[0047] Each NALU includes a header structure and a body. The header structure occupies 1 byte (8 bits). The header structure indicates whether the NAL unit can be discarded, the importance indication, and the NALU type. In the H.264 bitstream, each frame of data is a NALU. For example, in an embodiment of the present application, the auxiliary enhancement frame generated based on the target translation result is a custom data field in the supplementary enhancement information (SEI) that adds the target translation result and is encapsulated into a specific type of NALU.

[0048] To prevent the server from being overloaded with too many users and the slow download speed of users. Live broadcast application scenarios can use content delivery networks (CDNs) to distribute data. The content distribution network consists of two layers: the center and the edge. The edge layer servers are deployed in various places and across major operators. The physical distance between the edge servers and users is the shortest. The streaming media service cluster at the center layer is responsible for forwarding content. For example, according to the user's geographic location information, the nearest edge server is selected to provide push / pull streaming services for users.

[0049] The following is an introduction to the system architecture of the live broadcast translation method involved in this application.

[0050] See also Figure 1 , Figure 1 A schematic diagram of the architecture of a live broadcast translation system is shown. Figure 1 As shown, the live broadcast translation system 100 may include a content distribution network 110, a host terminal 130, and an audience terminal 150. The host terminal 130 and the audience terminal 150 may be terminal devices, such as smart phones, desktop computers, tablet computers, laptops, smart TVs, and vehicle-mounted devices, augmented reality (AR) and virtual reality (VR), etc., which are not limited here.

[0051] The content distribution network 110 may include an edge layer server and a streaming service cluster, which may include multiple streaming servers. The edge layer server and the streaming server may be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, cloud functions, cloud storage, cloud communications, network services, domain name services, middleware services, security services, blockchain, and big data and artificial intelligence platforms, without limitation here.

[0052] In some embodiments of the present application, after acquiring the live stream, the anchor terminal 130 can translate the live stream based on the live translation method of the present application to obtain a live stream carrying the translated content (i.e., the re-encoded live stream in the present application). The anchor terminal 130 can push the live stream carrying the translated content to the content distribution network 110, and then the audience terminal 150 can pull the stream from the content distribution network 110 to decode and obtain the audio and video carrying the subtitle content (the subtitle content includes at least the translated content, and may also include the translated content and the speech recognition content corresponding to the audio), and while playing the audio and video, the subtitle content is also displayed on the screen.

[0053] Understandably, Figure 1 It is only a schematic diagram of the architecture of a system provided in the embodiment of the present application. The architecture of the system described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Figure 1 The process of translating the live stream is that the anchor end translates it and then pushes the stream upstream. In other cases, the anchor end can push the live stream to the server end, and the live server end processes it according to the method of this application. Then, the live server end uploads the translated and re-encoded live stream to the content distribution network. After the audience end pulls the stream from the content distribution network and decodes it, the audio and video with the translated content are played on the screen.

[0054] See also Figure 2 , Figure 2 A diagram showing an application scenario of live broadcast translation is shown. Figure 2 As shown, the live broadcast translation method can be applied to a computer system 200, which can be applied to a network live broadcast scenario. The computer system 200 can include a server (or backend) 220 of a live broadcast service provider, a live broadcast terminal 210 on the user side, and an audience terminal 230. The content distribution network includes a streaming media service cluster 260, a first edge server 240 corresponding to the live broadcast terminal 210, and a second edge server 280 corresponding to the audience terminal 230.

[0055] In a live broadcast scenario, when the anchor uses the live broadcast terminal 210 for live broadcast, the live broadcast terminal 210 can encode the audio and video collected in real time into a live stream, and then push the live stream to the server 220. After receiving the live stream, the server can cache the real-time live stream, and perform live translation processing according to the method of the present application to obtain a re-encoded live stream, and push the re-encoded live stream to the first edge server 240.

[0056] Furthermore, the first edge layer server 240 can push the re-encoded live stream to the streaming media service cluster 260. After receiving the re-encoded live stream, the streaming media service cluster 260 can perform certain processing on the re-encoded live stream, for example, transcoding, that is, converting from one encoding format to another encoding format. This is because there are differences in the clients used by different viewers, and it is necessary to ensure that they can all watch the live broadcast normally. Then, the re-encoded live stream is pre-loaded to the edge server closest to the audience end, for example, the second edge server 280. The audience end 230 can pull the re-encoded live stream from the second edge server 280, decode the re-encoded live stream, and then play the decoded audio and video while displaying the corresponding translated subtitles of the audio on the video screen.

[0057] It should be noted that Figure 2 It is only an application scenario diagram of a system provided by an embodiment of the present application. The system architecture and application scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided by the embodiment of the present application. For example, the first edge server 240 can generally refer to one of multiple edge servers in a content distribution network, and the second edge server 280 can also generally refer to one of multiple edge servers in a content distribution network. This embodiment only uses the first edge server 240 and the second edge server 280 as examples. It is known to those skilled in the art that with the evolution of the system architecture, the technical solution provided by the embodiment of the present invention is also applicable to similar technical problems.

[0058] See also Figure 3 , Figure 3 The flowchart of the live broadcast translation method provided by one embodiment of the present application is shown. In this embodiment, the live broadcast translation method can be executed by the live broadcast service end or by the anchor end, which is not specifically limited here. Figure 3 As shown, the live broadcast translation method may include the following steps:

[0059] Step S110: Obtain a candidate live stream from the cached live stream; the candidate live stream refers to a live stream whose start time is the target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation content.

[0060] Usually, the host end can perform audio and video acquisition in real time when the host initiates a live broadcast. After that, the collected audio stream and video stream can be encoded to obtain the encoded code stream, that is, the live stream. Optionally, the audio stream and video stream can be pre-processed (such as beautifying, filtering or special effects on the video stream, echo cancellation or noise reduction on the audio stream, etc.) before encoding. The host end can push the encoded live stream to the server end based on the streaming media protocol. Among them, the streaming media protocol may include the Real-Time Messaging Protocol (RTMP) or the HTTP-based adaptive bit rate streaming media transmission protocol (HTTP Live Streaming, HLS), etc., which are not limited here.

[0061] As an implementation method, when receiving the live stream pushed by the anchor, the server can cache the live stream to facilitate subsequent processing such as translation of the live stream. For example, the server can cache the received live stream into a code stream cache pool, and then obtain a live stream to be translated from the cached live stream, that is, a candidate live stream. Specifically, the server can obtain the end time of the live stream corresponding to the last stable translation content, that is, the target timestamp.

[0062] Furthermore, the server can obtain a live stream as a candidate live stream from the cached live stream with the target timestamp as the starting time. The stable translation content refers to the translation content in a steady state, that is, the stable translation content refers to the translation content that will not change based on the subsequent live stream, or the stable translation content is the translation content corresponding to a semantically basically complete audio (such as the audio corresponding to a sentence), and its content will not change with the subsequent translation process.

[0063] For example, Figure 2 The host terminal 210 shown can collect audio and video during the live broadcast when the host initiates the live broadcast, and encode the audio stream and video stream collected in real time to obtain a live stream. After the host terminal 210 pushes the encoded live stream to the server terminal 220, the server terminal 220 can store the pushed live stream in the stream buffer pool. Furthermore, the server terminal 220 can obtain the target timestamp corresponding to the last stable translation content, for example, the target timestamp is 08:34.17, and then select the live stream starting from 08:34.17 from the stream buffer pool as the candidate live stream.

[0064] More specifically, a live stream with a start time of the target timestamp and an end time of the end time of the latest live stream cached in the stream cache pool can be obtained from the stream cache pool as a candidate live stream. For example, if the end time of the latest live stream cached in the stream cache pool is 08:34.23 and the target timestamp is 08:34.17, the live stream in the time period of 08:34.17 to 08:34.23 can be used as a candidate live stream. It can be understood that in this case, since the stream cache pool is caching new live streams in real time, the end times of the candidate live streams obtained from the stream cache pool twice are different.

[0065] Step S120: Translate the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream.

[0066] Specifically, when obtaining a candidate live stream, the server can decode the candidate live stream to obtain a corresponding audio stream, perform speech recognition on the audio stream to obtain corresponding speech recognition content, and then translate the speech recognition content to obtain a translation result, for example, translating English speech recognition content to obtain Chinese content.

[0067] In some embodiments, a speech recognition model and a text translation model may be deployed, wherein the speech recognition model and the text translation model may be models constructed by a neural network. Thus, after acquiring the candidate live stream, the audio stream in the candidate live stream (for ease of distinction, the audio stream is referred to as the candidate audio stream) is input into the speech recognition model for speech recognition, and the speech recognition content corresponding to the candidate audio stream is output, that is, the speech recognition content corresponding to the candidate live stream. Thereafter, the speech recognition content corresponding to the candidate live stream is input into the text translation model for text translation, and the translation content corresponding to the speech recognition content is output, that is, the translation result corresponding to the candidate live stream includes the translation content corresponding to the speech recognition content corresponding to the candidate live stream.

[0068] In some embodiments, a speech recognition model for performing speech recognition on audio in a first language may be deployed according to a first language (the first language is the language used by the anchor terminal for live broadcast), and a text translation model for translating text in the first language into text in a second language may be deployed according to the first language and a second language (the second language is the language to which the specified translated text belongs), for example, a text translation model for translating Mandarin Chinese into English, and a text translation model for translating English into German.

[0069] As an implementation method, the server may combine the translation content corresponding to the candidate live stream, the end time corresponding to the candidate live stream, and the target timestamp to obtain the translation result corresponding to the candidate live stream.

[0070] In some embodiments, after obtaining the translation content corresponding to the candidate live stream, the translation content corresponding to the candidate live stream, the speech recognition content corresponding to the candidate live stream, the end time corresponding to the candidate live stream, and the target timestamp can be combined to obtain the translation result corresponding to the candidate live stream. Of course, in other embodiments, the translation content corresponding to the candidate live stream and the speech recognition content corresponding to the candidate live stream can also be combined as the translation result corresponding to the candidate live stream.

[0071] Step S130: Determine the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold.

[0072] The target live stream refers to the live stream of the current stream to be pushed. In this application, for the sake of distinction, the timestamp representing the end time of the target live stream is called the target end timestamp.

[0073] In some embodiments, the duration of each live stream pushed can be set, assuming it is the first duration. If the end time of the last live stream pushed is the first time, then the live stream with the start time of the first time and the end time of the second time (the second time = the first time + the first duration) is obtained from the cached live stream (for example, the live stream cached in the code stream cache pool described above) as the target live stream to be pushed. It is worth mentioning that in the live broadcast scenario, the live stream of the same time period will not be repeatedly pushed, so the live streams pushed twice are different. The end time of the live stream pushed last time can usually be the start time of the live stream pushed next time, so as to ensure that the live streams pushed twice are different.

[0074] In the present application, since the candidate live stream is the live stream after the end time of the live stream corresponding to the previous stable translation content, that is, the start time of the candidate live stream is the target timestamp, and the end time is after the target timestamp, therefore, the start time of the candidate live stream is not later than the start time of the current target live stream to be pushed, that is, the start time of the candidate live stream can be earlier than the start time of the target live stream. If the previous translation content pushed is a stable translation content, the start time of the current candidate live stream can be equal to the start time of the target live stream.

[0075] Moreover, in the present application, the end time of the candidate live stream is not earlier than the end time of the target live stream, that is, the end time of the candidate live stream can be the same as the end time of the target live stream, or the end time of the candidate live stream can be later than the end time of the target live stream. In other words, the live time period corresponding to the target live stream is within the live time period corresponding to the candidate live stream, or the live time period corresponding to the target live stream is the same as the live time period corresponding to the candidate live stream.

[0076] The target translation result to be pushed refers to the translation result to be pushed along with the target live stream. In this application, the duration threshold is a positive number. Since the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold, the end time of the live stream corresponding to the target translation result is not later than the target end timestamp (i.e., the end time of the target live stream), and the difference between the end time of the live stream corresponding to the target translation result and the target end timestamp is greater than or equal to the duration threshold.

[0077] In a common live broadcast scenario, when the target live stream of the current stream to be pushed is read, the target live stream of the stream to be pushed is pushed in real time, and generally no additional waiting is required. In the present application, if the end time of the live stream corresponding to the target translation result is later than the target end timestamp, it indicates that after reading the target live stream of the current stream to be pushed, it continues to wait for a period of time. It is understandable that during this period of waiting, a new live stream is cached. In this case, the live stream obtained and the newly obtained live stream in the waiting time period can be combined as the data basis for translation. In this way, compared with only using the speech recognition content corresponding to the target live stream as the data basis for translation, in this case of the present application, more data basis for translation can be obtained. In the live broadcast scenario, the longer the duration corresponding to the live stream as the data basis for translation, the higher the probability of the audio stream including a complete sentence in the live stream, and correspondingly, the higher the accuracy of the translation result obtained by the speech recognition content translation based on the live stream.

[0078] As an implementation method, semantic analysis can be performed on the translation results corresponding to the candidate live streams to determine the target translation result. Figure 4 , step S130 may include:

[0079] Step S131: semantically analyze the translation result corresponding to the candidate live stream to obtain a semantic analysis result corresponding to the translation result, wherein the semantic analysis result is used to indicate whether there is stable translation content in the translation result.

[0080] As an implementation method, a semantic analysis may be performed on the translation result corresponding to the candidate live stream (e.g., the translation content in the translation result) based on natural language processing (NLP) technology. For example, a lexical level semantic analysis and / or a sentence level semantic analysis may be performed on the sentence content of the translation result, and based on the contextual semantics of the words, phrases, or short sentences in the sentence content, it is determined whether there is a stable translation content in the translation result.

[0081] Step S132: Compare the end time corresponding to the candidate live stream with the reference time; the reference time is equal to the sum of the target end timestamp and the duration threshold. By comparison, it is determined that the end time corresponding to the candidate live stream is less than the reference time, or is equal to the reference time.

[0082] Step S133: determining whether there is stable translation content in the translation result based on the semantic analysis result corresponding to the translation result.

[0083] If the end time corresponding to the candidate live stream is less than the reference time, and there is stable translation content in the translation result corresponding to the candidate live stream, execute step S134: use the translation result corresponding to the candidate live stream as the target translation result of the stream to be pushed.

[0084] For example, the duration threshold T thr = 5 seconds and the target end timestamp T stamp =14:32.15, then the reference time is T ref =T stamp +T thr =14:38.15. If the translation result corresponding to the candidate live stream includes the translation content of a sentence expressed by an audio with a duration of 3.23 seconds, and if the sentence expressed by the audio is a sentence, the translation result can be determined as a stable translation content after semantic analysis of the translation content of the sentence expressed by the audio, and the end time T corresponding to the candidate live stream is end =T stamp +3.23s=14:35.38, not exceeding the reference time T ref Therefore, the translation result corresponding to the candidate live stream can be used as the target translation result of the stream to be pushed.

[0085] If the end time corresponding to the candidate live stream is less than the reference time, and there is no stable translation content in the translation result corresponding to the candidate live stream, execute step S135: obtain a new candidate live stream from the cached live stream based on the target timestamp; then, return to execute step S120.

[0086] In step S135, the new candidate live stream obtained from the cached live stream based on the target timestamp is still a live stream with a start time of the target timestamp and an end time of the end time of the latest cached live stream. It can be understood that since the new live stream is cached in real time, when the target timestamp remains unchanged, the duration corresponding to the newly acquired candidate live stream is longer compared to the candidate live stream acquired last time, and the duration of the audio stream in the newly acquired candidate live stream is also longer, and the audio stream expresses more content. Therefore, in this way, the probability of obtaining stable translation content by translating based on the audio stream in the new candidate live stream with a longer duration will be higher. It can be understood that based on the acquired new candidate live stream, the translation processing is then performed in accordance with step S120, and then step S131 and subsequent steps are executed.

[0087] When the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result of the candidate live stream indicates that there is no stable translation content, the possible reason for the absence of stable translation content is that the duration corresponding to the candidate live stream is short, and the audio stream in the candidate live stream does not fully present a sentence of the anchor. In this case, if the translation result of the candidate live stream is pushed to the audience end along with the target live stream, the translation content displayed on the audience end may be inaccurate. In the subsequent process, the translation content displayed in the live broadcast screen may be modified many times, which causes the translation content seen by the audience to be frequently modified, resulting in a poor live broadcast experience for the audience and affecting the live broadcast effect. Therefore, in this case, the translation result of the candidate live stream is discarded, and a new candidate live stream is obtained for translation.

[0088] In the case where the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result of the candidate live stream indicates that there is no stable translation content, considering that the possible reason for the absence of stable translation content is that the duration corresponding to the candidate live stream is short, since the newly generated live stream is cached in real time, therefore, in this case, a candidate live stream with a longer duration is obtained from the cached live stream, so that a live audio stream with a longer duration can be provided for translation, and the probability of obtaining stable translation content later is higher. It can be understood that in the case where the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result of the candidate live stream indicates that there is no stable translation content, a new candidate live stream is obtained for translation. Since the target translation result has not been determined, the target translation stream is in a state of waiting to be pushed.

[0089] In some embodiments, a first interval duration between two consecutive acquisitions of candidate live streams can be set. When the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result indicates that there is no stable translation content, if the interval duration between the current time and the last time the candidate live stream was acquired reaches the first interval duration, a new candidate live stream is read from the cache.

[0090] If the end time corresponding to the candidate live stream is equal to the reference time, step S134 is executed: the translation result corresponding to the candidate live stream is used as the target translation result of the stream to be pushed.

[0091] For example, the duration threshold T thr = 5 seconds and the target end timestamp T stamp =14:32.15, then the reference time is T ref =T stamp +T thr =14:38.15, if the translation result corresponding to the candidate live stream is already 5 seconds long, then the end time corresponding to the candidate live stream is T end =T stamp +5s = 14:35.15, equal to the reference time T ref , the translation result corresponding to the candidate live stream can be determined as the target translation result of the stream to be pushed.

[0092] If the end time of the candidate live stream is equal to the reference time, the target live stream has been delayed for a certain period of time, and if the target live stream continues to be delayed, the real-time performance of the live stream will be greatly reduced for the user. Therefore, when the end time of the candidate live stream is equal to the reference time, regardless of whether there is stable translation content in the translation result of the candidate live stream, the translation result of the candidate live stream is determined as the target translation result to be pushed, so that the target translation result can be pushed along with the target live stream to avoid excessive delay in the live broadcast scenario.

[0093] In the above embodiment, a compromise is made between ensuring the real-time nature of the live broadcast and ensuring the accuracy of the translation results. If the translation result of the candidate live broadcast stream obtained does not include stable translation content, and the end time of the candidate live broadcast stream is less than the reference time, a new candidate live broadcast stream with a longer duration is obtained for translation, and the target live broadcast stream is delayed and pushed. When the end time of the candidate live broadcast stream is equal to the reference time, regardless of whether the translation result of the candidate live broadcast stream includes stable translation content, the translation result of the candidate live broadcast stream is used as the target translation result of the stream to be pushed, and the target translation result is pushed along with the target live broadcast stream. In this way, for the target live broadcast stream, the maximum delay in pushing the stream is the duration threshold. This solution achieves the sacrifice of the real-time nature of part of the live broadcast to improve the accuracy of the live broadcast translation. In this way, it can avoid the situation where the translation content is frequently modified on the audience side due to inaccurate translation content caused by the short duration of the audio stream in the live broadcast stream.

[0094] Furthermore, the method further includes: when the semantic analysis result corresponding to the translation result of the candidate live stream indicates that there is stable translation content in the translation result, executing step S136: storing the end time of the candidate live stream as the new target timestamp. In this case, regardless of whether the end time corresponding to the candidate live stream is less than the reference time or equal to the reference time, the end time of the candidate live stream is stored as the new target timestamp.

[0095] In the above embodiment, if the translation result of the candidate live stream includes stable translation content, the end time of the candidate live stream can be used as the new target timestamp, and the candidate live stream can be obtained according to the new target timestamp next time. In this way, it can be ensured that before the stable translation content appears, the audio stream in the live stream from the target timestamp to the start time of the target live stream is used for auxiliary translation, providing more data for translation to ensure the accuracy of the translation. When the translation result of the candidate live stream includes stable translation content, the translation content in this section of the translation result is generally considered to be accurate. In this way, in the subsequent process, a new candidate live stream for translation is obtained according to the new target timestamp (i.e., the end time of the candidate live stream), the new target timestamp can be skipped, and the live stream after the new target timestamp can be selected for translation, thereby reducing the translation workload and improving the translation speed.

[0096] As an implementation method, after determining the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed, the end time of the candidate live stream can be used as the new target timestamp.

[0097] When the translation results corresponding to the candidate live stream are semantically analyzed, the stable translation content in the translation results can be used as the target translation result for streaming. Moreover, when the end time corresponding to the candidate live stream is equal to the reference time, the translation content corresponding to the candidate live stream is directly used as the target translation result for streaming. This allows the target translation result to be obtained for streaming in the time period corresponding to each duration threshold. For example, the duration threshold is 5 seconds, and the 10-second candidate live stream includes a sentence. When the candidate live stream is translated to the 5th second, it is still unknown what the complete semantics is, that is, a stable translation content cannot be obtained, but the first 5 seconds of the translation still have certain semantics. For this reason, the translation content of the first 5 seconds can be used as the target translation result for streaming, and then the translation of the remaining live stream is continued to avoid the audience waiting for a long time and suddenly watching a large section of translation content. And because the stable translation content is a translation content in a steady state, it can ensure that the translation result of the live translation has a high accuracy. In this way, it can effectively avoid the long translation waiting time caused by the long translation of the complete sentence, while ensuring the accuracy of the live translation, it also improves the real-time performance of the live translation.

[0098] In the above embodiment, when the end time corresponding to the candidate live stream is equal to the reference time and the translation result corresponding to the candidate live stream does not include stable translation content, the translation result corresponding to the candidate live stream is determined as the target translation result of the stream to be pushed. In this case, the target timestamp is not updated. In this way, the next time the candidate live stream is obtained, it is still based on the target timestamp based on the last time the candidate live stream was obtained. In this way, compared with the candidate live stream obtained last time, the newly obtained candidate live stream has a longer duration, and the start time of both is the target timestamp. In this way, if the translation content in the translation result corresponding to the pushed candidate live stream is inaccurate, relatively accurate translation content can be obtained by translating based on the newly obtained candidate live stream with a longer duration. In this way, the more accurate translation content can be used to cover the inaccurate translation content of the previous push. Therefore, it is possible to transmit new and more accurate translation content to the audience for the previously inaccurate translation content, and replace the old translation content with the new translation content.

[0099] In some embodiments, the speech recognition content corresponding to the candidate live stream and the translation result corresponding to the candidate live stream can also be combined to determine whether the translation result corresponding to the candidate live stream has stable translation content. Specifically, the speech recognition content corresponding to the candidate live stream can be semantically analyzed to obtain a first semantic analysis result, which is used to indicate whether the speech recognition content corresponding to the candidate live stream is a complete sentence; if the speech recognition content corresponding to the candidate live stream is a complete sentence, the probability that the translation result corresponding to the candidate live stream has stable translation content is high, otherwise, if the speech recognition content corresponding to the candidate live stream is not a complete sentence, the probability that the translation result corresponding to the candidate live stream has stable translation content is low. Based on this principle, when the first semantic analysis result indicates that the speech recognition content corresponding to the candidate live stream is a complete sentence, the translation result corresponding to the candidate live stream is semantically analyzed to determine whether there is stable translation content in the translation result; otherwise, if the first semantic analysis result indicates that the speech recognition content corresponding to the candidate live stream is not a complete sentence, step 131 is not performed, and a new candidate live stream is obtained from the cached live stream based on the target timestamp for translation.

[0100] The accuracy and real-time performance of live translation are mutually exclusive. For example, if the duration of the live stream used for translation is longer, the accuracy of the translation will be higher. Correspondingly, the longer the waiting time for obtaining the live stream to be translated is, the lower the real-time performance of the live translation will be. In this application, a duration threshold is set, which is the maximum delayed push duration of the target live stream. Therefore, the size of the duration threshold can reflect the real-time performance of the live broadcast.

[0101] The size of the duration threshold can be set according to the real-time requirements of the live broadcast. It is understandable that for live broadcast scenarios where the host and the audience need to interact, the real-time requirements of the live broadcast are relatively high. In this case, a smaller duration threshold can be set to avoid the target live broadcast stream being delayed for too long. For live broadcast scenarios where there is little or no interaction between the host and the audience, the real-time requirements of the live broadcast are relatively low, and a larger duration threshold can be set to fully ensure the accuracy of the target translation results pushed to the audience end.

[0102] In some embodiments, scene application information may be obtained, the scene application information indicating a target live broadcast interaction level, the target live broadcast interaction level being the live broadcast interaction level of the current live broadcast scene (or the current target live broadcast room), and the target live broadcast interaction level may be preset by a user (e.g., a user who is a host). Thereafter, based on the correspondence between the live broadcast interaction level and the duration, the duration corresponding to the target live broadcast interaction level is determined as the duration threshold.

[0103] Among them, the target live broadcast interaction level is a quantitative expression of the real-time requirements for the current live broadcast scene. Scenario application information refers to information that can reflect the real-time requirements of the current live broadcast scene. The correspondence between the live broadcast interaction level and the duration can be pre-stored in a relational database. Optionally, the scenario application information can be pre-specified by the user at the terminal. For example, the anchor end can send the required size value for the real-time performance of the live broadcast translation directly set by the anchor to the server end as scenario application information. The scenario application information can also be information obtained by the server end from the anchor end to represent the amount of audience interaction, such as the barrage sending rate, and then the duration threshold is determined based on the barrage sending rate. The higher the sending rate, the more interactions there are, the higher the real-time requirements for the live broadcast translation, and the smaller the duration threshold can be.

[0104] In other embodiments, the duration prediction model can be used to predict the duration of the target audience's interaction data to obtain a corresponding duration threshold. The duration prediction model can be obtained by supervised learning of a preset classification network based on sample interaction data and label duration threshold. Among them, interaction data refers to data on the audience's participation in live interaction when watching live broadcasts, and may include the average barrage sending rate or average like rate of the audience when participating in different live broadcasts, etc., which are not limited here. Optionally, for different sample interaction data in the training set, the corresponding label duration threshold (Ground Truth) can be pre-calculated based on experiments. The preset classification network can be a deep neural network, for example, a combination of an embedding layer (Embedding Layer) and a multilayer perceptron (Multilayer Perceptron, MLP), etc.

[0105] Specifically, the duration prediction model is used to predict the duration of sample interaction data, and a predicted duration threshold is obtained. The target loss is calculated based on the predicted duration threshold and the label duration threshold corresponding to the sample interaction data, and then the preset classification network is iteratively trained for supervised learning based on the target loss until the preset classification network meets the training end condition, thereby obtaining a duration prediction model. In this way, the duration prediction model can be used to perform personalized duration predictions for the target audience, and a time threshold that matches the target audience's tolerance for their own delays is obtained, so that the target translation results pushed to the audience end are more accurate, and the real-time display of the target translation results better meets the personalized needs of the audience.

[0106] In other embodiments, a setting component of a duration threshold can be displayed at the audience end, and then the audience can set time thresholds of different durations according to their actual experience needs of watching live broadcasts when watching live broadcasts. For example, a setting button of a duration threshold is displayed on the live broadcast interface of the audience end, and when the audience clicks the setting button, a sliding bar for adjusting the size of the duration threshold is displayed, and the audience can adjust the duration threshold required for the audience live broadcast by sliding the sliding bar. In this way, by allowing the audience to directly set the duration threshold by themselves, the real-time display of the target translation result can be more in line with the current viewing needs of the audience, so that the target translation result pushed to the audience end is more accurate relative to the target audience, and the efficiency of adjusting the real-time and accuracy of the live translation is also improved.

[0107] Step S140: based on the target translation result, re-encode the target live stream to be pushed to obtain a re-encoded live stream.

[0108] When translating the candidate live stream, the candidate live stream has been decoded to obtain the corresponding audio and video stream and other decoding information. After adding the target translation result obtained by translation to the audio and video stream, an audio and video stream with the translated content is obtained. To this end, the audio and video stream with the translated content can be re-encoded to obtain a new audio and video stream for streaming.

[0109] As an implementation method, the target translation result can be added to the auxiliary enhancement frame as custom data of the frame corresponding to the additional enhancement information (i.e., the auxiliary enhancement frame), and then the auxiliary enhancement frame is inserted into the video stream of the target live stream to be pushed before the I frame or P frame, and then encoded into a new audio and video stream together with the target live stream, that is, the re-encoded live stream.

[0110] Step S150: Push the re-encoded live stream to display the translation content in the target translation result on the viewer side.

[0111] It can be understood that the re-encoded live stream includes the video stream and the audio stream in the target live stream, as well as the target translation result. After the re-encoded live stream is pushed to the audience end, the audience end can display the screen based on the video stream in the target live stream, and display the translated content in the target translation result as subtitles on the live screen, and play the corresponding audio stream synchronously while playing the video stream.

[0112] In some embodiments, the target translation result may also include speech recognition content corresponding to the live stream, so that both the speech recognition content and the translation content may be displayed as subtitles on the live screen at the viewer end.

[0113] In some embodiments, since the delay time of different target live streams may be different, if the re-encoded live stream is directly pushed after being re-encoded, the push may be discontinuous, that is, there may be no re-encoded live stream that can be pushed in some time periods. For example, taking a candidate live stream with a duration of 30 seconds as an example, if a stable translation content is obtained by translating the first 3 seconds and inserted into the audio and video stream, then another stable translation content can only be obtained after the end time of the candidate live stream.

[0114] If the live streams sent to the audience are all re-encoded live streams corresponding to the stable translation content, then after the re-encoded live streams of the stable translation content of the first 3 seconds are sent, there will be no live streams of the translated content to be sent for a period of time, which will cause the audience to experience screen freezes and affect the audience's viewing of the live broadcast. To this end, in some embodiments of the application, the re-encoded live stream can be cached and then pushed.

[0115] As an implementation method, if the time difference between the current time and the end time corresponding to the re-encoded live stream reaches a duration threshold, the target live stream is pushed. Specifically, the target live stream can be stored in a re-encoding cache pool. If the time difference between the current time and the end time corresponding to the re-encoded live stream reaches a duration threshold, the audio and video streams stored in the re-encoding cache pool can be pushed to the audience in real time. Among them, the end time corresponding to the re-encoded live stream is the end time of the target live stream corresponding to the re-encoded live stream. In this embodiment, it can be ensured that the stream is pushed while the re-encoded live stream is cached for a certain length of time, thereby ensuring the continuity and stability of the push stream.

[0116] For example, taking a candidate live stream with a duration of 30 seconds as an example, when the first 3 seconds of the stable translated content are re-encoded as a live stream, the re-encoded live stream is stored in the re-encoding cache pool, and the re-encoded live stream continues to be accumulated in the re-encoding cache pool until the time difference between the current time and the end time corresponding to the re-encoded live stream reaches the duration threshold, that is, the re-encoding cache pool has accumulated live streams with a duration of at least the duration threshold, and then the live stream stored in the re-encoding cache pool is pushed to the audience end in real time.

[0117] In this way, by setting a specific duration (that is, reaching the duration threshold) that a phased translation result must be given, and then by setting a certain duration (that is, reaching the duration threshold) of delayed sending, it is ensured that the re-encoding cache pool always has new re-encoded live streams replenished in real time, so as to achieve continuous streaming and avoid the situation where the live broadcast screen on the audience end is stuck due to discontinuous streaming.

[0118] For example, see Figure 5 , Figure 5A flow chart of live translation is shown. Figure 4 As shown, the live translation process may include a code stream cache pool, a translation selector, a translator, a controller, an uploader, and a re-encoding cache pool. Among them, the code stream cache pool can be used to cache the live stream pushed by the anchor end. The translation selector can be used to select a candidate live stream. The translator can be used to translate the candidate live stream. The controller can be used to determine the target translation result of the stream to be pushed. The re-encoding cache pool can cache the re-encoded live stream waiting to be pushed to the audience end.

[0119] As an implementation method, when the anchor initiates a live broadcast, the host can push the collected audio and video data to the server in real time after encoding, and then the server can cache the received encoded live stream into the code stream cache pool, and then the server can obtain a live stream from the code stream cache pool with the target timestamp as the starting time as the candidate live stream through the translation selector. Then the server can translate the speech recognition content corresponding to the candidate live stream through the translator and output the translation result corresponding to the candidate live stream.

[0120] Furthermore, the server can perform semantic analysis on the translation results corresponding to the candidate live streams through the controller, and then determine whether there is stable translation content in the translation results or whether the end time corresponding to the candidate live stream is equal to the reference time based on the semantic analysis results. If the end time corresponding to the candidate live stream does not exceed the reference time, and the semantic analysis result corresponding to the translation result indicates that there is stable translation content, the translation result corresponding to the candidate live stream is used as the target translation result of the stream to be pushed. If the end time corresponding to the candidate live stream is equal to the reference time, the translation result corresponding to the candidate live stream is determined as the target translation result of the stream to be pushed. If the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result indicates that there is no stable translation content, the translation selector obtains a new candidate live stream from the cached live stream based on the target timestamp, and continues to execute the translation processing steps.

[0121] Furthermore, the server can re-encode the target live stream to be pushed based on the target translation result. Specifically, the server can generate a corresponding auxiliary enhancement frame based on the target translation result, and insert the auxiliary enhancement frame into the video stream of the target live stream to be pushed, and then mix it with the audio stream and encode it into a re-encoded live stream. The server can cache the re-encoded live stream in the re-encoding cache pool. Furthermore, the server can upload the re-encoded live stream in the re-encoding cache pool to the content distribution network through the uploader when it is determined that the duration of the re-encoded live stream in the re-encoding cache pool is not less than the duration threshold. Furthermore, the audience can pull the stream from the content distribution network to obtain the re-encoded live stream, so as to decode and play the re-encoded live stream.

[0122] In this embodiment, a candidate live stream can be obtained from the cached live stream, and the candidate live stream refers to a live stream whose start time is the target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation content. Then, the speech recognition content corresponding to the candidate live stream is translated to obtain the translation result corresponding to the candidate live stream, and based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed, the target translation result of the stream to be pushed is determined.

[0123] Among them, the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold. Based on the target translation result, the target live stream to be pushed is re-encoded to obtain the re-encoded live stream, and the re-encoded live stream is pushed to display the translation content in the target translation result on the audience side. Therefore, under the condition that the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold, the target translation result to be pushed can be determined based on the translation result corresponding to the candidate live stream within a fixed duration and pushed first, so as to avoid the long waiting time for translation caused by the long translation of the complete sentence, and improve the real-time performance of the live translation. The translation result can be displayed synchronously with the video screen where the speaker is located.

[0124] See also Figure 6 , Figure 6 A flow chart of a live broadcast translation method provided by another embodiment of the present application is shown. In this embodiment, the live broadcast translation method can be executed by a server of a live broadcast service provider. Figure 6 The live translation method shown can be applied to Figure 7 The live translation process is shown. Figure 7 The live broadcast translation process shown includes the host pushing the stream to the server, which translates the live broadcast stream. The server uploads the translated code stream with subtitles to the CDN. The audience pulls the stream to the CDN for decoding, obtains the audio, video and subtitle content, and displays the subtitles on the screen while playing the audio and video. At the same time, the audience can also adjust the subtitles. Specifically, the live broadcast translation method may include the following steps:

[0125] Step S210: Obtain a candidate live stream from the cached live stream; the candidate live stream refers to a live stream whose start time is the target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation content.

[0126] Step S220: Translate the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream.

[0127] Step S230: Determine the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold.

[0128] Specifically, steps S210 and S230 may refer to the contents of step S110 and step S130 in the aforementioned embodiment, which will not be described in detail here.

[0129] Step S240: generating an auxiliary enhanced frame based on the target translation result.

[0130] Among them, the auxiliary enhancement frame refers to the additional enhancement information (SEI) frame. The additional enhancement information frame is a type of network abstraction layer data unit (NALU) in the audio and video coding stream that encapsulates auxiliary information into a specific type. It is also a data unit in video coding that meets network transmission. For example, in the H.264 code stream, images are organized in sequences. A sequence is a data stream after multiple frames of images are encoded, starting with an I frame and ending with the next I frame. A frame of an image can be divided into one or more slices (Slice), and a slice is composed of macroblocks (Macroblock). Macroblocks are the basic units of encoding processing. After the slice is encoded, it will be packaged into a NALU, that is, one frame of an image corresponds to one NALU.

[0131] See also Figure 8 , Figure 8 FIG. 1 is a schematic diagram showing a code stream format of a network abstraction layer data unit. Figure 8 As shown, the network abstraction layer data unit includes a start code (Start Code), a unit header (NALU Header) and a unit body (NALUPayload). Among them, the Start Code is used to separate two adjacent network abstraction layer data units and must be "0X 0000 00 01" or "00 00 01". The NALU Header is used to indicate the type of data it contains and other information.

[0132] For example, if the NALU Header is 0X06, the NALU Payload is filled in according to the data format requirements of SEI encoding. The NALU Header occupies 1 byte, and the detection of the sequence parameter set (SPS), picture parameter set (PPS), and I / P / B frames in H.264 is achieved through the NALU type in the NALU Header.

[0133] As an implementation method, the custom data field corresponding to the additional enhancement information can be determined based on the data encoding format. Then, the target translation result is added to the custom data field to obtain an auxiliary enhancement frame. For example, see Fig. 9 , Fig. 9 A schematic diagram showing a data format of additional enhancement information.

[0134] The header data of the additional enhancement information (NRI) is 0X06, the body type Payload type is 0X05, which means it complies with the H264 standard format, UUID is the business custom identification code, the custom data length is calculated from the custom data, the custom data is specified by the business party, and the end is aligned with 0X80. The target translation result is added to the field of the custom data in the SEI data format to obtain the auxiliary enhancement frame. After the viewer pulls the stream, it decodes according to the SEI data format to obtain the translated content for displaying the translated subtitles.

[0135] As another implementation method, the speech recognition content of the live stream corresponding to the target translation result is obtained, and the speech recognition content and the target translation result are added to the custom data field to obtain an auxiliary enhancement frame. In this way, after the viewer pulls the stream, it decodes it according to the SEI data format to obtain the translation content and the speech recognition content corresponding to the live stream for bilingual subtitle display.

[0136] Step S250: Encode the auxiliary enhanced frame into the target live stream to be pushed to obtain a re-encoded live stream.

[0137] As an implementation method, the target live stream to be pushed is decoded, the audio stream and the video stream in the target live stream to be pushed are obtained, and the auxiliary enhancement frame is inserted before the specified video frame in the video stream to obtain a reference video stream. Further, the audio stream and the reference video stream are re-encoded to obtain a re-encoded live stream.

[0138] The designated video frame is a key frame (I frame) or a forward predictive coding frame (P frame) in the video stream. Considering the requirements of the coding protocol, the auxiliary enhancement frame is inserted at the leading position of the I frame / P frame during encoding, which will not affect the decoding order of the re-encoded live stream.

[0139] Step S260: Push the re-encoded live stream to display the translation content in the target translation result on the viewer side.

[0140] As an implementation method, when the server pushes the re-encoded live stream, if it is determined that there is translation content that needs to be adjusted, it can send a translation adjustment instruction to the viewer, and the translation adjustment instruction controls the viewer to adjust the previously sent re-encoded live stream based on the newly sent re-encoded live stream. For example, for a candidate live stream, the i-th translation result in the previous 10s is Ⅰ{X1X3}, and the i+1-th translation result is Ⅱ{X1X2X3X4X5}. For this purpose, it is necessary to generate a translation adjustment instruction and send it to the viewer, so that the viewer replaces X3 in the second position of the translation result Ⅰ{X1X3} with X2X3, and fills X4X5 after X2.

[0141] In this way, the translated results can be adjusted again, which can improve the real-time performance of the live translation and the accuracy of the translation results. Specifically, step S260 can refer to the content of step S150 in the above embodiment, which will not be repeated here.

[0142] In this embodiment, a candidate live stream can be obtained from the cached live stream, wherein the candidate live stream refers to a live stream whose start time is the target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation content. The speech recognition content corresponding to the candidate live stream is translated and processed to obtain the translation result corresponding to the candidate live stream. Then, based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed, the target translation result of the stream to be pushed is determined; the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold.

[0143] Furthermore, an auxiliary enhancement frame can be generated based on the target translation result, including determining the custom data field corresponding to the additional enhancement information based on the data encoding format, and adding the target translation result to the custom data field to obtain an auxiliary enhancement frame. Then, the auxiliary enhancement frame is encoded into the target live stream of the stream to be pushed, and the re-encoded live stream is pushed to display the translation content in the target translation result at the audience end. Therefore, under the condition that the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold, the target translation result of the stream to be pushed can be determined based on the translation result corresponding to the candidate live stream within a fixed duration and pushed first, thereby improving the real-time performance of the live translation. At the same time, during encoding, the auxiliary enhancement frame is inserted in the leading position of the key frame or the forward prediction coding frame to avoid affecting the order in which the audience end decodes the audio and video stream. Ensure that the translation result and the video screen can be displayed synchronously.

[0144] See also Fig.10, which shows a structural block diagram of a live translation device 300 provided by an embodiment of the present application. The live translation device 300 includes: a live stream acquisition module 310, which is used to obtain a candidate live stream from a cached live stream; the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation content; a translation processing module 320, which is used to translate the speech recognition content corresponding to the candidate live stream to obtain the translation result corresponding to the candidate live stream; a result determination module 330, which is used to determine the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold; a re-encoding module 340, which is used to re-encode the target live stream to be pushed based on the target translation result to obtain a re-encoded live stream; a push stream module 350, which is used to push the re-encoded live stream to display the translation content in the target translation result on the audience end.

[0145] In some embodiments, the result determination module 330 may include: a semantic analysis unit and a first determination unit. The semantic analysis unit is used to perform semantic analysis on the translation result corresponding to the candidate live stream to obtain a semantic analysis result corresponding to the translation result; the semantic analysis result is used to indicate whether there is stable translation content in the translation result; the first determination unit is used to use the translation result corresponding to the candidate live stream as the target translation result of the stream to be pushed if the end time corresponding to the candidate live stream is less than the reference time and the semantic analysis result corresponding to the translation result indicates the existence of stable translation content; the reference time is equal to the sum of the target end timestamp and the duration threshold.

[0146] In some embodiments, the result determination module 330 may further include a second determination unit. The second determination unit is configured to determine the translation result corresponding to the candidate live stream as the target translation result of the stream to be pushed if the end time corresponding to the candidate live stream is equal to the reference time.

[0147] In some embodiments, the result determination module 330 may further include a return execution unit. The return execution unit is used to obtain a new candidate live stream from the cached live stream based on the target timestamp if the end time corresponding to the candidate live stream is less than the reference time and the semantic analysis result corresponding to the translation result indicates that there is no stable translation content, and return to the step of performing translation processing on the speech recognition content corresponding to the candidate live stream to obtain the translation result corresponding to the candidate live stream.

[0148] In some embodiments, the result determination module 330 may further include a timestamp determination unit. The timestamp determination unit is configured to store the end time of the candidate live stream as a new target timestamp.

[0149] In some embodiments, the re-encoding module 340 may include: an enhanced frame generation unit and a re-encoding unit. The enhanced frame generation unit is used to generate an auxiliary enhanced frame based on the target translation result; the re-encoding unit is used to encode the auxiliary enhanced frame into the target live stream of the stream to be pushed to obtain a re-encoded live stream.

[0150] In some embodiments, the enhanced frame generation unit may include: a determination subunit and an addition subunit. The determination subunit is used to determine the custom data field corresponding to the additional enhancement information based on the data encoding format; the addition subunit is used to add the target translation result to the custom data field to obtain an auxiliary enhanced frame.

[0151] In some embodiments, the adding subunit may be specifically used to: obtain the speech recognition content of the live stream corresponding to the target translation result; and add the speech recognition content and the target translation result to the custom data field to obtain an auxiliary enhanced frame.

[0152] In some embodiments, the re-encoding unit can be specifically used to: decode the target live stream of the stream to be pushed, obtain the audio stream and video stream in the target live stream of the stream to be pushed; insert the auxiliary enhancement frame before the specified video frame in the video stream to obtain a reference video stream; the specified video frame refers to a key frame or a forward prediction coding frame in the video stream; re-encode the audio stream and the reference video stream to obtain a re-encoded live stream.

[0153] In some embodiments, the streaming module 350 may be specifically used to: if the time difference between the current time and the end time corresponding to the re-encoded live stream reaches the duration threshold, the target live stream is streamed.

[0154] In some embodiments, the translation processing module 320 can be specifically used for: The translation processing module can be specifically used for: translating the speech recognition content corresponding to the candidate live stream to obtain the translation content corresponding to the candidate live stream; combining the translation content corresponding to the candidate live stream, the end time corresponding to the speech recognition content corresponding to the candidate live stream, and the target timestamp to obtain the translation result corresponding to the candidate live stream.

[0155] In some embodiments, the live broadcast translation device 300 may further include: an application information acquisition module and a duration threshold determination module. The application information acquisition module is used to acquire scenario application information, wherein the scenario application information indicates a target live broadcast interaction level; and the duration threshold determination module is used to determine the duration corresponding to the target live broadcast interaction level as the duration threshold based on the corresponding relationship between the live broadcast interaction level and the duration.

[0156] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0157] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0158] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0159] The solution provided by the present application can obtain a candidate live stream from a cached live stream, wherein the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation content. Then, the speech recognition content corresponding to the candidate live stream is translated to obtain a translation result corresponding to the candidate live stream, and based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed, the target translation result of the stream to be pushed is determined, wherein the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed the duration threshold.

[0160] Then, based on the target translation result, the target live stream to be pushed is re-encoded to obtain the re-encoded live stream, and the re-encoded live stream is pushed to display the translation content in the target translation result on the audience side. In this way, by limiting the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp to not exceed the duration threshold, the target translation result to be pushed can be determined based on the translation result corresponding to the candidate live stream, so that the target translation result can be obtained for pushing within the time period corresponding to the duration threshold, thereby improving the real-time performance of the live translation, and enabling the translation result to be displayed synchronously with the video screen where the speaker is located.

[0161] like Fig.11As shown, the embodiment of the present application also provides a computer device 400, which includes a processor 410, a memory 420, a power supply 430 and an input unit 440. The memory 420 stores computer program instructions. When the computer program instructions are called by the processor 410, the various method steps provided in the above embodiments can be implemented. Those skilled in the art can understand that the structure of the computer device shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0162] The processor 410 may include one or more processing cores. The processor 410 uses various interfaces and lines to connect various parts of the entire battery management system, and by running or executing instructions, programs, code sets or instruction sets stored in the memory 420, calls the data stored in the memory 420, executes various functions of the battery management system and processes data, and executes various functions and processes data of the computer device, thereby controlling the computer device as a whole. Optionally, the processor 410 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 410 can integrate one or more combinations of a central processing unit 410 (CPU), a graphics processing unit 410 (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 410, but may be implemented separately through a communication chip.

[0163] The memory 420 may include a random access memory 420 (Random Access Memory, RAM), and may also include a read-only memory 420 (Read-Only Memory). The memory 420 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 420 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the various method embodiments described below, etc. The data storage area may also store data (such as a phone book and audio and video data) created by the computer device during use. Accordingly, the memory 420 may also include a memory controller to provide the processor 410 with access to the memory 420.

[0164] The power supply 430 can be logically connected to the processor 410 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 430 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0165] The input unit 440 may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0166] Although not shown, the computer device 400 may also include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 410 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 420 according to the following instructions, and the processor 410 will run the application programs stored in the memory 420, thereby implementing the various method steps provided in the aforementioned embodiments.

[0167] like Fig.12 As shown, an embodiment of the present application further provides a computer-readable storage medium 500, in which computer program instructions 510 are stored. The computer program instructions 510 can be called by a processor to execute the method described in the above embodiment.

[0168] The computer readable storage medium can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium includes a non-volatile computer readable storage medium (Non-Transitory Computer-Readable Storage Medium). The computer readable storage medium 500 has a storage space for program codes that execute any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0169] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations provided in the above embodiments.

[0170] The above are only preferred embodiments of the present application, and are not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technical personnel in the field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present application. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A live broadcast translation method, characterized in that: The method comprises: Obtaining a candidate live stream from the cached live stream; the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation content; Translate the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream; Determine a target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a duration threshold; Based on the target translation result, re-encoding the target live stream to be pushed to obtain a re-encoded live stream; The re-encoded live stream is pushed to display the translation content in the target translation result on the viewer side.

2. The method according to claim 1, characterized in that The determining the target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed includes: Performing semantic analysis on the translation result corresponding to the candidate live stream to obtain a semantic analysis result corresponding to the translation result; the semantic analysis result is used to indicate whether there is stable translation content in the translation result; If the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result indicates that there is stable translation content, the translation result corresponding to the candidate live stream is used as the target translation result of the stream to be pushed; the reference time is equal to the sum of the target end timestamp and the duration threshold.

3. The method according to claim 2, characterized in that The step of determining a target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed further includes: If the end time corresponding to the candidate live stream is equal to the reference time, the translation result corresponding to the candidate live stream is determined as the target translation result of the stream to be pushed.

4. The method according to claim 2, characterized in that: The method further comprises: If the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result indicates that there is no stable translation content, a new candidate live stream is obtained from the cached live stream based on the target timestamp, and the step of translating the speech recognition content corresponding to the candidate live stream is returned to obtain the translation result corresponding to the candidate live stream.

5. The method according to claim 2, characterized in that: If the end time corresponding to the candidate live stream is less than the reference time, and the semantic analysis result corresponding to the translation result indicates that there is stable translation content, after taking the translation result corresponding to the candidate live stream as the target translation result of the stream to be pushed, the method further includes: The end time of the candidate live stream is stored as a new target timestamp.

6. The method according to any one of claims 1 to 5, characterized in that: The step of re-encoding the target live stream to be pushed based on the target translation result to obtain the re-encoded live stream comprises: Based on the target translation result, generating an auxiliary enhanced frame; The auxiliary enhanced frame is encoded into the target live stream of the stream to be pushed to obtain a re-encoded live stream.

7. The method according to claim 6, characterized in that The generating of the auxiliary enhanced frame based on the target translation result includes: Based on the data encoding format, determine the custom data field corresponding to the additional enhanced information; The target translation result is added to the custom data field to obtain an auxiliary enhanced frame.

8. The method according to claim 7, characterized in that The step of adding the target translation result to the custom data field to obtain an auxiliary enhanced frame includes: Obtaining the speech recognition content of the live stream corresponding to the target translation result; The speech recognition content and the target translation result are added to the custom data field to obtain an auxiliary enhanced frame.

9. The method according to claim 6 or 8, characterized in that: The step of encoding the auxiliary enhanced frame into the target live stream of the stream to be pushed to obtain a re-encoded live stream includes: Decoding the target live stream of the stream to be pushed, and obtaining the audio stream and the video stream in the target live stream of the stream to be pushed; Inserting the auxiliary enhancement frame before a specified video frame in the video stream to obtain a reference video stream; the specified video frame refers to a key frame or a forward prediction coding frame in the video stream; The audio stream and the reference video stream are re-encoded to obtain a re-encoded live stream.

10. The method according to claim 1, characterized in that The pushing of the re-encoded live stream includes: If the time difference between the current time and the end time corresponding to the re-encoded live stream reaches the duration threshold, the target live stream is pushed.

11. The method according to claim 1, characterized in that: The translating process of the speech recognition content corresponding to the candidate live stream to obtain the translation result corresponding to the candidate live stream includes: Translate the speech recognition content corresponding to the candidate live stream to obtain the translation content corresponding to the candidate live stream; The translation content corresponding to the candidate live stream, the end time corresponding to the candidate live stream, and the target timestamp are combined to obtain a translation result corresponding to the candidate live stream.

12. The method according to claim 1, characterized in that The method further comprises: Acquire scenario application information, where the scenario application information indicates a target live broadcast interaction level; Based on the correspondence between the live broadcast interaction level and the duration, the duration corresponding to the target live broadcast interaction level is determined as the duration threshold.

13. A live translation device, characterized in that: The device comprises: A live stream acquisition module, used to acquire a candidate live stream from the cached live stream; the candidate live stream refers to a live stream whose start time is a target timestamp and whose end time is after the target timestamp; the target timestamp refers to the end time of the live stream corresponding to the last stable translation content; A translation processing module, used to translate the speech recognition content corresponding to the candidate live stream to obtain a translation result corresponding to the candidate live stream; A result determination module is used to determine a target translation result of the stream to be pushed based on the translation result corresponding to the candidate live stream and the target end timestamp of the target live stream to be pushed; the time difference between the end time of the live stream corresponding to the target translation result and the target end timestamp does not exceed a duration threshold; A re-encoding module, used for re-encoding the target live stream to be pushed based on the target translation result to obtain a re-encoded live stream; The streaming module is used to stream the re-encoded live stream to display the translation content in the target translation result on the audience end.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 12.

15. A computer device, characterized in that: include: Memory; One or more processors coupled to the memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to execute the method according to any one of claims 1 to 12.

16. A computer program product or a computer program, characterized in that The computer program product or computer program includes computer instructions, which are stored in a storage medium; a processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions, so that the computer executes the method according to any one of claims 1 to 12.

Images