Audio broadcasting method, device and system, electronic equipment and storage medium

Abstract

The invention provides an audio broadcasting method, device and system, electronic equipment and a storage medium, and the method comprises the steps: carrying out the refreshing of a current service push information according to a push time node, a push time length and a current push time node of a previous synthesized audio frame when the previous synthesized audio frame sent by a synthesis engine is received; according to the phoneme chain and the current service push information, refreshing a synthesis speed of a synthesis engine to obtain a current synthesis speed; according to the phoneme chain and the current service push information, performing mute fragment configuration on the current synthesis audio frame synthesized according to the current synthesis speed to obtain an audio frame to be broadcasted; and pushing the audio frame to be broadcasted to an audio processing end for audio broadcasting. According to the invention, the dynamic adjustment of the phoneme-level synthesis speed and the dynamic configuration of the mute segment are finely and intelligently carried out, so that the overall fluency and real-time performance of audio broadcasting are improved, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to an audio broadcasting method, device, system, electronic equipment and storage medium. Background Art

[0002] In real-time audio broadcasting scenarios, the audio data transmission rate is affected by many factors, which can cause uneven arrival times of data frames at the front end, leading to pauses, freezes, backlogs, and other issues during audio broadcasting, resulting in poor user experience. Therefore, how to perform high-quality audio broadcasting to improve user experience is an important topic that needs to be studied urgently.

[0003] In the related art, a fixed playback rate is usually used or a larger playback buffer is set to store the transmitted data frames to optimize the audio broadcast strategy. However, when there is network delay or uneven data frame transmission, it is impossible to dynamically adjust the audio broadcast strategy, resulting in pauses or discontinuities in audio playback. Summary of the invention

[0004] The present invention provides an audio broadcast method, device, system, electronic device and storage medium, which are used to solve the defect in the prior art that the playback strategy cannot be adjusted when the network is delayed or the data frame transmission is uneven, resulting in pauses or discontinuities in the audio playback process, so as to improve the global fluency and real-time performance of the audio broadcast.

[0005] The present invention provides an audio broadcasting method, comprising: When receiving the last synthesized audio frame sent by the synthesis engine, refreshing the service push information according to the push time node and push time length of the last synthesized audio frame and the current push time node to obtain the current service push information; According to the phoneme chain and the current service push information, the synthesis speed of the synthesis engine is refreshed to obtain the current synthesis speed; the phoneme chain is constructed according to the phoneme text information synthesized and returned by the synthesis engine; According to the phoneme chain and the current service push information, a silent segment is configured for the current synthesized audio frame sent by the synthesis engine to obtain an audio frame to be broadcast; the current synthesized audio frame is synthesized by the synthesis engine according to the current synthesis speed; The audio frame to be broadcast is pushed to an audio processing end, and the audio processing end is used to broadcast the audio frame to be broadcast.

[0006] According to an audio broadcast method provided by the present invention, refreshing the service push information according to the push time node and push time length of the previous synthesized audio frame and the current push time node to obtain the current service push information includes: Summing the push time node and the push time length of the previous synthesized audio frame to obtain a first target time node; According to the comparison result between the current push time node and the first target time node, the service push information is refreshed to obtain the current service push information.

[0007] According to an audio broadcast method provided by the present invention, refreshing the service push information according to the comparison result between the current push time node and the first target time node to obtain the current service push information includes: When the current push time node is less than or equal to the first target time node, refreshing the service push information according to the first refresh strategy to obtain the current service push information; When the current push time node is greater than the first target time node, refreshing the service push information according to the second refresh strategy to obtain the current service push information; Among them, the first refresh strategy includes a strategy for keeping the push time node of the previous synthesized audio frame unchanged, and a strategy for updating the push time length of the previous synthesized audio frame to the sum of the push time length of the previous synthesized audio frame and the current push time length; the second refresh strategy includes a strategy for updating the push time node of the previous synthesized audio frame to the current push time node, and a strategy for updating the push time length of the previous synthesized audio frame to the current push time length.

[0008] According to an audio broadcast method provided by the present invention, the method further includes: The first target time node and the broadcast freeze time threshold are summed to obtain a second target time node; Summing the first target time node and the broadcast redundancy time threshold to obtain a third target time node; In the case where the current push time node is greater than the first target time node, if it is determined that the current push time node is less than or equal to the second target time node, determining that the second refresh strategy also includes a strategy of updating the broadcast jam identifier to be true; In the case where the current push time node is greater than the first target time node, if it is determined that the current push time node is greater than the third target time node, it is determined that the second refresh strategy also includes a strategy of updating the broadcast redundant identifier to be true.

[0009] According to an audio broadcasting method provided by the present invention, refreshing the synthesis speed of the synthesis engine according to the phoneme chain and the current service push information to obtain the current synthesis speed includes: According to the phoneme chain, the number of characters of the currently received synthesized audio frame is updated; Determine the number of characters of the currently sent unsynthesized audio frame according to the number of characters of the currently received synthesized audio frame; When it is determined that the number of characters of the currently sent unsynthesized audio frame is greater than or equal to a preset number, and when it is determined that the current broadcast state is a broadcast redundant state based on the current service push information, the synthesis speed of the synthesis engine is refreshed at a preset speed to obtain the current synthesis speed.

[0010] According to an audio broadcasting method provided by the present invention, the current synthesized audio frame sent by the synthesis engine is configured with a silent segment according to the phoneme chain and the current service push information to obtain an audio frame to be broadcasted, including: When it is determined that the number of characters of the currently sent unsynthesized audio frame is less than the preset number, and when it is determined that the current broadcast state is a broadcast blocking state according to the current service push information, according to the rhythmic pause identifier of each phoneme frame in the phoneme chain, a target phoneme frame having a rhythmic pause node is matched and obtained in the current synthesized audio frame; According to the rhythmic pause configuration position corresponding to the target phoneme frame, a silent segment is inserted into the current synthesized audio frame to obtain the audio frame to be broadcasted.

[0011] According to an audio broadcasting method provided by the present invention, the phoneme chain is constructed based on the following steps: Parsing the phoneme text information to obtain phoneme information of each word; splicing the phoneme information of each word in the phoneme text information in a manner of splicing different syllables of the same word to obtain a phoneme frame corresponding to each word; According to the sorting order of each word in the phoneme text information, a link is established between the phoneme frames corresponding to each word to obtain an initial phoneme chain; The pronunciation duration, synthesis time-consuming information and rhythmic pause identifier of the phoneme frame corresponding to each word are configured in the initial phoneme chain to obtain the phoneme chain.

[0012] According to an audio broadcast method provided by the present invention, the method further includes: In the case where the last synthesized audio frame is not received, obtaining the number of characters of the currently sent unsynthesized audio frame and the length of the currently pushed but unbroadcasted audio frame; Acquire the current broadcasting state according to the number of characters of the currently sent unsynthesized audio frame and the length of the currently pushed unbroadcasted audio frame; When it is determined that the current broadcast state is a broadcast redundant state, the synthesis speed of the synthesis engine is accelerated to obtain the current synthesis speed.

[0013] The present invention also provides an audio broadcasting device, comprising: A first refreshing unit is used to, when receiving a previous synthesized audio frame sent by the synthesis engine, refresh the service push information according to the push time node and push time length of the previous synthesized audio frame and the current push time node to obtain the current service push information; A second refreshing unit is used to refresh the synthesis speed of the synthesis engine according to the phoneme chain and the current service push information to obtain the current synthesis speed; the phoneme chain is constructed according to the phoneme text information synthesized and returned by the synthesis engine; An audio configuration unit, configured to perform a silent segment configuration on the current synthesized audio frame sent by the synthesis engine according to the phoneme chain and the current service push information, so as to obtain an audio frame to be broadcast; the current synthesized audio frame is synthesized by the synthesis engine according to the current synthesis speed; The audio broadcast unit is used to push the audio frame to be broadcast to the audio processing end, and the audio processing end is used to broadcast the audio frame to be broadcast.

[0014] The present invention also provides an audio broadcast system, comprising an audio processing terminal, a broadcast monitor and a synthesis engine; The broadcast monitor is connected to the audio processing end and the synthesis engine respectively, and is used to execute any of the audio broadcast methods described above.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned audio broadcast methods is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the audio broadcast method described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the audio broadcasting method described above is implemented.

[0018] The audio broadcast method, device, system, electronic device and storage medium provided by the present invention, after receiving the last synthesized audio frame sent by the synthesis engine, refresh the service push information in real time according to the push time node, push time length and current time node of the last synthesized audio frame, and according to the current service push information refreshed in real time and the phoneme chain constructed by the phoneme text information synthesized and returned by the synthesis engine, finely and intelligently perform dynamic adjustment of the synthesis speed at the phoneme level and dynamic configuration of the silent segments, so as to achieve more finely and intelligently perform dynamic optimization of the audio broadcast strategy, so as to more effectively cope with the adverse effects caused by network delay or uneven data frame transmission, thereby improving the overall fluency and real-time performance of the audio broadcast and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of the audio broadcasting method provided by the present invention.

[0021] Figure 2 It is a schematic diagram of the interaction of various components in the audio broadcasting system provided by the present invention.

[0022] Figure 3 It is a schematic diagram of the process of constructing the phoneme chain provided by the present invention.

[0023] Figure 4 This is one of the distribution diagrams of the audio broadcast time provided by the present invention.

[0024] Figure 5 This is the second distribution diagram of the audio broadcast time provided by the present invention.

[0025] Figure 6 This is the third distribution diagram of the audio broadcast time provided by the present invention.

[0026] Figure 7 This is the fourth distribution diagram of the audio broadcast time provided by the present invention.

[0027] Figure 8 It is a structural schematic diagram of the audio broadcasting device provided by the present invention.

[0028] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In the real-time audio broadcast scenario, the audio data transmission rate will be affected by many factors, such as network status, bandwidth fluctuations, data transmission delays, and the real-time processing speed of the synthesis engine. These factors may cause the data frames to arrive at the front end at an uneven time, causing pauses, freezes, backlogs and other problems during the audio broadcast process. This situation is particularly evident in high-demand real-time application scenarios, such as simultaneous voice interpretation and virtual voice assistants. These scenarios require continuous and stable audio output to ensure that users can receive information without obstacles. Therefore, it is urgent to provide an intelligent real-time audio broadcast method to cope with the adverse effects of data frame fluctuations, ensure smooth audio playback, no freezes and no backlogs, improve the quality of audio broadcasting, and enhance user experience.

[0031] Common audio broadcast systems mostly use a fixed playback rate. Regardless of how the data transmission situation changes, the audio output speed remains basically unchanged. Although this method is simple, when there is network delay or uneven data frame transmission, there will be pauses or discontinuities in the audio playback. In addition, to alleviate the problem of playback interruption, some systems set up a larger playback buffer to store the transmitted data frames, and read the content from the buffer when the data arrival is insufficient, thereby avoiding playback pauses. However, when there is network delay or uneven data frame transmission, this method cannot dynamically adjust the audio broadcast strategy in real time, and may still cause playback delays. Once the buffer is exhausted, pauses are still difficult to avoid. Therefore, it still has the problem of pauses or discontinuities in audio playback.

[0032] Although, some technical solutions attempt to dynamically adjust the audio playback rate. For example, the speech speed adjustment algorithm is used to accelerate the part with less transmission data to balance the problem of insufficient data; or when insufficient data arrives, silent segments are inserted to maintain the continuity of the audio. Although such solutions have improved the coherence of the broadcast to a certain extent, they usually cannot take into account the problems of unsmooth broadcast caused by data backlog and speech speed changes. In addition, the existing audio speed adjustment algorithms generally lack support for fine-grained adjustment at the phoneme level in practical applications, and it is difficult to accurately insert silence or accelerate adjustments on each phoneme chain, making it difficult to ensure global fluency.

[0033] In this regard, the present embodiment provides an audio broadcast method, which dynamically handles the backlog problem of content to be broadcast by predicting the relationship between the content to be broadcast and the current backlog; and based on the refined and intelligent dynamic adjustment of the synthesis speed at the phoneme level and the dynamic configuration of silent segments, so as to achieve the continuity and stability of audio content push, thereby improving the overall fluency and real-time performance of the audio broadcast.

[0034] Figure 1 The figure is a flow chart of the audio broadcast method provided by the present invention. The method can be applied to an audio broadcast system, which at least includes an audio processing terminal, a broadcast monitor and a synthesis engine.

[0035] Figure 2 is a schematic diagram of the interaction of various components in the audio broadcasting system provided by the present invention; Figure 2 As shown, the audio processing end is used to receive the voice data transmitted by the user, and convert or translate the voice data into text data, and transmit the text data as the data to be synthesized to the broadcast monitor; the broadcast monitor is used to receive the data to be synthesized transmitted by the upstream audio processing end, and by maintaining the phoneme chain and throughput content, judge in real time whether it is necessary to adjust the synthesis speed at present, so as to generate the corresponding synthesis speed, and send the synthesis speed and the data to be synthesized to the downstream synthesis engine, so that the downstream synthesis engine synthesizes the audio frame according to the synthesis speed and the data to be synthesized; in addition, the broadcast monitor is also used to receive the audio frames synthesized by the downstream synthesis engine, and by maintaining the phoneme chain and throughput content, make a backlog judgment in real time to determine whether it is necessary to insert rhythmic point silence at present, so as to control the push of the audio frame, and then push the synthesized audio frame to the audio processing end to realize the smooth broadcast of the audio frame, thereby realizing the dynamic optimization of the audio broadcast through the mutual cooperation of the various components in the audio broadcast system, so as to realize the coherence and stability of the audio content push, and then improve the overall fluency and real-time performance of the audio broadcast.

[0036] It should be noted that the method can be widely used in various audio broadcasting scenarios, including but not limited to voice simultaneous interpretation scenarios and virtual voice assistant scenarios, which are not specifically limited in this embodiment. The following describes the method provided in this embodiment using the voice simultaneous interpretation scenario as an example. For other scenarios, refer to this scenario to adaptively perform the audio broadcasting method.

[0037] like Figure 1 As shown, the method includes: step 110, step 120, step 130 and step 140.

[0038] Step 110, when the last synthesized audio frame sent by the synthesis engine is received, the service push information is refreshed according to the push time node and push time length of the last synthesized audio frame and the current push time node to obtain the current service push information.

[0039] Optionally, during the current audio broadcast process, the broadcast monitor may first determine whether the previous synthesized audio frame sent by the synthesis engine has been received. When the previous synthesized audio frame is received, the push time node P and push time length Dp of the previous synthesized audio frame are obtained according to the push record of the previous synthesized audio frame, and the current push time node N is loaded to dynamically determine the current broadcast status by combining the push time node P, push time length Dp and the current push time node N, and determine the corresponding service push information refresh strategy based on the current broadcast status, and dynamically refresh the service push information according to the corresponding service push information refresh strategy to obtain the current service push information, so as to dynamically adjust the synthesis speed based on the current service push information, thereby ensuring the continuity and stability of the audio content push.

[0040] The different broadcast states here correspond to different service push information refresh strategies, such as the first refresh strategy corresponding to the non-stuck broadcast state, the second refresh strategy corresponding to the stuck broadcast state, etc., wherein the first refresh strategy and the second refresh strategy contain different refresh strategies for the push time node P and the push time length Dp.

[0041] The method for obtaining the current broadcast status here includes performing multiple comparisons and judgments on the push time node and push time length of the previous synthetic audio frame, and the current push time node to obtain the current broadcast status; or, inputting the push time node and push time length of the previous synthetic audio frame, and the current push time node into a pre-trained recognition model, and the recognition model applies the push time node and push time length of the previous synthetic audio frame, and the current push time node to perform nonlinear calculations to identify the current broadcast status. The specific method can be determined according to actual needs, and this embodiment does not specifically limit this.

[0042] Step 120, refreshing the synthesis speed of the synthesis engine according to the phoneme chain and the current service push information to obtain the current synthesis speed; the phoneme chain is constructed according to the phoneme text information synthesized and returned by the synthesis engine.

[0043] The phoneme chain here is a chain data structure of multi-phoneme frames linked by parsing the phoneme text information returned by the synthesis engine. Each node can represent the pronunciation information of the synthesized word, including but not limited to pronunciation duration, synthesis time information and rhythmic pause identifier. The rhythmic pause identifier contains rhythmic pause types used to represent different types of rhythmic pauses, including but not limited to single-word pauses, word-level pauses (L1), short pauses (Sp in the sentence), complete sentence pauses (Sil), etc. Among them, phonemes refer to the index information of the synthesized audio returned by the synthesis engine, including the corresponding text of the synthesized audio and the pronunciation duration of each synthesized monosyllable.

[0044] Exemplarily, in some embodiments, the phoneme chain is constructed based on the following steps: Parsing the phoneme text information to obtain phoneme information of each word; splicing the phoneme information of each word in the phoneme text information in a manner of splicing different syllables of the same word to obtain a phoneme frame corresponding to each word; According to the sorting order of each word in the phoneme text information, a link is established between the phoneme frames corresponding to each word to obtain an initial phoneme chain; The pronunciation duration, synthesis time-consuming information and rhythmic pause identifier of the phoneme frame corresponding to each word are configured in the initial phoneme chain to obtain the phoneme chain.

[0045] Figure 3 Schematic diagram of the process of constructing the phoneme chain provided by the present invention. Figure 3 As shown, the specific construction process of the phoneme chain includes: initializing a queue; parsing the phoneme text information returned by the synthesis engine to obtain the phoneme information of each word; splicing the phoneme information of each word in the phoneme text information according to the splicing method of different syllables of the same word to obtain the phoneme frame (also called synthesized phoneme frame) corresponding to each word; establishing the link between the phoneme frames corresponding to each word according to the sorting order of each word in the phoneme text information to obtain the initial phoneme chain. And count the pronunciation time of each word to obtain the pronunciation time of the phoneme frame corresponding to each word; accumulate the total synthesis time information starting from the first frame to obtain the synthesis time information of the phoneme frame corresponding to each word; according to the rhythmic pause configuration information agreed upon by the synthesis, parse and obtain the identifiers allowed to be inserted as pauses and silences in the phoneme frame corresponding to each word, that is, the rhythmic pause identifier. Finally, by configuring the pronunciation duration, synthesis time information and rhythmic pause identifier of the phoneme frames corresponding to each word in the initial phoneme chain, the final complete phoneme chain can be obtained, so that the synthesis speed can be dynamically adjusted and the silent segments can be dynamically configured based on the phoneme chain. This will significantly improve the fluency and naturalness of the audio broadcast and ensure a smooth and uninterrupted broadcast process.

[0046] Optionally, after obtaining the current service push information, the phoneme chain constructed by combining the current service push information and the factor text information returned by the synthesis engine can be used to identify the current broadcast status and the number of characters in the unsynthesized audio frame, so as to dynamically refresh the synthesis speed of the synthesis engine based on the identified current broadcast status and the number of characters in the unsynthesized audio frame, so that the refreshed current synthesis speed matches the current broadcast speed, so as to improve the naturalness and fluency of the audio broadcast, and ensure the real-time nature of the audio broadcast, thereby enhancing the user experience.

[0047] Step 130, according to the phoneme chain and the current service push information, a silent segment is configured for the current synthesized audio frame sent by the synthesis engine to obtain an audio frame to be broadcast; the current synthesized audio frame is synthesized by the synthesis engine according to the current synthesis speed.

[0048] Optionally, after obtaining the current synthesis speed, the current data to be synthesized and the current synthesis speed received from the upstream audio processor can be transmitted to the synthesis engine, so that the synthesis engine can perform audio frame synthesis on the current data to be synthesized according to the current synthesis speed to obtain the current synthesized audio frame.

[0049] After receiving the current synthesized audio frame transmitted by the synthesis engine, the current broadcast status and the number of characters in the unsynthesized audio frame can be identified in combination with the current service push information and the phoneme chain, so as to determine whether it is necessary to insert a silent segment into the current synthesized audio frame based on the identified current broadcast status and the number of characters in the unsynthesized audio frame, and to configure the silent segment of the current synthesized audio frame sent by the synthesis engine according to the silent segment configuration strategy, so as to achieve fine control of the playback rhythm, so that the pause of the audio frame to be broadcast that needs to be broadcast is consistent with human voice habits, and avoid discordant playback caused by missing content, so as to improve the naturalness and fluency of the audio broadcast, ensure the real-time nature of the audio broadcast, and thus enhance the user experience. The silent segment configuration strategy here includes a strategy of adding silent segments or a strategy of not adding silent segments, and accordingly, the silent segment configuration includes adding silent segments or not adding silent segments.

[0050] Step 140: Push the audio frame to be broadcast to an audio processing end, and the audio processing end is used to broadcast the audio frame to be broadcast.

[0051] Optionally, after the audio frame to be broadcast is acquired, the audio frame to be broadcast can be pushed to the audio processing end, so that the audio processing end broadcasts the audio frame to be broadcast through the broadcast voice.

[0052] The method provided in this embodiment, after receiving the last synthesized audio frame sent by the synthesis engine, refreshes the service push information in real time according to the push time node, push time length and current time node of the last synthesized audio frame, and based on the current service push information refreshed in real time and the phoneme chain constructed by the phoneme text information synthesized and returned by the synthesis engine, performs dynamic adjustment of the synthesis speed at the phoneme level and dynamic configuration of the silent segments in a refined and intelligent manner, thereby achieving more refined and intelligent dynamic optimization of the audio broadcast strategy, so as to more effectively cope with the adverse effects brought about by network delays or uneven data frame transmission, thereby improving the overall fluency and real-time performance of the audio broadcast and enhancing the user experience.

[0053] In some embodiments, step 110 specifically includes: Summing the push time node and the push time length of the previous synthesized audio frame to obtain a first target time node; According to the comparison result between the current push time node and the first target time node, the service push information is refreshed to obtain the current service push information.

[0054] Optionally, during the service push information refresh process, the broadcast monitor may record two key information during the audio push process, namely, the push time node P and the push time length Dp of the previous synthesized audio frame. When the audio frame is pushed at the current push time node N, the service push information may be refreshed using the following refresh steps to obtain the current service push information: First, sum the push time node P and the push time length Dp to get the first target time node ,in, , this time node Represents the expected time point when the previous synthesized audio frame is fully played.

[0055] In addition, the current push time node N is obtained. And by comparing the current push time node N with the first target time node , to determine whether the current push time node is within the time window of the previous synthesized audio frame playback, or whether it has exceeded this time window.

[0056] Based on the comparison and judgment results, the current broadcast status is obtained, and whether the current audio broadcast is not stuck is determined according to the current broadcast status, and different service push information refresh strategies are determined according to whether the current audio broadcast is not stuck, and then the service push information is refreshed according to the corresponding service push information refresh strategy to obtain the current service push information. If the current audio broadcast is not stuck, the service push information refresh strategy is to keep part of the content in the previous service push information unchanged and refresh the other part; if the current audio broadcast is stuck, the service push information refresh strategy is to refresh all the content in the previous service push information to ensure that the subsequent audio broadcast can be closely connected to reduce stuck or delay.

[0057] The method provided in this embodiment sums the push time node and push time length of the previous synthesized audio frame and compares them with the current push time node, so as to be able to judge the current audio playback status in real time and accurately, and refresh the service push information accordingly, thereby ensuring that the audio broadcast can be closely connected and smooth, and effectively avoiding audio playback pauses or discontinuities caused by network delays or uneven data frame transmission.

[0058] In some embodiments, refreshing the service push information according to the comparison result between the current push time node and the first target time node to obtain the current service push information includes: When the current push time node is less than or equal to the first target time node, refreshing the service push information according to the first refresh strategy to obtain the current service push information; When the current push time node is greater than the first target time node, refreshing the service push information according to the second refresh strategy to obtain the current service push information; Among them, the first refresh strategy includes a strategy for keeping the push time node of the previous synthesized audio frame unchanged, and a strategy for updating the push time length of the previous synthesized audio frame to the sum of the push time length of the previous synthesized audio frame and the current push time length; the second refresh strategy includes a strategy for updating the push time node of the previous synthesized audio frame to the current push time node, and a strategy for updating the push time length of the previous synthesized audio frame to the current push time length.

[0059] Figure 4 This is one of the distribution diagrams of the audio broadcast time provided by the present invention; Figure 5 This is the second distribution diagram of the audio broadcast time provided by the present invention; Figure 6 This is the third distribution diagram of the audio broadcast time provided by the present invention; Figure 7This is the fourth distribution diagram of the audio broadcast time provided by the present invention.

[0060] Optionally, the current service push information refresh strategy further includes: like Figure 4 As shown, if the current push time node N is earlier than the first target time node , that is, N P+Dp, the current audio frame broadcast is not stuck, the current network status or data frame transmission is stable, and there is no need to adjust the time node. Therefore, the service push information is refreshed according to the first refresh strategy, specifically keeping the push time node of the previous synthesized audio frame. The push time length of the previous synthetic audio frame remains unchanged, and the push time length of the previous synthetic audio frame is updated to the sum of the push time length of the previous synthetic audio frame and the current push time length, that is, Dp=Dp+Dn. The distribution of the audio broadcast time in the current service push information updated accordingly is as follows: Figure 5 shown.

[0061] like Figure 6 As shown, if the current push time node N is later than the first target time node , that is, N P+Dp, the current audio frame broadcast has been stuck, and the playback of the previous synthesized audio frame has ended. It is necessary to calculate a new push time node and push time length from the current time, that is, it is necessary to synchronize the time node and the push time length. Therefore, at this time, the service push information is refreshed according to the second refresh strategy. Specifically, the push time node of the previous synthesized audio frame is updated to the current push time node, that is, P=N, and the push time length of the previous synthesized audio frame is updated to the current push time length, that is, Dp=Dn. The distribution of the audio broadcast time in the current service push information updated accordingly is as follows: Figure 7 The current push time node and the current push time length here are predicted and set based on factors such as the current network status, data frame size and playback speed.

[0062] The method provided in this embodiment dynamically refreshes the push time node and push time length in the service push information by adopting a corresponding refresh strategy based on the comparison result between the current push time node and the first target time node, thereby ensuring that the system accurately tracks the broadcast status in real time, so that in the subsequent broadcast process, the status of the audio broadcast can be judged more accurately, and the broadcast strategy can be adjusted accordingly, thereby ensuring that the audio broadcast can be closely connected and smooth, and effectively avoiding problems such as pauses, delays or jumps in audio playback.

[0063] In some embodiments, the method further comprises: Sum the first target time node and the broadcast lag time threshold to obtain a second target time node; Sum the first target time node and the broadcast redundancy time threshold to obtain a third target time node; When the current push time node is greater than the first target time node, if it is determined that the current push time node is less than or equal to the second target time node, it is determined that the second refresh policy further includes a policy of updating the broadcast lag identifier to true; When the current push time node is greater than the first target time node, if it is determined that the current push time node is greater than the third target time node, it is determined that the second refresh policy further includes a policy of updating the broadcast redundancy identifier to true.

[0064] The broadcast lag time threshold here is used to measure whether there is a lag situation during the broadcast; the broadcast redundancy time threshold is used to measure whether there is a broadcast redundancy situation during the broadcast. For the convenience of description, the broadcast lag time threshold is hereinafter defined as Ts (unit: millisecond ms), and the broadcast redundancy time threshold is defined as Tr (unit: millisecond ms).

[0065] Optionally, when it is determined that the current push time node is greater than the first target time node, the sum of the first target time node and the broadcast lag time threshold can be further calculated to obtain a second target time node P + Dp + Ts to determine whether there is a broadcast lag risk during the broadcast; and the sum of the first target time node and the broadcast redundancy time threshold is calculated to obtain a third target time node P + Dp + Tr to determine whether there is a broadcast redundancy risk during the broadcast.

[0066] Further, in order to update the broadcast status in real time, when it is determined that the current push time node is greater than the first target time node, in addition to updating the push time node of the previous synthesized audio frame to the current push time node and updating the push time length of the previous synthesized audio frame to the current push time length, it is also necessary to further determine whether the current push time node is less than or equal to the second target time node, or greater than the third target time node.

[0067] If it is further determined that the current push time node is less than or equal to the second target time node, that is, P + Dp < N <= P + Dp + Ts, it is determined that there is a broadcast lag risk during the broadcast. At this time, it is necessary to synchronously update the broadcast lag identifier maintained by the broadcast monitor to true (also called True).

[0068] If it is further determined that the current push time node is greater than the third target time node, that is, N>P+Dp+Tr, it is determined that there is a risk of broadcast redundancy in the broadcast process. At this time, it is necessary to synchronously update the broadcast redundancy identifier maintained by the broadcast monitor to true.

[0069] The method provided in this embodiment sets a broadcast freeze time threshold and a broadcast redundancy time threshold to monitor the front-end playback status when pushing audio frames, thereby realizing real-time identification of freeze and redundancy situations, so as to facilitate subsequent adjustment of the playback strategy and dynamic update of the playback status to ensure the continuity and stability of audio content push.

[0070] In some embodiments, step 130 specifically includes: According to the phoneme chain, the number of characters of the currently received synthesized audio frame is updated; Determine the number of characters of the currently sent unsynthesized audio frame according to the number of characters of the currently received synthesized audio frame; When it is determined that the number of characters of the currently sent unsynthesized audio frame is greater than or equal to a preset number, and when it is determined that the current broadcast state is a broadcast redundant state based on the current service push information, the synthesis speed of the synthesis engine is refreshed at a preset speed to obtain the current synthesis speed.

[0071] Optionally, after obtaining the phoneme chain, the currently received synthesized content can be refreshed according to the phoneme chain, thereby obtaining the number of characters of the currently received synthesized audio frame. The specific implementation logic includes: performing content matching according to the phoneme chain to determine the end of the corresponding character returned by the previous synthesized audio frame, so as to locate the position of the characters of the currently synthesized content, and then determining the currently received synthesized content according to the position of the characters of the currently synthesized content, and determining the number of characters of the currently received synthesized audio frame according to the number of characters contained in the currently received synthesized content. It should be noted that since there is no punctuation information in the phoneme information, a certain amount of redundancy can be designed when performing content matching. If the currently matched characters are correctly matched, backward retrieval is allowed; when the redundancy is reached and there is still no correct match, the character is ignored, and content matching is performed after the next audio frame is received.

[0072] After obtaining the number of characters of the currently received synthesized audio frame, the difference between the number of characters of the currently sent content to be synthesized and the number of characters of the currently received synthesized audio frame can be calculated to obtain the number of characters of the currently sent unsynthesized audio frame.

[0073] After obtaining the number of characters of the currently sent unsynthesized audio frame, the number of characters of the currently sent unsynthesized audio frame is compared with the preset number, and whether the current broadcast state is a broadcast redundant state is determined according to the current service push information. Here, the method of determining the current broadcast state according to the current service push information can be to determine whether the current broadcast state is a broadcast redundant state according to the update information of the push time node and the push time length in the current service push information, or to determine whether the current broadcast state is a broadcast redundant state according to the broadcast redundant identifier, which is specifically limited in this embodiment.

[0074] If it is determined that the number of characters in the currently sent unsynthesized audio frames is greater than or equal to the preset number, and the current broadcast state is a redundant broadcast state, it indicates that there is sufficient playback redundancy in the current front end and there is a lot of unsynthesized content. At this time, the synthesis parameters of the current queue of the synthesis engine can be refreshed through the specified synthesis speed, that is, the preset speed (accelerated speed) to obtain the current synthesis speed.

[0075] The method provided in this embodiment dynamically obtains the number of characters of the currently sent unsynthesized audio frame through the phoneme chain, and dynamically obtains the current broadcast status in real time through the current service push information, and dynamically refreshes the synthesis speed in combination with the number of characters of the currently sent unsynthesized audio frame and the current broadcast status, so as to realize more refined and intelligent dynamic optimization of the audio broadcast strategy, thereby improving the global fluency and real-time performance of the audio broadcast and enhancing the user experience.

[0076] In some embodiments, step 140 specifically includes: When it is determined that the number of characters of the currently sent unsynthesized audio frame is less than the preset number, and when it is determined that the current broadcast state is a broadcast blocking state according to the current service push information, according to the rhythmic pause identifier of each phoneme frame in the phoneme chain, a target phoneme frame having a rhythmic pause node is matched and obtained in the current synthesized audio frame; According to the rhythmic pause configuration position corresponding to the target phoneme frame, a silent segment is inserted into the current synthesized audio frame to obtain the audio frame to be broadcasted.

[0077] Optionally, after obtaining the number of characters of the currently sent unsynthesized audio frame based on the phoneme chain, the number of characters of the currently sent unsynthesized audio frame is compared with a preset number, and the current broadcast state is determined to be a broadcast blocking state based on the current service push information. The method of determining the current broadcast state based on the current service push information here can be to determine whether the current broadcast state is a broadcast blocking state based on the update information of the push time node and the push time length in the current service push information, or to determine whether the current broadcast state is a broadcast blocking state based on the broadcast redundancy identifier and the broadcast jam identifier, and this embodiment specifically defines this.

[0078] If it is determined that the number of characters of the currently sent unsynthesized audio frame is less than the preset number, and the current broadcast state is a broadcast congestion state, it indicates that the current front end is about to or has been blocked in playback and the unsynthesized content is insufficient. At this time, in order to ensure user experience, a silent segment needs to be inserted at an appropriate position to avoid broadcast interruption. The specific implementation steps are as follows: According to the rhythmic pause identifier of each phoneme frame in the phoneme chain, a target phoneme frame having a rhythmic pause node is matched and obtained in the current synthesized audio frame.

[0079] According to the rhythmic pause configuration position corresponding to the target phoneme frame, a silent segment of a preset length (the specific length can be adjusted according to user preference or system settings) is inserted into the corresponding position of the current synthesized audio frame to generate a new audio frame containing the silent segment, thereby obtaining the audio frame to be broadcast.

[0080] The method provided in this embodiment dynamically determines, based on the phoneme chain and the current service push information, that when the current front-end playback detects that playback congestion is about to occur or has occurred and there is insufficient synthesized content, the rhythmic pause node is dynamically determined through the phoneme chain to rationally insert silence, so that when playback is blocked, the playback content can be rationally segmented at the rhythmic pause nodes, thereby achieving fine control of the playback rhythm, to ensure the playback rhythm and naturalness of the synthesized audio, and to ensure that the pauses and playback rhythm of the synthesized audio are consistent with human voice habits, to avoid disharmonious playback and playback abruptness caused by missing content, thereby improving the user experience.

[0081] In some embodiments, the method further comprises: In the case where the last synthesized audio frame is not received, obtaining the number of characters of the currently sent unsynthesized audio frame and the length of the currently pushed but unbroadcasted audio frame; Acquire the current broadcasting state according to the number of characters of the currently sent unsynthesized audio frame and the length of the currently pushed unbroadcasted audio frame; When it is determined that the current broadcast state is a broadcast redundant state, the synthesis speed of the synthesis engine is accelerated to obtain the current synthesis speed.

[0082] Optionally, in the broadcast process, if the last synthesized audio frame is not received, that is, before the synthesized frame is delivered, then according to the interactive information between the broadcast monitor and the synthesis engine, the number of characters of the currently sent unsynthesized audio frame is determined, and according to the interactive information between the broadcast monitor and the audio processor, the length of the currently pushed unreported audio frame is determined. Thus, the number of characters of the currently sent unsynthesized audio frame and the length of the currently pushed unreported audio frame are combined to determine whether the current broadcast state is a broadcast redundant state. For example, the number of characters of the currently sent unsynthesized audio frame is compared with a quantity threshold, the length of the currently pushed unreported audio frame is compared with a length threshold, and when the number of characters of the currently sent unsynthesized audio frame exceeds the quantity threshold, and the length of the currently pushed unreported audio frame exceeds the length threshold, then the current broadcast state is determined to be a broadcast redundant state.

[0083] When the current broadcast state is known to be a broadcast redundant state according to the above judgment result, synthesis acceleration is triggered to accelerate the synthesis speed of the synthesis engine, thereby effectively reducing the playback waiting time and ensuring the continuity and smoothness of the broadcast process. The specific way of acceleration here can be to adjust the internal parameters of the synthesis engine, such as increasing the number of parallel processing threads, optimizing algorithms, etc., to improve synthesis efficiency.

[0084] The following describes the audio broadcast method provided in this embodiment by taking the voice simultaneous interpretation scenario as an example.

[0085] like Figure 2 As shown, in the scenario of simultaneous voice interpretation, the specific steps of the audio broadcast method include: The broadcast monitor receives the data to be synthesized sent by the upstream audio processing end, such as the deterministic result of translation; the broadcast monitor controls the synthesis speed, and sends the synthesis speed and the data to be synthesized to the downstream synthesis engine; the broadcast monitor receives the audio frames synthesized and returned by the downstream synthesis engine; the broadcast monitor performs backlog judgment to control the push of audio frames, that is, sends the audio frames to be broadcast to the audio processing end for audio broadcast.

[0086] Among them, before receiving the audio frame synthesized and returned by the synthesis engine, the broadcast monitor can determine whether the current broadcast state is a broadcast redundant state by combining the number of characters of the currently sent unsynthesized audio frame and the length of the currently pushed unbroadcast audio frame. If so, synthesis acceleration is triggered to speed up the synthesis speed.

[0087] When receiving the audio frame synthesized and returned by the synthesis engine, the main logic executed by the broadcast monitor includes: refreshing the current service push information, building a phoneme chain, and refreshing the currently received synthesis content.

[0088] For the steps of refreshing the currently pushed service information, the specific implementation logic includes: recording the push time node P and the push time length Dp of the previous synthesized audio frame, and obtaining the current push time node N and the push time length Dn; if the current push time node N is earlier than P+Dp, that is, N<=P+Dp, it indicates that the current front-end audio has not experienced stuttering, and the refreshed push information is: P remains unchanged, Dp=Dp+Dn; if the current time node N is later than P+Dp, that is, N>P+Dp, the current front-end audio playback has experienced stuttering, and the refreshed push information is: P=N, Dp=Dn; if P+Dp<N<=P+Dp+Ts, the refreshed push information is: set the broadcast stuttering identifier maintained by the broadcast monitor to True; if N>P+Dp+Tr, set the broadcast redundancy identifier maintained by the broadcast monitor to True.

[0089] For the steps of constructing a phoneme chain, the specific implementation logic includes: initializing a queue; parsing the phoneme text information returned by the synthesis engine to obtain the phoneme information of each character; splicing the phoneme information of each character in the phoneme text information in the way of splicing different syllables of the same character to obtain the phoneme frame corresponding to each character; establishing a link between the phoneme frames corresponding to each character according to the sorting order of each character in the phoneme text information to obtain an initial phoneme chain. And count the pronunciation duration of each character to obtain the pronunciation duration of the phoneme frame corresponding to each character; accumulate the total synthesis time-consuming information starting from the first frame to obtain the synthesis time-consuming information of the phoneme frame corresponding to each character; according to the prosodic pause configuration information agreed upon in the synthesis, parse and obtain the identifier that allows pauses and silences to be inserted in the phoneme frame corresponding to each character, that is, the prosodic pause identifier. Finally, configure the pronunciation duration, synthesis time-consuming information, and prosodic pause identifier of the phoneme frame corresponding to each character in the initial phoneme chain to obtain the final complete phoneme chain.

[0090] For the steps of refreshing the currently received synthesized content, the specific implementation logic includes: performing content matching based on the phoneme chain to determine the character ending corresponding to the previous synthesized audio frame, so as to locate the position of the character where the currently synthesized content is located, and then determine the currently received synthesized content based on the position of the character where the currently synthesized content is located. It should be noted that since there is no punctuation information in the phoneme information, when performing content matching, a certain amount of redundancy can be designed. If the currently matched character is a correct match, backward retrieval is allowed; if the correct match is not achieved after reaching the redundancy amount, the character is ignored and the content matching is performed again after the next audio frame is received.

[0091] Among them, when the audio frames synthesized and returned by the synthesis engine are received and the synthesized content and synthesis speed need to be sent, if the broadcast monitor identifies and determines that there is sufficient playback redundancy at the current front end and there is a lot of unsynthesized content based on the refreshed currently received synthesized content and the refreshed current service push information, it will refresh the synthesis parameters of the current queue of the synthesis engine by specifying the synthesis speed.

[0092] Among them, when receiving the audio frame synthesized and returned by the synthesis engine and when the audio frame needs to be pushed, the broadcast monitor identifies and determines that the current front end is about to or has already encountered playback congestion and insufficient unsynthesized content based on the refreshed currently received synthesized content and the refreshed current service push information. If the target phoneme frame with a rhythmic pause node in the current synthesized audio frame is further identified based on the phoneme chain, the silent segment difference is actively performed at the rhythmic pause configuration position corresponding to the target phoneme frame, so that the synthetic audio congestion played by the front end is optimized through the silent segment at a reasonable rhythmic pause point, thereby ensuring the global smoothness of the audio playback.

[0093] In summary, the method provided in this embodiment, through the dynamic control of the broadcast monitor and the fine management of the phoneme chain, can perform fine and intelligent dynamic adjustment of the synthesis speed at the phoneme level and dynamic configuration of silent segments, thereby achieving more fine and intelligent dynamic optimization of the audio broadcast strategy, so as to more effectively deal with the adverse effects of network delays or uneven data frame transmission, thereby improving the overall smoothness and real-time performance of the audio broadcast and enhancing the user experience.

[0094] The audio broadcast device provided by the present invention is described below. The audio broadcast device described below and the audio broadcast method described above can be referred to each other.

[0095] Figure 8 Schematic diagram of the structure of the audio broadcasting device provided by the present invention; Figure 8 As shown, the device comprises: The first refreshing unit 810 is used to, when receiving the last synthesized audio frame sent by the synthesis engine, refresh the service push information according to the push time node and push time length of the last synthesized audio frame and the current push time node to obtain the current service push information; The second refreshing unit 820 is used to refresh the synthesis speed of the synthesis engine according to the phoneme chain and the current service push information to obtain the current synthesis speed; the phoneme chain is constructed according to the phoneme text information synthesized and returned by the synthesis engine; The audio configuration unit 830 is used to configure a silent segment for the current synthesized audio frame sent by the synthesis engine according to the phoneme chain and the current service push information to obtain an audio frame to be broadcast; the current synthesized audio frame is synthesized by the synthesis engine according to the current synthesis speed; The audio broadcast unit 840 is used to push the audio frame to be broadcast to the audio processing end, and the audio processing end is used to broadcast the audio frame to be broadcast.

[0096] The device provided in this embodiment, after receiving the last synthesized audio frame sent by the synthesis engine, refreshes the service push information in real time according to the push time node, push time length and current time node of the last synthesized audio frame, and according to the current service push information refreshed in real time and the phoneme chain constructed by the phoneme text information synthesized and returned by the synthesis engine, performs dynamic adjustment of the synthesis speed at the phoneme level and dynamic configuration of the silent segments in a refined and intelligent manner, thereby achieving more refined and intelligent dynamic optimization of the audio broadcast strategy, so as to more effectively cope with the adverse effects brought about by network delays or uneven data frame transmission, thereby improving the overall fluency and real-time performance of the audio broadcast and enhancing the user experience.

[0097] In some embodiments, the first refresh unit is specifically configured to: Summing the push time node and the push time length of the previous synthesized audio frame to obtain a first target time node; According to the comparison result between the current push time node and the first target time node, the service push information is refreshed to obtain the current service push information.

[0098] In some embodiments, the first refresh unit is further configured to: When the current push time node is less than or equal to the first target time node, refreshing the service push information according to the first refresh strategy to obtain the current service push information; When the current push time node is greater than the first target time node, refreshing the service push information according to the second refresh strategy to obtain the current service push information; Among them, the first refresh strategy includes a strategy for keeping the push time node of the previous synthesized audio frame unchanged, and a strategy for updating the push time length of the previous synthesized audio frame to the sum of the push time length of the previous synthesized audio frame and the current push time length; the second refresh strategy includes a strategy for updating the push time node of the previous synthesized audio frame to the current push time node, and a strategy for updating the push time length of the previous synthesized audio frame to the current push time length.

[0099] In some embodiments, the first refresh unit is further configured to: The first target time node and the broadcast freeze time threshold are summed to obtain a second target time node; Summing the first target time node and the broadcast redundancy time threshold to obtain a third target time node; In the case where the current push time node is greater than the first target time node, if it is determined that the current push time node is less than or equal to the second target time node, determining that the second refresh strategy also includes a strategy of updating the broadcast jam identifier to be true; In the case where the current push time node is greater than the first target time node, if it is determined that the current push time node is greater than the third target time node, it is determined that the second refresh strategy also includes a strategy of updating the broadcast redundant identifier to be true.

[0100] In some embodiments, the second refresh unit is specifically configured to: According to the phoneme chain, the number of characters of the currently received synthesized audio frame is updated; Determine the number of characters of the currently sent unsynthesized audio frame according to the number of characters of the currently received synthesized audio frame; When it is determined that the number of characters of the currently sent unsynthesized audio frame is greater than or equal to a preset number, and when it is determined that the current broadcast state is a broadcast redundant state based on the current service push information, the synthesis speed of the synthesis engine is refreshed at a preset speed to obtain the current synthesis speed.

[0101] In some embodiments, the audio configuration unit is specifically configured to: When it is determined that the number of characters of the currently sent unsynthesized audio frame is less than the preset number, and when it is determined that the current broadcast state is a broadcast blocking state according to the current service push information, according to the rhythmic pause identifier of each phoneme frame in the phoneme chain, a target phoneme frame having a rhythmic pause node is matched and obtained in the current synthesized audio frame; According to the rhythmic pause configuration position corresponding to the target phoneme frame, a silent segment is inserted into the current synthesized audio frame to obtain the audio frame to be broadcasted.

[0102] In some embodiments, the device further includes a phoneme chain construction unit, specifically configured to: Parsing the phoneme text information to obtain phoneme information of each word; splicing the phoneme information of each word in the phoneme text information in a manner of splicing different syllables of the same word to obtain a phoneme frame corresponding to each word; According to the sorting order of each word in the phoneme text information, a link is established between the phoneme frames corresponding to each word to obtain an initial phoneme chain; The pronunciation duration, synthesis time-consuming information and rhythmic pause identifier of the phoneme frame corresponding to each word are configured in the initial phoneme chain to obtain the phoneme chain.

[0103] In some embodiments, the second refresh unit is further configured to: In the case where the last synthesized audio frame is not received, obtaining the number of characters of the currently sent unsynthesized audio frame and the length of the currently pushed but unbroadcasted audio frame; Acquire the current broadcasting state according to the number of characters of the currently sent unsynthesized audio frame and the length of the currently pushed unbroadcasted audio frame; When it is determined that the current broadcast state is a broadcast redundant state, the synthesis speed of the synthesis engine is accelerated to obtain the current synthesis speed.

[0104] The device provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0105] In some embodiments, this embodiment also provides an audio broadcast system, which includes an audio processing end, a broadcast monitor and a synthesis engine.

[0106] The audio processing end is used to convert the voice initiated by the user into the data to be synthesized; and to broadcast the audio frames to be broadcasted pushed by the broadcast monitor in real time; the synthesis engine is used to synthesize the audio frames according to the synthesis speed and the data to be synthesized sent by the broadcast monitor, and return the synthesized audio frames to the broadcast monitor; the broadcast monitor is used to execute the audio broadcast method provided in the above embodiments, which can be specifically referred to in Figure 1 The execution process shown will not be repeated here.

[0107] The system provided in this embodiment can perform fine-grained and intelligent dynamic adjustment of the synthesis speed at the phoneme level and dynamic configuration of silent segments through real-time interaction between the broadcast monitor and the audio processing end and the synthesis engine, so as to achieve more fine-grained and intelligent dynamic optimization of the audio broadcast strategy, so as to more effectively deal with the adverse effects of network delays or uneven data frame transmission, thereby improving the overall smoothness and real-time performance of the audio broadcast and enhancing the user experience.

[0108] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor (processor) 910 , a communication interface (Communications Interface) 920 , a memory (memory) 930 and a communication bus 940 , wherein the processor 910 , the communication interface 920 , and the memory 930 communicate with each other through the communication bus 940 . The processor 910 can call the logic instructions in the memory 930 to execute the audio broadcast method, which includes: when receiving the last synthesized audio frame sent by the synthesis engine, refreshing the service push information according to the push time node and push time length of the last synthesized audio frame and the current push time node to obtain the current service push information; refreshing the synthesis speed of the synthesis engine according to the phoneme chain and the current service push information to obtain the current synthesis speed; the phoneme chain is constructed according to the phoneme text information synthesized and returned by the synthesis engine; according to the phoneme chain and the current service push information, configuring the current synthesized audio frame sent by the synthesis engine with a silent segment to obtain the audio frame to be broadcast; the current synthesized audio frame is synthesized by the synthesis engine according to the current synthesis speed; pushing the audio frame to be broadcast to the audio processing end, and the audio processing end is used to broadcast the audio frame to be broadcast.

[0109] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0110] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the audio broadcast method provided by the above methods, which includes: when receiving the last synthesized audio frame sent by the synthesis engine, according to the push time node and push time length of the last synthesized audio frame, and the current push time node, refreshing the service push information to obtain the current service push information; according to the phoneme chain and the current service push information, refreshing the synthesis speed of the synthesis engine to obtain the current synthesis speed; the phoneme chain is constructed according to the phoneme text information synthesized and returned by the synthesis engine; according to the phoneme chain and the current service push information, configuring the current synthesized audio frame sent by the synthesis engine with a silent segment to obtain the audio frame to be broadcast; the current synthesized audio frame is synthesized by the synthesis engine according to the current synthesis speed; pushing the audio frame to be broadcast to the audio processing end, and the audio processing end is used to broadcast the audio frame to be broadcast.

[0111] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the audio broadcast method provided by the above-mentioned methods when the computer program is executed, the method comprising: upon receiving the last synthesized audio frame sent by the synthesis engine, refreshing the service push information according to the push time node and push time length of the last synthesized audio frame and the current push time node to obtain the current service push information; refreshing the synthesis speed of the synthesis engine according to the phoneme chain and the current service push information to obtain the current synthesis speed; the phoneme chain is constructed according to the phoneme text information synthesized and returned by the synthesis engine; configuring the current synthesized audio frame sent by the synthesis engine with a silent segment according to the phoneme chain and the current service push information to obtain the audio frame to be broadcast; the current synthesized audio frame is synthesized by the synthesis engine according to the current synthesis speed; pushing the audio frame to be broadcast to the audio processing end, and the audio processing end is used to broadcast the audio frame to be broadcast.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio broadcast method, characterized in that: include: When receiving the last synthesized audio frame sent by the synthesis engine, refreshing the service push information according to the push time node and push time length of the last synthesized audio frame and the current push time node to obtain the current service push information; According to the phoneme chain and the current service push information, the synthesis speed of the synthesis engine is refreshed to obtain the current synthesis speed; the phoneme chain is constructed according to the phoneme text information synthesized and returned by the synthesis engine; According to the phoneme chain and the current service push information, a silent segment is configured for the current synthesized audio frame sent by the synthesis engine to obtain an audio frame to be broadcast; the current synthesized audio frame is synthesized by the synthesis engine according to the current synthesis speed; The audio frame to be broadcast is pushed to an audio processing end, and the audio processing end is used to broadcast the audio frame to be broadcast.

2. The audio broadcast method according to claim 1, characterized in that: The refreshing of the service push information according to the push time node and push time length of the previous synthesized audio frame and the current push time node to obtain the current service push information includes: Summing the push time node and the push time length of the previous synthesized audio frame to obtain a first target time node; According to the comparison result between the current push time node and the first target time node, the service push information is refreshed to obtain the current service push information.

3. The audio broadcast method according to claim 2, characterized in that: The refreshing of the service push information according to the comparison result between the current push time node and the first target time node to obtain the current service push information includes: When the current push time node is less than or equal to the first target time node, refreshing the service push information according to the first refresh strategy to obtain the current service push information; When the current push time node is greater than the first target time node, refreshing the service push information according to the second refresh strategy to obtain the current service push information; Among them, the first refresh strategy includes a strategy of keeping the push time node of the previous synthesized audio frame unchanged, and a strategy of updating the push time length of the previous synthesized audio frame to the sum of the push time length of the previous synthesized audio frame and the current push time length; the second refresh strategy includes a strategy of updating the push time node of the previous synthesized audio frame to the current push time node, and a strategy of updating the push time length of the previous synthesized audio frame to the current push time length.

4. The audio broadcast method according to claim 3, characterized in that: The method further comprises: The first target time node and the broadcast freeze time threshold are summed to obtain a second target time node; Summing the first target time node and the broadcast redundancy time threshold to obtain a third target time node; In the case where the current push time node is greater than the first target time node, if it is determined that the current push time node is less than or equal to the second target time node, determining that the second refresh strategy also includes a strategy of updating the broadcast jam identifier to be true; In the case where the current push time node is greater than the first target time node, if it is determined that the current push time node is greater than the third target time node, it is determined that the second refresh strategy also includes a strategy of updating the broadcast redundant identifier to be true.

5. The audio broadcasting method according to any one of claims 1 to 4, characterized in that: The step of refreshing the synthesis speed of the synthesis engine according to the phoneme chain and the current service push information to obtain the current synthesis speed includes: According to the phoneme chain, the number of characters of the currently received synthesized audio frame is updated; Determine the number of characters of the currently sent unsynthesized audio frame according to the number of characters of the currently received synthesized audio frame; When it is determined that the number of characters of the currently sent unsynthesized audio frame is greater than or equal to a preset number, and when it is determined that the current broadcast state is a broadcast redundant state based on the current service push information, the synthesis speed of the synthesis engine is refreshed at a preset speed to obtain the current synthesis speed.

6. The audio broadcast method according to claim 5, characterized in that: The step of configuring a silent segment for the current synthesized audio frame sent by the synthesis engine according to the phoneme chain and the current service push information to obtain an audio frame to be broadcast includes: When it is determined that the number of characters of the currently sent unsynthesized audio frame is less than the preset number, and when it is determined that the current broadcast state is a broadcast blocking state according to the current service push information, according to the rhythmic pause identifier of each phoneme frame in the phoneme chain, a target phoneme frame having a rhythmic pause node is matched and obtained in the current synthesized audio frame; According to the rhythmic pause configuration position corresponding to the target phoneme frame, a silent segment is inserted into the current synthesized audio frame to obtain the audio frame to be broadcasted.

7. The audio broadcasting method according to any one of claims 1 to 4, characterized in that: The phoneme chain is constructed based on the following steps: Parsing the phoneme text information to obtain phoneme information of each word; splicing the phoneme information of each word in the phoneme text information in a manner of splicing different syllables of the same word to obtain a phoneme frame corresponding to each word; According to the sorting order of each word in the phoneme text information, a link is established between the phoneme frames corresponding to each word to obtain an initial phoneme chain; The pronunciation duration, synthesis time-consuming information and rhythmic pause identifier of the phoneme frame corresponding to each word are configured in the initial phoneme chain to obtain the phoneme chain.

8. The audio broadcasting method according to any one of claims 1 to 4, characterized in that: The method further comprises: In the case where the last synthesized audio frame is not received, obtaining the number of characters of the currently sent unsynthesized audio frame and the length of the currently pushed but unbroadcasted audio frame; Acquire the current broadcasting state according to the number of characters of the currently sent unsynthesized audio frame and the length of the currently pushed unbroadcasted audio frame; When it is determined that the current broadcast state is a broadcast redundant state, the synthesis speed of the synthesis engine is accelerated to obtain the current synthesis speed.

9. An audio broadcasting device, characterized in that: include: A first refreshing unit is used to, when receiving a previous synthesized audio frame sent by the synthesis engine, refresh the service push information according to the push time node and push time length of the previous synthesized audio frame and the current push time node to obtain the current service push information; A second refreshing unit is used to refresh the synthesis speed of the synthesis engine according to the phoneme chain and the current service push information to obtain the current synthesis speed; the phoneme chain is constructed according to the phoneme text information synthesized and returned by the synthesis engine; An audio configuration unit, configured to perform a silent segment configuration on the current synthesized audio frame sent by the synthesis engine according to the phoneme chain and the current service push information, so as to obtain an audio frame to be broadcast; the current synthesized audio frame is synthesized by the synthesis engine according to the current synthesis speed; The audio broadcast unit is used to push the audio frame to be broadcast to the audio processing end, and the audio processing end is used to broadcast the audio frame to be broadcast.

10. An audio broadcast system, characterized in that: It includes audio processing end, broadcast monitor and synthesis engine; The broadcast monitor is connected to the audio processing end and the synthesis engine respectively, and is used to execute the audio broadcast method as described in any one of claims 1 to 8.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the audio broadcast method according to any one of claims 1 to 8 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio broadcast method according to any one of claims 1 to 8 is implemented.

Images