Data processing method, device and system

Abstract

The invention provides a data processing method, device and system, and the method comprises the steps: receiving a first code stream and a second code stream, the first code stream comprising data obtained by encoding a text corresponding to first audio data, and the second code stream comprising data obtained by encoding the first audio data; under the condition that the reconstructed audio data are incomplete, text-to-speech processing is carried out on the reconstructed text data to obtain second audio data; and fusing the second audio data and the reconstructed audio data to obtain third audio data and outputting the third audio data, the reconstructed text data being obtained by decoding the first code stream, and the reconstructed audio data being generated based on the second code stream. Thus, even if the network state becomes poor, the receiving end can recover the complete audio data, so that the problems of jamming, word loss, even silence and the like are reduced, and the fluency of the audio in a real-time communication scene is improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data processing method, apparatus, and system. Background Technology

[0002] In real-time communications (RTC) scenarios, such as voice or video calls, video conferencing, and live interactive sessions, user A's voice is first captured by the microphone of their device A. Device A then encodes the captured audio data to obtain an audio stream, which is transmitted over the network to user B's device B. Upon receiving the audio stream, device B decodes it, reconstructs the audio data (i.e., rebuilds the audio data), and plays it back, allowing user B to hear user A's voice. The same process applies to user B transmitting their voice to user A, enabling real-time, two-way voice communication between the two parties.

[0003] In real-time communication, network conditions are a crucial factor determining audio communication quality. When network conditions deteriorate, issues such as packet loss can occur, severely impacting audio communication quality. Therefore, improving audio communication quality in real-time scenarios under degraded network conditions has become a significant technical challenge. Summary of the Invention

[0004] In view of this, this application provides a data processing method, apparatus, and system. This application can improve the audio communication quality (such as audio fluency, clarity, and fidelity) in real-time communication scenarios when network conditions deteriorate.

[0005] In a first aspect, this application provides a data processing method, which includes: firstly, receiving a first bitstream and a second bitstream, the first bitstream including data obtained by encoding text corresponding to first audio data, and the second bitstream including data obtained by encoding the first audio data; subsequently, in the case of incomplete reconstructed audio data, performing text-to-speech processing on the reconstructed text data to obtain second audio data; fusing the second audio data and the reconstructed audio data to obtain third audio data and outputting the third audio data, wherein the reconstructed text data is obtained by decoding the first bitstream, and the reconstructed audio data is generated based on the second bitstream.

[0006] Because the audio stream (i.e., the second stream) has a relatively high bitrate (approximately between 20kbps and 300kbps), the receiving end may be unable to receive the complete audio stream under poor or very poor network conditions, resulting in packet loss. In contrast, the text stream (i.e., the first stream) has a very low bitrate, generally not exceeding 100bps; even in extremely weak network conditions (such as 5kbps), the available bandwidth is more than sufficient for transmitting the text stream. Therefore, the probability of packet loss for the text stream is lower, and consequently, the receiving end has a higher probability of receiving the complete text stream or recovering the complete text stream. After receiving or recovering the complete text stream, the receiving end can perform text-to-speech (TTS) processing on the reconstructed text data obtained from decoding the complete text stream to obtain the complete second audio data. When the complete reconstructed audio data cannot be recovered based on the audio bitstream, the second audio data can be used to supplement the missing parts of the reconstructed audio data; thus, the audio data obtained for playback (i.e., the third audio data) is complete. In this way, even if the network conditions deteriorate, the receiving end can recover the complete audio data (i.e., the third audio data), thereby reducing problems such as stuttering, missing words, or even no sound, and improving the smoothness of audio in real-time communication scenarios.

[0007] When the receiving end cannot receive the complete text stream or cannot recover the complete text stream, the interruption duration of the second audio data is shorter than that of the reconstructed audio data. Therefore, by using the second audio data to supplement the missing parts in the reconstructed audio data, the interruption duration of the resulting third audio data is also shorter than that of the reconstructed audio data. Thus, it can also reduce problems such as stuttering, missing words, or even silence to a certain extent, and improve the smoothness of audio in real-time communication scenarios.

[0008] For example, the data processing method of the first aspect described above can be understood as follows: when a first bitstream is received and a second bitstream is received and the reconstructed audio data is incomplete, text-to-speech processing is performed on the reconstructed text data to obtain second audio data; the second audio data and the reconstructed audio data are fused to obtain third audio data and the third audio data is output.

[0009] For example, any one of the multiple terminal devices participating in real-time communication can execute this data processing method.

[0010] For example, the headers of the first and second bitstreams may include an identifier for identifying the type of bitstream. Thus, the first and second bitstreams can be distinguished by the value of this identifier in the bitstream header.

[0011] For example, the sending end (in some cases, the data forwarding unit can also be regarded as the sending end) can send the first bitstream and the second bitstream through different interfaces. In this way, the receiving end can distinguish between the first bitstream and the second bitstream based on the interface through which the bitstream is received.

[0012] For example, the first bitstream and the second bitstream can be two independent bitstreams, or they can be two sub-bitstreams in the same bitstream.

[0013] For example, the text corresponding to the first audio data can be obtained by performing speech recognition on the first audio data.

[0014] For example, the received second bitstream may include part or all of the data obtained from encoding the first audio data.

[0015] For example, a text-to-speech algorithm (such as a deep learning model) can be used to process the reconstructed text data into speech. Specifically, the TTS algorithm can be trained using reconstructed audio data determined based on historically received second bitstreams to continuously optimize it, making the timbre, tone, speech rate, and volume of the second audio data closer to the original audio data (i.e., the first audio data).

[0016] For example, outputting third audio data can be understood as playing third audio data.

[0017] According to the first aspect, the method further includes: if the second bitstream is not received, performing text-to-speech processing on the reconstructed text data to obtain second audio data and outputting the second audio data. In this way, even in extremely weak network conditions, the receiving end can hear the sending end user speaking.

[0018] For example, "if the second stream is not received" can be understood as "the first stream is received but the second stream is not received".

[0019] For example, outputting the second audio data can be understood as playing the second audio data.

[0020] According to the first aspect, or any implementation of the first aspect above, the method further includes: outputting reconstructed audio data when the reconstructed audio data is complete. Since the timbre, pitch, speech rate, volume, etc., of the second audio data generated by TTS processing differ from those of the original audio data, outputting the reconstructed audio data when the reconstructed audio data is complete ensures a better audio experience for the receiving user.

[0021] In this case, there is no need to perform TTS processing on the reconstructed text data.

[0022] For example, outputting reconstructed audio data can be understood as playing the reconstructed audio data.

[0023] According to the first aspect, or any implementation of the first aspect above, the second audio data and the reconstructed audio data are fused to obtain the third audio data, including: extracting the second segment corresponding to the start and end times of the missing first segment in the reconstructed audio data from the second audio data; and splicing the reconstructed audio data and the second segment based on the start and end times to obtain the third audio data. This ensures the integrity of the third audio data.

[0024] For example, spectral analysis can be used to analyze the reconstructed audio data to determine the start and end times of the first missing segment in the reconstructed audio data.

[0025] For example, the missing first segment in the reconstructed audio data can be one or more, and correspondingly, the second segment extracted from the second audio data can also be one or more; the number of second segments is the same as the number of first segments, and multiple second segments correspond one-to-one with multiple first segments.

[0026] For example, the difference between the start time of each second segment and the start time of its corresponding first segment is less than or equal to a threshold, and / or the difference between the end time of each second segment and the end time of its corresponding first segment is less than or equal to a threshold.

[0027] According to the first aspect, or any implementation of the first aspect above, the method further includes: determining whether the reconstructed audio data is complete.

[0028] Determining whether the reconstructed audio data is complete according to the first aspect, or any implementation of the first aspect above, includes: generating a third bitstream based on the second bitstream and the parsing result obtained by parsing the second bitstream, and the reconstructed audio data being obtained by decoding the third bitstream; if the third bitstream includes all the data in the data obtained by encoding the first audio data, the reconstructed audio data is complete; if the third bitstream includes only part of the data in the data obtained by encoding the first audio data, the reconstructed audio data is incomplete.

[0029] For example, an audio frame can be encoded into multiple data packets. The completeness of the third bitstream can be determined based on the number of data packets corresponding to each audio frame in the third bitstream, that is, whether the third bitstream includes all the data obtained by encoding the first audio data.

[0030] Another possible approach is to perform spectral analysis on the reconstructed audio data; then, based on the results of the spectral analysis (such as a spectrogram, which shows the energy distribution of the audio signal at different frequencies), it can be determined whether the reconstructed audio data is complete (or whether there are any interruptions in the reconstructed audio data).

[0031] For example, if a frequency component on the spectrogram suddenly disappears or decreases significantly at a certain point in time, it is determined that the reconstructed audio data was interrupted at that point in time.

[0032] For example, if the energy value at a certain time in the spectrogram suddenly drops or rises, it indicates that the reconstructed audio data was interrupted at that time.

[0033] For example, statistical analysis can be performed on the energy values ​​in the spectrogram, such as calculating the mean and variance of the energy values. If these statistical indicators change significantly at a certain point in time, it can be determined that the reconstructed audio data was interrupted at that point in time.

[0034] For example, an energy threshold can be preset, and when the energy value at a certain time point in the spectrogram is lower than the energy threshold, it can be determined that the reconstructed audio data has been interrupted at that time point.

[0035] In addition, time-domain analysis can be performed on the reconstructed audio data; for example, by observing the waveform, one can determine whether the reconstructed audio data is interrupted by observing whether the signal suddenly disappears at a certain point in time.

[0036] It should be understood that the energy distribution, frequency components, and energy abrupt changes in the spectrogram can also be observed, and time-domain analysis and statistical analysis can be combined to comprehensively determine whether there is an interruption in the reconstructed audio data.

[0037] For example, the third bitstream can be generated based on the second bitstream and the parsing result obtained from parsing the second bitstream as follows:

[0038] In one possible approach, the second bitstream is parsed to obtain a parsing result; if the parsing result indicates that the second bitstream contains redundancy, redundancy removal processing is performed on the second bitstream to obtain a third bitstream; if the parsing result indicates that the second bitstream does not contain redundancy, the second bitstream is identified as the third bitstream. This process is the same as the forward error / erasure correction (FEC) process at the receiver.

[0039] In one possible approach, the second bitstream is parsed to obtain the parsing result; when it is determined based on the parsing result that the second bitstream is missing data packets, a request for the missing data packets from the sender is sent; a fifth bitstream, which includes the missing data packets from the second bitstream, is received; the second and fifth bitstreams are then concatenated to obtain the third bitstream. It should be understood that the fifth bitstream may include some or all of the missing data packets from the second bitstream. This request for the missing data packets from the second bitstream can be implemented by sending an Automatic Repeat-reQuest (ARQ) request to the sender.

[0040] In one possible approach, the second bitstream is parsed to obtain the parsing result; based on the parsing result, when the missing data packets of the second bitstream are determined, the missing data packets of the second bitstream are predicted to obtain the sixth bitstream; the second bitstream and the sixth bitstream are then concatenated to obtain the third bitstream. It should be understood that the sixth bitstream may include some or all of the missing data packets from the second bitstream. This process is essentially the packet loss compensation (PLC) process at the receiving end.

[0041] According to the first aspect, or any implementation of the first aspect above, the first bitstream and the second bitstream are generated during real-time audio communication or real-time audio and video communication.

[0042] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0043] The system receives a fourth bitstream, decodes the fourth bitstream to obtain reconstructed video data, and outputs the reconstructed video data. The fourth bitstream includes the data obtained by encoding the video data corresponding to the first audio data.

[0044] For example, outputting reconstructed video data can be understood as playing the reconstructed video data.

[0045] Secondly, this application provides a data processing system, which includes a transmitter and a receiver, wherein:

[0046] The sending end is used for

[0047] If the network status between the sending end and the receiving end meets the first condition, the first bitstream and the second bitstream are sent; the first bitstream includes data obtained by encoding the text corresponding to the first audio data, and the second bitstream includes data obtained by encoding the first audio data;

[0048] Receiver, used for

[0049] Receive the first and second bitstreams;

[0050] In the case of incomplete reconstructed audio data, text-to-speech processing is performed on the reconstructed text data to obtain the second audio data; the second audio data and the reconstructed audio data are then fused to obtain the third audio data and played. The reconstructed text data is obtained by decoding the first bitstream, and the reconstructed audio data is generated based on the second bitstream.

[0051] Because the first bitrate is very low, even in extremely weak network conditions (such as 5kbps), the available bandwidth is more than sufficient for transmitting the text stream, thus reducing the probability of packet loss. Consequently, the receiving end has a higher probability of receiving the complete text stream or recovering it. Therefore, the second audio data obtained by the receiving end through TTS processing of the reconstructed text data from the decoded first bitrate has a higher probability of being complete or has a relatively shorter interruption duration. This second audio data can be directly used for playback or to supplement the reconstructed audio data, thereby ensuring relatively smooth audio playback at the receiving end.

[0052] Furthermore, existing technologies discard some non-critical frames or increase the audio compression rate when the network condition deteriorates, while the sending end of this application does not discard non-critical frames or increase the audio compression rate when the network condition deteriorates (i.e., the first condition is met); therefore, compared with the existing technology, the receiving end of this application has higher fidelity and clarity of the reconstructed audio data determined based on the second bitstream; consequently, the fidelity and clarity of the third audio data (obtained by splicing the reconstructed audio data and the second audio data) used for playback are also higher.

[0053] It should be noted that the sending end and the receiving end can be devices or applications.

[0054] According to the second aspect, the sending end is also used to send a first code stream when the network state between the sending end and the receiving end meets the second condition, wherein the network state that meets the first condition is better than the network state that meets the second condition.

[0055] The receiving end is also used to receive the first bitstream, perform text-to-speech processing on the reconstructed text data to obtain the second audio data, and play the second audio data.

[0056] In other words, when the network condition becomes very poor, the sender only transmits the first bit stream.

[0057] According to the second aspect, or any implementation of the second aspect above, the receiving end is also used to play the reconstructed audio data if the reconstructed audio data is complete.

[0058] According to the second aspect, or any implementation of the second aspect above, the receiving end is used to extract the second segment corresponding to the start and end times of the missing first segment in the reconstructed audio data from the second audio data; and to splice the reconstructed audio data and the second segment based on the start and end times to obtain the third audio data.

[0059] According to the second aspect, or any implementation of the second aspect above, the sending end is further configured to: collect first audio data; perform speech recognition on the first audio data to obtain text corresponding to the first audio data; encode the text corresponding to the first audio data to obtain a first bitstream; and encode the first audio data to obtain a second bitstream.

[0060] In this scenario, the first bitstream is generated by the sender and sent directly to the receiver.

[0061] It should be understood that the sending end can perform the step of "encoding the text corresponding to the first audio data to obtain the first bitstream" when the network status between the sending end and the receiving end meets the first condition.

[0062] According to the second aspect, or any implementation of the second aspect above, the sending end includes a first sub-sender and a second sub-sender, wherein:

[0063] The first sub-transmitter is used to collect the first audio data, encode the first audio data to obtain the second bitstream, and send the second bitstream to the second sub-transmitter.

[0064] The second sub-transmitter is used to receive the second bitstream; determine the reconstructed audio data based on the second bitstream; perform speech recognition on the reconstructed audio data to obtain the text corresponding to the first audio data; encode the text corresponding to the first audio data to obtain the first bitstream; and send the first bitstream and the second bitstream to the receiving end.

[0065] In this scenario, the first bitstream is generated by the second sub-sender and sent to the receiver.

[0066] It should be understood that the second sub-sender can perform the following steps, "based on the second bitstream, determine the reconstructed audio data; perform speech recognition on the reconstructed audio data to obtain the text corresponding to the first audio data; encode the text corresponding to the first audio data to obtain the first bitstream," when the network status between the sender and receiver meets the first condition.

[0067] It should be noted that the first and second sub-senders can be devices or applications.

[0068] According to the second aspect, or any implementation of the second aspect above, the sending end includes a first sub-sender and a second sub-sender, wherein:

[0069] The first sub-transmitter is used to collect first audio data; perform speech recognition on the first audio data to obtain the text corresponding to the first audio data; encode the text corresponding to the first audio data to obtain a first bitstream; encode the first audio data to obtain a second bitstream; and send the first bitstream and the second bitstream to the second sub-transmitter.

[0070] The second sub-sender is used to receive the first and second bitstreams and forward them to the receiving end.

[0071] In this scenario, the first bitstream is generated by the first sub-sender and forwarded to the receiver by the second sub-sender.

[0072] According to the second aspect, or any implementation of the second aspect above, the sending end is further configured to send a fourth bitstream when the network state between the sending end and the receiving end satisfies the third condition, the fourth bitstream including data obtained by encoding the video data corresponding to the first audio data.

[0073] According to the second aspect, or any implementation of the second aspect above, the sending end and the receiving end conduct real-time audio communication or real-time audio and video communication.

[0074] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0075] Thirdly, this application provides a data processing apparatus, which may include:

[0076] The receiving module is used to receive a first bitstream and a second bitstream. The first bitstream includes data obtained by encoding text corresponding to the first audio data, and the second bitstream includes data obtained by encoding the first audio data.

[0077] The text-to-speech processing module is used to perform text-to-speech processing on the reconstructed text data when the reconstructed audio data is incomplete, so as to obtain the second audio data. The reconstructed audio data is generated based on the second bitstream, and the reconstructed text data is obtained by decoding the first bitstream.

[0078] The fusion module is used to fuse the second audio data and the reconstructed audio data to obtain the third audio data;

[0079] The output module is used to output third-party audio data.

[0080] For example, the output module can be a playback module.

[0081] It should be understood that the data processing apparatus of this application can be used to execute the methods in the first aspect and any implementation thereof, which will not be elaborated here.

[0082] Fourthly, this application provides an electronic device, including: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, cause the electronic device to perform the method of the first aspect or any possible implementation thereof.

[0083] Fifthly, this application provides a chip including one or more interface circuits and one or more processors; the one or more processors receive or transmit data through the one or more interface circuits, and when the one or more processors execute computer instructions, cause the electronic device to perform the method in the first aspect or any possible implementation of the first aspect.

[0084] In a sixth aspect, this application provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the method of the first aspect or any possible implementation thereof.

[0085] In a seventh aspect, this application provides a computer program product, which includes computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the method in the first aspect or any possible implementation thereof.

[0086] Among them, the electronic equipment in aspects four through seven can refer to the second device.

[0087] Eighthly, this application provides an apparatus for storing a bitstream, the apparatus comprising: a receiver and at least one storage medium, the receiver being used to receive the bitstream; at least one storage medium being used to store the bitstream; the bitstream being the bitstream of the first aspect and any implementation thereof.

[0088] Ninthly, this application provides an apparatus for transmitting a bitstream, the apparatus comprising: a transmitter and at least one storage medium, the at least one storage medium being used to store the bitstream, the bitstream being the bitstream of the first aspect and any implementation thereof; the transmitter being used to obtain the bitstream from the storage medium and transmit the bitstream to an end-side device via the transmission medium.

[0089] In a tenth aspect, this application provides a system for distributing bitstreams, the system comprising: at least one storage medium for storing at least one bitstream, the at least one bitstream being a bitstream in the first aspect and any implementation thereof; and a streaming media device for obtaining a target bitstream from the at least one storage medium and sending the target bitstream to an end-side device, wherein the streaming media device includes a content server or a content distribution server.

[0090] For example, the bitstreams involved in the eighth, ninth and tenth aspects can refer to the first bitstream and the second bitstream in the first aspect and any implementation of the first aspect.

[0091] In one aspect, this application provides a bitstream, which includes a first bitstream and a second bitstream. The first bitstream includes data obtained by encoding text corresponding to first audio data, and the second bitstream includes data obtained by encoding the first audio data.

[0092] In a twelfth aspect, this application provides a bitstream comprising a first bitstream, a second bitstream, and a fourth bitstream, wherein the first bitstream comprises data obtained by encoding text corresponding to first audio data, the second bitstream comprises data obtained by encoding the first audio data, and the fourth bitstream comprises data obtained by encoding video data corresponding to the first audio data.

[0093] In a thirteenth aspect, this application provides a computer-readable storage medium storing a bitstream, the bitstream including a first bitstream and a second bitstream, the first bitstream including data obtained by encoding text corresponding to first audio data, and the second bitstream including data obtained by encoding the first audio data.

[0094] In a fourteenth aspect, this application provides a computer-readable storage medium storing a bitstream, the bitstream including a first bitstream, a second bitstream and a fourth bitstream, the first bitstream including data obtained by encoding text corresponding to first audio data, the second bitstream including data obtained by encoding the first audio data, and the fourth bitstream including data obtained by encoding video data corresponding to the first audio data.

[0095] In this embodiment, the electronic device, computer-readable storage medium, computer program product, chip, etc. are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above. Attached Figure Description

[0096] Figure 1A This is a schematic diagram illustrating one application scenario of an embodiment of this application;

[0097] Figure 1B This is a schematic diagram illustrating another application scenario of an embodiment of this application;

[0098] Figure 1C This is a schematic diagram illustrating another application scenario of this application embodiment;

[0099] Figure 1D This is a schematic diagram illustrating another application scenario of this application embodiment;

[0100] Figure 2A This is a schematic diagram of a data processing system 200 according to an embodiment of this application;

[0101] Figure 2B This is a schematic diagram of another data processing system 200 according to an embodiment of this application;

[0102] Figure 2C This is a schematic diagram of another data processing system 200 according to an embodiment of this application;

[0103] Figure 3 This is a schematic diagram of a code stream transmission process 300 according to an embodiment of this application;

[0104] Figure 4 This is a schematic diagram of a data processing procedure 400 according to an embodiment of this application;

[0105] Figure 5 This is a schematic diagram of another data processing procedure 500 according to an embodiment of this application;

[0106] Figure 6 This is a schematic diagram of a data processing apparatus 600 according to an embodiment of this application;

[0107] Figure 7 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0108] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0109] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0110] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0111] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0112] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0113] In the embodiments of this application, the modules / components shown in the framework diagram (or structural diagram or system diagram) are merely examples of this application. The actual framework (or structure or system) may include more or fewer modules / components than those shown in the diagram, or may have different component configurations. Furthermore, the various components / modules shown in the diagrams may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0114] Figure 1A This is a schematic diagram illustrating one application scenario of an embodiment of this application. Figure 1A This illustrates a video call (or audio-visual call) scenario.

[0115] Figure 1A The image shows a scenario where user A and user B are having a video call through an instant messaging application.

[0116] During a video call between user A and user B, mobile phone A can use its camera to capture user A's image, obtaining video data A. Then, mobile phone A can play video data A in video playback window 101, and simultaneously encode video data A to obtain video stream A. While capturing user A's image with the camera, mobile phone A can also use its microphone to capture user A's voice, obtaining audio data A. This audio data A can then be encoded to obtain audio stream A. Mobile phone A can then send both video stream A and audio stream A to mobile phone B. After decoding video data A from video stream A, mobile phone B can play the decoded video data A in video playback window 104; and after decoding audio data A from audio stream A, it can play the decoded audio data A through its speaker.

[0117] Similarly, mobile phone B can play video data B (video data obtained by capturing the image of user B through mobile phone B's camera) in video playback window 103, and send video data B and audio data B (audio data obtained by capturing user B's voice through mobile phone B's microphone) to mobile phone A in the same way. Mobile phone A can display the decoded video data B in video playback window 102 and play the decoded audio data B through its speaker. In this way, real-time video calls between user A and user B can be realized.

[0118] It should be understood that User A and User B can also conduct video calls through instant messaging pages, instant messaging tools provided by game platforms, etc.

[0119] Figure 1B This is a schematic diagram illustrating another application scenario of this application embodiment. Figure 1B This illustrates a scenario of a voice call (or audio call).

[0120] Figure 1B The example shown is a scenario where user A and user B are having a voice call via telephone dialing.

[0121] During a voice call between user A and user B, phone A can use its microphone to capture user A's voice, obtaining audio data A. Then, it can encode audio data A to obtain audio stream A. Phone A can then send audio stream A to phone B. Phone B can decode audio data A from audio stream A and play the decoded audio data A through its speaker.

[0122] Similarly, mobile phone B can send audio data B (audio data obtained by capturing user B's voice through mobile phone B's microphone) to mobile phone A in the same way described above. Mobile phone A can then play the decoded audio data B through its speaker. In this way, real-time audio calls between user A and user B can be achieved.

[0123] It should be understood that User A and User B can also make voice calls through the aforementioned instant messaging applications, instant messaging pages, and instant messaging tools provided by the game platform.

[0124] Figure 1C This is a schematic diagram illustrating another application scenario of this application embodiment. Figure 1C This illustrates a video / audio conferencing scenario.

[0125] Figure 1C Users located in different locations can conduct video / audio conferences through terminal devices. The following explanation uses video conferencing as an example.

[0126] Figure 1C In this scenario, a user located at location E can create a video conference using a terminal device. Once a user at location E joins the video conference, that user can be called the host, and the terminal device used by the host to join the video conference can be called participant terminal 1. Subsequently, participant A at location A can join the video conference through participant terminal 2, participant B at location B can join through participant terminal 3, participant C at location C can join through participant terminal 4, participant D at location D can join through participant terminal 5, and so on.

[0127] During the video conference, participating terminal 1 plays video data 1 (video data obtained by capturing the image of the host through the camera of participating terminal 1) in a local video playback window. It can also send video data 1 and audio data 1 (audio data obtained by capturing the host's voice through the microphone of participating terminal 1) to multiple participating terminals, including participating terminal 2, participating terminal 3, participating terminal 4, and participating terminal 5, in the same manner. Each of these participating terminals, after decoding video data 1 from video stream 1, can play the decoded video data 1 in a local video playback window; and after decoding audio data 1 from audio stream 1, it can play the decoded audio data 1 through its speaker. Similarly, the same applies to other participating terminals, which will not be elaborated further. This enables multi-terminal real-time video conferencing.

[0128] Figure 1DThis is a schematic diagram illustrating another application scenario of this application embodiment. Figure 1D The image shows a live-streaming interaction scenario.

[0129] Figure 1D The middle image shows a scene where four streamers, A, B, C, and D, are conducting a live broadcast together.

[0130] Figure 1D In this scenario, during a live stream using phone A, streamer A can send connection requests to streamers B's phone B, C's phone C, and D's phone D. After streamer B receives the request on phone B, phone B can respond to streamer B's action and connect with phone A. Once the connection is successful, streamer B can join streamer A's live stream. Similarly, phones C and D can connect with phone A in the same way, allowing streamers C and D to join streamer A's live stream.

[0131] In a live streaming session, phone A plays video data A (video data obtained from the image of the broadcaster captured by phone A's camera) in a local video playback window. Similarly, it can send video data A and audio data A (audio data obtained from the broadcaster's voice captured by phone A's microphone) to phones B, C, and D respectively. Each of phones B, C, and D, after decoding video data A from video stream A, can play the decoded video data A in a local video playback window; and after decoding audio data A from audio stream A, it can play the decoded audio data A through its speaker. The same applies to other phones, which will not be elaborated further. Therefore, phone A can also play video data B (video data obtained from the image of the broadcaster captured by phone B's camera), video data C (video data obtained from the image of the broadcaster captured by phone C's camera), and video data D (video data obtained from the image of the broadcaster captured by phone D's camera) in three other video playback windows. This enables multi-device live streaming.

[0132] It should be understood that this application may also include other real-time audio communication scenarios or real-time audio and video communication scenarios.

[0133] Since deteriorating network conditions (such as reduced available bandwidth) can affect the quality of audio communication during real-time communication, the sending end (in some cases, the data forwarding unit can also be considered as the sending end) of this application can generate text corresponding to the audio data and send the bitstream of the text corresponding to the audio data to the receiving end. In this way, when the receiving end determines that it cannot recover complete audio data based on the received audio bitstream, it can supplement the missing parts of the audio data recovered from the received audio bitstream with audio data obtained by text-to-speech processing of the text corresponding to the audio data, thereby improving the quality of audio communication.

[0134] The following describes a data processing system that can be used for real-time audio communication or real-time audio-video communication. It should be noted that the data processing system of this application may include a sending end and a receiving end; wherein, the sending end and the receiving end may be devices or application programs. The following embodiments use devices as examples to illustrate this.

[0135] Figure 2A This is a schematic diagram of a data processing system 200 according to an embodiment of this application. Figure 2A In this embodiment, the sending end is the first device 201, and the receiving end is the second device 202.

[0136] Figure 2A The data processing system 200 includes a first device 201 and a second device 202.

[0137] The first device 201 may include a first SDK (203) for real-time audio communication / real-time audio and video communication (Software Development Kit, SDK is a set of tools to help developers create software applications), the first SDK (203) may include: an ASR (Automatic Speech Recognition) module 21, a text encoding module 22, an audio acquisition module 23, an audio encoding module 24, a video acquisition module 25, a video encoding module 26, and a transmission module 27.

[0138] The second device 202 may include a second SDK (204) for real-time audio communication / real-time audio-video communication. The second SDK (204) may include: a text decoding module 31, a TTS processing module 32, an audio preprocessing module 33, an audio decoding module 34, a decision module 35, a video decoding module 36, a first playback module 37, and a second playback module 38. The audio preprocessing module 33 may include a Forward Error / Erasure Correction (FEC) module, an Automatic Repeat-reQuest (ARQ) module, and a Packet Loss Concealment (PLC) module.

[0139] During real-time audio communication / real-time audio-video communication, the audio acquisition module 23 and the video acquisition module 25 can acquire data synchronously. The audio acquisition module 23 acquires sound to obtain first audio data and outputs it to the audio encoding module 24 and the ASR module 21, respectively. The video acquisition module 25 acquires video to obtain video data and outputs it to the video encoding module 26. It should be understood that the first audio data corresponds to the video data.

[0140] Subsequently, on one hand, the ASR module 21 can perform speech recognition on the first audio data to obtain the text corresponding to the first audio data, and output the text corresponding to the first audio data to the text encoding module 22; the text encoding module 22 can encode the text corresponding to the first audio data (such as variable-length character encoding (such as UTF-8 encoding)) to obtain the first bitstream (also called the text bitstream) and output the first bitstream to the transmission module 27. On the other hand, the audio encoding module 24 can perform audio encoding on the first audio data (such as Opus encoding) to obtain the second bitstream (also called the audio bitstream) and output the second bitstream to the transmission module 27. And the video encoding module 26 can perform video encoding on the video data (such as H.264 / H.245 / H.266 encoding, etc.) to obtain the fourth bitstream (also called the video bitstream) and output the fourth bitstream to the transmission module 27.

[0141] Subsequently, the transmission module 27 can send the first, second, and fourth bitstreams to the second device 202. Furthermore, the transmission module 27 can also be used to determine the network status between the first device 201 and the second device 202; when the transmission module 27 determines that the network status between the first device 201 and the second device 202 is poor, it can use FEC technology to add redundancy to the second bitstream; or, upon receiving an ARQ request, it can retransmit the second bitstream; thus, the packet loss rate of data packets in the second bitstream transmitted to the second device 202 can be reduced.

[0142] After the second device 202 receives the second bitstream, the audio preprocessing module 33 can preprocess the second bitstream to obtain the third bitstream. For example, the FEC module can use FEC technology to process the second bitstream to obtain the third bitstream. Alternatively, the ARQ module can send an ARQ request to the first device 201, and then generate the third bitstream based on the requested bitstream and the second bitstream. Another example is the PLC module, which can perform packet loss compensation (PLC) on the second bitstream to obtain the third bitstream. Afterwards, the audio preprocessing module 33 outputs the third bitstream to the audio decoding module 34. The audio decoding module 34 can decode the third bitstream to obtain reconstructed audio data and output the reconstructed audio data to the decision module. It should be noted that the reconstructed audio data is different from the first audio data; the audio decoding method used by the audio decoding module 34 corresponds to the audio encoding method used by the audio encoding module 24.

[0143] After the second device 202 receives the first bitstream, the file decoding module 31 can perform text decoding on the first bitstream to obtain reconstructed text data and output the reconstructed text data to the TTS processing module 32. It should be noted that the reconstructed text data output by the file decoding module 31 is the same as the text output by the ASR module 21; the decoding method used by the file decoding module 31 corresponds to the encoding method used by the text encoding module 22.

[0144] For example, the TTS processing module 32 can determine whether TTS processing is needed on the reconstructed text data based on the instructions from the audio preprocessing module 33 / decision module 35. When the instruction received by the TTS processing module 32 is a first instruction, the TTS processing module 32 can perform TTS processing on the reconstructed text data to obtain second audio data and output the second audio data to the decision module 35. When the instruction received by the TTS processing module 32 is a second instruction, the TTS processing module 32 may not perform TTS processing on the reconstructed text data.

[0145] The decision module 35 can be used to decide whether to output the reconstructed audio data, the second audio data, or the fusion result of the reconstructed audio data and the second audio data to the first playback module 37. The first playback module 31 can play the audio data input to the decision module 35. The first playback module 31 can be an audio playback module; for example, an audio playback module can include a speaker, a sound card, etc.

[0146] After the second device 202 receives the fourth bitstream, the video decoding module 36 can perform video decoding on the fourth bitstream to obtain reconstructed video data and output the reconstructed video data to the second playback module 38. The second playback module 38 can be a video playback module; for example, a video playback module may include a display screen, a graphics card, etc. It should be noted that the reconstructed video data output by the video decoding module 36 may be the same as or different from the video data output by the video acquisition module 25; the video decoding method used by the video decoding module 36 corresponds to the video encoding method used by the video acquisition module 25.

[0147] In addition, the text decoding module 31 can also output the reconstructed text data to the second playback module 38, which can display the corresponding reconstructed text data (i.e., display subtitles) during the playback of the reconstructed video data.

[0148] Figure 2B This is a schematic diagram of another data processing system 200 according to an embodiment of this application. Figure 2B In one embodiment, the transmitting end includes a third device 205 (i.e., the first sub-transmitting end mentioned above) and a fourth device 206 (i.e., the second sub-transmitting end mentioned above), and the receiving end is a second device 202.

[0149] Figure 2B The data processing system 200 includes a third device 205, a fourth device 206, a fifth device 208, and a second device 202.

[0150] The third device 205 may include a first SDK (203) for real-time audio communication / real-time audio and video communication, which may include: an audio acquisition module 23, an audio encoding module 24, a video acquisition module 25, a video encoding module 26, and a transmission module 27.

[0151] The fourth device 204 may include a Selective Forwarding Unit (SFU) 207 (a Selective Forwarding Unit can be considered as an example of a data forwarding unit), and the SFU (207) may include: a transmission module 41.

[0152] The fifth device 208 may include: an audio preprocessing module 33, an audio decoding module 34, an ASR module 21, and a text encoding module 22.

[0153] Figure 2B The process of the third device 205 generating the second and fourth bitstreams can be referred to Figure 2AThe process of the first device 201 generating the second and fourth bitstreams will not be described in detail here. After the second and fourth bitstreams are generated, the transmission module 27 of the third device 205 can send the second and fourth bitstreams to the fourth device 206. In addition, the transmission module 27 can also be used to determine the network status between the third device 205 and the fourth device 206; when the transmission module 27 determines that the network status between the third device 205 and the fourth device 206 is poor, redundancy can be added to the second bitstream using FEC technology; or the second bitstream can be retransmitted when an ARQ request is received; in this way, the packet loss rate of data packets in the second bitstream transmitted to the fourth device 206 can be reduced.

[0154] The transmission module 41 of the fourth device 206 can send the second bit stream to the fifth device 208.

[0155] After receiving the second bitstream, the fifth device 208 performs audio preprocessing 33 to obtain a third bitstream and outputs it to the audio decoding module 34. The audio decoding module 34 decodes the third bitstream to obtain reconstructed audio data and outputs it to the ASR module 21. The ASR module 21 performs speech recognition on the reconstructed audio data to obtain the text corresponding to the reconstructed audio data (which can also be considered the text corresponding to the first audio data) and outputs this text to the text encoding module 22. The text encoding module 22 encodes the text corresponding to the reconstructed audio data to obtain a first bitstream and sends it to the fourth device 206.

[0156] Subsequently, the transmission module 41 of the fourth device 206 can send the first, second, and fourth bitstreams to the second device 202. Furthermore, the transmission module 41 can also be used to determine the network status between the fourth device 206 and the second device 202. When the transmission module 41 determines that the network status between the fourth device 206 and the second device 202 is poor, it can use FEC technology to add redundancy to the second bitstream; or, upon receiving an ARQ request, it can retransmit the second bitstream. This can reduce the packet loss rate of data packets in the second bitstream transmitted to the second device 202.

[0157] Figure 2B The processing procedures of the second device 202 for the first, second, and fourth bitstreams can be referred to... Figure 2A The processing of the first, second, and fourth bitstreams by the second device 202 will not be described in detail here.

[0158] It should be noted that the audio preprocessing module 33, audio decoding module 34, ASR module 21 and text encoding module 22 can also be integrated into the SFU, and this application does not impose any restrictions on this.

[0159] contrast Figure 2A and Figure 2B It can be seen that, Figure 2A Voice recognition is performed by the first device 201. Figure 2B The fourth device 206 calls the fifth device 208 to perform speech recognition (or the fourth device 206 performs speech recognition (not shown)).

[0160] Figure 2C This is a schematic diagram of another data processing system 200 according to an embodiment of this application. Figure 2C In one embodiment, the transmitting end includes a third device 205 (i.e., the first sub-transmitting end mentioned above) and a fourth device 206 (i.e., the second sub-transmitting end mentioned above), and the receiving end is a second device 202.

[0161] Figure 2C The data processing system 200 includes a third device 205, a fourth device 206, and a second device 202.

[0162] Figure 2C The modules included in the third device 205 and the processing procedures of each module can be found in [reference]. Figure 2A The modules included in the first device 201 and the processing procedures of each module will not be described in detail here.

[0163] Figure 2C The modules included in the second device 202 and the processing procedures of each module can be found in [reference]. Figure 2A The modules included in the second device 202 and the processing procedures of each module will not be described in detail here.

[0164] Figure 2C The fourth device 206 includes a transmission module 41, which is used only for data forwarding.

[0165] contrast Figure 2A and Figure 2C It can be seen that, Figure 2A The first device 201 directly sends the first, second, and fourth bitstreams to the second device 202; while Figure 2C The third device 205 forwards the first, second, and fourth streams to the second device 202 via the fourth device 206.

[0166] It should be understood that this application does not limit whether the first bitstream is generated by the sending end or by the data forwarding unit (including both cases where the data forwarding unit generates the first bitstream locally and cases where the data forwarding unit calls other services / other devices to generate the first bitstream).

[0167] It should be understood that among multiple terminal devices conducting real-time audio communication or real-time audio and video communication, any one of the terminal devices can act as a sender (i.e., the first or third device) or a receiver (i.e., the second device).

[0168] The following is Figure 2B Taking the data processing system 200 as an example, the strategy for the fourth device to transmit the first, second, and fourth bit streams is explained.

[0169] If video data is encoded using the H.264 encoding method, the resulting fourth bitrate is generally around 300Kbps to 4Mbps (depending on encoding parameters such as resolution and frame rate).

[0170] Based on a 16kHz sampling rate, 16-bit depth, mono, and 20ms frame length, the data size of each audio frame = bit depth * sampling rate * frame length * channels / (1000 * 8) = 16 * 16000 * 20 * 1 / (1000 * 8) = 640 bytes. After Opus encoding (estimated at a compression ratio of 10:1), the bitrate of one audio frame = (640B * 8 / 10) * (1000 / 20) = 25.6 kbps; that is, the bitrate of the second stream = 25.6 kbps. It should be understood that higher sampling rates, more channels, etc., will lead to a higher bitrate for the second stream.

[0171] If we assume a normal human speaking speed of 180 Chinese characters per minute, or 3 Chinese characters per second, and each Chinese character uses 3 bytes for UTF-8 encoding, then the bitrate of 3 Chinese characters after UTF-8 encoding is 3 Chinese characters * 3 bytes * 8 = 72 bps. In other words, the bitrate of the first stream is 72 bps; and the bitrate of the first stream is independent of factors such as audio sampling rate and number of channels.

[0172] A comparison shows that the bitrate of the fourth stream is higher than that of the second stream, and the bitrate of the second stream is higher than that of the first stream. Based on this, one design concept for the fourth device's transmission stream is to transmit the first, second, and fourth streams when network conditions are good; to transmit the first and second streams when network conditions are poor; and to transmit the first stream when network conditions are very poor. Because the bitrate of the first stream is very low, even in extremely weak network conditions (such as 5kbps), the available bandwidth is more than sufficient for transmitting the text stream, thus reducing the probability of packet loss. Therefore, the receiving end has a higher probability of receiving the complete text stream or recovering it. In this way, the second audio data obtained by the receiving end through TTS processing of the reconstructed text data from decoding the first stream is also complete or has a relatively short interruption duration. The second audio data can be directly used for playback or to supplement the reconstructed audio data, thereby ensuring relatively smooth audio playback at the receiving end.

[0173] Figure 3 This is a schematic diagram of a bitstream transmission process 300 according to an embodiment of this application. The bitstream transmission process 300 can be executed by the transmission module 41 in the fourth device 206. The fourth device 206 can execute the following S301 to S307 each time a bitstream is transmitted.

[0174] S301, the fourth device determines whether the network status between the fourth device and the second device meets the third condition.

[0175] For example, network status can be represented by at least one of the following network parameters: network transmission bandwidth, network transmission throughput, blocked data volume, or packet loss rate.

[0176] Network bandwidth refers to the data transmission rate, that is, the amount of data a network can transmit per unit of time. It is usually expressed in units such as bits per second (bps), kilobits per second (kbps), megabits per second (Mbps), or gigabits per second (Gbps).

[0177] Network throughput refers to the amount of data successfully transmitted per unit of time. It is usually expressed in bits per second (bps) or other equivalent units.

[0178] Congestion Data Volume: This refers to the amount of data that cannot be transmitted immediately due to insufficient network resources (such as bandwidth and buffer space) during network congestion. When the data traffic in the network exceeds the network's processing capacity, congestion occurs, causing data packets to queue and wait for transmission; this waiting data volume is the congestion data volume.

[0179] Packet loss rate: This refers to the ratio between the number of data packets lost during network transmission and the total number of data packets sent. Packet loss can be caused by network congestion, routing errors, hardware failures, or other reasons.

[0180] In one possible approach, the third condition may include at least one of the following: network transmission bandwidth is greater than (or equal to) a first bandwidth threshold, network transmission throughput is greater than (or equal to) a first throughput threshold, and the amount of blocked data is less than (or equal to) a first data amount threshold or the packet loss rate is less than (or equal to) a first packet loss rate threshold.

[0181] In one possible approach, the third condition could be that the weighted values ​​of multiple network parameters are greater than (or equal to) a first threshold. The weights corresponding to each network parameter can be set as needed, and this application does not impose any restrictions on this.

[0182] It should be understood that network status can also be represented by other network parameters, and this application does not impose any restrictions on this. The third condition can be set according to the type of network parameter used to represent the network status.

[0183] For example, when the network status between the fourth device 206 and the second device 202 meets the third condition, it indicates that the network status is better, and the fourth device 206 can execute S302; when the network status between the fourth device 206 and the second device 202 does not meet the third condition, it indicates that the network status is not better, and the fourth device 206 can execute S303.

[0184] S302, the fourth device sends the first bitstream, the second bitstream, and the fourth bitstream to the second device.

[0185] In other words, when the network conditions are relatively good, the fourth device 206 can send the first bitstream, the second bitstream, and the fourth bitstream to the second device 202.

[0186] In one possible approach, the first, second, and fourth streams can be three independent streams.

[0187] In one possible approach, any two or three of the first, second, and fourth bitstreams can be sub-bitstreams within the same bitstream.

[0188] S303, the fourth device determines whether the network status between the fourth device and the second device meets the first condition.

[0189] In one possible approach, the first condition may include at least one of the following: the network transmission bandwidth is less than (or equal to) a first bandwidth threshold and greater than (or equal to) a second bandwidth threshold; the network transmission throughput is less than (or equal to) a first throughput threshold and greater than (or equal to) a second bandwidth threshold; the amount of blocked data is greater than (or equal to) a first data amount threshold and less than (or equal to) a second data amount threshold; or the packet loss rate is greater than (or equal to) a first packet loss rate threshold and less than (or equal to) a second packet loss rate threshold.

[0190] Among them, the second bandwidth threshold is less than the first bandwidth threshold, the second throughput threshold is less than the first throughput threshold, the second data volume threshold is greater than the first data volume threshold, and the second packet loss rate threshold is greater than the first packet loss rate threshold.

[0191] In one possible approach, the first condition could be that the weighted values ​​of multiple network parameters are less than (or equal to) a first threshold and greater than (or equal to) a second threshold. The second threshold is less than the first threshold.

[0192] It should be understood that the first condition can be set according to the type of network parameters used to represent the network status.

[0193] When the network status between the fourth device 206 and the second device 202 does not meet the third condition, the fourth device 206 can determine whether the network status between the fourth device 206 and the second device 202 meets the first condition. When the network status between the fourth device 206 and the second device 202 meets the first condition, it indicates that the network status is poor, and the fourth device 206 can execute S304; when the network status between the fourth device 206 and the second device 202 does not meet the first condition, it indicates that the network status is very poor, and the fourth device 206 can execute S305.

[0194] S304, the fourth device sends the first and second bit streams to the second device.

[0195] In other words, when network conditions are poor, the fourth device 206 can send the first and second bitstreams to the second device 202. In this way, when network conditions become poor, available bandwidth can be reserved for audio and text bitstreams as much as possible to ensure audio communication quality.

[0196] For example, the first bitstream and the second bitstream can be two independent bitstreams, or two sub-bitstreams within the same bitstream.

[0197] S305, the fourth device determines whether the network status between the fourth device and the second device meets the second condition.

[0198] In one possible approach, the second condition may include at least one of the following: network transmission bandwidth is less than (or equal to) a second bandwidth threshold and greater than (or equal to) a third bandwidth threshold; network transmission throughput is less than (or equal to) a second throughput threshold and greater than (or equal to) a third bandwidth threshold; the amount of blocked data is greater than (or equal to) a second data amount threshold and less than (or equal to) a third data amount threshold; or the packet loss rate is greater than (or equal to) a second packet loss rate threshold and less than (or equal to) a third packet loss rate threshold.

[0199] Among them, the third bandwidth threshold is less than the second bandwidth threshold, the third throughput threshold is less than the second throughput threshold, the third data volume threshold is greater than the second data volume threshold, and the third packet loss rate threshold is greater than the second packet loss rate threshold.

[0200] In one possible approach, the second condition could be that the weighted values ​​of multiple network parameters are less than (or equal to) a second threshold and greater than (or equal to) a third threshold. The third threshold is less than the second threshold.

[0201] It should be understood that the second condition can be set according to the type of network parameters used to represent the network status.

[0202] When the network status between the fourth device 206 and the second device 202 does not meet the first condition, the fourth device 206 can determine whether the network status between the fourth device 206 and the second device 202 meets the second condition. When the network status between the fourth device 206 and the second device 202 meets the second condition, it indicates that the network status is very poor, and the fourth device 206 can execute S306; when the network status between the fourth device 206 and the second device 202 does not meet the second condition, it indicates that there is basically no network between the fourth device 206 and the second device 202, and at this time, the fourth device 206 can execute S307.

[0203] S306, the fourth device sends the first bitstream to the second device.

[0204] In other words, when the network conditions are poor, the bitstream sent by the fourth device 206 to the second device 202 only includes the first bitstream. In this way, when the network conditions become very poor, as much available bandwidth as possible can be reserved for the text bitstream, so as to ensure that the receiving end can hear the sending end's voice.

[0205] S307, the fourth device stops sending bitstreams to the second device.

[0206] For example, the bitstream in S307 may include any one of the first bitstream, the second bitstream, and the fourth bitstream.

[0207] In other words, when there is essentially no network connection between the fourth device 206 and the second device 202, the fourth device 206 can pause the transmission of the bitstream to the second device 202. After executing S307, it can return to execute S301.

[0208] It should be noted that the network state that satisfies the third condition is due to the network state that satisfies the first condition, and the network state that satisfies the first condition is due to the network state that satisfies the second condition.

[0209] It should be understood that, Figure 2C The fourth device 206 can also determine, based on the network status between the fourth device 206 and the second device 202, whether to send the first bitstream, the first bitstream and the second bitstream, or the first bitstream, the second bitstream and the fourth bitstream to the second device 202.

[0210] It should be understood that, Figure 2B and Figure 2C The third device can also determine, based on the network status between the third device 205 and the fourth device 206, whether to send the first bitstream, the first bitstream and the second bitstream, or the first bitstream, the second bitstream and the fourth bitstream to the fourth device 206.

[0211] It should be understood that, Figure 2A The first device 201 can also determine, based on the network status between the first device 201 and the second device 202, whether to send the first bitstream, the first bitstream and the second bitstream, or the first bitstream, the second bitstream and the fourth bitstream to the second device 202.

[0212] It should be understood that during real-time audio communication, the fourth device 206 can determine whether the network status between the fourth device 206 and the second device 202 meets the first condition. When the network status between the fourth device 206 and the second device 202 meets the first condition, the fourth device 206 sends the first bitstream and the second bitstream to the second device 202. When the network status between the fourth device 206 and the second device 202 does not meet the first condition, the fourth device 206 can determine whether the network status between the fourth device 206 and the second device 202 meets the second condition. When the fourth device 206 can determine that the network status between the fourth device 206 and the second device 202 meets the second condition, the fourth device 206 sends the first bitstream to the second device 202. When the fourth device 206 can determine that the network status between the fourth device 206 and the second device 202 does not meet the second condition, the fourth device 206 can pause sending the bit stream to the second device 202 (the bit stream can be the first bit stream or the second bit stream); then the fourth device 206 can determine again whether the network status between the fourth device 206 and the second device 202 meets the first condition, and so on.

[0213] Furthermore, existing technologies discard some non-critical frames or increase the audio compression rate when the network condition deteriorates, while the transmitting end of this application does not discard non-critical frames or increase the audio compression rate when the network condition deteriorates. Therefore, compared with the existing technology, the receiving end of this application has higher fidelity and clarity of the reconstructed audio data determined based on the second bitstream; consequently, the third audio data (obtained by splicing the reconstructed audio data and the second audio data) used for playback also has higher fidelity and clarity.

[0214] The following describes the data processing process of the second device 202 on the received bitstream.

[0215] Figure 4 This is a schematic diagram of a data processing procedure 400 according to an embodiment of this application. The data processing procedure 400 can be executed by the second device 202.

[0216] S401, receive a first bitstream and a second bitstream, the first bitstream including data obtained by encoding the text corresponding to the first audio data, and the second bitstream including data obtained by encoding the first audio data.

[0217] Based on the bitstream transmission process 300, it can be seen that the bitstream received by the second device 202 includes the following situations:

[0218] In one possible scenario, the second device 202 only receives the first bitstream.

[0219] In one possible scenario, the second device 202 receives only the first and second bitstreams.

[0220] In one possible scenario, the second device 202 receives the first bitstream, the second bitstream, and the fourth bitstream.

[0221] For example, after determining that the second bitstream has been received, the second device 202 then determines whether the reconstructed audio data generated based on the second bitstream is complete.

[0222] For example, the audio preprocessing module 33 in the second device 202 can determine whether the second bitstream has been received, or other modules in the second device 202, such as the communication module, can determine this. Figure 2B (Not shown in the image) Determine whether the second bitstream has been received.

[0223] If the reconstructed audio data is complete, it can be determined that there is no need to supplement it with second audio data obtained by performing TTS processing on the reconstructed text data obtained from decoding the first bitstream. In this case, the decision module 35 of the second device 202 can output the reconstructed audio data to the first playback module 37, which will then play the reconstructed audio data. If the reconstructed audio data is incomplete, it can be determined that there is a need to supplement it with second audio data obtained by performing TTS processing on the reconstructed text data obtained from decoding the first bitstream. In this case, step S402 can be executed.

[0224] In one possible approach, the decision module 35 can perform spectral analysis on the reconstructed audio data; then, based on the results of the spectral analysis (such as a spectrogram, which shows the energy distribution of the audio signal at different frequencies), it can determine whether the reconstructed audio data is complete (or whether the reconstructed audio data is interrupted).

[0225] For example, if a frequency component on the spectrogram suddenly disappears or decreases significantly at a certain point in time, it is determined that the reconstructed audio data was interrupted at that point in time.

[0226] For example, if the energy value at a certain time in the spectrogram suddenly drops or rises, it indicates that the reconstructed audio data was interrupted at that time.

[0227] For example, statistical analysis can be performed on the energy values ​​in the spectrogram, such as calculating the mean and variance of the energy values. If these statistical indicators change significantly at a certain point in time, it can be determined that the reconstructed audio data was interrupted at that point in time.

[0228] For example, an energy threshold can be preset, and when the energy value at a certain time point in the spectrogram is lower than the energy threshold, it can be determined that the reconstructed audio data has been interrupted at that time point.

[0229] In addition, time-domain analysis can be performed on the reconstructed audio data; for example, by observing the waveform, one can determine whether the reconstructed audio data is interrupted by observing whether the signal suddenly disappears at a certain point in time.

[0230] It should be understood that the energy distribution, frequency components, and energy abrupt changes in the spectrogram can also be observed, and time-domain analysis and statistical analysis can be combined to comprehensively determine whether there is an interruption in the reconstructed audio data.

[0231] In one possible approach, the audio preprocessing module 33 can determine whether the reconstructed audio data is complete by judging whether the third bitstream generated based on the second bitstream is complete; the specific process will be explained later. Judging whether the third bitstream is complete can mean judging whether the third bitstream contains all the data obtained from encoding the first audio data.

[0232] S402, in the case of incomplete reconstructed audio data, perform text-to-speech processing on the reconstructed text data to obtain the second audio data; fuse the second audio data and the reconstructed audio data to obtain the third audio data and output the third audio data. The reconstructed text data is obtained by decoding the first bitstream, and the reconstructed audio data is generated based on the second bitstream.

[0233] For example, if the decision module 35 or the audio preprocessing module 33 determines that the reconstructed audio data is incomplete, a first instruction can be sent to the TTS processing module 32. After receiving the first instruction, the TTS processing module 32 can perform text-to-speech processing on the reconstructed text data obtained by the text decoding module 31 from the first bitstream to obtain the second audio data and output the second audio data to the decision module 35.

[0234] Subsequently, the decision module 35 can fuse the second audio data and the reconstructed audio data to obtain the third audio data and output the third audio data to the first playback module 37; the process of the decision module 35 fusing the second audio data and reconstructing the audio data will be explained later. In this way, complete third audio data can be obtained, and thus the audio data played by the first playback module 37 is complete, ensuring the smoothness of the audio.

[0235] It should be noted that "outputting third audio data" in S402 can be understood as playing third audio data.

[0236] Figure 5 This is a schematic diagram of another data processing procedure 500 according to an embodiment of this application. Data processing procedure 500 describes the fusion process of the second audio data and the reconstructed audio data, based on data processing procedure 400. Data processing procedure 500 is executed by the second device 202.

[0237] S501, Receive the first bitstream, the first bitstream includes data obtained by encoding the text corresponding to the first audio data.

[0238] For example, S501 can be described with reference to the above description of S401, and will not be repeated here.

[0239] After S501, S502, S503 and S514 can be executed.

[0240] S502 decodes the first bitstream to obtain the reconstructed text data.

[0241] For example, the first bitstream can be decoded by the text decoding module 31 to obtain reconstructed text data and output the reconstructed text data to the TTS processing module 32.

[0242] S503 determines whether the second bitstream has been received.

[0243] In one possible approach, the headers of the first, second, and fourth streams include identifiers for identifying the stream type. Thus, the audio preprocessing module 33 of the second device 202 can distinguish between the first, second, and fourth streams by the value of the identifier in the header.

[0244] In one possible approach, the fourth device 206 sends the first, second, and fourth bitstreams through different interfaces. In this way, the communication module of the second device 202 can distinguish between the first, second, and fourth bitstreams based on the interface through which the bitstreams are received.

[0245] When the second bitstream is received, the second device 202 can execute S504. When the second bitstream is not received, the second device 202 can execute S507, and after executing S507, execute S513.

[0246] For example, when the audio preprocessing module 33 or the communication module determines that the second bitstream has been received, the audio preprocessing module 33 or the communication module may send a first instruction to the TTS processing module 32, which is used to instruct the TTS processing module 32 to perform TTS processing, that is, to execute S507.

[0247] It should be understood that this application does not restrict the execution order of S502 and S503.

[0248] S504, based on the second bitstream and the parsing result obtained from parsing the second bitstream, determines the third bitstream.

[0249] For example, the audio preprocessing module 33 can parse the second bitstream to obtain the parsing result; then, based on the parsing result and the second bitstream, a third bitstream is generated.

[0250] For example, the FEC module can parse the second bitstream to obtain a parsing result; then, the FEC module can determine whether the second bitstream has redundancy based on the parsing result; when it is determined that the second bitstream has redundancy, it can perform redundancy removal processing on the second bitstream to obtain the third bitstream. When it is determined that the second bitstream does not have redundancy, the second bitstream can be identified as the third bitstream.

[0251] For example, encoding an audio frame typically yields three data packets: packet 1, packet 2, and packet 3. To prevent packet loss, the sending end adds redundancy to the data packets corresponding to this audio frame, generating six data packets: packet 1, packet 2, packet 3, and packet 4. In other words, this audio frame corresponds to six data packets in the second bitstream.

[0252] When the second device 202 receives 4, 5, or 6 data packets corresponding to the audio frame in the second bitstream, it can be determined that the audio frame in the second bitstream has redundancy. In this case, the data packets corresponding to the audio frame in the second bitstream can be deredundantized, and the data packets of the audio frame after redundancy processing can be used as the data packets corresponding to the audio frame in the third bitstream. For example, when the second device 202 receives the following data packets corresponding to the audio frame in the second bitstream: data packet 1, data packet 2, and data packet 3, then data packet 1 can be discarded, and data packets 1, 2, and 3 can be used as the data packets corresponding to the audio frame in the third bitstream (at this time, the data packets corresponding to the audio frame in the third bitstream are complete).

[0253] When the second device 202 receives two or three data packets corresponding to the audio frame in the second bitstream, it can determine whether there are any identical data packets among the two or three data packets corresponding to the audio frame. If there are identical data packets among the two or three data packets corresponding to the audio frame, it can be determined that the audio frame in the second bitstream has redundancy. In this case, the data packets corresponding to the audio frame in the second bitstream can be deredundantized, and the data packets of the audio frame after redundancy processing can be used as the data packets corresponding to the audio frame in the third bitstream (at this time, the data packets corresponding to the audio frame in the third bitstream are incomplete). Otherwise, it is determined that the audio frame in the second bitstream does not have redundancy. In this case, the data packets corresponding to the audio frame in the second bitstream can be used as the data packets corresponding to the audio frame in the third bitstream (at this time, the data packets corresponding to the audio frame in the third bitstream may be complete or incomplete). For example, when the second device 202 receives the following data packets corresponding to the audio frame in the second bitstream: data packet 1, data packet 1, and data packet 2, data packet 1 can be discarded, and data packets 1 and 2 can be used as the data packets corresponding to the audio frame in the third bitstream. When the second device 202 receives data packets corresponding to the audio frame in the second bitstream, such as data packet 1 and data packet 2, then data packet 1 and data packet 2 are used as the data packets corresponding to the audio frame in the third bitstream.

[0254] When the second device 202 receives only one data packet corresponding to the audio frame in the second bitstream, it can be determined that there is no redundancy in the audio frame in the second bitstream. At this time, the data packet corresponding to the audio frame in the second bitstream can be used as the data packet corresponding to the audio frame in the third bitstream (at this time, the data packet corresponding to the audio frame in the third bitstream is incomplete).

[0255] For example, the ARQ module can parse the second bitstream to obtain a parsing result. Then, when the ARQ module determines missing data packets in the second bitstream based on the parsing result, it can send an ARQ request to the fourth device 206, which includes an identifier of the missing data packets. If the ARQ module receives a fifth bitstream (which includes all or part of the missing data packets), it can concatenate the second and fifth bitstreams to obtain a third bitstream (in this case, the data packets corresponding to the audio frame in the third bitstream may be complete or incomplete). If the ARQ module does not receive the fifth bitstream, it uses the second bitstream as the third bitstream (in this case, the data packets corresponding to the audio frame in the third bitstream are incomplete).

[0256] For example, the PLC module can parse the second bitstream received this time to obtain the parsing result; then, the PLC module can determine the missing data packets in the second bitstream received this time based on the parsing result. Subsequently, the PLC module can predict the missing data packets in the second bitstream received this time based on the previously received second bitstreams to obtain the sixth bitstream; the second bitstream and the sixth bitstream are then concatenated to obtain the third bitstream.

[0257] It should be understood that since the data packets predicted by the PLC module are typically less than 100ms, the number of data packets contained in the sixth stream may be the same as or different from the number of missing data packets in the second stream. Thus, the generated third stream may or may not be complete.

[0258] In one possible approach, any one of the FEC module, ARQ module, or PLC module is enabled.

[0259] In one possible approach, any two or more of the FEC module, ARQ module, or PLC module are enabled. In this case, any two or more of the FEC module, ARQ module, or PLC module can compare the completeness of their respective generated third bitstreams and output the third bitstream with the highest completeness to the audio decoding module 34.

[0260] After S504, S505 and S506 can be executed.

[0261] The S505 decodes the third bitstream to obtain the reconstructed audio data.

[0262] For example, the audio decoding module 34 can decode the third bitstream to obtain reconstructed audio data; then, the audio decoding module 34 can output the reconstructed audio data to the decision module 35.

[0263] S506, determine whether the third bitstream includes all the data obtained from encoding the first audio data.

[0264] Among them, it is determined whether the third bitstream includes all the data obtained from encoding the first audio data, that is, whether the third bitstream is complete.

[0265] For example, the audio preprocessing module 33 can determine whether the third bitstream is complete. Specifically, during the generation of the third bitstream, the FEC module and ARQ module determine whether the third bitstream is complete based on the completeness of the data packets corresponding to each audio frame in the third bitstream. If there are audio frames in the third bitstream with incomplete corresponding data packets, it indicates that the third bitstream is incomplete; if there are no audio frames with incomplete corresponding data packets, it indicates that the third bitstream is complete. During the generation of the third bitstream, the PLC module can determine whether the sixth bitstream contains fewer data packets than the missing data packets in the second bitstream; if the sixth bitstream contains fewer data packets than the missing data packets in the second bitstream, it can determine that the third bitstream is incomplete; otherwise, it can determine that the third bitstream is complete.

[0266] When the third bitstream includes only a portion of the data obtained from encoding the first audio data (i.e., the third bitstream is incomplete), the second device 202 can execute S507 to S511. When the third bitstream includes all the data obtained from encoding the first audio data (i.e., the third bitstream is complete), the second device 202 can execute S512. Furthermore, when the third bitstream includes all the data obtained from encoding the first audio data (i.e., the third bitstream is complete), the audio preprocessing module 33 can send a second instruction to the TTS processing module 32, the second instruction being used to instruct the TTS processing module 32 not to perform TTS processing.

[0267] For example, this application does not limit the execution order of S505 and S506.

[0268] S507, perform text-to-speech processing on the reconstructed text data to obtain the second audio data.

[0269] For example, when the TTS processing module 32 receives the first instruction, it can perform text-to-speech processing on the reconstructed text data to obtain the second audio data and output the second audio data to the decision module 35.

[0270] For example, when the decision module 35 receives the reconstructed audio data and the second audio data, it can execute the following steps S508 to S510:

[0271] S508, determine the start and end times of the first missing segment in the reconstructed audio data.

[0272] For example, the decision module 35 can perform spectrum analysis on the reconstructed audio data to obtain the spectrum analysis results of the reconstructed audio data; then, the decision module 35 can determine the start and end times of the missing segment (hereinafter referred to as the first segment) in the reconstructed audio data based on the spectrum analysis results of the reconstructed audio data; for details, please refer to the above description, which will not be repeated here.

[0273] For example, the first segment may be one or more.

[0274] S509, extract the second segment corresponding to the start and end times from the second audio data.

[0275] For example, the decision module 35 can extract a segment (hereinafter referred to as the second segment) from the second audio data to supplement the missing first segment in the reconstructed audio data based on the start and end times of the missing first segment in the reconstructed audio data.

[0276] For example, there may be one or more second segments, and the number of first and second segments may be the same. For a given first segment, a second segment can be extracted from the second audio data based on the start and end times corresponding to that first segment.

[0277] S510 reconstructs audio data and the second segment based on the start and end times to obtain the third audio data.

[0278] In one possible approach, the decision module 35 can filter the reconstructed audio data and the second segment separately, and then splice the filtered reconstructed audio data and the filtered second segment together to obtain the third audio data.

[0279] In one possible approach, the decision module 35 can first splice and reconstruct the audio data and the second segment to obtain the third audio data; then, the third audio data is filtered.

[0280] S511, playing the third audio data.

[0281] In one possible approach, when the audio preprocessing module 33 determines that the reconstructed audio data is incomplete, the audio preprocessing module 33 can also send a third instruction to the decision module 35. After receiving the third instruction, the decision module 35 can output the third audio data it generates to the first playback module 37, which will then play the third audio data.

[0282] In one possible approach, when the decision module 35 determines that the reconstructed audio data is incomplete, after generating the third audio data, the decision module 35 can output the third audio data to the first playback module 37, which will then play the third audio data.

[0283] S512, play the reconstructed audio data.

[0284] In one possible approach, when the audio preprocessing module 33 determines that the reconstructed audio data is complete, the audio preprocessing module 33 can also send a fourth instruction to the decision module 35. After receiving the fourth instruction, the decision module 35 can output the reconstructed audio data to the first playback module 37, which will then play the reconstructed audio data.

[0285] In one possible approach, when the decision module 35 determines that the reconstructed audio data is complete, the decision module 35 can output the reconstructed audio data to the first playback module 37, which will then play the reconstructed audio data.

[0286] In one possible approach, if the decision module 35 fails to acquire the second audio data within a first preset time period, the decision module 35 can output the reconstructed audio data to the first playback module 37, which will then play the reconstructed audio data.

[0287] S513, play the second audio data.

[0288] In one possible approach, when the audio preprocessing module 33 / communication module determines that the bitstream does not contain the second bitstream, the audio preprocessing module 33 / communication module can also send a fifth instruction to the decision module 35. After receiving the fifth instruction, the decision module 35 can output the second audio data to the first playback module 37, and the first playback module 37 will execute S513.

[0289] For example, if the decision module 35 fails to acquire the reconstructed audio data within the second preset time period, the decision module 35 can output the second audio data to the first playback module 37, and the first playback module 37 can play the second audio data.

[0290] S514, if a fourth bitstream is also received, then the fourth bitstream is decoded to obtain the reconstructed video data.

[0291] For example, when the communication module receives the fourth bitstream, it can output the fourth bitstream to the video decoding module 36; the video decoding module 26 can use the fourth bitstream to obtain reconstructed video data and output the reconstructed video data to the second playback module 38.

[0292] S515, play the reconstructed video data.

[0293] For example, the second playback module 38 can play the reconstructed video data.

[0294] Figure 6This is a schematic diagram of a data processing apparatus 600 according to an embodiment of this application. The data processing apparatus 600 can be used to execute the methods of the foregoing embodiments; therefore, the beneficial effects it can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0295] For example, the data processing apparatus 600 may include:

[0296] The receiving module 601 is used to receive a first bitstream and a second bitstream. The first bitstream includes data obtained by encoding text corresponding to the first audio data, and the second bitstream includes data obtained by encoding the first audio data.

[0297] The text-to-speech processing module 602 is used to perform text-to-speech processing on the reconstructed text data to obtain the second audio data when it is determined that the reconstructed audio data is incomplete. The reconstructed audio data is generated based on the second bitstream, and the reconstructed text data is obtained by decoding the first bitstream.

[0298] The fusion module 603 is used to fuse the second audio data and the reconstructed audio data to obtain the third audio data when it is determined that the reconstructed audio data is incomplete.

[0299] Output module 604 is used to output third audio data when it is determined that the reconstructed audio data is incomplete.

[0300] Among them, the fusion module 603 may belong to the decision module 35 mentioned above, and the output module 60,4 may refer to the first playback module 37 mentioned above.

[0301] It should be noted that the data processing device 600 may be the second device described above, or it may be a part of the second device described above.

[0302] For example, the text-to-speech processing module 602 is further configured to perform text-to-speech processing on the reconstructed text data to obtain the second audio data if it is determined that the bitstream does not include the second bitstream.

[0303] Output module 604 is also used to output second audio data.

[0304] For example, the output module 604 is also configured to output the reconstructed audio data when it is determined that the reconstructed audio data is complete.

[0305] For example, the fusion module 603 is used to determine the start and end times of the missing first segment in the reconstructed audio data; extract the second segment corresponding to the start and end times from the second audio data; and splice the reconstructed audio data and the second segment to obtain the third audio data.

[0306] For example, the device may further include a judgment module for determining whether the reconstructed audio data is complete. The judgment module may be part of the decision module 35 described above, or it may also be part of the audio preprocessing module 33 described above.

[0307] The judgment module is used to generate a third bitstream based on the second bitstream and the parsing result obtained from parsing the second bitstream, and to reconstruct the audio data obtained by decoding the third bitstream; if the third bitstream includes all the data in the data obtained by encoding the first audio data, the reconstructed audio data is complete; if the third bitstream includes only part of the data obtained by encoding the first audio data, the reconstructed audio data is incomplete.

[0308] For example, the first and second streams are generated during real-time communication or real-time audio and video communication.

[0309] For example, the receiving module 601 is further configured to receive a fourth bitstream, the fourth bitstream including data obtained by encoding video data corresponding to the first audio data; the device may also include a video decoding module for decoding the fourth bitstream to obtain reconstructed video data; and the output module 604 is further configured to output the reconstructed video data.

[0310] In one example, Figure 7 The schematic block diagram illustrating an embodiment of the present application shows an apparatus 700. The apparatus 700 may include a processor 701 and a transceiver 702, and optionally, a memory 703.

[0311] The various components of device 700 are coupled together via bus 704, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus 704 in the figure.

[0312] Optionally, the memory 703 can be used to store instructions from the foregoing method embodiments. The processor 701 can be used to execute the instructions in the memory 703, and to control the transceiver 702 to receive signals and transmit signals.

[0313] The device 700 may be an electronic device or a chip of an electronic device in the above method embodiments. The electronic device may include any one of a first device, a second device, a third device, a fourth device, or a fifth device.

[0314] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0315] This application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the above-described related method are executed to achieve the steps of the method in the above embodiments. The interface circuit is a transceiver 702.

[0316] This embodiment also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the methods described in the above embodiments. Exemplarily, the computer-readable storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0317] This embodiment also provides a computer program product containing computer instructions that, when executed by a computer or processor, cause the computer to perform the aforementioned steps to implement the methods described in the above embodiments. Exemplarily, the computer program product can be stored in random access memory (RAM), flash memory, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, read-only optical discs (CD-ROMs), or any other form of storage medium well known in the art.

[0318] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0319] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0320] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0321] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0322] Any content in the various embodiments of this application, as well as any content in the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0323] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data processing method, characterized by, The method comprises: receiving a first code stream and a second code stream, the first code stream comprising data obtained by encoding text corresponding to first audio data, and the second code stream comprising data obtained by encoding the first audio data; in a case where reconstructed audio data is incomplete, performing text-to-speech processing on reconstructed text data to obtain second audio data, and fusing the second audio data and the reconstructed audio data to obtain third audio data and output the third audio data, the reconstructed text data being obtained by decoding the first code stream, and the reconstructed audio data being generated based on the second code stream.

2. The method of claim 1, wherein, The method further comprises: if the second code stream is not received, performing text-to-speech processing on the reconstructed text data to obtain the second audio data and output the second audio data.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: in a case where the reconstructed audio data is complete, outputting the reconstructed audio data.

4. The method according to any one of claims 1 to 3, characterized in that, The fusing the second audio data and the reconstructed audio data to obtain third audio data comprises: extracting a second segment corresponding to a start time and an end time of a first segment missing in the reconstructed audio data from the second audio data based on the start time and the end time; splicing the reconstructed audio data and the second segment based on the start time and the end time to obtain the third audio data.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: judging whether the reconstructed audio data is complete.

6. The method of claim 5, wherein, The judging whether the reconstructed audio data is complete comprises: generating a third code stream based on the second code stream and a parsing result obtained by parsing the second code stream, the reconstructed audio data being obtained by decoding the third code stream; in a case where the third code stream comprises all data in the data obtained by encoding the first audio data, the reconstructed audio data is complete; in a case where the third code stream comprises part of the data obtained by encoding the first audio data, the reconstructed audio data is incomplete.

7. The method according to any one of claims 1 to 6, characterized in that, The first code stream and the second code stream are generated in a process of real-time audio communication or real-time audio and video communication.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: receiving a fourth code stream, decoding the fourth code stream to obtain reconstructed video data, and outputting the reconstructed video data, the fourth code stream comprising data obtained by encoding video data corresponding to the first audio data.

9. A data processing system, characterized by The data processing system comprises a sending end and a receiving end, wherein: the sending end is configured to in a case where a network state between the sending end and the receiving end satisfies a first condition, send a first code stream and a second code stream, the first code stream comprising data obtained by encoding text corresponding to first audio data, and the second code stream comprising data obtained by encoding the first audio data; the receiving end is configured to receive the first code stream and the second code stream; in a case where reconstructed audio data is incomplete, perform text-to-speech processing on reconstructed text data to obtain second audio data, and fuse the second audio data and the reconstructed audio data to obtain third audio data and play the third audio data, the reconstructed text data being obtained by decoding the first code stream, and the reconstructed audio data being generated based on the second code stream.

10. The data processing system of claim 9, wherein the sending end is further configured to send the first code stream in a case where a network state between the sending end and the receiving end meets a second condition, the network state meeting the first condition being superior to the network state meeting the second condition; the receiving end is further configured to receive the first code stream; perform text-to-speech processing on the reconstructed text data to obtain the second audio data and play the second audio data.

11. The data processing system according to claim 9 or 10, characterized in that, the receiving end is further configured to play the reconstructed audio data in a case where the reconstructed audio data is complete.

12. The data processing system according to any one of claims 9 to 11, characterized in that, the receiving end is configured to extract, from the second audio data, a second segment corresponding to a start time and an end time of a first segment missing in the reconstructed audio data based on the start time and the end time; splice the reconstructed audio data and the second segment based on the start time and the end time to obtain the third audio data.

13. The data processing system according to any of claims 9 to 12, characterized in that, the sending end is configured to collect the first audio data; perform speech recognition on the first audio data to obtain text corresponding to the first audio data; encode the text corresponding to the first audio data to obtain the first code stream; encode the first audio data to obtain the second code stream.

14. The data processing system according to any of claims 9 to 12, characterized in that, the sending end comprises a first sub-sending end and a second sub-sending end, wherein: the first sub-sending end is configured to collect the first audio data, encode the first audio data to obtain the second code stream, and send the second code stream to the second sub-sending end; the second sub-sending end is configured to receive the second code stream, determine the reconstructed audio data based on the second code stream, perform speech recognition on the reconstructed audio data to obtain the text corresponding to the first audio data, encode the text corresponding to the first audio data to obtain the first code stream, and send the first code stream and the second code stream to the receiving end.

15. The data processing system according to any of claims 9 to 12, characterized in that, the sending end comprises a first sub-sending end and a second sub-sending end, wherein: the first sub-sending end is configured to collect the first audio data, perform speech recognition on the first audio data to obtain the text corresponding to the first audio data, encode the text corresponding to the first audio data to obtain the first code stream, and encode the first audio data to obtain the second code stream; and the second sub-sending end is configured to receive the first code stream and the second code stream, and forward the first code stream and the second code stream to the receiving end.

16. The data processing system according to any of claims 9 to 15, characterized in that, the sending end is further configured to send a fourth code stream in a case where a network state between the sending end and the receiving end meets a third condition, the fourth code stream comprising data obtained by encoding video data corresponding to the first audio data.

17. The data processing system according to any of claims 9 to 16, characterized in that, the sending end and the receiving end perform real-time audio communication or real-time audio-video communication.

18. A data processing apparatus, characterized by the data processing apparatus comprises: a receiving module configured to receive a first code stream and a second code stream, the first code stream comprising data obtained by encoding text corresponding to first audio data, and the second code stream comprising data obtained by encoding the first audio data; The text-to-speech processing module is configured to perform text-to-speech processing on the reconstructed text data to obtain second audio data in a case where the reconstructed audio data is incomplete, the reconstructed audio data being generated based on the second code stream, and the reconstructed text data being obtained by decoding the first code stream; The fusion module is configured to fuse the second audio data and the reconstructed audio data to obtain third audio data; The output module is configured to output the third audio data.

19. An electronic device, comprising: The electronic device comprises: a memory and a processor, the memory being coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device performs the method in any one of claims 1 to 8.

20. A chip, characterized by The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method in any one of claims 1 to 8 are performed.

21. A computer-readable storage medium, characterized in that, The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method in any one of claims 1 to 8 are performed.

22. A computer program product, characterised in that, The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method in any one of claims 1 to 8 are performed.

Images