Streaming transmission method of audio and video files and related equipment

Abstract

The invention relates to the technical field of audio and video transmission, and provides an audio and video file streaming transmission method and related equipment, and the method comprises the steps: splitting a to-be-transmitted audio and video file, and obtaining an audio file and a video file; setting a video decoding parameter according to the video file, and decoding the video file based on the video decoding parameter to obtain a video decoding result; setting an audio decoding parameter according to the audio file, and decoding the audio file based on the audio decoding parameter to obtain an audio decoding result; respectively compressing the video decoding result and the audio decoding result based on the network state of the audio and video sending end to obtain a video compression data stream and an audio compression data stream; and sending the video compression data stream and the audio compression data stream to an audio and video receiving end. The method provided by the invention can reduce the delay and resource consumption of audio and video transmission.

Description

Technical Field

[0001] This application relates to the field of audio and video transmission technology, and in particular to a method and related equipment for streaming audio and video files. Background Technology

[0002] With the improvement of network bandwidth and the advancement of technology, audio and video transmission technology will continue to develop. In the future, more efficient encoding algorithms and lower-latency transmission protocols may emerge to meet the growing demands of real-time audio and video applications. Currently, audio and video file transmission mainly employs the following technical solutions:

[0003] 1. Server-based file upload and download solution: Users need to upload audio and video files to the server first, and then the recipient downloads and plays them from the server.

[0004] 2. Streaming solutions based on traditional streaming media protocols such as RTMP (Real-Time Messaging Protocol) for live source push and HLS (HTTP Live Streaming) for live viewing: require dedicated streaming media servers for forwarding and processing.

[0005] 3. Progressive download scheme based on Hypertext Transfer Protocol (HTTP): Download files step by step via HTTP and play them while downloading.

[0006] Existing solutions have the following drawbacks:

[0007] The system is highly dependent on servers, requiring the deployment and maintenance of dedicated server infrastructure, which increases system complexity and cost; it suffers from high latency, as data needs to be relayed through servers, increasing network hops and processing delays; it has poor cross-platform compatibility, as different platforms have varying levels of support for streaming media protocols, requiring targeted development; it consumes a lot of resources, as servers need to handle a large number of concurrent streams, placing high demands on bandwidth and computing resources; it lacks real-time performance, as traditional solutions struggle to achieve low-latency real-time interaction; and its data segmentation strategies are simplistic, with existing solutions often using fixed-size data segments that do not consider the dynamic changes in audio and video content characteristics and network conditions, resulting in high data transmission latency.

[0008] It is evident that current audio and video transmission technologies suffer from high latency and high resource consumption. Summary of the Invention

[0009] This application provides a method and related equipment for streaming audio and video files, which can solve the problems of high latency and high resource consumption in audio and video transmission.

[0010] In a first aspect, embodiments of this application provide a method for streaming audio and video files, applied at an audio and video sending end, the streaming method comprising:

[0011] The audio and video files to be transmitted are split into audio files and video files; the video file includes multiple image frames, and the audio file includes the audio data corresponding to the video file.

[0012] The video decoding parameters are set according to the video file, and the video file is decoded based on the video decoding parameters to obtain the video decoding result; the video decoding result includes multiple decoded image frames;

[0013] The audio decoding parameters are set according to the audio file, and the audio file is decoded based on the audio decoding parameters to obtain the audio decoding result;

[0014] Based on the network status of the audio and video sending end, the video decoding result and the audio decoding result are compressed respectively to obtain the video compressed data stream and the audio compressed data stream.

[0015] The video compressed data stream and the audio compressed data stream are sent to the audio and video receiving end.

[0016] Optionally, video decoding parameters include the color format for each image frame;

[0017] Set the video decoding parameters according to the video file, including:

[0018] Calculate the color richness of the video file, and the color saturation of each image frame in the video file;

[0019] The color format of each image frame is set based on color richness and color saturation of each image frame.

[0020] Optionally, calculate the color richness of the video file, including:

[0021] Through the formula:

[0022] ;

[0023] Calculate color richness ;

[0024] in, This represents the average color saturation of the video file. This represents the variance of color saturation in a video file. This represents the color entropy value of the video file.

[0025] Optionally, based on color richness and color saturation of each image frame, the color format of each image frame can be set, including:

[0026] For each image frame, perform the following steps:

[0027] If the color richness is greater than the first richness threshold and the color saturation of the image frame is greater than the first saturation threshold, then the color format of the image frame is set to YUV420.

[0028] If the color richness is less than or equal to the first richness threshold and greater than the second richness threshold, and the color saturation of the image frame is less than or equal to the first saturation threshold and greater than the second saturation threshold, then the color format of the image frame is set to YUV422; the first richness threshold is greater than the second richness threshold, and the first saturation threshold is greater than the second saturation threshold.

[0029] If the color richness is less than the second richness threshold and the color saturation of the image frame is less than the second saturation threshold, then the color format of the image frame is set to YUV444.

[0030] Optionally, the video file is decoded based on video decoding parameters to obtain the video decoding result, including:

[0031] Identify the frame category of each image frame in the video file; the frame category is keyframe, forward reference frame, or bidirectional reference frame.

[0032] For each image frame, if the image frame is a keyframe, the first decoder is used to decode the image frame based on the video decoding parameters to obtain the decoded image frame. If the image frame is a forward reference frame or a bidirectional reference frame, the second decoder is used to decode the image frame based on the video decoding parameters to obtain the decoded image frame. The number of parameters of the first decoder is larger than that of the second decoder.

[0033] All decoded image frames are combined into one file to obtain the video decoding result.

[0034] Optionally, the audio file is decoded based on the audio decoding parameters to obtain the audio decoding result, including:

[0035] Using an audio decoder, the audio file is decrypted based on the audio decoding parameters to obtain the initial decoded audio;

[0036] Determine whether the initial decoded audio channel is stereo and whether the sampling rate is equal to the preset sampling rate;

[0037] If so, channel mapping is performed on the initial decoded audio to obtain the audio decoding result;

[0038] Otherwise, the initial decoded audio is resampled to obtain the audio decoding result.

[0039] Optionally, channel mapping is performed on the initial decoded audio to obtain the audio decoding result, including:

[0040] The right channel audio signal of the initial decoded audio is delayed and phase-modulated to obtain extended audio;

[0041] Multichannel mixing and phase modulation are performed on the extended audio to obtain the audio decoding result;

[0042] Optionally, the initial decoded audio is resampled to obtain the audio decoding result, including:

[0043] The initial decoded audio is divided into multiple audio segments;

[0044] For each audio segment, calculate the spectral centroid of the audio segment. If the spectral centroid is less than the first preset frequency, the audio segment is divided into a speech segment. If the spectral centroid is greater than or equal to the first preset frequency but less than the second preset frequency, the audio segment is divided into a mixed segment. If the spectral centroid is greater than the second preset frequency, the audio segment is divided into a music segment. The first preset frequency is less than the second preset frequency.

[0045] The first interpolation method is used to interpolate each speech segment to obtain the interpolated audio segment of each speech segment. The second interpolation method is used to interpolate each mixed segment to obtain the interpolated audio segment of each mixed segment. The third interpolation method is used to interpolate each music segment to obtain the interpolated audio segment of each music segment.

[0046] All interpolated audio segments are concatenated to obtain the audio decoding result.

[0047] Optionally, based on the network status of the audio and video transmitters, the video decoding results and audio decoding results are compressed separately to obtain compressed video data streams and compressed audio data streams, including:

[0048] The audio decoding result is segmented according to the network status to obtain multiple audio data blocks, and all audio data blocks are compressed to obtain an audio compressed data stream.

[0049] Set the priority of each decoded image frame in the video decoding result, and compress all decoded image frames according to the network status and all priorities to obtain the audio compressed data stream.

[0050] Secondly, embodiments of this application provide a streaming device for audio and video files, comprising:

[0051] The splitting module is used to split the audio and video files to be transmitted into audio files and video files; the video file includes multiple image frames, and the audio file includes the audio data corresponding to the video file.

[0052] The first decoding module is used to set video decoding parameters according to the video file, and decode the video file based on the video decoding parameters to obtain the video decoding result; the video decoding result includes multiple decoded image frames;

[0053] The second decoding module is used to set audio decoding parameters according to the audio file, and decode the audio file based on the audio decoding parameters to obtain the audio decoding result;

[0054] The compression module is used to compress the video decoding result and the audio decoding result respectively based on the network status of the audio and video sending end, so as to obtain the video compressed data stream and the audio compressed data stream;

[0055] The sending module is used to send compressed video data streams and compressed audio data streams to the audio / video receiving end.

[0056] Thirdly, embodiments of this application provide an audio / video transmitting end, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for streaming audio / video files.

[0057] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for streaming audio and video files.

[0058] The above-mentioned solution in this application has the following beneficial effects:

[0059] In the embodiments of this application, the audio and video files to be transmitted are split into audio and video files. Then, video decoding parameters are set according to the video file, and the video file is decoded based on these parameters to obtain a video decoding result. Similarly, audio decoding parameters are set according to the audio file, and the audio file is decoded based on these parameters to obtain an audio decoding result. Then, based on the network status of the audio / video sending end, the video and audio decoding results are compressed respectively to obtain a compressed video data stream and a compressed audio data stream. Finally, the compressed video and audio data streams are sent to the audio / video receiving end. Setting corresponding decoding parameters based on the video and audio files takes into account the inherent properties of the video and audio files, improving the rationality and practicality of the decoding parameters. Decoding the video and audio files using reasonable decoding parameters improves decoding reliability. Compressing the video and audio decoding results based on the network status of the audio / video sending end effectively considers the network conditions of the sending end, reducing file transmission latency. Directly sending the compressed file to the audio / video receiving end eliminates the need for server relay, enabling real-time audio / video interaction, reducing network hops and processing latency, and further reducing audio / video transmission latency and resource consumption.

[0060] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This application provides a method and related equipment for streaming audio and video files according to an embodiment of the present application;

[0063] Figure 2 A schematic diagram of the structure of an audio / video file streaming device provided in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of the structure of an audio / video transmitting end provided in an embodiment of this application. Detailed Implementation

[0065] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0066] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0067] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0068] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0069] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0070] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0071] To address the issue of low reliability in existing audio and video transmission methods, this application provides a streaming method for audio and video files. This method involves splitting the audio and video file to be transmitted into an audio file and a video file. Then, video decoding parameters are set based on the video file, and the video file is decoded based on these parameters to obtain a video decoding result. Next, audio decoding parameters are set based on the audio file, and the audio file is decoded based on these parameters to obtain an audio decoding result. Then, based on the network status of the audio / video sending end, the video decoding result and the audio decoding result are compressed respectively to obtain a compressed video data stream and a compressed audio data stream. Finally, the compressed video data stream and the compressed audio data stream are sent to the audio / video receiving end. Specifically, the system sets corresponding decoding parameters based on video and audio files, taking into account the inherent properties of these files. This improves the rationality and practicality of the decoding parameters. Decoding video and audio files using reasonable parameters enhances decoding reliability. Furthermore, the system compresses the video and audio decoding results based on the network conditions of the audio / video sender, effectively considering the network status and reducing file transmission latency. The compressed files are then sent directly to the audio / video receiver without server intermediaries, enabling real-time audio / video interaction. This reduces network hops and processing latency, further minimizing audio / video transmission latency and resource consumption.

[0072] The following is an exemplary description of the audio and video file streaming method provided in this application.

[0073] like Figure 1 As shown, the audio and video file streaming method provided in this application includes the following steps:

[0074] Step 11: Split the audio and video files to be transmitted to obtain audio files and video files.

[0075] The aforementioned audio and video files to be transmitted are videos that need to be transmitted. The video files include multiple image frames and are compressed. The audio files include audio data corresponding to the video files and are also compressed.

[0076] In some embodiments of this application, the aforementioned audio and video sending end can be a terminal such as a live streaming device. Tools such as a file selector of the audio and video sending end can be used to obtain the audio and video files to be transmitted. An extractor can be used to split the audio and video files to be transmitted to obtain audio files and video files.

[0077] For example, create two independent multimedia extractors, such as audioExtractor and videoExtractor, to process audio and video tracks respectively. Set the same data source for both extractors using audioExtractor.setDataSource(filePath) and videoExtractor.setDataSource(filePath) to achieve parallel processing of the same file, obtaining audio and video data.

[0078] Step 12: Set the video decoding parameters according to the video file, and decode the video file based on the video decoding parameters to obtain the video decoding result.

[0079] The video decoding result above is the raw video data stream. The video decoding parameters include the color format of each image frame.

[0080] In some embodiments of this application, the steps described above—setting video decoding parameters based on the video file and decoding the video file based on the video decoding parameters to obtain the video decoding result—include:

[0081] The first step is to calculate the color richness of the video file and the color saturation of each image frame in the video file.

[0082] Specifically, through the formula:

[0083] ;

[0084] Calculate color richness .

[0085] in, This represents the average color saturation of the video file. This represents the variance of color saturation in a video file. This represents the color entropy value of the video file.

[0086] For example, the color saturation of an image frame can be calculated using the Hue Saturation Value (HSV) color space analysis method.

[0087] The second step is to set the color format for each image frame based on color richness and color saturation.

[0088] Specifically, for each image frame, the following steps are performed:

[0089] If the color richness is greater than the first richness threshold and the color saturation of the image frame is greater than the first saturation threshold, then the color format of the image frame is set to YUV420.

[0090] If the color richness is less than or equal to the first richness threshold and greater than the second richness threshold, and the color saturation of the image frame is less than or equal to the first saturation threshold and greater than the second saturation threshold, then the color format of the image frame is set to YUV422; the first richness threshold is greater than the second richness threshold, and the first saturation threshold is greater than the second saturation threshold.

[0091] If the color richness is less than the second richness threshold and the color saturation of the image frame is less than the second saturation threshold, then the color format of the image frame is set to YUV444.

[0092] It's important to note that YUV is a color format widely used in video compression, image processing, and digital video. It represents color by separating luminance (Y) and chrominance (UV) information. YUV420 means that every four pixels share a pair of chrominance components (U and V), implying that the sampling rate of the chrominance components is 1 / 4 that of the luminance components. YUV422 means that every two pixels share a pair of chrominance components (U and V), implying that the sampling rate of the chrominance components is 1 / 2 that of the luminance components. YUV444 means that each pixel has independent luminance (Y), chrominance U, and chrominance V components, implying that the sampling rates of the luminance and chrominance components are the same.

[0093] For example, the first richness threshold can be 0.8, the second richness threshold can be 0.5, the first saturation threshold can be 0.7, and the second saturation threshold can be 0.3. In other cases, the color format of the image frame is set to YUV420.

[0094] It should be noted that, in order to avoid efficiency loss due to frequent switching, after setting the color format for an image frame, multiple consecutive image frames can use the same color format. Then, the color format of subsequent multiple image frames can be analyzed through the above process. If multiple consecutive image frames have the same color format and are different from the previous color format, then switch to that color format.

[0095] In some embodiments of this application, the video decoding parameters also include resolution. The optimal output resolution is dynamically selected based on the detail density and motion intensity of the video content; the original resolution is maintained for static content, while the resolution is appropriately reduced for fast-moving scenes to improve smoothness.

[0096] The third step is to identify the frame category of each image frame in the video file.

[0097] The frame type is keyframe, forward reference frame, or bidirectional reference frame.

[0098] For example, the frame category of an initial image frame can be identified through the network abstraction layer unit (NALU) header and the slice header information of the image frame.

[0099] The fourth step involves each image frame separately. If the image frame is a keyframe, the first decoder is used to decode the image frame based on the video decoding parameters to obtain the decoded image frame. If the image frame is a forward reference frame or a bidirectional reference frame, the second decoder is used to decode the image frame based on the video decoding parameters to obtain the decoded image frame.

[0100] The number of parameters in the first decoder is greater than that in the second decoder (i.e., the number of parameters in the decoding algorithm of the decoder, which can reflect the performance of the decoder; the higher the number of parameters, the higher the performance of the decoder).

[0101] For example, during decoding, image frames are decoded into their corresponding color formats. The first decoder can be a decoder equipped with high-performance buffers and optimization algorithms (such as a High Efficiency Video Coding (HEVC) video decoder), and the second decoder can be a decoder employing a lightweight decoding strategy (such as an Advanced Video Coding (AVC) video decoder). Data sharing between decoders is achieved through a high-speed circular buffer, avoiding data copying overhead.

[0102] The fifth step is to combine all the decoded image frames into one file to obtain the video decoding result.

[0103] It should be noted that all decoded image frames are integrated according to the order of the image frames in the video file.

[0104] Step 13: Set the audio decoding parameters according to the audio file, and decode the audio file based on the audio decoding parameters to obtain the audio decoding result.

[0105] The above audio decoding result is the decoded audio information, i.e., the compressed audio data stream.

[0106] In some embodiments of this application, the steps of setting audio decoding parameters based on the audio file and decoding the audio file based on the audio decoding parameters to obtain the audio decoding result include:

[0107] The first step is to set the audio decoding parameters according to the audio file.

[0108] For example, you can use MediaExtractor.readSampleData() to read audio files in a loop and set the audio decoding parameters.

[0109] The above audio decoding parameters are used to describe the file format of the decoded audio.

[0110] The second step is to use an audio decoder to decrypt the audio file based on the audio decoding parameters to obtain the initial decoded audio.

[0111] For example, the MediaCodec decoder decodes an audio file into PCM format based on audio decoding parameters to obtain the initial decoded audio.

[0112] The third step is to determine whether the initial decoded audio channel is stereo and whether the sampling rate is equal to the preset sampling rate (e.g., 48kHz).

[0113] If so, channel mapping is performed on the initial decoded audio to obtain the audio decoding result.

[0114] Otherwise, the initial decoded audio is resampled to obtain the audio decoding result.

[0115] It should be noted that the above steps for channel mapping of the initial decoded audio to obtain the audio decoding result include:

[0116] The right channel audio signal of the initial decoded audio is delayed and phase-modulated to obtain extended audio. Then, the extended audio is multi-channel mixed and phase-modulated to obtain the audio decoding result.

[0117] For example, the left channel audio signal of the initial decoded audio remains unchanged. The process of delaying and phase modulating the right channel audio signal is as follows: the right channel audio signal is modulated by delaying the original audio signal by 0.5 milliseconds and then multiplying it by an attenuation factor (0.95), resulting in the extended audio corresponding to the initial decoded audio. Multichannel mixing can be achieved by linearly superimposing the multichannels. For example, when a center channel exists, the left channel output is equal to 0.707 times the front left channel plus 0.5 times the center channel plus 0.707 times the surround left channel, and the right channel output is equal to 0.707 times the front right channel plus 0.5 times the center channel plus 0.707 times the surround right channel (when only left and right channels exist, multichannel mixing can be omitted, and phase modulation can be performed directly). Then, the Hilbert transform can be used to perform phase modulation on the multichannel mixed audio, ensuring that the stereo width transform function is equal to the negative imaginary unit multiplied by the frequency sign function, thus obtaining the audio decoding result.

[0118] The steps described above for resampling the initial decoded audio to obtain the audio decoding result include:

[0119] First, the initial decoded audio is divided into multiple audio segments.

[0120] For example, the audio can be divided according to the sampling points of the initial decoded audio, such as each audio segment consisting of 1024 sampling points.

[0121] Then, for each audio segment, the spectral centroid of the audio segment is calculated. If the spectral centroid is less than the first preset frequency, the audio segment is divided into a speech segment. If the spectral centroid is greater than or equal to the first preset frequency but less than the second preset frequency, the audio segment is divided into a mixed segment. If the spectral centroid is greater than the second preset frequency, the audio segment is divided into a music segment. The first preset frequency is less than the second preset frequency.

[0122] Then, the first interpolation method is used to interpolate each speech segment to obtain the interpolated audio segment of each speech segment. The second interpolation method is used to interpolate each mixed segment to obtain the interpolated audio segment of each mixed segment. The third interpolation method is used to interpolate each music segment to obtain the interpolated audio segment of each music segment.

[0123] Finally, all interpolated audio segments are concatenated to obtain the audio decoding result.

[0124] It should be noted that when splicing interpolated audio segments, all interpolated audio segments are spliced ​​into one file according to the order of the audio segments in the audio file.

[0125] For example, the spectral centroid is calculated based on all frequency components and amplitudes of each audio segment. Specifically, each frequency component is multiplied by its corresponding amplitude, the sum of these multiplications is calculated, and then divided by the sum of all amplitudes to obtain the spectral centroid of the audio segment. The first preset frequency can be 1000Hz, and the second preset frequency can be 4000Hz. The first interpolation method can be cubic Hermite interpolation, the second interpolation method can be a sinc function-based interpolation method (Kaiser window weighted sinc interpolation), and the third interpolation method can be overlapping and summed frequency domain interpolation. The first interpolation method is computationally simple and suitable for real-time processing, but its accuracy is relatively low. The second interpolation method has higher accuracy and provides better results. The third interpolation method efficiently processes long signals and is suitable for long-term sequence signals.

[0126] Step 14: Based on the network status of the audio and video sending end, compress the video decoding result and the audio decoding result respectively to obtain the video compressed data stream and the audio compressed data stream.

[0127] The aforementioned network conditions include the round-trip latency, packet loss rate, and bandwidth utilization of the audio and video transmitters at the current moment.

[0128] In some embodiments of this application, the steps of compressing the video decoding result and the audio decoding result respectively based on the network status of the audio and video transmitting end to obtain the video compressed data stream and the audio compressed data stream include:

[0129] The first step is to segment the audio decoding result according to the network status to obtain multiple audio data blocks, and then compress all audio data blocks to obtain a compressed audio data stream.

[0130] For example, when the round-trip time is less than the first delay threshold (e.g., 50ms) and the packet loss rate is less than the first packet loss rate threshold (e.g., 0.1%), an aggressive segmentation strategy is adopted: the audio data block size is equal to the base block size multiplied by one, minus the difference of the spectral complexity factor multiplied by 0.3.

[0131] When the network round-trip time (RTT) is between the first and second delay thresholds (e.g., 50-150ms), a balanced segmentation strategy is adopted: the block size is dynamically adjusted according to the energy distribution of the audio decoding result, and smaller blocks are used in areas with drastic energy changes.

[0132] When the network RTT is greater than the second delay threshold (e.g., 150ms) or the packet loss rate is greater than the second packet loss rate threshold (1%), a conservative segmentation strategy is adopted: the data block size is increased to reduce transmission overhead, and forward error correction coding is introduced.

[0133] It should be noted that an encoder can be used to compress each audio data block, and all compressed audio data blocks can be combined to obtain an audio compressed data stream.

[0134] The second step is to set the priority of each decoded image frame in the video decoding result, and compress all decoded image frames according to the network status and all priorities to obtain a compressed video data stream.

[0135] For example, if the decoded image frame is a keyframe, it has the highest priority; if it is a forward reference frame, it has the medium priority; and if it is a bidirectional reference frame, it has the lowest priority. When the network is congested (e.g., packet loss rate greater than a preset packet loss rate threshold, round-trip latency greater than a preset latency threshold, or bandwidth utilization greater than a preset utilization rate), the lowest priority image frames are discarded first, sacrificing data quality to improve data transmission efficiency and ensure smooth video playback. Then, the video decoding results are compressed in descending order of priority.

[0136] It should be noted that context-based Real-time Transport Protocol (RTP) header compression technology can be used, specifically:

[0137] The first step is packet header analysis: The standard RTP packet header is 12 bytes, while the extended packet header can be up to 20 bytes. We analyze the changing patterns of the packet header fields.

[0138] Step 2, Compression Strategy:

[0139] Serial number compression: Utilizing the incremental characteristic of serial numbers, only the incremental value is transmitted, resulting in 1-2 bytes after compression.

[0140] Timestamp compression: Audio timestamps have a fixed step size of 960 (48kHz, 20ms), while video timestamps are calculated based on the frame rate and are compressed to 2-3 bytes.

[0141] Synchronous Source Reuse (SSRC): The SSRC remains unchanged within the same session and is not transmitted again after the connection is established.

[0142] When decompressing compressed data: the receiving end maintains the context state and reconstructs the complete RTP header based on the compression identifier.

[0143] In other words, when compressing files, the process first analyzes the changing patterns of the standard RTP header and its extended fields to identify which fields are predictable. Then, a compression strategy is developed using the temporal characteristics of these fields: incremental values ​​are transmitted for incrementing sequence numbers, compressed values ​​are transmitted after calculating timestamps at fixed steps or frame rates, and the session-invariant SSRC field is reused to reduce redundant transmissions. During decompression, the receiving end relies on established context information and compression identifiers to recover the complete RTP header, thereby significantly reducing bandwidth overhead without compromising transmission accuracy.

[0144] In some embodiments of this application, in order to improve the efficiency of file transfer, the compression bitrate is controlled based on the network status: the network quality index is defined as equal to the first weight coefficient multiplied by 1 minus the packet loss rate, plus the second weight coefficient multiplied by 1 minus the RTT and divided by the maximum RTT value, plus the third weight coefficient multiplied by the bandwidth utilization rate, wherein the first weight coefficient is 0.4, the second weight coefficient is 0.3, and the third weight coefficient is 0.3.

[0145] The adjusted bitrate is equal to the base bitrate multiplied by the quality factor and then by the congestion factor. The base bitrate is equal to the video resolution multiplied by the frame rate and then by the bitrate coefficient, where the coefficient is 0.1 for 720p and 0.2 for 1080p. The congestion factor is the larger of 0.1 and 1 minus twice the packet loss rate. When the packet loss rate exceeds 20%, the bitrate is forcibly reduced through the above process, and the bitrate is set to the calculated modulation bitrate value.

[0146] If the network quality index is greater than 0.8 and the packet loss rate is less than 1% for 5 consecutive sampling periods (i.e., the period of the preset modulation code rate), the code rate will be increased.

[0147] If the network quality index is less than 0.6 or the packet loss rate is greater than 3%, the bitrate will be reduced.

[0148] The increase is equal to the smaller of 1.5 and the result of multiplying the current bitrate by 1.2; the decrease is equal to the larger of 0.5 and the result of multiplying the current bitrate by 0.8.

[0149] The new bitrate is equal to 0.7 times the target bitrate plus 0.3 times the current bitrate, to avoid drastic bitrate fluctuations.

[0150] It should be noted that, in order to avoid quality fluctuations caused by frequent bitrate switching, a bitrate switching buffer mechanism was designed. Bitrate increase: It is only executed after 3 consecutive seconds of good network conditions (such as packet loss rate being less than the preset minimum packet loss rate threshold), and a gradual increase (20% each time) is adopted. Bitrate decrease: It is executed immediately when the network conditions deteriorate (such as packet loss rate being greater than the preset maximum packet loss rate threshold), and a rapid decrease (40% each time) is adopted. Stability protection: The same bitrate level is maintained for at least 5 seconds to avoid fluctuations.

[0151] For example, when compressing video data, if color space conversion is required, different color space conversion strategies are selected based on the content of the image frames:

[0152] For image frames with rich colors (such as color saturation greater than 0.7), a high-precision conversion algorithm is used, and a lookup table is used to accelerate the calculation.

[0153] For image frames with monotonous colors (such as color saturation less than 0.3), a simplified conversion algorithm is used to reduce computational overhead;

[0154] For image frames with rapid motion (such as those with inter-frame differences greater than a threshold), prioritize conversion speed and employ parallel processing.

[0155] The system also implements Single Instruction Multiple Data (SIMD) optimization, which utilizes the vector instructions of the Central Processing Unit (CPU) to process multiple pixels simultaneously, improving conversion efficiency by more than 40%.

[0156] For example, memory pool allocation optimization is implemented within the buffer of the audio / video sending end:

[0157] 1. Memory pool initialization: Pre-allocate 32MB of contiguous memory space, and divide it into block sizes according to powers of 2 (4KB, 8KB, 16KB, 32KB, 64KB, 128KB, 256KB, 512KB, 1MB, 2MB).

[0158] 2. Memory allocation algorithm:

[0159] Calculate the required memory size: The size of the buffer storing color data (RGBA buffer) is equal to the video width multiplied by the video height and then multiplied by 4 bytes;

[0160] Round up to the nearest power of 2: Allocate a size equal to a power of 2, which is the logarithm of the required size to the base 2, rounded up.

[0161] Obtain memory blocks from the free list at the corresponding level; if the level is empty, split from a higher level.

[0162] 3. Memory reclamation strategy:

[0163] Implement a reference counting mechanism, where each memory block maintains a reference counter;

[0164] When the reference count is 0, it is not released immediately, but added to the Least Recently Used (LRU) cache queue;

[0165] When the cache queue is full, the least recently used memory block is released according to the LRU policy;

[0166] 4. Memory fragmentation management:

[0167] Regularly check the memory fragmentation rate, which is equal to the difference between the total free memory and the maximum contiguous free memory, divided by the total free memory.

[0168] When the fragmentation rate exceeds 30%, start a background defragmentation thread;

[0169] Scrambling algorithm: Merge adjacent free blocks and reorganize the memory layout.

[0170] Step 15: Send the video compressed data stream and the audio compressed data stream to the audio and video receiving end.

[0171] The aforementioned audio and video receiving end is a terminal that plays the audio and video files sent by the audio and video sending end mentioned above, such as a user's mobile phone or other devices.

[0172] For example, an end-to-end P2P connection can be established through RTCPeerConnection. The compressed audio data stream is sent in real-time via the audio track of RTCRTpSender, and the compressed video data stream is sent via the video track. The audio and video receiver receives both the compressed audio and video data streams, decompresses them, and plays them.

[0173] In some embodiments of this application, in order to reduce file transmission latency, transmission parameters are set based on network status: the sending window size is equal to the smaller of the receiving window and the congestion window, wherein the congestion window is dynamically adjusted according to the packet loss rate; the retransmission timeout is equal to the smoothed RTT plus 4 times the RTT deviation, wherein the smoothed RTT is equal to 0.875 times the smoothed RTT plus 0.125 times the measured RTT; the packet sending interval is equal to the packet size divided by the available bandwidth, which is dynamically adjusted according to the bandwidth utilization rate; and the Broadband Delay Product (BBR) algorithm is used to optimize the sending rate by probing the bandwidth and RTT.

[0174] In another embodiment of this application, a synchronization algorithm based on timestamp interpolation compensation is designed to ensure audio and video synchronization:

[0175] 1. Timestamp generation: A 64-bit timestamp is generated using a high-precision timer with microsecond-level accuracy, and audio and video frames share the same clock source.

[0176] 2. Synchronization Deviation Calculation:

[0177] The audio-video discrepancy equals the video timestamp minus the audio timestamp.

[0178] Deviation threshold: ±40ms is acceptable; if it exceeds the range, synchronization correction will be initiated.

[0179] Correction strategy: Discard image frames when the deviation is greater than 40ms, and repeat audio frames when the deviation is less than -40ms.

[0180] 3. Adaptive buffer control:

[0181] The buffer size equals the base buffer plus the network jitter factor multiplied by the jitter value;

[0182] The base buffer is equal to 3 frames (approximately 100ms), and the network jitter factor is equal to 2.0.

[0183] When the buffer utilization exceeds 80%, the frame dropping mechanism is activated.

[0184] 4. Playback rate fine-tuning: Gradually correct long-term accumulated synchronization deviations by adjusting the playback rate (±2%).

[0185] It is worth mentioning that by setting corresponding decoding parameters based on video and audio files, the inherent attributes of video and audio files are taken into account, improving the rationality and practicality of the decoding parameters. Decoding video and audio files based on reasonable decoding parameters can improve the reliability of decoding. Compression of video and audio decoding results based on the network status of the audio and video sending end effectively takes into account the network conditions of the audio and video sending end, reduces file transmission latency, and directly sends the compressed file to the audio and video receiving end without going through the server relay, realizing real-time audio and video interaction, reducing network hops and processing latency, and further reducing audio and video transmission latency and resource consumption.

[0186] Furthermore, compared with the prior art, this application has the following significant technical advantages:

[0187] 1. Server independence: Based entirely on P2P architecture, there is no need to deploy a dedicated streaming media server, which greatly reduces system deployment costs and maintenance complexity.

[0188] 2. Ultra-low latency transmission: Through RTP packet header compression and adaptive transmission control, end-to-end latency can be reduced to 100-300ms, which is 60-80% lower than traditional server relay solutions.

[0189] 3. Cross-platform consistency: Based on Web standard technologies, a single codebase can support PCs, mobile devices, different operating systems, and browsers simultaneously, improving development efficiency by more than 50%.

[0190] 4. Decoder load balancing: Employs multi-level decoder scheduling, improving decoding efficiency by 40% and supporting more complex audio and video format processing.

[0191] 5. Audio processing technology innovation: Based on segmented resampling algorithm and channel processing technology, advanced algorithms such as Hermite interpolation and Kaiser window sinc interpolation are adopted, which improves audio quality by 60% compared with traditional linear interpolation.

[0192] 6. Adaptive Bitrate Optimization: Based on network state prediction, the bitrate control mechanism improves bandwidth utilization efficiency by 45% compared to traditional fixed bitrate schemes through network quality index quantification and congestion factor calculation, while ensuring stable visual quality.

[0193] 7. Audio Intelligent Segmentation Optimization: Based on the adaptive segmentation strategy of spectrum analysis and network status, combined with boundary detection of audio content awareness, audio transmission latency is reduced by 35% and sound quality loss is reduced by 50%.

[0194] 8. Memory Management Innovation: Based on the buddy allocator-based memory pool management mechanism, memory utilization efficiency is improved by 60% and video processing performance is improved by 40% through reference counting and LRU caching strategies.

[0195] 9. Precise synchronization mechanism: Based on the synchronization algorithm of timestamp difference compensation, the audio and video synchronization accuracy reaches ±40ms through microsecond-level timestamp and playback speed fine adjustment, which is 70% more accurate than the traditional solution.

[0196] 10. Intelligent resource scheduling: Dynamic allocation and load balancing of client computing resources can reduce server resource consumption by 85% compared to centralized server processing solutions, while increasing the overall system throughput by 30%.

[0197] 11. Transmission protocol optimization: RTP header compression and adaptive transmission control, with a header compression rate of 70% and a transmission efficiency improvement of 35% compared to standard WebRTC.

[0198] The test results obtained from testing the method of this application are as follows:

[0199] End-to-end latency: Average latency is 150ms, with 95% of requests having a latency of less than 300ms, a 70% reduction compared to traditional solutions;

[0200] Connection success rate: P2P direct connection success rate reaches 92%, and the success rate through TURN relay reaches 99.5%;

[0201] Transmission quality: 96% smooth playback rate for 1080p video and 99% for 720p video;

[0202] Bandwidth optimization: Saves 35% of bandwidth consumption compared to traditional server relay solutions.

[0203] Audio and video format support: Supports mainstream encoding formats such as H.264, H.265, AAC, and MP3, with a format coverage rate of 98%.

[0204] Network environment adaptability: It can work normally in different network environments such as 4G / 5G / WiFi.

[0205] The following is an exemplary description of the audio and video file streaming device provided in this application.

[0206] like Figure 2 As shown, this application embodiment provides an audio / video file streaming transmission device 200, which includes:

[0207] The splitting module 201 is used to split the audio and video files to be transmitted into audio files and video files; the video file includes multiple image frames, and the audio file includes the audio data corresponding to the video file.

[0208] The first decoding module 202 is used to set video decoding parameters according to the video file, and decode the video file based on the video decoding parameters to obtain the video decoding result; the video decoding result includes multiple decoded image frames;

[0209] The second decoding module 203 is used to set audio decoding parameters according to the audio file, and decode the audio file based on the audio decoding parameters to obtain the audio decoding result;

[0210] The compression module 204 is used to compress the video decoding result and the audio decoding result respectively based on the network status of the audio and video sending end, so as to obtain the video compressed data stream and the audio compressed data stream.

[0211] The sending module 205 is used to send the video compressed data stream and the audio compressed data stream to the audio and video receiving end.

[0212] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0213] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0214] like Figure 3 As shown, an embodiment of this application provides an audio / video transmitter, wherein the audio / video transmitter D10 of this embodiment includes: at least one processor D100 ( Figure 3 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 executes the computer program D102 to implement the steps in any of the above method embodiments.

[0215] Specifically, when the processor D100 executes the computer program D102, it splits the audio and video files to be transmitted into audio and video files. Then, it sets video decoding parameters according to the video files and decodes the video files based on the video decoding parameters to obtain video decoding results. Next, it sets audio decoding parameters according to the audio files and decodes the audio files based on the audio decoding parameters to obtain audio decoding results. Then, based on the network status of the audio and video sending end, it compresses the video decoding results and audio decoding results respectively to obtain compressed video data streams and compressed audio data streams. Finally, it sends the compressed video data streams and compressed audio data streams to the audio and video receiving end. Specifically, the system sets corresponding decoding parameters based on video and audio files, taking into account the inherent properties of these files. This improves the rationality and practicality of the decoding parameters. Decoding video and audio files using reasonable parameters enhances decoding reliability. Furthermore, the system compresses the video and audio decoding results based on the network conditions of the audio / video sender, effectively considering the network status and reducing file transmission latency. The compressed files are then sent directly to the audio / video receiver without server intermediaries, enabling real-time audio / video interaction. This reduces network hops and processing latency, further minimizing audio / video transmission latency and resource consumption.

[0216] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0217] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0218] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0219] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0220] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a streaming method apparatus / terminal device for audio and video files, recording media, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0221] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0222] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0223] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention.

Claims

1. A method for streaming audio and video files, characterized in that, Applied to the audio and video sending end, the method for streaming the audio and video files includes: The audio and video files to be transmitted are split into audio files and video files; the video file includes multiple image frames, and the audio file includes the audio data corresponding to the video file. Based on the video file, video decoding parameters are set, and the video file is decoded based on the video decoding parameters to obtain a video decoding result; the video decoding result includes multiple decoded image frames; The audio decoding parameters are set according to the audio file, and the audio file is decoded based on the audio decoding parameters to obtain the audio decoding result; Based on the network status of the audio and video sending end, the video decoding result and the audio decoding result are compressed respectively to obtain a video compressed data stream and an audio compressed data stream. The video compressed data stream and the audio compressed data stream are sent to the audio and video receiving end.

2. The streaming method according to claim 1, characterized in that, The video decoding parameters include the color format of each image frame; The step of setting video decoding parameters according to the video file includes: Calculate the color richness of the video file and the color saturation of each image frame in the video file; Based on the color richness and color saturation of each image frame, the color format of each image frame is set.

3. The streaming method according to claim 2, characterized in that, The calculation of the color richness of the video file includes: Through the formula: ； Calculate color richness ; in, This represents the average color saturation of the video file. This represents the variance of the color saturation of the video file. This represents the color entropy value of the video file.

4. The streaming method according to claim 3, characterized in that, The step of setting the color format for each image frame based on the color richness and the color saturation of each image frame includes: For each of the image frames, the following steps are performed: If the color richness is greater than the first richness threshold and the color saturation of the image frame is greater than the first saturation threshold, then the color format of the image frame is set to YUV420. If the color richness is less than or equal to the first richness threshold and greater than the second richness threshold, and the color saturation of the image frame is less than or equal to the first saturation threshold and greater than the second saturation threshold, then the color format of the image frame is set to YUV422; the first richness threshold is greater than the second richness threshold, and the first saturation threshold is greater than the second saturation threshold. If the color richness is less than the second richness threshold and the color saturation of the image frame is less than the second saturation threshold, then the color format of the image frame is set to YUV444.

5. The streaming method according to claim 1, characterized in that, The process of decoding the video file based on the video decoding parameters to obtain the video decoding result includes: Identify the frame category of each image frame in the video file; the frame category is a keyframe, a forward reference frame, or a bidirectional reference frame. For each image frame, if the image frame is a keyframe, the first decoder is used to decode the image frame based on the video decoding parameters to obtain the decoded image frame; if the image frame is a forward reference frame or a bidirectional reference frame, the second decoder is used to decode the image frame based on the video decoding parameters to obtain the decoded image frame; the parameter magnitude of the first decoder is greater than that of the second decoder. All decoded image frames are combined into one file to obtain the video decoding result.

6. The streaming method according to claim 1, characterized in that, The process of decoding the audio file based on the audio decoding parameters to obtain the audio decoding result includes: The audio file is decrypted using an audio decoder based on the audio decoding parameters to obtain the initial decoded audio. Determine whether the initial decoded audio channel is stereo and whether the sampling rate is equal to the preset sampling rate; If so, channel mapping is performed on the initial decoded audio to obtain the audio decoding result; Otherwise, the initial decoded audio is resampled to obtain the audio decoding result.

7. The streaming method according to claim 6, characterized in that, The step of performing channel mapping on the initial decoded audio to obtain the audio decoding result includes: The right channel audio signal of the initial decoded audio is delayed and phase-modulated to obtain extended audio; The extended audio is multichannel mixed and phase modulated to obtain the audio decoding result.

8. The streaming method according to claim 7, characterized in that, The step of resampling the initial decoded audio to obtain the audio decoding result includes: The initial decoded audio is divided into multiple audio segments; For each audio segment, the spectral centroid of the audio segment is calculated. If the spectral centroid is less than a first preset frequency, the audio segment is divided into a speech segment. If the spectral centroid is greater than or equal to the first preset frequency but less than a second preset frequency, the audio segment is divided into a mixed segment. If the spectral centroid is greater than the second preset frequency, the audio segment is divided into a music segment. The first preset frequency is less than the second preset frequency. The first interpolation method is used to interpolate each speech segment to obtain the interpolated audio segment of each speech segment. The second interpolation method is used to interpolate each mixed segment to obtain the interpolated audio segment of each mixed segment. The third interpolation method is used to interpolate each music segment to obtain the interpolated audio segment of each music segment. All interpolated audio segments are concatenated to obtain the audio decoding result.

9. The streaming method according to claim 1, characterized in that, Based on the network status of the audio / video transmitting end, the video decoding result and the audio decoding result are compressed respectively to obtain a video compressed data stream and an audio compressed data stream, including: The audio decoding result is segmented according to the network state to obtain multiple audio data blocks, and all audio data blocks are compressed to obtain an audio compressed data stream; The priority of each decoded image frame in the video decoding result is set, and all decoded image frames are compressed according to the network status and all priorities to obtain an audio compressed data stream.

10. An audio / video transmitting end, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for streaming audio and video files as described in any one of claims 1 to 9.

Images