Data transmission method, sending end and receiving end

Abstract

The application discloses a data transmission method, a sending end and a receiving end, and relates to the field of audio and video signal transmission, and comprises the following steps: encoding original data to obtain a plurality of video frames; determining data levels to which the video frames belong according to video features of the video frames, wherein the data levels comprise a basic layer and an enhanced layer; sending the video frames corresponding to the basic layer to the receiving end through a first path; and sending the video frames corresponding to the enhanced layer to the receiving end through a second path, wherein the time delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path. Through layered transmission, the application preferentially guarantees low delay and high priority delivery of the basic layer video, and improves the quality of data transmission as a whole.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio and video signal transmission, in particular to a data transmission method, a sending end and a receiving end. BACKGROUND

[0002] The current AVoIP (Audio-Visual over Internet Protocol) technology mainly dynamically selects an optimal path for audio and video data transmission according to real-time detection of network bandwidth and time delay, so as to realize smooth and low-delay transmission.

[0003] However, when a network anomaly occurs, key frame loss and other situations are likely to occur, which may cause subsequent prediction frames to be unable to be correctly restored and decoded, resulting in mosaic, blurred pictures or audio lag in the finally restored video picture, and thus causing the quality of audio and video data transmission to decrease and the user experience to decrease. SUMMARY

[0004] The main purpose of the present application is to provide a data transmission method, a sending end and a receiving end, aiming to solve the technical problem of how to improve the transmission quality of data transmission.

[0005] To achieve the above-mentioned purpose, the present application provides a data transmission method, which is applied to a sending end, and the method comprises the following steps:

[0006] encoding original data to obtain a plurality of video frames;

[0007] determining a data level to which each video frame belongs according to the video features of each video frame, wherein the data level comprises a basic layer and an enhanced layer;

[0008] sending the video frames corresponding to the basic layer to a receiving end through a first path;

[0009] sending the video frames corresponding to the enhanced layer to the receiving end through a second path, wherein the time delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path.

[0010] In addition, to achieve the above-mentioned purpose, the present application further provides a data transmission method, which is applied to a receiving end, and the method comprises the following steps:

[0011] receive a plurality of video frames sent by a client through a preset first path and a preset second path, wherein the video frames are obtained by encoding original data by the client, the first path is used to send video frames of a data level of a base layer, the second path is used to send video frames of a data level of an enhancement layer, the data level of each video frame is determined according to a video feature of each video frame, the time delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path;

[0012] recombine and decode each video frame to obtain a restored video.

[0013] In addition, to achieve the above object, the application further provides a data transmission device, which is applied to a sending end and comprises:

[0014] an encoding module, configured to encode original data to obtain a plurality of video frames;

[0015] a layering module, configured to determine a data level to which each video frame belongs according to a video feature of each video frame, wherein the data level comprises a base layer and an enhancement layer;

[0016] a first sending module, configured to send video frames corresponding to the base layer to a receiving end through a first path;

[0017] a second sending module, configured to send video frames corresponding to the enhancement layer to the receiving end through a second path, wherein the time delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path.

[0018] In addition, to achieve the above object, the application further provides a data transmission device, which is applied to a receiving end and comprises:

[0019] a receiving module, configured to receive a plurality of video frames sent by a client through a preset first path and a preset second path, wherein the video frames are obtained by encoding original data by the client, the first path is used to send video frames of a data level of a base layer, the second path is used to send video frames of a data level of an enhancement layer, the data level of each video frame is determined according to a video feature of each video frame, the time delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path;

[0020] a decoding module, configured to recombine and decode each video frame to obtain a restored video.

[0021] In addition, to achieve the above object, the application further provides a sending end, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the data transmission method.

[0022] In addition, to achieve the above object, the application further provides a receiving end, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the data transmission method.

[0023] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the data transmission method.

[0024] In addition, to achieve the above object, the application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the data transmission method.

[0025] The one or more technical solutions provided by the application have at least the following technical effects: first, the sending end encodes the original data to convert the continuous media stream into a discrete frame sequence that is convenient for network transmission, so as to perform hierarchical processing; then, the data level to which each video frame belongs is determined according to the video features of each video frame, the importance of the video content is layered, and the differential transmission of the video frames of different data levels is realized; then, the video frames corresponding to the base layer are sent to the receiving end through the first path with lower delay and lower bandwidth, so as to ensure that the receiving end can quickly and stably receive the base layer data necessary to maintain video continuous playback and basic quality, thereby effectively reducing the initial loading time and the risk of freezing; at the same time, the video frames corresponding to the enhancement layer are sent to the receiving end through the second path with higher bandwidth, so as to fully utilize the high-bandwidth channel to transmit the enhancement layer data for improving the quality, and finally combine the base layer data to restore a higher-quality video picture. The application intelligently layers the original data before transmission, and according to the real-time state of the network, preferentially guarantees the low-delay transmission of the base layer to maintain video continuity, and optimizes the detail transmission by utilizing the high-bandwidth demand characteristics of the enhancement layer, so as to dynamically adapt to the network condition and maximize the use of available network resources; and the independent transmission of the first path and the second path realizes the load sharing and isolation of the video frames on different paths, avoids the overall quality degradation caused by the congestion of a single path, especially guarantees the low delay and high priority delivery of the base layer video, thereby improving the quality of data transmission as a whole and providing stable, continuous and high-quality audio and video experience for users. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating an embodiment of the data transmission method of this application.

[0029] Figure 2 A schematic diagram of the overall flow of the data transmission method provided in Embodiment 3 of this application;

[0030] Figure 3 This is a schematic diagram of the module structure of the data transmission device according to Embodiment 1 of this application;

[0031] Figure 4 This is a schematic diagram of the module structure of another data transmission device according to an embodiment of this application;

[0032] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment of the sending end involved in the data transmission method in this application embodiment;

[0033] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment of the receiving end involved in the data transmission method in this application embodiment.

[0034] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0035] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0036] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0037] When transmitting audio and video using AVOIP technology, keyframe loss can easily occur when there are sudden network anomalies. This may result in subsequent predicted frames not being correctly restored and decoded, leading to issues such as pixelation, blurry images, or audio stuttering in the final video, thus degrading the quality of audio and video transmission and the user experience.

[0038] The application provides a solution, according to the video features of each video frame, the video content is divided into a base layer and an enhancement layer, the importance of the video content is layered, and differential transmission of video frames of different contents is realized; then, according to parameters such as a delay and a bandwidth of each network path, a transmission path corresponding to the base layer and the enhancement layer is determined, wherein the base layer is preferentially allocated a first path with a low delay to ensure real-time performance and continuity of the video, and the enhancement layer is allocated a high-throughput path (a second path) with a high bandwidth weight to carry large traffic; then, according to a real-time state of the network, low-delay transmission of the base layer is preferentially ensured to maintain video continuity, and high-bandwidth demand characteristics of the enhancement layer are utilized to optimize detail transmission, so that network conditions can be dynamically adapted, available network resources can be maximized, and low-delay and high-priority delivery of the base layer video are ensured, and the quality of data transmission is improved.

[0039] It should be noted that the execution subject of the embodiment relates to a sending end and a receiving end, wherein the sending end or the receiving end can be an electronic device with data processing, network communication and program running functions, for example, a tablet computer, a personal computer, a mobile phone and the like. The sending end refers to an electronic device for sending original data, which is used to collect, encode and package original audio and video signals into a network transmissible format and then push the original audio and video signals to a network node. The receiving end refers to an electronic device for receiving data, which is used to receive a code stream or a data packet from the network, perform operations such as unpacking, deduplication, sorting, decoding and synchronization, and render to a screen and / or a loudspeaker.

[0040] Hereinafter, the sending end is taken as an example to illustrate the embodiment and each of the following embodiments. Based on this, the application embodiment provides a data transmission method, which is described below with reference to Figure 1 Figure 1 The flowchart of the first embodiment of the data transmission method of the application is shown in FIG. 1.

[0041] In the embodiment, the data transmission method includes steps S10-S40.

[0042] Step S10, encoding the original data to obtain a plurality of video frames.

[0043] The original data refers to a data stream that has not been compressed and encoded, which can include audio data and video data, such as analog audio signals collected by a microphone and analog video signals collected by a camera.

[0044] ​A video frame refers to a basic unit of video data after encoding, usually including an independent key frame (I frame) that can represent a complete image, a prediction frame (P frame) that records the difference information of the previous frame, and a bidirectional prediction frame (B frame) that records the difference between the previous and subsequent frames.

[0045] Exemplarily, the video data in the original data can be processed by a video encoder to remove redundant information in the original video data and encode the video stream in the original video data into a plurality of video frames.

[0046] In step S20, the data level to which each video frame belongs is determined according to the video features of each video frame, wherein the data level includes a base layer and an enhancement layer.

[0047] Video features refer to various attribute information in a video frame that can reflect its content, structure, motion, etc., which can be extracted and analyzed by image processing and computer vision algorithms, such as color histogram, texture feature, motion vector, etc.

[0048] Data level refers to a logical label used to identify the importance and transmission priority of data for hierarchical management, which can be mainly divided into a base layer and an enhancement layer.

[0049] The base layer (Base Layer) refers to the minimum data combination containing key information that ensures the basic understanding and availability of data, while the enhancement layer (Enhancement Layer) refers to a data set that provides additional information to improve data quality or enrich data content based on the base layer.

[0050] Exemplarily, after the video frames are divided, the base layer contains the minimum data set that the receiving end absolutely needs to reconstruct an identifiable and coherent video sequence, usually including I frames and a few key P frames; while the enhancement layer contains additional data for enhancing the video quality of the base layer, usually including most P frames, B frames, and possibly high-frequency detail information.

[0051] Exemplarily, feature extraction and analysis can be performed on each video frame by a video encoder, and video frames with rich video features are divided into the base layer, for example, video frames with high motion vectors and rich video features, such as large movements of characters or dynamic changes in the background, can be determined as video frames of the base layer.

[0052] It can be understood that by dividing the video frames into the base layer and the enhancement layer based on the video features, the transmission strategy can be dynamically adjusted according to the network conditions to achieve differentiated transmission and thus improve the network resource utilization.

[0053] In one possible implementation, the video features include motion vectors, histogram difference, and gradient variance mean, and step S20 includes:

[0054] Step S21: If the motion vector of a video frame is greater than the first threshold or the histogram difference is greater than the second threshold, the video frame is determined to be a video frame belonging to the base layer.

[0055] Motion vectors are parameters used to describe the direction and magnitude of motion of an image block (macroblock) from the previous frame to the current frame. They are usually obtained by the encoder during the P / B frame prediction stage and are used to quantify the motion activity of that frame.

[0056] A histogram is a statistical graph of pixel intensity distribution in an image; while histogram difference is a scalar value calculated by comparing the brightness (and / or color) histograms of the current frame with those of the previous frame. It is used to measure the overall change in brightness and color distribution between two frames, thereby determining whether a scene switch has occurred between the two frames.

[0057] Both the first and second thresholds are preset or dynamically adaptively adjusted critical parameters. The first threshold is used to compare with the motion vector of the video frame to determine whether a scene change has occurred. The second threshold is used to compare with the histogram difference of the video frame to determine whether a scene change has occurred. Both values ​​can be preset by the user or adjusted automatically according to network conditions. For example, when network latency is high, the threshold can be increased appropriately to reduce the number of video frames in the base layer and prioritize the normal transmission of base layer video frames.

[0058] For example, when the video frame is a P-frame or B-frame, the transmitting end can directly parse the motion vectors of each macroblock in the video frame from the encoder's output data, and perform frame-level statistical aggregation to obtain the motion vector of the video frame. Then, this motion vector is compared with a first threshold. If the motion vector is greater than the first threshold, it indicates that the video frame is a high-motion video frame, and it is classified into the base layer to ensure the continuity of the reconstructed video and avoid abrupt changes in video object movement. Alternatively, when the video frame is an I-frame, the "hist_diff" parameter output by the encoder can be directly read and determined as the histogram difference of the video frame. If the histogram difference is greater than a second threshold, it indicates that a scene switch has occurred between the video frame and the previous video frame, and this video frame is determined as a base layer video frame to ensure the continuity of the reconstructed video and avoid abrupt changes in scene movement.

[0059] Step S22: If the motion vector of the video frame is less than or equal to the first threshold, the histogram difference is less than or equal to the second threshold, and the mean gradient variance is less than or equal to the third threshold, then the video frame is determined to be a video frame belonging to the enhancement layer.

[0060] The gradient variance mean is a value obtained by calculating the variance of the gradient values of each local region in the video frame and then taking the average, and is used to reflect the richness and change of image details and edges in the video frame. The higher the value, the more complex the scene.

[0061] The third threshold is also a critical parameter that is dynamically and adaptively adjusted, and is used to determine the scene complexity of the current video frame.

[0062] Exemplarily, the sending end can directly use the SSIM (Structural Similarity Index) or BRISQUE (Blind / Reference-less Image Spatial Quality Evaluator) to determine the scene complexity of the video frame, but the resource consumption of the two is large; and when the resource is limited, the gradient variance value of the video frame in the encoder can be directly read, the mean value thereof is calculated, and the mean value is compared with the third threshold. If the gradient variance mean value of the video frame is greater than the third threshold, it indicates that the current scene complexity is high, and if the gradient variance mean value of the video frame is less than or equal to the third threshold, it indicates that the scene complexity is low.

[0063] Optionally, in the case that the motion vector of the video frame is less than or equal to the first threshold, the histogram difference is less than or equal to the second threshold, and the gradient variance mean is greater than the third threshold, the video frame is determined to be a video frame belonging to the base layer.

[0064] Exemplarily, for any video frame, in the case that the motion vector of the video frame is less than or equal to the first threshold and the histogram difference is less than or equal to the second threshold, it can be determined that the video frame neither appears high motion change nor scene switching; and the gradient variance mean of the video frame can be further used to determine whether the video frame belongs to the base layer or the enhancement layer. If the gradient variance mean is greater than the third threshold, it indicates that the scene complexity of the video frame is low, and even if the network is poor and the frame is lost, it will not affect the picture continuity of the restored video, so the video frame can be determined as a video frame of the enhancement layer.

[0065] In the embodiment, the importance of the video frame to maintaining the picture continuity is determined according to the video features such as the motion vector, the histogram difference and the gradient variance mean of the video frame, so that the accuracy of the video frame layering can be ensured, and the high-quality layered transmission is realized.

[0066] In step S30, the video frame corresponding to the base layer is sent to the receiving end through the first path.

[0067] In step S40, the video frame corresponding to the enhancement layer is sent to the receiving end through the second path, wherein the time delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path.

[0068] The first path represents a network path for transmitting the base layer data, and the second path represents a network path for transmitting the enhancement layer. The first path and the second path can be the same network path or two completely independent paths, and the embodiment does not make specific limitations on this.

[0069] For example, probe data packets can be continuously or periodically sent to all available network paths, and the current time delay and available bandwidth of each network path are measured. Then, the network path with lower time delay is determined as the first path, and the network path with higher available bandwidth is determined as the second path.

[0070] It can be understood that, based on the completely different transmission requirements (such as low delay, high bandwidth, etc.) of the base layer and the enhancement layer data, the base layer data with high priority and low delay requirement is guided to the most stable path to ensure the coherence and the lowest acceptable quality of the video stream, and the enhancement layer data with high bandwidth requirement is guided to the path with the largest capacity, which can provide the highest video clarity and details as much as possible when the network condition allows, thereby realizing the maximum utilization of network resources and improving the overall quality and user perception of video data transmission.

[0071] Optionally, between sending the video frames, the video frames can be packaged to obtain video packets corresponding to the video frames, and the video packets corresponding to the video frames of the base layer are sent to the receiving end through the first path, and the video packets corresponding to the video frames of the enhancement layer are sent to the receiving end through the second path.

[0072] Packaging refers to the process of encapsulating data (such as video frame data, audio frame data) of a specific format according to certain protocols and rules, adding necessary header information (such as sequence number, timestamp, etc.), so as to become a data packet that meets the transmission or storage requirements. Common network protocols include: RTP (Real-time Transport Protocol, Real-time Transport Protocol), SRTP (Secure RTP, Secure Real-time Transport Protocol), etc. The video packet refers to a data unit formed by packaging the video frame, which contains video frame data and its corresponding header information. The receiving end can analyze and process the video frame data according to the header information in the video packet.

[0073] For example, in the process of packaging the video frame, first, a packet header is constructed according to a preset network protocol, including steps of assigning a sequence number to the video frame, writing a timestamp according to the packaging time, and writing a base layer or enhancement layer identifier in the extension field of the packet header; and then the packet header and the video frame are encapsulated into a video packet.

[0074] In an implementation, step S10 comprises:

[0075] Step A10, encoding the original data to obtain a plurality of audio frames;

[0076] An audio frame refers to a basic unit of audio encoding, including compressed samples of all audio channels in an extremely short time (such as tens of milliseconds).

[0077] Exemplarily, the audio data in the original data can be processed by an audio encoder to be divided into individual audio frames.

[0078] The data transmission method further comprises:

[0079] Step A20, performing voice activity detection on each audio frame to obtain a detection result of each audio frame;

[0080] Voice activity detection (VAD) can be used to detect whether there is valid voice in an audio signal. In this embodiment, a voice activity detection method is used to detect whether there is human voice in an audio frame. The detection result can be a result indicating whether there is human voice in the audio frame.

[0081] Optionally, for any audio frame to be processed, the short-time energy and zero-crossing rate of the audio signal in the frame are calculated by a VAD algorithm; in the case where both the short-time energy and the zero-crossing rate of the frame are greater than a preset threshold, it is determined that the detection result of the frame is that there is human voice.

[0082] Exemplarily, the short-time energy of a frame of audio signal can be determined by calculating the sum of squares of the amplitudes of the sample points in the frame; the zero-crossing rate of a frame of audio signal can be determined by calculating the number of sign changes of the sample points in the frame.

[0083] Step A30, in the case where the detection result of the audio frame is that there is human voice, determining that the data level to which the audio frame belongs is the base layer;

[0084] Step A40, in the case where the detection result of the audio frame is that there is no human voice, determining that the data level to which the audio frame belongs is the enhancement layer;

[0085] Exemplarily, since the base layer contains the minimum data set absolutely necessary for the receiving end to reconstruct an identifiable and coherent audio sequence, and the enhancement layer contains additional data for enhancing audio quality, the audio frames detected to contain human voice are divided into the high-priority base layer, and the audio frames without human voice (silence or pure noise) are divided into the enhancement layer, thereby achieving differentiation of the importance of audio data, so as to realize differentiated transmission of audio data.

[0086] Optionally, after obtaining the detection result of whether the audio frame contains human voice, the audio frame containing human voice can be further determined as a candidate frame, and keyword detection is performed on each candidate frame to determine a candidate frame containing a preset keyword as a target frame. Furthermore, each target frame can be determined as an audio frame belonging to the base layer, and other audio frames other than the target frames can be determined as audio frames belonging to the enhancement layer.

[0087] For example, the process of keyword detection includes: for any candidate frame, Mel Frequency Cepstral Coefficents (MFCC) or Log-Mel Spectrogram are extracted from the candidate frame, and are used as a feature vector representing human auditory characteristics; then, the feature vector is input into a preset lightweight deep neural network or convolutional neural network, and a confidence score of the feature vector compared with a preset keyword is output; and then, a candidate frame with a confidence score greater than a preset confidence threshold is determined as a target frame.

[0088] In step A50, the audio frame corresponding to the base layer is sent to the receiving end through the first path, and the audio frame corresponding to the enhancement layer is sent to the receiving end through the second path.

[0089] It can be understood that the audio frame corresponding to the base layer is transmitted through the first path with low delay and stable transmission, which ensures the accurate transmission of key audio information; at the same time, the audio frame corresponding to the enhancement layer is transmitted through the second path with high bandwidth, which can provide as much audio detail as possible when the network condition allows, and the two together improve the quality of audio data transmission.

[0090] Optionally, the audio frame can also be packaged before being sent, and the specific processing process is the same as the packaging and sending process of the video frame in step S40, and thus will not be described again.

[0091] The embodiment provides a data transmission method. The sending end divides the video frame and the audio frame into a base layer and an enhancement layer according to the content characteristics of the video frame and the audio frame, so as to realize differentiated data transmission. Then, the video frame / audio frame corresponding to the base layer is sent to the receiving end through a first path with low delay, so as to ensure the stable transmission of the base layer data and ensure that the receiving end can decode a coherent and usable video / audio stream, effectively avoiding problems such as video / audio lag and discontinuity, thereby improving the quality of audio / video data transmission. At the same time, the video frame / audio frame corresponding to the enhancement layer is sent to the receiving end through a second path with high bandwidth, so that the overall scene details of the audio / video can be enriched when the network condition is good, and the overall quality of the audio / video data transmission and the user's viewing experience are improved.

[0092] Based on the first embodiment, a second embodiment of the data transmission method is proposed, which further comprises the following steps before step S30 in the embodiment:

[0093] In step S301, the health score of each network path is determined according to the network state parameters of each network path and the corresponding parameter weights, wherein the network state parameters include at least one of the delay, the jitter, the packet loss rate, the available bandwidth and the stability, the delay weight of the base layer is higher than that of the enhancement layer, the jitter weight of the base layer is higher than that of the enhancement layer, the packet loss rate weight of the base layer is higher than that of the enhancement layer, the stability weight of the base layer is lower than that of the enhancement layer, and the bandwidth weight of the base layer is lower than that of the enhancement layer.

[0094] The network path refers to a logical or physical communication channel that the data packet may pass through from the sending end to the receiving end, which can be an IP (Internet Protocol) route formed by different routing protocols, or different physical network interfaces (such as 5G cellular network, Wi-Fi, wired Ethernet) or a logical interface after binding.

[0095] The network state parameter refers to a set of indexes obtained by active or passive detection of the network, which are used to quantify the transmission quality of the path, mainly including: delay, jitter, packet loss rate, available bandwidth and stability; wherein the delay refers to the one-way or round-trip time of the data packet from the sending end to the receiving end, and the jitter refers to the variation of the delay, i.e. the variance of the time interval between consecutive data packets, the packet loss rate represents the proportion of the lost data packets in the total sent packets, the available bandwidth refers to the maximum data transmission efficiency that the network path can provide, and the stability refers to the ability of the network path to maintain normal transmission state within a period of time, which is determined based on the degree of change of other network state parameters (such as bandwidth, delay, packet loss rate, etc.).

[0096] The parameter weight refers to the coefficient assigned to each network state parameter in the health score calculation formula; the size of the weight determines the relative importance of the parameter in the health score of each network path.

[0097] The health score refers to a comprehensive measurement value for quantifying the current transmission quality of the network path, which can be obtained by weighted calculation of the network state parameters.

[0098] Optionally, the scores of the network state parameters and the corresponding parameter weights of the network state parameters can be multiplied and added to obtain the health score of each network path, wherein the network state parameters include at least one of the delay, the jitter, the packet loss rate, the available bandwidth and the stability.

[0099] For example, the formula for calculating the health score is as follows:

[0100]

[0101] in, , , , , These represent the latency weight, jitter weight, packet loss rate weight, bandwidth weight, and availability weight, respectively; f(D) represents the latency processing function, used to convert latency into a numerical value that quantifies the latency level; f(J) and f(P) represent the jitter and packet loss rate processing functions, respectively, used to evaluate the impact of jitter and packet loss on data transmission quality; f(AvailBW) represents the available bandwidth processing function, used to evaluate the bandwidth conditions of the network path; f(Stability) represents the stability processing function, used to evaluate the stability and consistency of the network path.

[0102] The health scores for the base layer and the enhancement layer can be set with two different sets of parameter weights. The latency weight of the base layer is higher than that of the enhancement layer, the jitter weight of the base layer is higher than that of the enhancement layer, the packet loss rate weight of the base layer is higher than that of the enhancement layer, the bandwidth weight of the base layer is lower than that of the enhancement layer, and the stability weight of the base layer is lower than that of the enhancement layer.

[0103] Step S302: Based on the health score of each network path, determine the first path and the second path from each network path.

[0104] For example, probe packets can be continuously or periodically sent to all available network paths to measure the current latency and available bandwidth of each network path. For each network path, health scores are calculated for the network path corresponding to the base layer and the enhancement layer using two sets of weight configuration data, respectively, where the latency weight of the base layer is higher than that of the enhancement layer, and the bandwidth weight of the base layer is lower than that of the enhancement layer. Then, the network path with the highest health score corresponding to the weight configuration data of the base layer is determined as the first path, and the network path with the highest health score corresponding to the weight configuration data of the enhancement layer is determined as the second path.

[0105] It can be understood that by setting different parameter weights according to the characteristics of the base layer and the enhancement layer, the adaptation degree of each network path to the video frames of different levels can be more accurately evaluated, the base layer data is guided to a high-reliability path (the first path) with low latency, low jitter and low packet loss rate, and the enhancement layer data is guided to a large-capacity path (the second path) with high bandwidth and high stability, thereby realizing the optimal pairing between the network path capability and the data transmission demand, and improving the quality of audio and video data transmission.

[0106] In a possible implementation, the first path includes a main path and a secondary path, and step S302 includes:

[0107] In step S3021, in the case that the video frame belongs to the base layer, the main path and the secondary path are determined according to the health score of each network path, wherein the main path is used to transmit all video packets corresponding to the video frame, and the secondary path is used to transmit the redundant data of the video frame according to a preset redundancy replication ratio.

[0108] In a possible implementation, the sending end can sort according to the health score, select two transmission paths for the same base layer video frame, determine the network path with a higher score as the main path and the network path with a lower score as the secondary path, and then transmit all data packets corresponding to the video frame through the main path, determine the redundant packets in proportion to the redundancy replication ratio from all data packets corresponding to the video frame, and transmit the redundant packets through the secondary path. When the main path has a problem, the secondary path can provide certain data backup and supplement, thereby improving the reliability and fault tolerance of video transmission.

[0109] The redundancy replication ratio is a preset parameter used to determine how many percentages of the original data packets are copied and transmitted through the secondary path, and by setting different redundancy replication ratios, the reliability and bandwidth utilization of transmission can be balanced.

[0110] For example, for the video frame of the base layer, the sending end can query the health scores of all network paths, select the one with the highest score as the main path and the one with the second highest score as the secondary path, then put all video packets corresponding to the video frame into the sending queue of the main path, at the same time, according to the preset redundancy replication ratio, copy the corresponding proportion of packets from all video packets corresponding to the video frame according to a rule (such as selecting one every few packets or randomly), and put them into the sending queue of the secondary path, then transmit all original video packets to the receiving end through the main path, and transmit the redundant video packets obtained by copying to the receiving end through the secondary path. After receiving the video packets transmitted by the main path and the secondary path, the receiving end can perform deduplication according to the sequence numbers of the video packets.

[0111] It can be understood that the base layer data is transmitted through the double paths, so that even if the main path is lost or interrupted, the receiving end can still recover the complete data from the redundant packets sent by the secondary path with a high probability, thereby reducing the possibility of base layer video frame loss caused by network burst exception, reducing the occurrence of video lag, frame loss and other phenomena, and improving the quality and reliability of audio and video transmission. At the same time, through the preset redundant replication ratio, the secondary path does not send all the video packets, but reasonably uses the bandwidth resources on the premise of ensuring a certain fault tolerance, which improves the reliability of transmission and avoids excessive occupation of bandwidth, and improves the utilization efficiency of the overall bandwidth.

[0112] In step S3022, in the case that the video frame belongs to the enhancement layer, the video frame is disassembled into multiple network abstraction layer units, and the second path corresponding to each network abstraction layer unit is determined through the weighted round-robin scheduling algorithm and the health score of each network path.

[0113] The network abstraction layer unit (NALU) is a syntax structure defined in the H.264, H.265 and other video coding standards, which represents the basic encapsulation unit of the encoded video data.

[0114] The weighted round-robin (WRR) scheduling algorithm is an algorithm for task allocation among multiple resources (such as network paths), which allocates tasks to each resource in a certain order according to the weight value of each resource. The higher the weight value, the higher the probability of being allocated to the task. In this embodiment, the resource refers to the network path, and the weight value is determined according to the health score of each network path.

[0115] For example, for the video frame of the enhancement layer, the sending end can read its code stream through the parser, and divide it into multiple independent NALUs according to the start code of the NALU; then, the health scores of the enhancement layer of all network paths are obtained, and the health scores of the network paths are used as their weights; then, through the WRR algorithm, the network paths are sequentially and circularly pointed, and the paths are allocated to each NALU according to their weights. The higher the weight, the higher the probability of being selected, and the more frequent the number.

[0116] Optionally, the FEC (Forward Error Correction) can be used to add redundant codes to each NALU, so that even if there is a packet loss in the transmission process of each NALU, other NALUs can also recover the complete video frame according to the redundant codes without retransmission, thereby improving the quality and efficiency of the enhancement layer video transmission.

[0117] It can be understood that for less important enhancement layer video frames, they are disassembled into smaller, independent transmission units (network abstraction layer units), and then using a weighted round robin scheduling algorithm, according to the health score of each path, the small units are dispersed to multiple paths for transmission at the same time, which can effectively aggregate the bandwidth of multiple paths, thereby efficiently and quickly transmitting large capacity enhancement layer data.

[0118] In a possible implementation, the data transmission method further includes:

[0119] Step E10, respectively determining the short-term fluctuation value and the long-term trend value of each network state parameter in the main path and the second path by a preset second-order exponentially weighted moving average model;

[0120] The exponentially weighted moving average (EWMA) model refers to a statistical method for calculating the weighted average value of time series data. The core idea is to assign an exponentially decaying weight to historical data over time, with closer data points having higher weights and more distant data points having lower weights. The formula is as follows: wherein is the EWMA value at time t, is the observation value of a certain network state parameter (such as delay, packet loss rate, etc.) at the current time, is a smoothing coefficient (0.05 <0.3), which determines the speed of weight decay.

[0121] The second-order EWMA model calculates the weighted average value by assigning different weights to data at different times, not only considering the relationship between the current data point and the previous time data point, but also further considering the second-order dynamic characteristics of the data points over time, thereby more effectively capturing the changing trend and fluctuation of the data.

[0122] The short-term fluctuation value refers to the EWMA value with a relatively large smoothing coefficient of the observation value at the current time, reflecting the rapid fluctuations of the network; and the long-term trend value refers to the EWMA value with a relatively small smoothing coefficient of the observation value at the current time, reflecting the overall trend of the network.

[0123] Exemplarily, the calculation formula of the second-order EWMA model of any network state parameter is as follows:

[0124]

[0125]

[0126] wherein, is the observation value of a certain network state parameter at the current time; and respectively represent the short-term fluctuation value and the long-term trend value of a certain network status parameter. and respectively represent the short-term fluctuation value and the long-term trend value of a certain network status parameter.

[0127] Step E20, in the case that there is at least one network status parameter whose difference between the short-term fluctuation value and the long-term trend value is greater than the preset fluctuation threshold value in the main path, increasing the redundant replication ratio of the secondary path of the base layer;

[0128] The fluctuation threshold value refers to a preset threshold value for determining whether the short-term fluctuation is severe enough to trigger the emergency mechanism.

[0129] Exemplarily, the sending end can monitor the change of each network status parameter of the main path in real time; then, according to the observation value of each network status parameter and the second-order EWMA model, the short-term fluctuation value and the long-term trend value of each network status parameter are calculated, and the difference between them is further calculated; then, the difference is compared with the preset fluctuation threshold value, if there is at least one network status parameter whose difference is greater than the fluctuation threshold value, it means that the network status of the current main path is deteriorating, and the emergency mechanism is automatically triggered, and the redundant replication ratio of the secondary path is appropriately increased, for example, the redundant replication ratio of the secondary path of the base layer is adjusted to 100%, that is, a complete copy is created for each base layer data packet and sent through the secondary path, so that even in the case of data loss or damage due to high load in the main path, the redundant data of the secondary path can be used for recovery, ensuring the integrity of the key data.

[0130] Step E30, in the case that there is at least one network status parameter whose difference between the short-term fluctuation value and the long-term trend value is greater than the fluctuation threshold value in the second path, adjusting the second path to a preset backup path.

[0131] The backup path refers to a predetermined backup network path; when the currently used second path fails, its performance decreases or meets a certain trigger condition, the sending end will automatically switch to this backup path to ensure the continuity and stability of data transmission. In the present embodiment, the backup path can be determined as a network path whose health score determined based on the weight configuration of the enhancement layer is lower than the original second path, or the health score of each network path can be determined based on the current network status parameters of each network path, and the backup path can be determined according to the re-determined health score, which is not specifically limited in the present embodiment.

[0132] ​Exemplarily, the sending end can monitor the change of each network state parameter of the second path in real time; the triggering process of the emergency mechanism is the same as that in step E20, and thus is not described herein again; then, after triggering the emergency mechanism, the second path is adjusted to the backup path with the suboptimal health score, so as to guarantee the continuity of the enhanced layer data transmission.

[0133] Optionally, in the case that the difference between the short-term fluctuation value and the long-term trend value of at least one network state parameter of the second path is greater than the fluctuation threshold, in addition to switching the backup path, the redundancy ratio of the FEC of the second path can also be increased; and the change of each network state parameter is continuously detected, and in the case that the difference of at least one network state parameter is greater than the fluctuation threshold for a time period greater than a preset time period, the backup path is switched to.

[0134] It can be understood that by analyzing the short-term fluctuation and the long-term trend of the parameters, the abnormal signs of the path can be perceived in advance before serious packet loss or delay occurs, so that the unstable network path can be quickly avoided, the transmission interruption caused by the abnormality of a single path can be effectively avoided, and the stability of the audio / video transmission can be improved.

[0135] In a feasible implementation, the data transmission method further includes:

[0136] By a preset change point detection algorithm, it is determined whether each network state parameter of the main path and the second path has a change point; in the case that at least one network state parameter of the main path has a change point, the redundancy copy ratio of the secondary path of the base layer is adjusted to 100%; in the case that at least one network state parameter of the second path has a change point, the second path is adjusted to a preset backup path.

[0137] The change point detection algorithm refers to an algorithm for identifying a point at which a statistical characteristic (such as mean, variance) of time series data suddenly changes, which can analyze the data sequence to determine whether the distribution rule of the data after a certain point is significantly and continuously deviated from that before the point; the point can be referred to as a change point, that is, a data point at which the distribution rule of the data is deviated. Common change point detection algorithms include CUSUM (Cumulative Sum) detection algorithm, Page-Hinkley detection algorithm, etc.

[0138] Exemplarily, after detecting the change point, an emergency mechanism is triggered immediately, and the specific emergency mechanism can refer to the specific implementation modes of steps E20 and E30, and thus is not described herein again.

[0139] In this embodiment, the fluctuation of the network state parameter is predicted at the sending end through the second-order EWMA model and / or the change point detection algorithm, so that the network state can be judged and actively scheduled in advance, and the stability and continuity of the audio and video data transmission are improved. The second-order EWMA model can be used for trend prediction and early congestion trend sensing by hundreds of milliseconds. The change point detection algorithm can be used to capture sudden abnormalities and achieve rapid response. By combining the second-order EWMA model and the change point detection algorithm, the end-to-end delay can be stabilized within 50 ms in a metropolitan area network / campus network environment, and the audio and video can remain smooth without obvious lag under 2% burst packet loss.

[0140] Based on the first and / or second embodiments, a third embodiment of the data transmission method of the present application is proposed. In this embodiment, the data transmission method is applied to the receiving end, and the data transmission method comprises:

[0141] Step A10, receiving a plurality of video frames sent by the client through a preset first path and a preset second path, wherein the video frames are obtained by encoding the original data by the client, the first path is used to send video frames of a basic layer, the second path is used to send video frames of an enhancement layer, the data level of each video frame is determined according to the video characteristics of each video frame, the time delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path;

[0142] Optionally, when receiving the data packets (including video packets and audio packets) of the first path and the second path, different reordering windows can be set respectively, wherein the reordering window for receiving the basic layer data from the first path is smaller than the reordering window for receiving the enhancement layer data from the second path.

[0143] Illustratively, the reordering window of the basic layer data can be set to 12-24 ms, and the reordering window of the enhancement layer data can be set to 40-80 ms, allowing a larger delay. Within the reordering window, the data packets in the same video frame / audio frame can be sorted according to the sequence numbers of the data packets. In addition, in the sorting process, the data packets can be further de-duplicated according to the sequence numbers of the data packets.

[0144] Illustratively, in order to avoid congestion, a data packet decodable deadline can also be set in the reordering window of the basic layer; for example, for a 30 frames per second (fps) video, the deadline can be set to 33 milliseconds; for a 60 fps video, the deadline can be set to 16.7 milliseconds. If a data packet cannot complete reordering and de-duplication processing before the deadline, it cannot be guaranteed to be decoded on time, so the data packet will be discarded to avoid affecting subsequent decoding and playing.

[0145] Step A20, reorganizing and decoding each video frame to obtain a restored video.

[0146] The restored video refers to a sequence of video frames consistent with the original video data of the sending end after reorganization and decoding processing by the receiving end.

[0147] Exemplarily, the sending end sends the video frames of the base layer to the preset receiving end through the low-delay first path and sends the video frames of the enhancement layer to the receiving end through the high-bandwidth second path; then, after receiving the video frames, the receiving end first reorders (reorganizes) each audio frame according to the sequence number of each audio frame and removes the redundancy according to the sequence number; then, the receiving end can decode the reorganized video frames to obtain the restored video.

[0148] In a feasible implementation, the data transmission method further comprises:

[0149] Step A30, the receiving client receives the plurality of audio frames sent by the sending client through the first path and the second path;

[0150] Step A40, reorganizing and decoding each audio frame to obtain a restored audio;

[0151] The restored audio refers to an audio signal consistent with the audio data in the original data of the sending end after reorganization and decoding processing by the receiving end.

[0152] Exemplarily, after receiving the plurality of audio frames sent by the receiving client, the receiving end can recombine each audio frame into a continuous audio stream according to the original order relationship according to the header information (such as the sequence number, timestamp, etc.) corresponding to each audio frame; then, the reorganized audio stream is decoded by an audio decoder to obtain the restored audio.

[0153] The video frame and the audio frame each include a presentation timestamp and a sending timestamp, and the data transmission method further comprises:

[0154] Step A50, determining an audio frame with the same presentation timestamp as the presentation timestamp of the video frame as a reference audio frame;

[0155] The presentation timestamp (PTS) refers to information used to identify the time point at which the media data (such as a video frame or an audio frame) should be presented when played, which is usually expressed in time units (such as milliseconds, microseconds, etc.).

[0156] It can be understood that the video alignment is realized by correcting the stable transmission of the audio frame as a reference, which ensures the stability and accuracy of the audio-visual synchronization.

[0157] Step A60, determining the local presentation time of the corresponding restored video frame in the restored video according to the presentation timestamp and the offset time of the video frame, and determining the local presentation time of the corresponding restored audio frame in the restored audio according to the presentation timestamp and the offset time of the reference audio frame, wherein the offset time is determined according to the sending timestamp and the local receiving time;

[0158] The sending timestamp refers to the time marked when the data is sent out at the sending end, usually based on the system clock of the sending end, and is used to record the time information of data sending. The local receiving time refers to the receiving time recorded by the receiving end when the data arrives at the receiving end, based on the system clock of the receiving end.

[0159] The offset time refers to the time difference between the sending end and the receiving end caused by network transmission delay, device processing time difference and other factors; it can be calculated according to the difference between the sending timestamp and the local receiving time of the last sent data packet corresponding to the video frame / audio frame, and is used to measure the time delay experienced by the video frame / audio frame from sending to receiving.

[0160] The local presentation time refers to the time point at which the restored video frame and audio frame should actually be presented on the receiving end device, which can be calculated comprehensively according to the presentation timestamp, offset time and other factors, and is used to ensure that the audio and video can be played in the correct time sequence and rhythm at the receiving end.

[0161] Exemplarily, for a video frame / audio frame, its corresponding local presentation time wherein pts represents the presentation timestamp, represents the offset time.

[0162] Exemplarily, before calculating the local presentation time, the offset time can also be processed by Kalman filtering or moving average to avoid introducing jittery "drag / catch", for example, the offset time is wherein, represents the sending timestamp, represents the local receiving time, and E represents Kalman filtering processing.

[0163] Step A33, determining the time deviation of the restored video frame as the difference between the offset time of the video frame and the offset time of the reference audio frame;

[0164] The time deviation is used to measure the degree of inconsistency between video and audio in time; by calculating the time deviation, the synchronization between video and audio can be understood, providing a basis for subsequent adjustment.

[0165] Exemplarily, the calculation formula of the time deviation is as follows:

[0166]

[0167] wherein, and respectively represent the presentation timestamps of the video frame and the base audio frame, and respectively represent the local presentation times of the video frame and the base audio frame.

[0168] Step A34, according to the time deviation, adjusting the local presentation time of the restored video frame until the difference between the local presentation time of the restored video frame and the local presentation time of the restored audio frame is less than or equal to the preset deviation threshold, so as to synchronize the restored video and the restored audio.

[0169] The deviation threshold refers to the maximum range value of the allowed difference between the local presentation times of the audio and video preset in advance; when the difference between the local presentation times of the audio and video is less than or equal to this threshold, it can be considered that the audio and video have reached a synchronized state; when the difference is greater than this threshold, adjustment needs to be continued. For example, in a normal network state, the threshold can be set to 20 ms; or in a network congestion state, the threshold can be set to 50 ms.

[0170] Exemplarily, the time deviation can be directly determined as the adjustment range of the local presentation time of the restored video frame, to ensure the audio-visual synchronization of the current video frame and audio frame, but the operation for other video frames is time-consuming and complex.

[0171] Exemplarily, in order to realize the overall audio-visual synchronization, the restored audio is slightly resampled to correct the drift (±100~300ppm), or the video is slightly adjusted in advance / after presentation by ≤1ms, to avoid audio-visual synchronization. For example, according to the time deviation drift, the fine-tuning sampling rate of the restored audio can be determined as follows:

[0172]

[0173] wherein, kp is a gain coefficient for controlling the correction amplitude of the audio sampling rate, the clamp function is used to ensure that the correction does not exceed ±300ppm, and base_rate is the nominal audio sampling rate when the audio acquisition device acquires audio; further, the fine-tuning sampling rate can be sent to the audio resampler or clock fine-tuning to offset the audio stream and video stream at a very small rate. The video end only makes a small presentation in advance / after presentation by less than 1ms, and the main alignment is completed by the audio end, so as to continuously suppress the lip difference without multiple adjustments for each video frame.

[0174] Exemplarily, the restored video and the restored audio after synchronization can be combined to obtain the restored audio-video, that is, the complete audio-video content finally presented to the user.

[0175] In this embodiment, after receiving the video packets and audio packets sent by the sending end, reorganization and decoding are performed, and the restored video and restored audio obtained by decoding are synchronized to improve the audio-video synchronization experience of the user. In the synchronization process, the local presentation time of the audio and video is calculated respectively, and the transmission delay difference caused by different network paths is accurately compensated based on the playback scheduling of the local clock of the receiving end, thereby solving the problem of audio-video asynchronization and ensuring the immersive experience of audio-video synchronization.

[0176] For example, in order to facilitate understanding of the implementation process of the data transmission method obtained after combining the above-mentioned embodiment one, please refer to Figure 2 , Figure 2 An overall flowchart of a data transmission method is provided, and specifically:

[0177] First, the sending end performs B101 to obtain the original audio-video, and performs B102 to encode and content feature analyze the original audio-video to obtain the video frames and audio frames corresponding to the original audio-video, and determine the video features of each video frame, such as motion vector, histogram difference, gradient variance mean, etc., and voice activity detection can also be performed on the audio frames; then, B103 is performed to perform stream layering according to the content feature analysis result, and each video frame and audio frame is divided into a base layer or an enhancement layer, for example, the video frame with a motion vector and a histogram difference greater than the corresponding threshold is divided into the base layer, and the audio frame with a voice activity detection result of existing voice is divided into the base layer; at the same time, B104 can be performed to perform path health score and path scheduling, and the health score of each network path can be determined according to the weighted sum of the network state parameters (including delay and bandwidth, etc.) on each network path and the corresponding parameter weight. Since the base layer and the enhancement layer are provided with two different sets of parameter weights, the delay weight of the base layer is higher than that of the enhancement layer, and the bandwidth weight of the base layer is lower than that of the enhancement layer, so the transmission path of the base layer is determined as a low-delay path (corresponding to the first path), and the transmission path of the enhancement layer is determined as a high-bandwidth path (corresponding to the second path); then, the video frames and audio frames corresponding to the base layer are sent to the receiving end through the low-delay path, and the video frames and audio frames corresponding to the enhancement layer are sent to the receiving end through the high-bandwidth path. Then, the receiving end performs B105 to reorganize and decode the received video frames and audio frames, respectively. The receiving end can reorganize and de-duplicate according to the sequence number of each data packet, decode the data packets corresponding to the reorganized audio and the data packets corresponding to the video, respectively, obtain the restored video and the restored audio, and synchronize the restored audio and the restored video to obtain the restored audio-video; finally, B106 is performed to output the restored audio-video to a preset interactive interface for the user to watch.

[0178] For example, the key parameters and timing budget based on the above-mentioned data transmission method are given as follows:

[0179] For the data of the base layer, the packaging delay of the sending side is 1-3 ms, the copy processing needs 0.2-0.8 ms, and the one-way round-trip delay under the city area or the park area is less than or equal to 6 ms; the receiving rearrangement window is usually set to 12-24 ms, no FEC redundancy is set or 0-10% FEC redundancy is set, and the deviation threshold is usually set to 20 ms.

[0180] For the data of the enhancement layer, the packaging delay of the sending side is 3-8 ms, the copy processing needs 1-3 ms, and the one-way round-trip delay under the city area or the park area is less than or equal to 12 ms; the receiving rearrangement window is usually set to 40-80 ms, 10-25% FEC redundancy is set, and the deviation threshold is usually set to 50 ms.

[0181] Exemplarily, the sending end or the receiving end can be implemented based on a SoC (System-on-Chip) or an FPGA (Field-Programmable Gate Array) to accelerate the processing efficiency; the following improvements can also be made:

[0182] For the sending end, the NIC (Network Interface Card) can be combined with the DMA (Direct Memory Access) technology, the data is directly transmitted from the memory to the network interface card at the transmission end, bypassing the traditional CPU (Central Processing Unit) processing, and higher efficient data transmission is provided; the RaptorQ acceleration can also be implemented in the FPGA IP core to improve the real-time error correction capability and reduce the influence of packet loss on transmission.

[0183] For the receiving end, the hardware queue can be used for deduplication and reordering of data packets (including video packets and audio packets) to improve the recombination efficiency of the data packets; the hardware acceleration module can also be set to accelerate the data recovery of the FEC technology; the digital signal processor (Digital Signal Processor, DSP) unit can also be used to resample the audio to achieve more accurate audio-visual synchronization.

[0184] By using the data transmission method, the base layer data required by the 1080p60 (1920x1080 resolution + 60 frames per second) demand in the metropolitan area network can be transmitted to the receiving end with a low delay of less than 35ms; even in the case of 2% packet loss and 5ms jitter in a single path, the restored video picture display has no noticeable lag; even in the case of 3 congestion in a 10 second window, the time deviation of audio and picture synchronization can be controlled within 20ms; and in the case of 1 path disconnection and recovery within 100ms, the complex strategy of the base layer can ensure the normal display of the picture. It can be seen that the data transmission method has the advantages of stable low delay, anti-packet loss, anti-interruption, etc.

[0185] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data transmission method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0186] The present application also provides a data transmission device, please refer to Figure 3 , the data transmission device is applied to the sending end, and the device comprises:

[0187] The encoding module 10 is configured to encode the original data to obtain a plurality of video frames.

[0188] The layering module 20 is configured to determine the data level to which each video frame belongs according to the video features of each video frame, wherein the data level comprises a base layer and an enhancement layer.

[0189] The first sending module 30 is configured to send the video frames corresponding to the base layer to the receiving end through the first path.

[0190] The second sending module 40 is configured to send the video frames corresponding to the enhancement layer to the receiving end through the second path, wherein the delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path.

[0191] The data transmission device provided by the present application adopts the data transmission method in the above embodiment, which can solve the technical problem of how to improve the transmission quality of data transmission. Compared with the prior art, the data transmission device provided by the present application has the same beneficial effects as the data transmission method provided by the above embodiment, and the other technical features in the data transmission device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0192] The present application also provides another data transmission device, please refer to Figure 4 , the data transmission device is applied to the receiving end, and the device comprises:

[0193] The receiving module 50 is configured to receive a plurality of video frames sent by the client through a preset first path and a preset second path, wherein the video frames are obtained by encoding original data by the client, the first path is used to send video frames with a data level of a base layer, and the second path is used to send video frames with a data level of an enhancement layer, the data level of each video frame is determined according to a video feature of the video frame, the time delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path.

[0194] The decoding module 60 is configured to recombine and decode each video frame to obtain a restored video.

[0195] The data transmission device provided by the embodiment of the present application adopts the data transmission method in the above embodiment, and can solve the technical problem of how to improve the transmission quality of data transmission. Compared with the prior art, the data transmission device provided by the present application has the same beneficial effects as the data transmission method provided by the above embodiment, and other technical features in the data transmission device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0196] The embodiment of the present application provides a sending end, which comprises at least one processor and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data transmission method in the above embodiment one.

[0197] Reference will be made to Figure 5 which shows a structural diagram of a sending end suitable for implementing the embodiment of the present application. The sending end in the embodiment of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 5 The sending end shown is only an example, and should not bring any limitation to the functions and use range of the embodiment of the present application.

[0198] As Figure 5As shown, the sending end can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded from a storage device 1003 into a random access memory 1004. Various programs and data required for the operation of the sending end are also stored in the random access memory 1004. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other by a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the sending end to communicate wirelessly or by wire with other devices to exchange data. Although the sending end with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can alternatively be implemented or possessed.

[0199] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0200] The sending end provided by the embodiments of the present application adopts the data transmission method in the above-mentioned embodiments, and can solve the technical problem of how to improve the transmission quality of data transmission. Compared with the prior art, the sending end provided by the present application has the same beneficial effects as the data transmission method provided by the above-mentioned embodiments, and other technical features in the sending end are the same as the features disclosed in the previous embodiment method, which will not be described here.

[0201] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any appropriate way in one or more embodiments or examples.

[0202] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0203] The embodiment of the present application provides a receiving end, which comprises at least one processor, and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data transmission method in the above embodiment one.

[0204] Reference will be made to the following description of the embodiments of the present application, taken in conjunction with the accompanying drawings, in which Figure 6 which shows a structural diagram of a receiving end suitable for implementing the embodiment of the present application. The receiving end in the embodiment of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle terminals (such as vehicle navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 6 The receiving end shown is merely an example, and should not bring any limitation to the function and use range of the embodiment of the present application.

[0205] As Figure 6As shown, the receiving end can include a processing device 2001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory 2002 or loaded from the storage device 2003 into a random access memory 2004. Various programs and data required for the operation of the receiving end are also stored in the random access memory 2004. The processing device 2001, the read-only memory 2002, and the random access memory 2004 are connected to each other by a bus 2005. An input / output interface 2006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 2006: input devices 2007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 2008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 2003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 2009. The communication device 2009 can allow the receiving end to communicate wirelessly or by wire with other devices to exchange data. Although the receiving end with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0206] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication device, or installed from the storage device 2003, or installed from the read-only memory 2002. When the computer program is executed by the processing device 2001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0207] The receiving end provided by the embodiments of the present disclosure adopts the data transmission method in the above embodiments, and can solve the technical problem of how to improve the transmission quality of data transmission. Compared with the prior art, the receiving end provided by the present disclosure has the same beneficial effects as the data transmission method provided by the above embodiments, and other technical features in the receiving end are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0208] It should be understood that parts of the present disclosure can be realized by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.

[0209] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0210] The embodiment of the present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, and the computer readable program instructions are used to execute the data transmission method in the above embodiment.

[0211] The computer readable storage medium provided by the embodiment of the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0212] The above computer readable storage medium can be contained in the sending end / receiving end, or can exist separately and not be assembled into the sending end / receiving end.

[0213] The above computer readable storage medium carries one or more programs, and when the one or more programs are executed by the sending end, the sending end is caused to: encode original data to obtain a plurality of video frames; determine a data level to which each video frame belongs according to a video feature of each video frame, wherein the data level includes a base layer and an enhancement layer; send the video frame corresponding to the base layer to the receiving end through a first path; and send the video frame corresponding to the enhancement layer to the receiving end through a second path, wherein the time delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path.

[0214] The computer readable storage medium stores one or more programs, which, when executed by the receiving end, cause the receiving end to: receive a plurality of video frames sent by a client through a preset first path and a preset second path, wherein the video frames are obtained by encoding original data by the client, the first path is used to send video frames of a base layer, the second path is used to send video frames of an enhancement layer, the data level of each video frame is determined according to a video feature of each video frame, the time delay of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path; and recombine and decode each video frame to obtain a restored video.

[0215] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0216] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It is also noted that each block in the block diagrams and / or flow diagrams and combinations of blocks in the block diagrams and / or flow diagrams can be implemented by special-purpose hardware-based systems that perform the specified functions or operations, or combinations of special-purpose hardware and computer instructions.

[0217] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0218] The readable storage medium provided by the embodiments of the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the data transmission method described above, and can solve the technical problem of improving the transmission quality of audio and video transmission. Compared with the prior art, the computer readable storage medium provided by the embodiments of the present application has the same beneficial effects as the data transmission method described above, and will not be described here.

[0219] The embodiments of the present application also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the data transmission method described above are implemented.

[0220] The computer program product provided by the embodiments of the present application can solve the technical problem of improving the transmission quality of audio and video transmission. Compared with the prior art, the computer program product provided by the embodiments of the present application has the same beneficial effects as the data transmission method described above, and will not be described here.

[0221] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation, direct / indirect application in other related technical fields based on the technical concept of the present application, and the contents of the present application and the accompanying drawings are included in the patent protection scope of the present application.

Claims

1. A data transmission method, characterized in that, The data transmission method is applied at the sending end, and the method includes: The raw data is encoded to obtain multiple video frames; Based on the video features of each video frame, the data level to which each video frame belongs is determined, wherein the data level includes a base layer and an enhancement layer; The video frame corresponding to the base layer is sent to the receiving end via the first path; The video frames corresponding to the enhancement layer are sent to the receiving end through the second path, wherein the latency of the first path is lower than the latency of the second path, and the bandwidth of the first path is lower than the bandwidth of the second path. The video features include motion vectors, histogram differences, and mean gradient variance. The step of determining the data level to which each video frame belongs based on its video features includes: If the motion vector of the video frame is greater than a first threshold or the histogram difference is greater than a second threshold, the data layer to which the video frame belongs is determined to be the base layer. If the motion vector of the video frame is less than or equal to a first threshold, the histogram difference is less than or equal to a second threshold, and the mean gradient variance is less than or equal to a third threshold, the data layer to which the video frame belongs is determined to be an enhancement layer.

2. The data transmission method as described in claim 1, characterized in that, Before the step of sending the video frame corresponding to the base layer to the receiving end via the first path, the method further includes: Based on the network status parameters and corresponding parameter weights of each network path, a health score for each network path is determined. The network status parameters include at least one of latency, jitter, packet loss rate, available bandwidth, and stability. The latency weight of the base layer is higher than that of the enhancement layer, the jitter weight of the base layer is higher than that of the enhancement layer, the packet loss rate weight of the base layer is higher than that of the enhancement layer, the stability weight of the base layer is lower than that of the enhancement layer, and the bandwidth weight of the base layer is lower than that of the enhancement layer. The first path and the second path are determined from the network paths based on the health scores of each network path.

3. The data transmission method as described in claim 2, characterized in that, The first path includes a primary path and a secondary path. The step of determining the first path and the second path from the network paths based on the health scores of each network path includes: When the video frame belongs to the base layer, the main path and the secondary path are determined according to the health score of each network path. The main path is used to send the video frame in full, and the secondary path is used to send the copied data of the video frame according to a preset redundancy copy ratio. When the video frame belongs to the enhancement layer, the video frame is decomposed into multiple network abstraction layer units, and the second path corresponding to each network abstraction layer unit is determined by a weighted round-robin scheduling algorithm and the health score of each network path.

4. The data transmission method as described in claim 3, characterized in that, The data transmission method further includes: The short-term fluctuation value and long-term trend value of each network state parameter in the main path and the second path are determined by using a preset second-order exponential weighted moving average model. If the difference between the short-term fluctuation value and the long-term trend value of at least one network state parameter in the main path is greater than a preset fluctuation threshold, the redundancy replication ratio of the secondary path in the base layer is increased. If the difference between the short-term fluctuation value and the long-term trend value of at least one network state parameter in the second path is greater than the fluctuation threshold, the second path will be adjusted to a preset backup path.

5. The data transmission method as described in claim 1, characterized in that, The step of encoding the raw data includes: The raw data is encoded to obtain multiple audio frames; The data transmission method further includes: Speech activity detection is performed on each of the audio frames to obtain the detection results for each audio frame; If the detection result of the audio frame indicates the presence of human voice, the data layer to which the audio frame belongs is determined to be the basic layer; If the detection result of the audio frame indicates that there is no human voice, the data layer to which the audio frame belongs is determined to be the enhancement layer; The audio frame corresponding to the base layer is sent to the receiving end via the first path, and the audio frame corresponding to the enhancement layer is sent to the receiving end via the second path.

6. A data transmission method, characterized in that, The data transmission method is applied at the receiving end, and the method includes: The receiving client sends multiple video frames through a preset first path and a preset second path. The video frames are obtained by the client encoding the original data. The first path is used to send video frames with a data layer of the basic layer, and the second path is used to send video frames with a data layer of the enhancement layer. The data layer of each video frame is determined according to the video characteristics of each video frame. The latency of the first path is lower than that of the second path, and the bandwidth of the first path is lower than that of the second path. The video frames are reassembled and decoded to obtain the restored video; The video features include motion vectors, histogram differences, and mean gradient variance. The step of determining the data level of each video frame based on the video features of each video frame includes: If the motion vector of the video frame is greater than a first threshold or the histogram difference is greater than a second threshold, the data layer to which the video frame belongs is determined to be the base layer. If the motion vector of the video frame is less than or equal to a first threshold, the histogram difference is less than or equal to a second threshold, and the mean gradient variance is less than or equal to a third threshold, the data layer to which the video frame belongs is determined to be an enhancement layer.

7. The data transmission method as described in claim 6, characterized in that, The data transmission method further includes: Receive multiple audio frames sent by the client through the first path and the second path; Each audio frame is reconstructed and decoded to obtain the restored audio; Both the video frame and the audio frame include a presentation timestamp and a transmission timestamp, and the data transmission method further includes: The audio frame whose presentation timestamp is the same as that of the video frame is determined as the reference audio frame; Based on the presentation timestamp and offset time of the video frame, the local presentation time of the corresponding restored video frame within the restored video is determined, and based on the presentation timestamp and offset time of the reference audio frame, the local presentation time of the corresponding restored audio frame within the restored audio is determined, wherein the offset time is determined based on the sending timestamp and the local receiving time. The difference between the offset time of the video frame and the offset time of the reference audio frame is determined as the time deviation of the restored video frame; Based on the time deviation, the local rendering time of the restored video frame is adjusted until the difference between the local rendering time of the restored video frame and the local rendering time of the restored audio frame is less than or equal to a preset deviation threshold, so as to synchronize the restored video and the restored audio.

8. A transmitter, characterized in that, The transmitting end includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data transmission method as described in any one of claims 1 to 5.

9. A receiving end, characterized in that, The receiving end includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data transmission method as described in claim 6 or 7.

Images