Video data transmission method and device

Abstract

The embodiment of the present application provides a method and device for transmitting video data, which relates to the technical field of streaming media transmission, and can ensure, to a certain extent, the smooth transmission of important information in the video data during the transmission process. The method includes: detecting the transmission quality of N links; determining, according to the transmission quality of the N links, the video frames to be transmitted corresponding to the N links respectively; wherein, when the transmission quality of the kth link meets the preset conditions, the video frames to be transmitted corresponding to the kth link are all the video frames of the above-mentioned first video data; when the transmission quality of the kth link does not meet the preset conditions, the video frames to be transmitted corresponding to the kth link are part of all the video frames of the above-mentioned first video data; wherein the above-mentioned video frames to be transmitted include all the key data of the first video data; and transmitting the video frames to be transmitted corresponding to the N links respectively in the above-mentioned N links.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of streaming media transmission, and in particular to a method and device for transmitting video data. Background Art

[0002] Video is increasingly used in real life, such as video calls, live webcasts, and near-field projection. The transmission of video data is an important factor affecting the quality of video data playback.

[0003] At present, in the process of video data transmission, the video transmitter transmits data packets of the video data on two links respectively to realize redundant transmission of the video data, for example, all data packets of the video data are transmitted on the first link, and all data packets of the video data are also transmitted on the second link. Of the above two links, one link can be the main link and the other link can be the auxiliary link. The data packets transmitted on the above two links are backups for each other. When the data packets of one link are lost, the video receiver can obtain the lost data packets from the data packets transmitted on the other link to successfully complete the decoding of the video data.

[0004] However, when the bandwidth of at least one of the two links used to transmit video data is limited (i.e., one of the two transmission links is strong and the other is weak), the link with limited bandwidth may lose part of the video data when transmitting the video data. Therefore, the two links can no longer back up each other. In this case, if part of the video data is lost due to other factors (for example, the receiving end is restarted) during the transmission of video data by the link with unlimited bandwidth, and the part of the video data is the same as the video data lost by the link with limited bandwidth, and the video data lost by the two links includes important information of the video data, then the video receiving end will experience problems such as freeze, screen distortion and high latency when playing the received video data. Summary of the invention

[0005] The embodiments of the present application provide a method and device for transmitting video data, which can ensure, to a certain extent, normal transmission of important information in the video data during the transmission process.

[0006] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0007] In a first aspect, an embodiment of the present application provides a method for transmitting video data, the method comprising: detecting the transmission quality of N links; wherein the N links are used to transmit first video data, the N links are links in different networks, the first video data includes multiple video frames, and N is a positive integer greater than or equal to 2; according to the transmission quality of the N links, determining the video frames to be transmitted corresponding to the N links respectively; wherein, when the transmission quality of the kth link meets the preset conditions, the video frames to be transmitted corresponding to the kth link are all the video frames of the first video data; when the transmission quality of the kth link does not meet the preset conditions, the video frames to be transmitted corresponding to the kth link are part of all the video frames of the first video data; wherein the value of k is any one of 1, 2, ...N, and the video frames to be transmitted include all key data of the first video data; and transmitting the video frames to be transmitted corresponding to the N links respectively in the N links.

[0008] In the video data transmission method provided by the embodiment of the present application, the video transmitter detects the transmission quality of N links used to transmit the first video data, and determines the video frames to be transmitted corresponding to the N links respectively according to the transmission quality of the N links, wherein, when the transmission quality of the kth link meets the preset conditions, all the video frames in the first video data are used as the video frames to be transmitted of the kth link, and when the transmission quality of the kth link does not meet the preset conditions, part of the video frames in the first video data are used as the video frames to be transmitted of the kth link; and the above-determined video frames to be transmitted include all the key data in the first video data. In this way, when the transmission quality of the kth link is poor, this part of the video frames in the first video data transmitted on the kth link includes all the key data in the first video data, thereby ensuring the normal transmission of the key data in the first video data to a certain extent, thereby solving the problems of freeze, screen distortion and high latency when the video receiving end plays the received video data.

[0009] In a possible implementation, the above-mentioned detection of the transmission quality of the kth link specifically includes: obtaining feedback information of the kth link, the feedback information including transmission delay and / or number of retransmissions; and determining the transmission quality of the kth link based on the above-mentioned feedback information of the kth link.

[0010] When the above feedback information is the transmission delay, if the transmission delay of the kth link is less than the preset delay, the transmission quality of the kth link meets the preset conditions; if the transmission delay of the kth link is greater than or equal to the preset delay, the transmission quality of the kth link does not meet the above preset conditions.

[0011] When the above feedback information is the number of retransmissions, if the number of retransmissions of the kth link is less than the preset number, the transmission quality of the kth link meets the above preset conditions; if the number of retransmissions of the kth link is greater than or equal to the preset number, the transmission quality of the kth link does not meet the above preset conditions.

[0012] When the above feedback information is the transmission delay and the number of retransmissions, if the transmission delay of the k-th link is less than the preset delay, and the number of retransmissions of the k-th link is less than the preset number, then the transmission quality of the k-th link meets the preset conditions; if the transmission delay of the k-th link is greater than or equal to the preset threshold, and / or the number of retransmissions of the k-th link is greater than or equal to the preset number, then the transmission quality of the k-th link does not meet the preset conditions.

[0013] In one possible implementation, when the transmission quality of the kth link does not meet the preset conditions, the video frames to be transmitted corresponding to the kth link are determined, specifically including: according to the preset ratio corresponding to the kth link, in the order of the priority of the video frames of the first video data from high to low, selecting some video frames in the video frames of the above-mentioned first video data as the video frames to be transmitted corresponding to the kth link.

[0014] The above preset ratios meet the following requirements: Among them, η represents the preset ratio, Num 1 Indicates the number of non-key video frames in the video frame of the first video data, Num 2 Represents the total number of video frames of the first video data; wherein the non-key video frame refers to a video frame including non-key data.

[0015] In the video data transmission method provided by the embodiment of the present application, the video transmitter detects the transmission quality of the kth link used to transmit the first video data, and determines the video frame to be transmitted corresponding to the kth link according to the transmission quality of the kth link, wherein, when the transmission quality of the kth link meets the preset conditions, all the video frames in the first video data are used as the video frames to be transmitted of the kth link, and when the transmission quality of the kth link does not meet the preset conditions, part of the video frames in the first video data are used as the video frames to be transmitted of the kth link; and the above-determined video frames to be transmitted include all the key data in the first video data. In this way, when the transmission quality of the kth link is poor, this part of the video frames in the first video data transmitted on the kth link includes all the key data in the first video data, thereby ensuring the normal transmission of the key data in the first video data to a certain extent, thereby solving the problems of freeze, screen distortion and high latency when the video receiving end plays the received video data.

[0016] In one possible implementation, after the video frames to be transmitted corresponding to the N links are transmitted respectively in the N links, a video data transmission method provided in an embodiment of the present application further includes: detecting the transmission quality of the kth link; and adjusting the preset ratio corresponding to the kth link according to the transmission quality of the kth link.

[0017] In a possible implementation, adjusting the preset ratio corresponding to the k-th link according to the transmission quality of the k-th link specifically includes: when the transmission quality of the k-th link meets the preset conditions, increasing the preset ratio; when the transmission quality of the k-th link does not meet the preset conditions, reducing the preset ratio.

[0018] In the video data transmission method provided in the embodiment of the present application, when the transmission quality of k links meets the preset conditions, the video sending end increases the preset ratio; that is, the transmission quality of the kth link is better, and the video sending end increases the preset ratio so that the video sending end sends more video frames in the video data to the video receiving end, thereby enabling the video receiving end to play high-quality video images based on more video frames in the video data.

[0019] In a possible implementation, a video data transmission method provided by an embodiment of the present application further includes: determining, based on a preset ratio corresponding to the adjusted k-th link, a video frame to be transmitted corresponding to the K-th link of the second video data; and transmitting the video frame to be transmitted corresponding to the k-th link in the above-mentioned k-th link.

[0020] In a possible implementation, a video data transmission method provided by an embodiment of the present application also includes: performing frequency domain layered encoding on the first video data to obtain video frames of the first video data, the video frames of the first video data including at least one basic frame and an extended frame of a key frame, and a plurality of basic frames and extended frames of forward frames; non-key video frames including extended frames of key frames, basic frames of forward frames and extended frames of forward frames, and the key data of the above-mentioned first video data is the basic frame of the key frame.

[0021] In a possible implementation, a video data transmission method provided by an embodiment of the present application also includes: the priorities of the video frames of the first video data are, from high to low, the basic frame of the key frame, the basic frame of the forward frame, the extended frame of the key frame, and the extended frame of the forward frame.

[0022] In a possible implementation, a video data transmission method provided by an embodiment of the present application also includes: the i-th video frame in the video frame of the first video data corresponding to the first link is the basic frame of the first video frame, and the i-th video frame in the video frame of the first video data corresponding to the second link is the extended frame of the first video frame; the first link and the second link are two different links among N links.

[0023] In summary, the arrangement order of multiple video frames of the first video data transmitted on different links is: in the video frame queues corresponding to two different links, the video frames with the same position (or the same number or index) are the basic frames and extended frames of the same video frame. The video sending end transmits the video frames in parallel on the two links according to this arrangement order of the video frames, and can quickly send the basic frames and extended frames of the video frames to the video receiving end. For the video receiving end, the basic frames and extended frames of the video frames can also be quickly aggregated to obtain decoded video frames.

[0024] In a possible implementation, a video data transmission method provided by an embodiment of the present application also includes: determining a retransmitted video frame set, which is the intersection of the video frames to be retransmitted corresponding to each of the above-mentioned N links; transmitting the video frames in the above-mentioned retransmitted video frame set in the j-th link in a queue-jumping transmission manner; wherein the transmission quality of the j-th link is higher than the transmission quality of other links, and the other links are links among the above-mentioned N links except the j-th link.

[0025] In a possible implementation, the transmitting of the video frames in the retransmitted video frame set in the j-th link includes: transmitting the video frames in the retransmitted video frame set in the j-th link in order of priority of the video frames in the retransmitted video frame set from high to low.

[0026] In a possible implementation, transmitting the video frames in the retransmitted video frame set in the j-th link in a queue-jumping transmission manner includes: inserting the retransmitted video frame set into the first position of the to-be-transmitted video frame queue corresponding to the j-th link.

[0027] In a possible implementation, before transmitting the to-be-transmitted video frames respectively corresponding to the N links in the N links, the method further includes: encrypting the to-be-transmitted video frames respectively corresponding to the N links.

[0028] In an embodiment of the present application, on the one hand, the video sending end transmits the video frames in the retransmitted video frame set in a queue-jumping manner, and gives priority to transmitting the video frames in the retransmitted video frame set, so that the video receiving end can obtain the lost video frames as soon as possible, improve the speed of aggregating video frames at the video receiving end, and realize rapid aggregation of video frames; on the other hand, when the above-mentioned video sending end transmits the video frames in the retransmitted video frame set, the video sending end transmits the video frames in the retransmitted video frame set in order from high to low priority. Since the higher the priority of the video frame, the more important the data information included in it, therefore, retransmitting the video frame according to the priority can ensure that the important information in the first video data is not lost to a certain extent, so as to ensure the playback quality of the video data.

[0029] In a second aspect, the video data transmission device provided by the embodiment of the present application includes: a detection module, a determination module and a transmission module; the detection module is used to detect the transmission quality of N links; wherein the N links are used to transmit the first video data, the N links are links in different networks, the above-mentioned first video data includes multiple video frames, and N is a positive integer greater than or equal to 2; the determination module is used to determine the video frames to be transmitted corresponding to the N links respectively according to the transmission quality of the N links; wherein, when the transmission quality of the kth link meets the preset conditions, the video frames to be transmitted corresponding to the kth link are all video frames of the above-mentioned first video data; when the transmission quality of the kth link does not meet the preset conditions, the video frames to be transmitted corresponding to the kth link are all part of the video frames of the above-mentioned first video data; wherein the value of k is any one of 1, 2, ...N, and the above-mentioned video frames to be transmitted include all key data of the first video data; the transmission module is used to transmit the video frames to be transmitted corresponding to the N links respectively in the above-mentioned N links.

[0030] In a possible implementation, the video data transmission device provided in the embodiment of the present application further includes: an acquisition module; the acquisition module is used to obtain feedback information of the kth link, and the feedback information includes transmission delay and / or number of retransmissions.

[0031] The determination module is used to determine the transmission quality of the kth link according to the feedback information of the kth link.

[0032] When the feedback information is the transmission delay, if the transmission delay of the kth link is less than the preset delay, the transmission quality of the kth link meets the preset condition; if the transmission delay of the kth link is greater than or equal to the preset delay, the transmission quality of the kth link does not meet the preset condition.

[0033] When the feedback information is the number of retransmissions, if the number of retransmissions of the kth link is less than the preset number, the transmission quality of the kth link meets the preset conditions; if the number of retransmissions of the kth link is greater than or equal to the preset number, the transmission quality of the kth link does not meet the preset conditions.

[0034] When the feedback information is the transmission delay and the number of retransmissions, if the transmission delay of the k-th link is less than the preset delay, and the number of retransmissions of the k-th link is less than the preset number, then the transmission quality of the k-th link meets the preset conditions; if the transmission delay of the k-th link is greater than or equal to the preset threshold, and / or the number of retransmissions of the k-th link is greater than or equal to the preset number, then the transmission quality of the k-th link does not meet the preset conditions.

[0035] In one possible implementation, the video data transmission device provided in an embodiment of the present application includes: a control module; a control module, which is used to select part of the video frames of the first video data as the video frames to be transmitted corresponding to the kth link according to a preset ratio corresponding to the kth link and in order of priority of the video frames of the above-mentioned first video data from high to low.

[0036] The above preset ratios meet the following requirements: Among them, η represents the preset ratio, Num 1 Indicates the number of non-key video frames in the video frame of the first video data, Num 2 Represents the total number of video frames of the first video data; wherein, the non-key video frame refers to a video frame including non-key data.

[0037] In one possible implementation, the video data transmission device provided in the embodiment of the present application includes: an adjustment module; a detection module for detecting the transmission quality of the kth link; and an adjustment module for adjusting a preset ratio corresponding to the kth link according to the transmission quality of the kth link.

[0038] In one possible implementation, the video data transmission device provided by the embodiment of the present application, wherein the adjustment module is specifically used to increase the preset ratio when the transmission quality of the kth link meets the preset conditions; and to reduce the preset ratio when the transmission quality of the kth link does not meet the preset conditions.

[0039] In a possible implementation, the above-mentioned determination module is also used to determine the video frames to be transmitted corresponding to the K links of the second video data based on the preset ratio corresponding to the adjusted k-th link; the above-mentioned transmission module is also used to transmit the video frames to be transmitted corresponding to the k-th link in the above-mentioned k-th link.

[0040] In one possible implementation, the video data transmission device provided by the embodiment of the present application includes: an encoding module; the encoding module is used to perform frequency domain layered encoding on the first video data to obtain video frames of the first video data, and the video frames of the first video data include at least one basic frame and an extended frame of a key frame, and a plurality of basic frames and extended frames of forward frames; the above-mentioned non-key video frames include extended frames of key frames, basic frames of forward frames and extended frames of forward frames, and the key data of the above-mentioned first video data is the basic frame of the key frame.

[0041] In one possible implementation, an embodiment of the present application provides a video data transmission device, wherein the priorities of video frames of the first video data are, from low to high, an extended frame of a forward frame, an extended frame of a key frame, a basic frame of a forward frame, and a basic frame of a key frame.

[0042] In one possible implementation, an embodiment of the present application provides a video data transmission device, wherein the i-th video frame in the video frames of the first video data corresponding to the first link is a basic frame of the first video frame, and the i-th video frame in the video frames of the first video data corresponding to the second link is an extended frame of the first video frame; the first link and the second link are two different links among the above-mentioned N links.

[0043] In one possible implementation, the video data transmission device provided by the embodiment of the present application also includes: a queue-jumping module; the above-mentioned determination module is also used to determine a retransmitted video frame set, which is the intersection of the video frames to be retransmitted corresponding to each of the above-mentioned N links; the queue-jumping module is used to transmit the video frames in the above-mentioned retransmitted video frame set in the j-th link in a queue-jumping transmission mode; wherein the transmission quality of the j-th link is higher than the transmission quality of other links, and the other links are links among the above-mentioned N links except the above-mentioned j-th link.

[0044] In one possible implementation, the video data transmission device provided by an embodiment of the present application, wherein the transmission module is also used to transmit the video frames in the retransmitted video frame set in the jth link in order of the priority of the video frames in the retransmitted video frame set from high to low.

[0045] In a possible implementation, the queue-inserting module is specifically configured to insert the retransmitted video frame set into the first position of the queue of video frames to be transmitted corresponding to the j-th link.

[0046] In a possible implementation, the video data transmission device provided in the embodiment of the present application further includes an encryption module; the encryption module is used to encrypt the video frames to be transmitted corresponding to the N links respectively.

[0047] In a third aspect, a video data transmission device provided in an embodiment of the present application includes a memory and a processor, wherein the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the video data transmission device executes the method described in the first aspect and any one of its possible implementation methods.

[0048] In a fourth aspect, an embodiment of the present application provides a computer storage medium, comprising computer instructions, which, when executed on a video data transmission device, enables the video data transmission device to execute the method described in the first aspect and any one of its possible implementation methods.

[0049] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the first aspect and any one of its possible implementations.

[0050] It should be understood that the beneficial effects achieved by the technical solutions of the second to fifth aspects of the embodiments of the present application and the corresponding possible implementation methods can refer to the technical effects of the first aspect and its corresponding possible implementation methods mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a screen projection method provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of a scenario using a distributed camera provided in an embodiment of the present application Figure 1 ;

[0053] Figure 3 A schematic diagram of a scenario using a distributed camera provided in an embodiment of the present application Figure 2 ;

[0054] Figure 4 A video data transmission process diagram provided in an embodiment of the present application Figure 1 ;

[0055] Figure 5 A video data encoding method provided in an embodiment of the present application is shown in FIG. Figure 1 ;

[0056] Figure 6 A video data transmission method provided in an embodiment of the present application is shown in FIG. Figure 1 ;

[0057] Figure 7 A schematic diagram of a multi-path transmission architecture provided in an embodiment of the present application Figure 1 ;

[0058] Figure 8 A schematic diagram of a video data transmission device provided in an embodiment of the present application Figure 1 ;

[0059] Fig. 9 A video data transmission method provided in an embodiment of the present application is shown in FIG. Figure 1 ;

[0060] Fig.10 A video data transmission method provided in an embodiment of the present application is shown in FIG. Figure 2 ;

[0061] Fig.11 A schematic diagram of sorting multiple video frames provided in an embodiment of the present application Figure 1 ;

[0062] Fig.12 A video data transmission method provided in an embodiment of the present application is shown in FIG. Figure 3 ;

[0063] Fig.13 A video data transmission method provided in an embodiment of the present application is shown in FIG. Figure 4 ;

[0064] Fig.14 A video data transmission method provided in an embodiment of the present application is shown in FIG. Figure 5 ;

[0065] Fig.15 A schematic diagram of a video data transmission device provided in an embodiment of the present application Figure 1 ;

[0066] Fig.16 A schematic diagram of a video data transmission device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0067] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0068] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first link and a second link are used to distinguish links in different networks rather than to describe a specific order of the links.

[0069] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0070] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple video frames refers to two or more video frames.

[0071] With the development of technology, streaming media (such as video) is more and more widely used in real life. In daily life, video data transmission is involved in both near-field projection scenarios and distributed camera scenarios.

[0072] In the near-field projection scenario, projection methods can include: mirror coordination, heterogeneous collaboration, and reverse control, etc.

[0073] Mirror coordination: refers to the real-time synchronous transmission of the video screen on the projection side to the projection side, so that it can be played synchronously on the projection side.

[0074] Exemplary, reference Figure 1 In (a), the large screen 101 is the projected screen side, and the mobile phone 102 is the projecting screen side. The projection mode of the mobile phone 102 and the large screen 101 is mirror coordination. For example, when using the mobile phone 102 to make a video call, the mobile phone 102 transmits the video screen during the video call (specifically, the video data corresponding to the video screen) to the large screen 101, and the large screen 101 can synchronously play the video screen of the mobile phone 102.

[0075] Heterogeneous collaboration: refers to synchronously transmitting a task running in the background of the projection side to the projection side, so that it can be played synchronously on the projection side. At this time, the projection side can perform other tasks.

[0076] Exemplary, reference Figure 1 In (b), the large screen 103 is the projected screen side, and the mobile phone 104 is the projecting screen side. The screen projection relationship between the mobile phone 104 and the large screen 103 is heterogeneous collaboration. The mobile phone 104 displays the chat interface (the chat interface displayed when the mobile phone runs a social application) and the video playback interface (the interface displayed when the mobile phone runs a video playback application) in a split screen manner. The mobile phone 104 can transmit the video picture displayed on the split screen 2 to the large screen 103, and the large screen 103 synchronously plays the video picture displayed on the mobile phone 104. After the screen projection is successful, the user can perform corresponding operations on the mobile phone 104 and put the video playback application in the background, so that the mobile phone 104 displays the chat interface in full screen.

[0077] Reverse control: refers to transmitting the picture displayed on the projection side to the projection side, so that the picture is displayed in real time on the projection side. Then, the user performs relevant operations on the projection side, so that the projection side can respond to the operation and perform the corresponding task.

[0078] refer to Figure 1 In (c), the computer 105 is the screen-projected side, and the mobile phone 106 is the screen-projecting side. The screen projection method between the mobile phone 106 and the computer 105 is reverse control. The mobile phone 106 synchronously transmits the video screen displayed on the display screen of the mobile phone 106 to the computer 105, and the video screen is displayed on the display screen of the computer 105. Then, the user can operate the displayed content on the computer 105, and the mobile phone 106 can respond to the user's operation on the computer 105. For example, the user opens a certain video software on the mobile phone 106, and the mobile phone 106 displays an interface including multiple video resources. After the mobile phone 106 and the computer 105 establish a reverse control screen projection connection, the computer 105 also synchronously displays the interface including multiple video resources. Then, when the user opens a video resource among the multiple video resources on the computer 105, the mobile phone 106 can respond to the opening operation of the video resource, and the mobile phone 106 displays the video screen corresponding to the video resource. At the same time, the video screen displayed on the mobile phone 106 is also synchronously transmitted to the computer 105, so that the computer 105 displays the video screen.

[0079] In the screen projection scenario described above, the screen projection side is the sender of video data, and the screen projection side is the receiver of video data.

[0080] In the distributed camera scenario, distributed cameras can be used to realize video calls, remote control of mobile phone photography, mobile phone live broadcasts, and cross-device dual-view recording, etc.

[0081] Figure 2 (a) is a schematic diagram of a scenario of using a distributed camera for a video call, wherein the camera on the large screen 201 and the mobile phone 202 form a distributed camera. Before using the mobile phone 202 for a video call, the mobile phone 202 establishes a communication connection with the camera on the large screen 201. Since the pixel of the camera on the large screen 201 is higher than the pixel of the built-in camera of the mobile phone 202, when the mobile phone 202 is used for a video call, the camera on the large screen 201 is used as the camera for collecting video images during the video call. Specifically, the video images are collected by the camera on the large screen 201 and synchronously transmitted to the mobile phone 202, and the mobile phone 202 displays the video images in real time.

[0082] In the above video call scenario, the large screen 201 is the transmitter of the video data, and the mobile phone 202 is the receiver of the video data.

[0083] Figure 2(b) is a schematic diagram of a scenario in which a distributed camera is used to remotely control a mobile phone to take photos, wherein a smart watch 203 and a mobile phone 204 form a distributed camera; when the mobile phone 204 is placed at a remote end to take photos, the mobile phone 204 establishes a communication connection with the smart watch 203, and the camera of the mobile phone 204 transmits the collected preview image to the smart watch 203, which is synchronously displayed on the smart watch 203, and then the user can operate the smart watch 203 to control the mobile phone 204 to take photos. For example, if the user thinks that the image in the preview interface currently displayed on the smart watch 203 is better, the user can select the "take photo" button in the smart watch 203, and the mobile phone 204 responds to the "take photo" operation and collects images.

[0084] In the above scenario of remotely controlling a mobile phone to take photos, the mobile phone 204 is the sender of video data, and the smart watch 203 is the receiver of video data.

[0085] Figure 3 (a) is a schematic diagram of a scenario of using a distributed camera for mobile phone live broadcast, wherein the rear camera of mobile phone 301 and mobile phone 302 form a distributed camera, and the pixel of the rear camera of mobile phone 301 is higher than the front camera of mobile phone 302. In the live broadcast scenario, after mobile phone 302 establishes a communication connection with mobile phone 301, a live broadcast APP is run on mobile phone 302, and the rear camera of mobile phone 301 captures the live broadcast picture (such as scenery or anchor), and then mobile phone 301 transmits the captured live broadcast picture to mobile phone 302 in real time, and mobile phone 302 plays the live broadcast picture synchronously.

[0086] In the above mobile phone live broadcast scenario, mobile phone 301 is the sender of video data, and mobile phone 302 is the receiver of video data.

[0087] Figure 3 (b) is a schematic diagram of a scenario in which a distributed camera is used for cross-device dual-view recording, wherein a mobile phone 303, a mobile phone 305 and a camera 304 form a distributed camera, and the mobile phone 303 and the camera 304 respectively establish communication connections with the mobile phone 305. After starting the dual-view recording, the mobile phone 303 transmits the close-up image of the target scene captured by the camera of the mobile phone 303 to the mobile phone 305. At the same time, the camera 304 transmits the long-view image of the target scene captured to the mobile phone 305. After the mobile phone 305 receives the images sent by the mobile phone 303 and the camera 304, the close-up image and the long-view image are displayed on the display screen of the mobile phone 305 in a split-screen manner to realize the dual-view recording.

[0088] In the above cross-device dual-view video recording scenario, the mobile phone 303 and the camera 304 are both senders of video data, and the mobile phone 305 is a receiver of video data.

[0089] It can be understood that in the embodiment of the present application, the video data is a video stream, including multiple video frames, each of which is an image, and therefore, the video frame can also be called an image frame.

[0090] Combined with the above multiple scenarios involving video data transmission, refer to Figure 4 The transmission process diagram of video data shown in the figure, first, the transmitter encodes (preprocesses) the collected video data (including multiple video frames), and the commonly used encoding methods include time domain encoding and frequency domain encoding; secondly, the transmitter encrypts the encoded video data and sends the encrypted video data to the receiver. When the receiver receives the encrypted video data, first, it reversely decrypts the encrypted video data according to the encryption method of the transmitter to obtain the decrypted video data; then, it reversely decodes the video data according to the encoding method of the transmitter to obtain the decoded video data, and finally, the receiver renders the decoded video data before playing it.

[0091] Commonly used video data encoding methods may include time domain encoding and frequency domain encoding.

[0092] In the embodiment of the present application, the forward search frame is referred to as the forward frame for short, and will not be described in the following embodiments.

[0093] Time domain coding: The collected video frames can be encoded by time domain coding to obtain small forward frames (i.e. small P frames) and large forward frames (large P frames) of the video frames, such as Figure 5 As shown in (a), when the receiving end decodes a large P frame, it needs to rely on the decoding result of the previous large P frame of the large P frame and the small P frame corresponding to the large P frame.

[0094] Frequency domain coding: It encodes the collected video frames into basic frames and extended frames according to the frequency domain distribution of the video data, such as Figure 5 As shown in (b) in the figure; a base frame of a video frame includes low-frequency data of the video frame, and an extended frame of a video frame includes high-frequency data of the video frame. The sizes (data volume) of the base frame and the extended frame are different. Specifically, the size ratio of the base frame to the extended frame is between 1:4 and 1:8. In one case, the sum of the sizes of the base frame and the extended frame of a video frame can be 1.2 times that of the video frame.

[0095] Currently, multiple links (i.e., multi-path transmission) can be used to transmit video data, such as Figure 6As shown, the video transmitter transmits all data packets of the video data to be transmitted (i.e., data packets 1 to 8) on the first link, and also transmits all data packets of the video data to be transmitted on the second link, wherein the data packets transmitted on the first link and the second link back up each other, and when the data packets of one link are lost, the video receiver can obtain the lost data packets from the data packets transmitted on the other link to successfully complete the decoding of the video data.

[0096] However, when the bandwidth of the second link of the above two links is limited, the second link will lose some video data packets when transmitting the data packets of the video data to be transmitted. If the first link also loses some video data packets due to other factors (for example, the receiving end is restarted) during the process of transmitting the data packets of the video data to be transmitted, and the part of the video data packets lost by the second link intersects with the part of the video data packets lost by the first link, and the video data packets of the intersection may contain important information of the video data to be transmitted, resulting in the receiving end being unable to normally decode the video data to be transmitted, which will cause the video receiving end to experience problems such as freeze, screen distortion and high latency when playing the received video data.

[0097] The above two multi-path transmission methods are implemented by the data flow module in the multi-path transmission architecture, such as Figure 7 As shown, the data packet transmission process is specifically as follows: after the acquisition module of the sending end obtains the data packet sequence, it is connected to the receiving end through the communication module, and then the data stream module transmits the data packet sequence obtained by the above acquisition module to the receiving end in the form of a character stream or a byte stream, wherein the method of transmitting the data packet sequence from the sending end to the receiving end can be any one of the above-mentioned multi-path transmission methods.

[0098] In the multi-path transmission method in the embodiment of the present application, the transmission of the same video data on different links is performed by parallel transmission on multiple links, which will not be described in detail later.

[0099] Based on the problem that the important information in the video data cannot be smoothly transmitted during the video data transmission process in the prior art, the embodiment of the present application provides a method and device for transmitting video data, wherein the video transmission device or the sending end of the video data detects the transmission quality of N links; the N links are used to transmit video data, and the N links are links in different networks; the video data includes video frames of first video data, and N is a positive integer greater than or equal to 2; the video transmission device determines the video frames to be transmitted corresponding to the N links according to the transmission quality of the N links; wherein, when the transmission quality of the kth link meets the preset conditions, the video frames to be transmitted corresponding to the kth link are the video frames of the above-mentioned first video data; when the transmission quality of the kth link does not meet the preset conditions, the video frames to be transmitted corresponding to the kth link are part of the video frames of the above-mentioned first video data; wherein the value of k is any one of 1, 2, ...N, and the above-mentioned determined video frames to be transmitted include all key data in the video frames of the first video data; the video transmission device transmits the video frames to be transmitted corresponding to the N links in the N links. The technical solution provided in the embodiments of the present application can ensure the smooth transmission of key data (ie, important information) in the video data to a certain extent.

[0100] Exemplarily, the video transmission device in the embodiments of the present application can be a tablet computer, a mobile phone, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), and other devices. The embodiments of the present application do not impose any special restrictions on the specific form of the video transmission device.

[0101] The execution subject of the video data transmission method provided in the embodiment of the present application may be a device supporting video data transmission (hereinafter referred to as a video transmission device), please refer to Figure 8 In the embodiment of the present application, the video transmission device is a mobile phone 800 as an example to introduce the video transmission device provided by the embodiment of the present application. Figure 8 The illustrated cell phone 800 is merely one example of a video transmission device, and the cell phone 800 may have more or fewer components than shown, may combine two or more components, or may have a different configuration of components. Figure 8 The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0102] like Figure 8As shown, the mobile phone 800 may include a processor 810, an external memory interface 820, an internal memory 821, a universal serial bus (USB) interface 830, a charging management module 840, a power management module 841, a battery 842, an antenna 1, an antenna 2, a radio frequency module 850, a communication module 860, an audio module 870, a speaker 870A, a receiver 870B, a microphone 870C, an earphone interface 870D, a sensor module 880, a button 890, a motor 891, an indicator 892, a camera 893, a display screen 894, and a subscriber identification module (SIM) card interface 895, etc.

[0103] Among them, the above-mentioned sensor module 880 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.

[0104] The processor 810 may include one or more processing units. For example, the processor 810 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0105] The controller can be the nerve center and command center of the mobile phone 800. The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0106] The processor 810 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 810 is a high-speed cache memory, which can store instructions or data that the processor 810 has just used or cyclically used. If the processor 810 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 810, and thus improves the efficiency of the system.

[0107] In some embodiments, the processor 810 may include an interface. The interface may include an Inter-Integrated Circuit (I2C) interface, an inter-integrated circuit Sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0108] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDL) and a serial clock line (SCL). In some embodiments, the processor 810 may include multiple groups of I2C buses. The processor 810 may be coupled to the touch sensor 280K, the charger, the flash, the camera 893, etc. through different I2C bus interfaces. For example: the processor 810 may be coupled to the touch sensor 280K through the I2C interface, so that the processor 810 communicates with the touch sensor 280K through the I2C bus interface, thereby realizing the touch function of the mobile phone 800.

[0109] The I2S interface can be used for audio communication. In some embodiments, the processor 810 can include multiple groups of I2S buses. The processor 810 can be coupled to the audio module 870 via the I2S bus to achieve communication between the processor 810 and the audio module 870. In some embodiments, the audio module 870 can transmit an audio signal to the communication module 860 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0110] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 870 and the communication module 860 can be coupled via a PCM bus interface. In some embodiments, the audio module 870 can also transmit audio signals to the communication module 860 via the PCM interface to realize the function of answering a call via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication, and the sampling rates of the two interfaces are different.

[0111] The MIPI interface can be used to connect the processor 810 with peripheral devices such as the display screen 894 and the camera 893. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 810 and the camera 893 communicate via the CSI interface to realize the shooting function of the mobile phone 800. The processor 810 and the display screen 894 communicate via the DSI interface to realize the display function of the mobile phone 800.

[0112] The interface connection relationship between the modules illustrated in the embodiment of the present application is only for illustrative purposes and does not constitute a structural limitation on the mobile phone 800. The mobile phone 800 may adopt different interface connection methods in the embodiment of the present application, or a combination of multiple interface connection methods.

[0113] The charging management module 840 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 840 may receive charging input from a wired charger through the USB interface 830. In some wireless charging embodiments, the charging management module 840 may receive wireless charging input through a wireless charging coil of the mobile phone 800. While the charging management module 840 is charging the battery 842, it may also power the mobile phone 800 through the power management module 841.

[0114] The power management module 841 is used to connect the battery 842, the charging management module 840 and the processor 810. The power management module 841 receives input from the battery 842 and / or the charging management module 840, and supplies power to the processor 810, the internal memory 821, the external memory interface 820, the display screen 894, the camera 893, and the communication module 860. The power management module 841 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some embodiments, the power management module 841 can also be set in the processor 810. In some embodiments, the power management module 841 and the charging management module 840 can also be set in the same device.

[0115] The wireless communication function of the mobile phone 800 can be realized through antenna 1, antenna 2, radio frequency module 850, communication module 860, modem and baseband processor.

[0116] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 800 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, a cellular network antenna can be reused as a wireless local area network diversity antenna. In some embodiments, the antenna can be used in combination with a tuning switch.

[0117] The RF module 850 can provide a communication processing module for wireless communication solutions including 2G / 3G / 4G / 5G, etc., applied to the mobile phone 800. The RF module 850 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The RF module 850 receives electromagnetic waves from the antenna 1, and performs filtering, amplification, and other processing on the received electromagnetic waves, and transmits them to the modem for demodulation.

[0118] The communication module 860 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the mobile phone 800. For example, in the embodiment of the present application, the mobile phone 800 can access the Wi-Fi network through the wireless communication module 860.

[0119] The wireless communication module 860 may be one or more devices integrating at least one communication processing module. The wireless communication module 860 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 810. The wireless communication module 860 may also receive a signal to be sent from the processor 810, modulate the signal, amplify the signal, and convert the signal into an electromagnetic wave and radiate it via the antenna 2.

[0120] The mobile phone 800 implements the display function through a GPU, a display screen 894, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 894 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 810 may include one or more GPUs, which execute program instructions to generate or change display information.

[0121] The display screen 894 is used to display images, videos, etc. The display screen 894 includes a display panel. For example, in an embodiment of the present application, the display screen 894 can be used to display the application interface of the first application, such as a device sharing interface, a device search interface, and a QR code scanning interface.

[0122] The mobile phone 800 can realize the shooting function through the ISP, the camera 893, the video codec, the GPU, the display screen and the application processor.

[0123] The camera 893 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the mobile phone 800 may include 1 or N cameras 893, where N is a positive integer greater than 1.

[0124] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the mobile phone 800 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0125] The video codec is used to compress or decompress digital video. The mobile phone 800 may support one or more video codecs. In this way, the mobile phone 800 can play or record videos in multiple encoding formats. In this application, the mobile phone 800 uses a frequency domain encoder to encode video data.

[0126] The external memory interface 820 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 800. The external memory card communicates with the processor 810 through the external memory interface 820 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0127] The internal memory 821 may be used to store computer executable program codes, which may include instructions. The processor 810 executes various functional applications and data processing of the mobile phone 800 by running the instructions stored in the internal memory 821. For example, in an embodiment of the present application, the processor 810 may execute the instructions stored in the internal memory 821, and the internal memory 821 may include a program storage area and a data storage area.

[0128] The mobile phone 800 can implement audio functions such as music playing and recording through the audio module 870, the speaker 870A, the receiver 870B, the microphone 870C, the earphone interface 870D, and the application processor.

[0129] The audio module 870 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 870 can also be used to encode and decode audio signals. In some embodiments, the audio module 870 can be arranged in the processor 810, or some functional modules of the audio module 870 can be arranged in the processor 810.

[0130] Speaker 870A, also called "horn", is used to convert audio electrical signals into sound signals.

[0131] The receiver 870B, also called a "handset", is used to convert audio electrical signals into sound signals.

[0132] Microphone 870C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0133] The earphone interface 870D is used to connect a wired earphone and can be a USB interface 830 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0134] The key 890 includes a power button, a volume button, etc. The key 890 can be a mechanical key or a touch key. The mobile phone 800 receives the key 890 input and generates a key signal input related to the user settings and function control of the mobile phone 800.

[0135] Motor 891 can generate vibration prompts. Motor 891 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0136] Indicator 292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0137] The SIM card interface 895 is used to connect a SIM card. The SIM card can be connected to or disconnected from the mobile phone 800 by inserting the SIM card interface 895 or pulling the SIM card interface 895 out. The mobile phone 800 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 895 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0138] The methods in the following embodiments can all be implemented in a video transmission device having the above hardware structure. The video transmission device is a video transmitter. In the following embodiments, the above video transmitter is Figure 8 Taking the mobile phone 800 shown as an example, the method of the embodiment of the present application is explained.

[0139] The present application embodiment provides a video data transmission method, which is applied to a video sending end, such as Fig. 9 As shown, the video data transmission method may include S910-S930.

[0140] S910: The video transmitting end detects transmission quality of N links used to transmit first video data.

[0141] In an embodiment of the present application, when a video transmitter sends video data to a video receiver, the video transmitter may send the video data to the video receiver through N links respectively, where N is a positive integer greater than or equal to 2.

[0142] Based on the description of the above embodiment, it can be known that the first video data sent by the above video sending end to the video receiving end includes multiple video frames, and the first video data comes from a video segment in the video to be transmitted. For example, a video segment can be a group of pictures (GOP) or multiple groups of pictures, and a video segment can also be a video segment divided by other methods, which is not specifically limited in the embodiments of the present application.

[0143] Exemplarily, one GOP consists of one key frame (eg, I frame) and 29 forward search frames (P frames).

[0144] It should be noted that the N links are links in different networks, that is, the N links belong to different networks. For example, the value of N is 2, and the two links for transmitting video data are link 1 and link 2, where link 1 can be a link in WIFI network 1, and link 2 can be a link in WIFI network 2.

[0145] In an embodiment of the present application, factors affecting the transmission quality of the link may include the transmission delay of the link, the number of retransmissions, etc. The video transmitter can determine the transmission quality of the link based on at least one of the above-mentioned factors, which will be described in detail in the following embodiments.

[0146] S920: The video transmitter determines the to-be-transmitted video frames corresponding to the N links respectively according to the transmission qualities of the N links.

[0147] In the embodiment of the present application, one link among N links is used for explanation, for example, the kth link is used as an example to explain the above S920, wherein the value of k is any value among 1, 2, ... N. It should be understood that the video frame to be transmitted corresponding to a link refers to the video frame to be transmitted on the link.

[0148] Specifically, the video sending end determines whether the transmission quality of the kth link meets the preset conditions. When the transmission quality of the kth link meets the preset conditions, it determines that the video frames to be transmitted corresponding to the kth link are all video frames of the first video data; when the transmission quality of the kth link does not meet the preset conditions (that is, the transmission quality of the kth link is poor), it determines that the video frames to be transmitted corresponding to the kth link are partial video frames of the first video data, wherein the video frames to be transmitted include all key data in the above-mentioned first video data (key data can be understood as important data in the first video data or data that is indispensable for decoding the first video data).

[0149] In an embodiment of the present application, after determining the video frames to be transmitted corresponding to the N links respectively, the video transmitter encrypts the video frames to be transmitted corresponding to the N links respectively, wherein the encryption method can be hash algorithm encryption or symmetric encryption algorithm, and the specific encryption algorithm used is not limited in this application.

[0150] Exemplarily, assuming that the first video data transmitted on two links (link 1 and link 2, respectively) includes 10 video frames, the video transmitter first determines whether the transmission quality of link 1 meets the preset conditions. If the transmission quality of link 1 meets the preset conditions, the above 10 video frames are the video frames to be transmitted on link 1; if link 1 does not meet the preset conditions, some of the above 10 video frames are determined as the video frames to be transmitted on link 1, for example, 6 of the above 10 video frames are determined as the video frames to be transmitted on link 1. Similarly, the video transmission device also determines whether the transmission quality of link 2 meets the preset conditions. If the transmission quality of link 2 meets the preset conditions, the above 10 video frames are determined as the video frames to be transmitted on link 2; if link 2 does not meet the preset conditions, some of the above 10 video frames are determined as the video frames to be transmitted on link 2, for example, 8 of the above 10 video frames are determined as the video frames to be transmitted on link 2.

[0151] In an embodiment of the present application, when the transmission quality of the link does not meet the preset conditions, the video transmitter transmits part of the video frames in the first video data on the link, which can reduce the transmission pressure of the link and improve the transmission efficiency of the link.

[0152] S930: The video transmitter transmits the to-be-transmitted video frames corresponding to the N links respectively on the N links.

[0153] In the embodiment of the present application, the video transmitter transmits the to-be-transmitted video frames corresponding to each of the N links in parallel on N different links.

[0154] Combined with the example in S920, the video frames to be transmitted on link 1 are 6 video frames of the first video data, and the video frames to be transmitted on link 2 are 8 video frames of the first video data. In this way, the video transmitter transmits the above 6 video frames on link 1 and the above 8 video frames on link 2 in parallel.

[0155] In the video data transmission method provided by the embodiment of the present application, the video transmitter detects the transmission quality of N links used to transmit the first video data, and determines the video frames to be transmitted corresponding to the N links respectively according to the transmission quality of the N links, wherein, when the transmission quality of the kth link meets the preset conditions, all the video frames in the first video data are used as the video frames to be transmitted of the kth link, and when the transmission quality of the kth link does not meet the preset conditions, part of the video frames in the first video data are used as the video frames to be transmitted of the kth link; and the above-determined video frames to be transmitted include all the key data in the first video data. In this way, when the transmission quality of the kth link is poor, this part of the video frames in the first video data transmitted on the kth link includes all the key data in the first video data, thereby ensuring the smooth transmission of the key data in the first video data to a certain extent, thereby solving the problems of freeze, screen distortion and high latency when the video receiving end plays the received video data.

[0156] Optional, combined Fig. 9 ,like Fig.10 As shown, before S910, the video data transmission method provided in the embodiment of the present application also includes S1010.

[0157] S1010: A video transmitter performs frequency domain hierarchical coding on first video data to obtain multiple video frames of the first video data.

[0158] Optionally, the multiple video frames of the above-mentioned first video data are video frames obtained after the video sending end encodes the first video data in the captured video stream. Combined with the description of the above-mentioned embodiment, it can be seen that the first video data (unencoded) includes at least one key frame and multiple forward frames.

[0159] In one implementation, the first video data can be encoded using a frequency domain layered coding method to obtain multiple video frames of the first video data. Specifically, for any key frame in the first video data, a base frame of the key frame and an extended frame of the key frame are obtained after frequency domain layered coding. Similarly, for any forward frame in the first video data, a base frame of the forward frame and an extended frame of the forward frame are obtained after frequency domain layered coding. In this way, the multiple video frames obtained after the first video data is encoded include a base frame and an extended frame of at least one key frame, and a base frame and an extended frame of multiple forward frames.

[0160] For a detailed description of the frequency domain layered coding method, reference may be made to the related description of the above embodiment, which will not be repeated here.

[0161] In an embodiment of the present application, multiple video frames of the first video data transmitted on each of the N links may be arranged in a certain order, and the arrangement order of the multiple video frames corresponding to each of the N links may be the same or different.

[0162] Optionally, in one implementation, taking the first link and the second link among the N links as examples, the arrangement order of the multiple video frames corresponding to the first link is different from the arrangement order of the multiple video frames corresponding to the second link. Specifically, among the multiple video frames of the first video data, the i-th video frame corresponding to the first link is the base frame of the first video frame, and the i-th video frame corresponding to the second link is the extended frame of the first video frame; wherein the first link and the second link are two different links among the above N links. Optionally, the first video frame can be a key frame or a forward frame.

[0163] For example, Fig.11 A schematic diagram of the ordering of multiple video frames on a first link and multiple video frames on a second link, wherein Fig.11 In the example, the first video frame on the first link is a base frame of a key frame (which can be called the first key frame), and the first video frame on the second link is an extended frame of the first key frame. For another example, the fourth video frame on the first link is an extended frame of a forward frame (which can be called the first forward frame), and the fourth video frame on the second link is the base frame of the first forward frame.

[0164] In summary, the arrangement order of multiple video frames of the first video data transmitted on different links is: in the video frame queues corresponding to two different links, the video frames with the same position (or the same number or index) are the basic frames and extended frames of the same video frame. The video sending end transmits the video frames in parallel on the two links according to this arrangement order of the video frames, and can quickly send the basic frames and extended frames of the video frames to the video receiving end. For the video receiving end, the basic frames and extended frames of the video frames can also be quickly aggregated to obtain decoded video frames.

[0165] Based on the description of the above embodiments, in the process of the video sending end transmitting the video frames to be transmitted on multiple links in parallel, one or more video frames in the video frames to be transmitted may be lost simultaneously on the above multiple links due to some factors (for example: the receiving end is restarted or the receiving end fails to consume the message sent by the sending end in time, causing the message to time out). Then the video receiving end cannot obtain the lost video frames from multiple links. In order to solve this problem, the video sending end needs to start a retransmission mechanism to retransmit the above one or more lost video frames to the video receiving end.

[0166] Optional, combined Fig.10 ,like Fig.12 As shown, in the case where the same or multiple video frames are lost on multiple links at the same time, the video data transmission method provided in the embodiment of the present application also includes S1210-S1220.

[0167] S1210: The video transmitter determines a set of retransmitted video frames.

[0168] The above-mentioned retransmitted video frame set is the intersection of the video frames to be retransmitted corresponding to each of the N links.

[0169] Exemplarily, assuming that the value of N is 2, the set of video frames to be retransmitted in link 1 (i.e., video frames lost on link 1) includes basic frames of P1 frames, extended frames of P3 frames, and extended frames of P4 frames; the set of video frames to be retransmitted in link 2 (i.e., video frames lost on link 2) includes basic frames of P1 frames, extended frames of P2 frames, and extended frames of P4 frames. Take the intersection of the set of video frames to be retransmitted in link 1 and the set of video frames to be retransmitted in link 2, and use the intersection as the set of retransmitted video frames, which includes: basic frames of P1 frames and extended frames of P4 frames.

[0170] S1220. The video transmitter transmits the video frames in the retransmitted video frame set in the j-th link in a queue-jumping transmission mode.

[0171] The transmission quality of the j-th link is higher than the transmission quality of other links among the N links except the j-th link, and the j-th link is the link with the highest transmission quality among the N links.

[0172] The upload queue-jumping transmission mode is to insert the video frames in the retransmitted video frame set into the video frame queue being transmitted by the current link, so as to give priority to transmitting the video frames in the retransmitted video frame set.

[0173] Optionally, in an embodiment of the present application, the video transmitter may insert the video frames in the retransmitted video frame set into the first position of the video frame queue being transmitted by the current link, that is, the video transmitter temporarily stops transmitting the video frames in the current video frame queue, and gives priority to transmitting the video frames in the retransmitted video frame set. After the transmission of the video frames in the retransmitted video frame set is completed, the video transmitter continues to transmit the video frames in the above video frame queue.

[0174] Optionally, the method by which the video transmitter transmits video frames in a retransmitted video frame set in the jth link specifically includes: the video transmitter transmits the video frames in the retransmitted video frame set in the jth link in order of the priorities of the video frames in the retransmitted video frame set from high to low.

[0175] In an embodiment of the present application, the priorities of the above-mentioned video frames are from low to high: extended frame of the forward frame, extended frame of the key frame, basic frame of the forward frame, and basic frame of the key frame, wherein the higher the priority, the more important the video data in the video frame.

[0176] Based on the above example, the retransmitted video frame set determined by the video transmitter includes the basic frame of the P1 frame and the extended frame of the P4 frame, and the transmission quality of link 1 is better than the transmission quality of link 2 in the two links (e.g., link 1 and link 2) for transmitting the video frames to be transmitted. For link 1, assuming that the video frames included in the current video frame queue are the basic frame of the P20 frame, the extended frame of the P20 frame, the basic frame of the P21 frame, the extended frame of the P21 frame, ... the 100 basic frame of the P frame, and the extended frame of the P100 frame, the video transmitter suspends the transmission of the video frames in the current video frame queue (i.e., suspends the transmission of the basic frame of the P20 frame, etc.), and the video transmitter transmits the basic frame of the P1 frame and the extended frame of the P4 frame in the order of the priority of the video frames in the retransmitted video frame set from high to low (the priority of the basic frame of the P1 frame is higher than the priority of the extended frame of the P4 frame).

[0177] In an embodiment of the present application, on the one hand, the video sending end transmits the video frames in the retransmitted video frame set in a queue-jumping manner, and gives priority to transmitting the video frames in the retransmitted video frame set, so that the video receiving end can obtain the lost video frames as soon as possible, improve the speed of aggregating video frames at the video receiving end, and realize rapid aggregation of video frames; on the other hand, when the above-mentioned video sending end transmits the video frames in the retransmitted video frame set, the video sending end transmits the video frames in the retransmitted video frame set in order from high to low priority. Since the higher the priority of the video frame, the more important the data information included in it, therefore, retransmitting the video frame according to the priority can ensure that the important information in the first video data is not lost to a certain extent, so as to ensure the playback quality of the video data.

[0178] It should be noted that, in the embodiment of the present application, the method for transmitting video data on each link by the video transmitter is similar. The following describes the method for transmitting video data in detail by taking the kth link as an example. Fig.13 As shown, the method for transmitting video data on the kth link includes: S1301-S1308.

[0179] S1301. A video transmitter performs frequency domain hierarchical coding on first video data to obtain multiple video frames of the first video data.

[0180] It should be understood that the method for the video transmitter to perform frequency domain hierarchical coding on the first video data to obtain multiple video frames of the first video data can be referred to the relevant description of S1010 in the above embodiment, which will not be repeated here.

[0181] S1302: The video transmitter obtains feedback information of the kth link.

[0182] In an embodiment of the present application, the feedback information of the link may include transmission delay and / or number of retransmissions. Optionally, the transmission delay may be any one of the average transmission delay of the video frame, the sum of the transmission delays of a predetermined number of video frames, or the transmission delay of the current video frame.

[0183] S1303: The video transmitter determines the transmission quality of the kth link according to the feedback information of the kth link.

[0184] In the embodiment of the present application, the video transmitter determines whether the transmission quality of the kth link meets the preset condition based on the feedback information of the kth link. Specifically:

[0185] When the feedback information of the k-th link is the transmission delay, if the transmission delay of the k-th link is less than the preset delay, the transmission quality of the k-th link meets the preset condition; if the transmission delay of the k-th link is greater than or equal to the preset delay, the transmission quality of the k-th link does not meet the preset condition.

[0186] When the feedback information of the k-th link is the number of retransmissions, if the number of retransmissions of the k-th link is less than the preset number, the transmission quality of the k-th link meets the preset conditions; if the number of retransmissions of the k-th link is greater than or equal to the preset number, the transmission quality of the k-th link does not meet the preset conditions.

[0187] When the feedback information of the k-th link is the transmission delay and the number of retransmissions, if the transmission delay of the k-th link is less than the preset delay, and the number of retransmissions of the k-th link is less than the preset number, then the transmission quality of the k-th link meets the preset conditions; if the transmission delay of the k-th link is greater than or equal to the preset threshold, and / or the number of retransmissions of the k-th link is greater than or equal to the preset number, then the transmission quality of the k-th link does not meet the preset conditions.

[0188] It should be understood that in the embodiment of the present application, if the transmission quality of the kth link meets the preset condition, it indicates that the transmission quality of the kth link is good, and if the transmission quality of the kth link does not meet the preset condition, it indicates that the transmission quality of the kth link is poor.

[0189] S1304: The video transmitter determines the to-be-transmitted video frame corresponding to the k-th link according to the transmission quality of the k-th link.

[0190] With reference to the description of S920, the video frames to be transmitted corresponding to the kth link may be all video frames of the first video data, or may be part of the video frames of the first video data. Specifically, S1304 includes S1304a or S1304b.

[0191] S1304a: When the transmission quality of the kth link meets a preset condition, the video frames to be transmitted corresponding to the kth link are all video frames of the first video data.

[0192] S1304b. When the transmission quality of the kth link does not meet the preset conditions, the video sending end selects part of the video frames of the first video data as the video frames to be transmitted corresponding to the kth link according to the preset ratio corresponding to the kth link and in descending order of the priority of the video frames of the first video data.

[0193] The above preset ratios meet the following requirements: Among them, η represents the preset ratio, Num 1 Indicates the number of non-key video frames in the video frame of the first video data, Num 2 Represents the total number of video frames of the first video data; wherein the non-key video frame refers to a video frame including non-key data.

[0194] Optionally, the non-key video frames in the first video data include extended frames of key frames, base frames of forward frames and extended frames of forward frames. When the transmission quality of a certain link is poor, the video transmitter can choose to discard non-key video frames in the video frames of the first video data, and the non-key video frames have little impact on the video quality.

[0195] It should be understood that since the base frame of the key frame contains important information in the key frame, it is stipulated in the embodiment of the present application that the base frame of the key frame must be transmitted in the link as the video frame to be transmitted (that is, the base frame of the key frame is a video frame that cannot be discarded). The priorities of the video frames of the above-mentioned first video data are from high to low: the base frame of the key frame, the base frame of the forward frame, the extended frame of the key frame, and the extended frame of the forward frame.

[0196] Exemplarily, assuming that the video frames of the first video data include 100 video frames, the preset ratio is 40% (that is, when the link quality is poor, 60 video frames need to be discarded), and the 100 video frames include 48 extended frames of forward frames, 2 extended frames of key frames, 48 ​​basic frames of forward frames, and 2 basic frames of key frames. When the transmission quality of the kth link does not meet the preset conditions, the video transmitter selects 38 basic frames of forward frames and 2 basic frames of key frames as the video frames to be transmitted corresponding to the kth link according to the priority of the video frames of the first video data from high to low.

[0197] S1305. The video transmitter transmits the to-be-transmitted video frame corresponding to the k-th link on the k-th link.

[0198] In the video data transmission method provided by the embodiment of the present application, the video transmitter detects the transmission quality of the kth link used to transmit the first video data, and determines the video frames to be transmitted corresponding to the kth link according to the transmission quality of the kth link, wherein, when the transmission quality of the kth link meets the preset conditions, all the video frames in the first video data are used as the video frames to be transmitted of the kth link, and when the transmission quality of the kth link does not meet the preset conditions, part of the video frames in the first video data are used as the video frames to be transmitted of the kth link; and the above-determined video frames to be transmitted include all the key data in the first video data. In this way, when the transmission quality of the kth link is poor, this part of the video frames in the first video data transmitted on the kth link includes all the key data in the first video data, thereby ensuring the smooth transmission of the key data in the first video data to a certain extent, thereby solving the problems of freeze, screen distortion and high latency when the video receiving end plays the received video data.

[0199] S1306: The video transmitter detects the transmission quality of the kth link.

[0200] It should be noted that the above S1306 is executed after the video transmitting end has completed transmitting the to-be-transmitted video frame corresponding to the kth link on the kth link.

[0201] In the embodiment of the present application, the detailed description of the method for the video transmitter to detect the transmission quality of the kth link can refer to the relevant description of the above embodiment, which will not be repeated here.

[0202] S1307: The video transmitter adjusts the preset ratio corresponding to the kth link according to the transmission quality of the kth link.

[0203] In an embodiment of the present application, the video transmitter adjusts the preset ratio according to the transmission quality of the kth link and uses it for subsequent video data transmission. For example, the adjusted preset ratio is used to transmit the second video data, wherein the second video data and the first video data are derived from two adjacent video segments of the same video to be transmitted.

[0204] It should be understood that in the embodiment of the present application, the transmission quality of the kth link is continuously detected in the above manner, and the preset ratio is dynamically adjusted according to the transmission quality of the kth link, which can ensure the smooth transmission of the video frames to be transmitted to a certain extent and improve the utilization rate of the kth link.

[0205] Combination Fig.13 ,like Fig.14 As shown, the above S1307 includes S1401-S1403.

[0206] S1401. The video transmitter determines whether the transmission quality of the kth link meets a preset condition.

[0207] In the embodiment of the present application, a detailed description of the method by which the video transmitter determines whether the transmission quality of the kth link meets the preset conditions can be referred to the relevant description of S1303 above, which will not be repeated here.

[0208] S1402: When the transmission quality of the kth link meets a preset condition, the video transmitting end increases a preset ratio.

[0209] When the transmission quality of k links meets the preset conditions, the video sending end increases the preset ratio; that is, the transmission quality of the kth link is good, and the video sending end increases the preset ratio so that the video sending end sends more video frames in the video data to the video receiving end, thereby improving the playback quality of the video data decoded by the video receiving end.

[0210] S1403: When the transmission quality of the kth link does not meet a preset condition, the video transmitting end reduces a preset ratio.

[0211] When the transmission quality of the kth link does not meet the preset conditions, the video sending end reduces the preset ratio. That is to say, when the transmission quality of the kth link is poor, the proportion of non-critical video frames in the video frames to be transmitted is reduced, so that the video sending end reduces the number of video data frames sent to the video receiving end, thereby reducing the transmission pressure of the kth link and improving the transmission efficiency of the kth link.

[0212] S1308. The video transmitter selects some video frames of the second video data as the video frames to be transmitted corresponding to the kth link according to the preset ratio corresponding to the adjusted kth link and in descending order of priority of the video frames of the second video data.

[0213] In the embodiment of the present application, after the video sending end determines the video frame to be transmitted corresponding to the k-th link, the video sending end transmits the video frame to be transmitted corresponding to the k-th link on the k-th link.

[0214] Accordingly, an embodiment of the present application provides a video data transmission device, which is used to execute each step in the above-mentioned video data transmission method. The embodiment of the present application can divide the functional modules of the video data transmission device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. The division of modules in the embodiment of the present application is schematic, which is only a logical functional division. There may be other division methods in actual implementation.

[0215] In the case of dividing each functional module into corresponding functional modules, Fig.15 A possible structural diagram of the video data transmission device involved in the above embodiment is shown. Fig.15 As shown, the video data transmission device includes a detection module 1501, a determination module 1502 and a transmission module 1503.

[0216] The detection module 1501 is used to detect the transmission quality of N links, for example, executing S910 in the above method embodiment.

[0217] The determination module 1502 is used to determine the to-be-transmitted video frames respectively corresponding to the N links according to the transmission qualities of the N links, for example, executing S920 in the above method embodiment.

[0218] The transmission module 1503 is used to transmit the to-be-transmitted video frames corresponding to the N links respectively on the N links, for example, executing S930 in the above method embodiment.

[0219] Optionally, the video data transmission device provided in the embodiment of the present application also includes: an encoding module 1507; the encoding module 1507 is used to perform frequency domain layered encoding on the first video data to obtain multiple video frames of the first video data, for example, executing S1010 and S1301 in the above method embodiment.

[0220] Optionally, the above-mentioned determination module 1502 is also used to determine a set of retransmitted video frames, for example, to execute S1210 in the above-mentioned method embodiment.

[0221] Optionally, the video data transmission device provided in the embodiment of the present application also includes: a queue-jumping module 1508; the queue-jumping module 1508 is used to transmit the video frames in the retransmitted video frame set in the jth link in a queue-jumping transmission manner, for example, executing S1220 in the above method embodiment.

[0222] Optionally, the video data transmission device provided in the embodiment of the present application further includes: an acquisition module 1504; the acquisition module 1504 is used to obtain feedback information of the kth link, for example, executing S1302 in the above method embodiment.

[0223] Optionally, the determination module 1502 is further configured to determine the transmission quality of the kth link according to the feedback information of the kth link, for example, executing S1303 in the method embodiment.

[0224] Optionally, the determination module 1502 is further configured to determine the to-be-transmitted video frame corresponding to the k-th link according to the transmission quality of the k-th link. For example, S1304 in the above method embodiment is executed.

[0225] Optionally, the video data transmission device provided in the embodiment of the present application also includes: a control module 1505; the control module 1505 is used to select, when the transmission quality of the kth link does not meet the preset conditions, the video sending end according to the preset ratio corresponding to the kth link, in the order of the priority of the video frames of the above-mentioned first video data from high to low, part of the video frames in the video frames of the first video data as the video frames to be transmitted corresponding to the kth link; for example, execute S1304b in the above-mentioned method embodiment.

[0226] Optionally, the transmission module 1503 is further configured to transmit the to-be-transmitted video frame corresponding to the k-th link on the k-th link; for example, executing S1305 in the method embodiment.

[0227] Optionally, the detection module 1501 is further used to detect the transmission quality of the kth link, for example, executing S1306 in the above method embodiment.

[0228] Optionally, the video data transmission device provided in the embodiment of the present application also includes: an adjustment module 1506; the adjustment module 1506 is used by the video sending end to adjust the preset ratio corresponding to the kth link according to the transmission quality of the kth link, for example, executing S1307, S1402 and S1403 in the above method embodiment.

[0229] Optionally, the control module 1505 is also used to select part of the video frames of the second video data as the video frames to be transmitted corresponding to the kth link according to the preset ratio corresponding to the adjusted kth link and in descending order of priority of the video frames of the second video data; for example, executing S1308 in the above method embodiment.

[0230] Each module of the above-mentioned video data transmission device can also be used to perform other actions in the above-mentioned method embodiment. All relevant contents of each step involved in the above-mentioned method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0231] In the case of using an integrated unit, the structural diagram of the video data transmission device provided in the embodiment of the present application is as follows: Fig.16 As shown. Fig.16 In the embodiment, the video data transmission device includes: a processing module 1601 and a communication module 1602. The processing module 1601 is used to control and manage the actions of the video data transmission device, for example, to execute the steps executed by the above-mentioned detection module 1501, determination module 1502, acquisition module 1504, control module 1505, encoding module 1507 and queue-jumping module 1508, and / or to execute other processes of the technology described herein. The communication module 1602 is used to support the interaction between the video data transmission device and other devices, for example, to execute the steps executed by the above-mentioned transmission module 1503, and / or to execute other processes of the technology described herein. Fig.16 As shown, the video data transmission device may further include a storage module 1603, and the storage module 1603 is used to store program codes of the video data transmission device and video frames to be transmitted, etc.

[0232] The processing module 1601 may be a processor or a controller, for example Figure 8 The communication module 1602 may be a transceiver, an RF circuit or a communication interface, for example Figure 8 The mobile communication module 850 and / or the wireless communication module 860 in the storage module 1603 can be a memory, for example Figure 8 The internal memory 821 in.

[0233] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) mode or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a magnetic disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0234] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0235] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0236] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0238] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, or all or part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk, and other media that can store program codes.

[0239] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for transmitting video data, characterized in that: include: Detecting the transmission quality of N links; wherein the N links are used to transmit first video data, the N links are links in different networks, the first video data includes multiple video frames, and N is a positive integer greater than or equal to 2; According to the transmission quality of the N links, determine the video frames to be transmitted corresponding to the N links respectively; wherein, when the transmission quality of the kth link meets the preset condition, the video frames to be transmitted corresponding to the kth link are all video frames of the first video data; wherein the value of k is any one of 1, 2, ..., N, and the video frames to be transmitted include all key data of the first video data; When the transmission quality of the k-th link does not meet the preset condition, according to the preset ratio corresponding to the k-th link, in descending order of priority of the video frames of the first video data, select some video frames of the video frames of the first video data as the video frames to be transmitted corresponding to the k-th link; Wherein, the preset ratio satisfies: Wherein, n represents the preset ratio, Num1 represents the number of non-key video frames in the video frames of the first video data, and Num2 represents the total number of video frames of the first video data; wherein the non-key video frame refers to a video frame including non-key data; Transmitting the to-be-transmitted video frames respectively corresponding to the N links in the N links; Detecting the transmission quality of the k-th link; According to the transmission quality of the k-th link, the preset ratio corresponding to the k-th link is adjusted.

2. The method according to claim 1, characterized in that: Detecting the transmission quality of the kth link includes: Obtaining feedback information of the kth link, the feedback information including transmission delay and / or number of retransmissions; Determining the transmission quality of the k-th link according to the feedback information of the k-th link; When the feedback information is a transmission delay, if the transmission delay of the k-th link is less than a preset delay, the transmission quality of the k-th link meets the preset condition; if the transmission delay of the k-th link is greater than or equal to the preset delay, the transmission quality of the k-th link does not meet the preset condition; When the feedback information is the number of retransmissions, if the number of retransmissions of the k-th link is less than a preset number, the transmission quality of the k-th link meets the preset condition; if the number of retransmissions of the k-th link is greater than or equal to the preset number, the transmission quality of the k-th link does not meet the preset condition; When the feedback information is the transmission delay and the number of retransmissions, if the transmission delay of the k-th link is less than the preset delay, and the number of retransmissions of the k-th link is less than the preset number, then the transmission quality of the k-th link meets the preset condition; if the transmission delay of the k-th link is greater than or equal to the preset threshold, and / or the number of retransmissions of the k-th link is greater than or equal to the preset number, then the transmission quality of the k-th link does not meet the preset condition.

3. The method according to claim 1, characterized in that According to the transmission quality of the k-th link, adjusting the preset ratio corresponding to the k-th link includes: When the transmission quality of the kth link meets the preset condition, increasing the preset ratio; When the transmission quality of the kth link does not meet the preset condition, the preset ratio is reduced.

4. The method according to claim 3, characterized in that The method further comprises: Determine the to-be-transmitted video frames corresponding to the K links of the second video data according to the adjusted preset ratio corresponding to the k-th link; The to-be-transmitted video frame corresponding to the k-th link is transmitted in the k-th link.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The first video data is subjected to frequency domain hierarchical encoding to obtain video frames of the first video data, wherein the video frames of the first video data include a base frame and an extended frame of at least one key frame, and base frames and extended frames of multiple forward frames; the non-key video frames include extended frames of key frames, base frames of forward frames and extended frames of forward frames, and the key data of the first video data is the base frame of the key frame.

6. The method according to claim 5, characterized in that The priorities of the video frames of the first video data are, from high to low, the base frame of the key frame, the base frame of the forward frame, the extended frame of the key frame, and the extended frame of the forward frame.

7. The method according to any one of claims 1 to 4, characterized in that The i-th video frame in the video frames of the first video data corresponding to the first link is a basic frame of the first video frame, and the i-th video frame in the video frames of the first video data corresponding to the second link is an extended frame of the first video frame; the first link and the second link are two different links among the N links.

8. The method according to any one of claims 1 to 4 and 6, characterized in that: The method further comprises: Determine a retransmitted video frame set, where the retransmitted video frame set is an intersection of the to-be-retransmitted video frames corresponding to the N links; The video frames in the retransmitted video frame set are transmitted in the jth link in a queue-jumping transmission mode; wherein the transmission quality of the jth link is higher than the transmission quality of other links, and the other links are links among the N links except the jth link.

9. The method according to claim 8, characterized in that Transmitting the video frames in the retransmitted video frame set in the jth link, comprising: The video frames in the retransmitted video frame set are transmitted in the jth link in descending order of priority.

10. The method according to claim 9, characterized in that Transmitting the video frames in the retransmitted video frame set in the j-th link in a queue-jumping transmission manner, comprising: The retransmitted video frame set is inserted into the first position of the queue of video frames to be transmitted corresponding to the j-th link.

11. The method according to any one of claims 1 to 4, 6 and 9, characterized in that: Before transmitting the to-be-transmitted video frames respectively corresponding to the N links in the N links, the method further includes: The video frames to be transmitted corresponding to the N links are encrypted.

12. A video data transmission device, characterized in that: It comprises a memory and a processor, wherein the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the processor, the video data transmission device executes the method as described in any one of claims 1 to 11.

13. A computer storage medium, characterized in that: The method comprises computer instructions, which, when executed on a video data transmission device, cause the video data transmission device to execute the method according to any one of claims 1 to 11.

14. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

Images